Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pipeline run tests for DVC #867

Merged
merged 3 commits into from
Jan 13, 2023
Merged

Add pipeline run tests for DVC #867

merged 3 commits into from
Jan 13, 2023

Conversation

andycui97
Copy link
Contributor

Description

Add tests for DVC pipelines to test that they actually run in tests/test_pipeline_run_dvc.py.
Aldo changes default flavor for DVC to be StagePerArtifact.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Added tests (non-breaking change)

How Has This Been Tested?

  • Locally and included as part of github actions.

@andycui97 andycui97 marked this pull request as ready for review January 9, 2023 20:48
@@ -0,0 +1,4 @@
dvc==2.38.1
pandas
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we install the additional packages (panda, scikit-learn and SQLAlchemy) using the pipeline generated requirements file? As additional virtualenv.run() command after 'git init' and 'dvc init'?

@pd-t
Copy link
Contributor

pd-t commented Jan 10, 2023

@andycui97 I'm fine with the changes, I think the comment above is obsolete!

@andycui97
Copy link
Contributor Author

@andycui97 I'm fine with the changes, I think the comment above is obsolete!

Sg! Just waiting on @aayan636 to review internally as well.

.github/workflows/dvc.yml Outdated Show resolved Hide resolved
.github/workflows/dvc.yml Show resolved Hide resolved
Makefile Show resolved Hide resolved
airflow-requirements.txt Show resolved Hide resolved
lineapy/plugins/base_pipeline_writer.py Show resolved Hide resolved
tests/test_pipeline_run_airflow.py Outdated Show resolved Hide resolved
tests/test_pipeline_run_dvc.py Outdated Show resolved Hide resolved
tests/test_pipeline_run_dvc.py Outdated Show resolved Hide resolved
tests/test_pipeline_run_dvc.py Show resolved Hide resolved
tests/test_pipeline_run_ray.py Outdated Show resolved Hide resolved
@andycui97 andycui97 merged commit 0726d56 into main Jan 13, 2023
@andycui97 andycui97 deleted the dvctest branch January 13, 2023 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants