LIN-632, LIN-618 Ray Integration #862
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andycui97
force-pushed
the
ray_andycui97
branch
from
49dcf6c
to
65c7e77
Compare
andycui97
force-pushed
the
ray_andycui97
branch
from
1377ffc
to
310d0e5
Compare
andycui97
force-pushed
the
ray_andycui97
branch
from
994fc40
to
5399e61
Compare
lionsardesai
approved these changes
Dec 23, 2022
Done, synced up and made sure housing example with inter task communication works. |
lazargugleta
pushed a commit
to lazargugleta/lineapy
that referenced
this pull request
Dec 28, 2022
# Description Adds support for ray dags. Setting framework= RAY in to_pipeline now produces a dag file that can be run using ray. Ray DAGS currently use `.remote()` instead of the new [alpha workflow](https://docs.ray.io/en/latest/workflows/index.html) API with `.bind()`. No setup and teardown tasks are added because Ray also does not have a mechanism to specify execution order to allow them to be run in the correct places. All sink nodes in the graph are materialized since RAY does not have a "run whole dag" API Fixes # (issue) LIN-632, LIN-618 ## Type of change Please delete options that are not relevant. - [X] New feature (non-breaking change which adds functionality) # How Has This Been Tested? Snapshot and e2e tests added following airflow example including a new github action to run Ray tests.
1 task
andycui97
added a commit
that referenced
this pull request
Jan 6, 2023
# Description Adds new RAY workflow as the default for RAY pipelines. This change modifies the behavior described in #862 * Ray now supports `use_workflows`, setting to True (which is also default) uses the new [Ray workflows API ](https://docs.ray.io/en/latest/workflows/index.html) * Adds ability to configure the runtime_env and storage options in ray.init() * Updated tests since some workflows are not representable in new Ray workflows API Fixes LIN-738 ## Type of change Please delete options that are not relevant. - [X] New feature (non-breaking change which adds functionality) # How Has This Been Tested? Ray tests updated. Run with housing example using both "use_workflows" as True and False to ensure both new and old DAGs run. [LIN-738]: https://linea.atlassian.net/browse/LIN-738?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds support for ray dags. Setting framework= RAY in to_pipeline now produces a dag file that can be run using ray.
Ray DAGS currently use
.remote()instead of the new alpha workflow API with.bind().No setup and teardown tasks are added because Ray also does not have a mechanism to specify execution order to allow them to be run in the correct places.
All sink nodes in the graph are materialized since RAY does not have a "run whole dag" API
Fixes # (issue)
LIN-632, LIN-618
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Snapshot and e2e tests added following airflow example including a new github action to run Ray tests.