Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DSIP-49][Workflow] Rerun across workflow or project with dependency automatically #16194

Open
2 tasks done
Tracked by #14102
zhuxt2015 opened this issue Jun 20, 2024 · 0 comments
Open
2 tasks done
Tracked by #14102
Assignees
Labels

Comments

@zhuxt2015
Copy link
Contributor

zhuxt2015 commented Jun 20, 2024

Search before asking

  • I had searched in the DSIP and found no similar DSIP.

Motivation

There are three workflow in the one project ,such as ods,dwd,dws. in workflow ods, it include two task A,B. in workflow dwd,it include two task C,D. in workflow dws ,it include one task E. one of the dependency relationship describe as follow:
A->C->E
If the calculation logic of task A is wrong or the task failed, the downstream tasks C and E need to be rerun, and A, C and E can only be rerun manually, instead of automatically starting C and E when A finished.

Design Detail

Through t_ds_process_task_relation, t_ds_process_definition, t_ds_task_definition three tables can get the task lineage, then all downstream tasks of task A can be found, Then we can rerun the task layer by layer according to the dependency hierarchy. for example, the downstream of A has task C, and the downstream of C is E, then when A completes the rerun, start to rerun C, wait for C to run completely, and start to rerun E.

Compatibility, Deprecation, and Migration Plan

  1. If there is a running instance of the rerun task, you need to close it first to avoid unnecessary errors caused by running the same task instance at the same time.
  2. You can only rerun the workflow of your own execution permission

Test Plan

No response

Code of Conduct

@zhuxt2015 zhuxt2015 added DSIP Waiting for reply Waiting for reply labels Jun 20, 2024
@zhuxt2015 zhuxt2015 changed the title [DSIP-48][Workflow] Rerun across workflow or project with dependency automatically [DSIP-49][Workflow] Rerun across workflow or project with dependency automatically Jun 20, 2024
@SbloodyS SbloodyS removed the Waiting for reply Waiting for reply label Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants