-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Asset bundle run_job_task fails #812
Comments
same here - we're hoping to ideally:
|
Thank for reporting the issue, We're aware of the problem and are working on a fix. We will keep this thread updated |
We are in the exact same position, wanted to drop a note here in case you needed anyone else to test the solution. Happy to run a specific branch/commit if you'd like! |
Same issue. Posted a description of the issue on the Databricks Community forum. |
Thanks, Tim - I just posted this issue to the community forum and hadn't got back here yet! |
We're working on a fix for the issue and will keep this thread updated. Unfortunately, it's not as simple as a mere type change, due to where the type information for the bundle schema comes from. We expect this to take ~2 weeks. We'll keep you posted. |
@andrewnester, any updates on this? |
Work is in progress and coming along. I expect the fix for this to take another week or so. What's underpinning this fix is a change in the way we're loading and processing the bundle configuration. Currently, we're loading it directly into the typed Go structures and this prevents us from storing a string (e.g. |
@pietern thanks for working on this! any update on the timeline? |
track |
1 similar comment
track |
Can we please get an update on when the fix is expected? |
Yes we are in the same boat. We need to be able to reference another job by name as well. Hope this issue is still alive. |
Hey all, development for this issue is very much alive, though it's taking longer to finalize. If you're keen to follow along, there are two places to look:
Note that because the change is fundamental (thus ~risky), we're making sure it doesn't regress existing use cases. Thanks for your patience. |
Hey everyone! Exciting news: the change to make "run_job_task" work has been merged and released in CLI version 0.214.0. As an example, it can be used in this form
Closing the issue as this functionality now works in DABs |
Does this work in the json file? |
@virtualdvid what do you mean by that? |
@andrewnester we use json files to deploy workflows in several environments. Here you guys are showing a yaml file example. I'm wondering if I can use "run_job_task": {
"job_id": "{{resources.jobs.job_a.id}}"
}, |
Ah, I see, if you're referring to JSON I guess you're using |
It looks to me like this issue still exists. resources:
jobs:
sync_earnings:
name: Sync earnings
max_concurrent_runs: 1
tasks:
...
e2e_tests:
name: E2E tests
max_concurrent_runs: 1
tasks:
- task_key: mock_tlr
existing_cluster_id: ${var.cluster_id}
notebook_task:
source: GIT
notebook_path: tests/syncs/Mock Sync TurnoverLineResults
- task_key: run_sync_earnings
depends_on:
- task_key: mock_tlr
run_job_task:
job_id: ${resources.jobs.sync_earnings.id} I have also tried using a variable such as
If I run If I replace the job id with the hardcoded job id, that works fine. |
@thisisthemurph which CLI version are you using? |
@andrewnester thank you for quickly pointing out I am dumb. Currently on |
@andrewnester thank you, updating to the latest version fixed the issue. |
I get an error when trying to deploy a workflow job wich contains a "Run Job" Task:
databricks.yml
The validation step is successful, but when trying to deploy, I get the following error:
The job ID is missing in the bundle.tf.json:
I assume the reason for this is, that terraform expects a string as job ID and not a int. But quoting the ID results in a different error:
The text was updated successfully, but these errors were encountered: