Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed printing the tasks in job output in DAG execution order #377

Merged
merged 2 commits into from
May 8, 2023

Conversation

andrewnester
Copy link
Contributor

@andrewnester andrewnester commented May 8, 2023

Fixes #259

Changes

Sort task output in an execution order based on task end time

Tests

Added TestTaskJobOutputOrderToString unit test.

Bundle configuration used for manual testing

bundle:
  name: multiple-tasks

workspace:
  host: ***

environments:
  development: {
    default: true
  }

  production: {}

resources:
  jobs:
    my_job_with_file_task:
      name: "[${bundle.environment}] Job with multiple tasks"

      tasks:
        - task_key: Task1
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

        - task_key: Task2
          depends_on:
            - task_key: Task1
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

        - task_key: Task3
          depends_on:
            - task_key: Task2
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

        - task_key: Task4
          depends_on:
            - task_key: Task1
          existing_cluster_id: ***
          spark_python_task:
            python_file: ./test.py

Output

> bricks bundle run my_job_with_file_task

Run URL: ***

2023-05-08 16:08:00 "[development] Job with multiple tasks" RUNNING 
2023-05-08 16:08:23 "[development] Job with multiple tasks" TERMINATED SUCCESS 
Output:
=======
Task Task1:
Hello

=======
Task Task4:
Hello

=======
Task Task2:
Hello

=======
Task Task3:
Hello

@andrewnester andrewnester requested review from pietern and a team May 8, 2023 13:58
Copy link
Contributor

@pietern pietern left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! 🚀

LGTM bar the comment.

)

type TaskOutput struct {
Name string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the task key so can be named TaskKey instead to mirror the API objects (or simply Key).

@andrewnester andrewnester requested a review from pietern May 8, 2023 14:12
@andrewnester andrewnester merged commit 1916bc9 into main May 8, 2023
@andrewnester andrewnester deleted the output-dag-order branch May 8, 2023 14:35
@shreyas-goenka shreyas-goenka mentioned this pull request May 15, 2023
shreyas-goenka added a commit that referenced this pull request May 15, 2023
## Changes
## 0.0.32
* Add support for variables in bundle config. Introduces 4 ways of
setting variable values, which in decreasing order of priority are:
([#383](https://github.com/databricks/bricks/pull/383))([#359](https://github.com/databricks/bricks/pull/359)).
	1. Command line flag. For example: `--var="foo=bar"`
	2. Environment variable. eg: BUNDLE_VAR_foo=bar
	3. Default value as defined in the applicable environments block
	4. Default value defined in variable definition
* Make the git details bundle config block optional
([#372](#372)).
* Fix api post integration tests
([#371](#371)).
* Fix table of content by removing not required top-level item
([#366](#366)).
* Fix printing the tasks in job output in DAG execution order
([#377](#377)).
* Improved error message when 'bricks bundle run' is executed before
'bricks bundle deploy'
([#378](#378)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Print the tasks in job output in DAG execution order
2 participants