Added transformation mutator for Python wheel task for them to work on DBR <13.1 #635

andrewnester · 2023-08-03T14:08:48Z

Changes

Note: this PR relies on sync.include functionality from here: #671

Added transformation mutator for Python wheel task for them to work on DBR <13.1

Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1

In order to make Python wheel work correctly on DBR < 13.1 we do the following:

Build and upload python wheel as usual
Transform python wheel task into special notebook task which does the following
a. Installs all necessary wheels with %pip magic
b. Executes defined entry point with all provided parameters
Upload this notebook file to workspace file system
Deploy transformed job task

This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up

Tests

bundle.yml

bundle:
  name: wheel-task

workspace:
  host: ****

resources:
  jobs:
    test_job:
      name: "[${bundle.environment}] My Wheel Job"
      tasks:
        - task_key: TestTask
          existing_cluster_id: "***"
          python_wheel_task:
            package_name: "my_test_code"
            entry_point: "run"
            parameters: ["first argument","first value","second argument","second value"]
          libraries:
          - whl: ./dist/*.whl

Output

andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job
Run URL: ***

2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS 
Output:
=======
Task TestTask:
Hello from my func
Got arguments v1:
['python', 'first argument', 'first value', 'second argument', 'second value']

…n DBR <13.1

bundle/python/transform.go

bundle/artifacts/artifacts.go

bundle/bundle.go

bundle/config/root.go

bundle/deploy/files/sync.go

bundle/python/transform.go

libs/fileset/glob.go

libs/sync/sync.go

bundle/python/transform.go

bundle/python/transform_test.go

bundle/python/transform.go

#674) ## Changes Fixes #673 It also includes a change for `libraries` from #635 to get the list of wheel tasks

mgyucht

May be a good idea to wait for @pietern to get back, but this looks good to me. A couple small (non-blocking) suggestions from me.

bundle/config/mutator/trampoline.go

pietern · 2023-08-30T10:07:13Z

bundle/config/mutator/trampoline.go

+	remotePath := path.Join(b.Config.Workspace.FilesPath, filepath.ToSlash(internalDirRel), notebookName)
+
+	task.NotebookTask = &jobs.NotebookTask{
+		NotebookPath: remotePath,


Can you comment on how we expect the wheel args, or task params, to be passed in the template?

As a reference example, in Python trampoline we do it with sys.argv as you can see here
https://github.com/databricks/cli/pull/635/files#diff-8ddf2564fcd580399d61df5283d0c0c7b614c476e0803f63f9f785b62dc69a04R23

I'd expect it should work similarly for task params

bundle/python/transform_test.go

bundle/python/transform.go

CLI: * Fixed --environment flag ([#705](#705)). Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)).

Also see #635.

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

…on DBR <13.1 (#635) ## Changes ***Note: this PR relies on sync.include functionality from here: #671 Added transformation mutator for Python wheel task for them to work on DBR <13.1 Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1 In order to make Python wheel work correctly on DBR < 13.1 we do the following: 1. Build and upload python wheel as usual 2. Transform python wheel task into special notebook task which does the following a. Installs all necessary wheels with %pip magic b. Executes defined entry point with all provided parameters 3. Upload this notebook file to workspace file system 4. Deploy transformed job task This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up ## Tests bundle.yml ```yaml bundle: name: wheel-task workspace: host: **** resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" parameters: ["first argument","first value","second argument","second value"] libraries: - whl: ./dist/*.whl ``` Output ``` andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job Run URL: *** 2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS Output: ======= Task TestTask: Hello from my func Got arguments v1: ['python', 'first argument', 'first value', 'second argument', 'second value'] ```

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

Bundles: * Support cluster overrides with cluster_key and compute_key ([#696](#696)). * Allow referencing local Python wheels without artifacts section defined ([#703](#703)). * Fixed --environment flag ([#705](#705)). * Correctly identify local paths in libraries section ([#702](#702)). * Fixed path joining in FindFilesWithSuffixInPath ([#704](#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([#635](#635)). Internal: * Add a foundation for built-in templates ([#685](#685)). * Test transform when no Python wheel tasks defined ([#714](#714)). * Pin Terraform binary version to 1.5.5 ([#715](#715)). * Cleanup after "Add a foundation for built-in templates" ([#707](#707)). * Filter down to Python wheel tasks only for trampoline ([#712](#712)). * Update Terraform provider schema structs from 1.23.0 ([#713](#713)).

…on DBR <13.1 (databricks#635) ## Changes ***Note: this PR relies on sync.include functionality from here: databricks#671 Added transformation mutator for Python wheel task for them to work on DBR <13.1 Using wheels upload to Workspace file system as cluster libraries is not supported in DBR < 13.1 In order to make Python wheel work correctly on DBR < 13.1 we do the following: 1. Build and upload python wheel as usual 2. Transform python wheel task into special notebook task which does the following a. Installs all necessary wheels with %pip magic b. Executes defined entry point with all provided parameters 3. Upload this notebook file to workspace file system 4. Deploy transformed job task This is also beneficial for executing on existing clusters because this notebook always reinstall wheels so if there are any changes to the wheel package, they are correctly picked up ## Tests bundle.yml ```yaml bundle: name: wheel-task workspace: host: **** resources: jobs: test_job: name: "[${bundle.environment}] My Wheel Job" tasks: - task_key: TestTask existing_cluster_id: "***" python_wheel_task: package_name: "my_test_code" entry_point: "run" parameters: ["first argument","first value","second argument","second value"] libraries: - whl: ./dist/*.whl ``` Output ``` andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job Run URL: *** 2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS Output: ======= Task TestTask: Hello from my func Got arguments v1: ['python', 'first argument', 'first value', 'second argument', 'second value'] ``` Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

## Changes Fixes issue introduced in databricks#635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle. Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

Bundles: * Support cluster overrides with cluster_key and compute_key ([databricks#696](databricks#696)). * Allow referencing local Python wheels without artifacts section defined ([databricks#703](databricks#703)). * Fixed --environment flag ([databricks#705](databricks#705)). * Correctly identify local paths in libraries section ([databricks#702](databricks#702)). * Fixed path joining in FindFilesWithSuffixInPath ([databricks#704](databricks#704)). * Added transformation mutator for Python wheel task for them to work on DBR <13.1 ([databricks#635](databricks#635)). Internal: * Add a foundation for built-in templates ([databricks#685](databricks#685)). * Test transform when no Python wheel tasks defined ([databricks#714](databricks#714)). * Pin Terraform binary version to 1.5.5 ([databricks#715](databricks#715)). * Cleanup after "Add a foundation for built-in templates" ([databricks#707](databricks#707)). * Filter down to Python wheel tasks only for trampoline ([databricks#712](databricks#712)). * Update Terraform provider schema structs from 1.23.0 ([databricks#713](databricks#713)). Signed-off-by: Arpit Jasapara <arpit.jasapara@databricks.com>

stawo · 2023-09-19T13:19:10Z

@andrewnester , @pietern , I believe this PR introduced some breaking behavior. See #783

Added transformation mutator for Python wheel task for them to work o…

78b75f6

…n DBR <13.1

andrewnester requested review from pietern and shreyas-goenka August 3, 2023 14:08

pietern reviewed Aug 10, 2023

View reviewed changes

use sync include and templates

2be69af

andrewnester requested a review from pietern August 14, 2023 15:37

andrewnester changed the title ~~Added transformation mutator for Python wheel task for them to work on DBR <13.1~~ Added Sync.Include and Sync.Excluded + Added transformation mutator for Python wheel task for them to work on DBR <13.1 Aug 14, 2023

pietern reviewed Aug 16, 2023

View reviewed changes

andrewnester changed the title ~~Added Sync.Include and Sync.Excluded + Added transformation mutator for Python wheel task for them to work on DBR <13.1~~ Added transformation mutator for Python wheel task for them to work on DBR <13.1 Aug 16, 2023

andrewnester added 2 commits August 16, 2023 15:50

removed sync include code

2e3c5ca

add tests

a9ed5df

andrewnester requested a review from pietern August 16, 2023 14:35

andrewnester mentioned this pull request Aug 16, 2023

Do not try auto detect Python package if no Python wheel tasks defined #674

Merged

pietern reviewed Aug 17, 2023

View reviewed changes

github-merge-queue bot pushed a commit that referenced this pull request Aug 17, 2023

Do not try auto detect Python package if no Python wheel tasks defined (

4694832

#674) ## Changes Fixes #673 It also includes a change for `libraries` from #635 to get the list of wheel tasks

andrewnester added 3 commits August 17, 2023 17:50

moved to separate function

476439c

Merge branch 'main' into transform-wheel-task

b10d50f

more reusable code

aeb77d9

andrewnester requested a review from pietern August 18, 2023 09:27

andrewnester enabled auto-merge August 18, 2023 09:38

andrewnester requested a review from mgyucht August 23, 2023 15:06

mgyucht approved these changes Aug 25, 2023

View reviewed changes

bundle/config/mutator/trampoline.go Show resolved Hide resolved

bundle/config/mutator/trampoline.go Outdated Show resolved Hide resolved

bundle/config/mutator/trampoline.go Outdated Show resolved Hide resolved

andrewnester added this pull request to the merge queue Aug 25, 2023

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Aug 25, 2023

kartikgupta-db reviewed Aug 29, 2023

View reviewed changes

bundle/config/mutator/trampoline.go Outdated Show resolved Hide resolved

pietern reviewed Aug 30, 2023

View reviewed changes

andrewnester added 2 commits August 30, 2023 12:24

Merge branch 'main' into transform-wheel-task

36732ac

addressed feedback

de391b2

andrewnester requested review from pietern and kartikgupta-db August 30, 2023 11:04

Allow entry point from any group

d56aa06

pietern approved these changes Aug 30, 2023

View reviewed changes

andrewnester added this pull request to the merge queue Aug 30, 2023

Merged via the queue into main with commit 12368e3 Aug 30, 2023
4 checks passed

andrewnester deleted the transform-wheel-task branch August 30, 2023 12:32

andrewnester mentioned this pull request Aug 30, 2023

Release v0.203.3 #711

Closed

pietern added a commit that referenced this pull request Aug 30, 2023

Filter down to Python wheel tasks only for trampoline

cc364a8

Also see #635.

pietern mentioned this pull request Aug 30, 2023

Filter down to Python wheel tasks only for trampoline #712

Merged

github-merge-queue bot pushed a commit that referenced this pull request Aug 30, 2023

Filter down to Python wheel tasks only for trampoline (#712)

ca2f1dc

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

andrewnester mentioned this pull request Aug 30, 2023

Release v0.203.3 #716

Merged

lennartkats-db pushed a commit that referenced this pull request Sep 1, 2023

Filter down to Python wheel tasks only for trampoline (#712)

2c9d687

## Changes Fixes issue introduced in #635. ## Tests Added new unit test to confirm correct behavior. Manually deployed sample bundle.

stawo mentioned this pull request Sep 15, 2023

Python wheel based tasks no longer deployed correctly #783

Closed

andrewnester mentioned this pull request Nov 21, 2023

bundle run fails for non unity-catalog workspace #1000

Closed

andrewnester mentioned this pull request Dec 11, 2023

Python code is not updated when using Python wheel tasks with existing Cluster ID in the task definition #1050

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added transformation mutator for Python wheel task for them to work on DBR <13.1 #635

Added transformation mutator for Python wheel task for them to work on DBR <13.1 #635

andrewnester commented Aug 3, 2023 •

edited

Loading

mgyucht left a comment

pietern Aug 30, 2023

andrewnester Aug 30, 2023

stawo commented Sep 19, 2023

Added transformation mutator for Python wheel task for them to work on DBR <13.1 #635

Added transformation mutator for Python wheel task for them to work on DBR <13.1 #635

Conversation

andrewnester commented Aug 3, 2023 • edited Loading

Changes

Tests

mgyucht left a comment

Choose a reason for hiding this comment

pietern Aug 30, 2023

Choose a reason for hiding this comment

andrewnester Aug 30, 2023

Choose a reason for hiding this comment

stawo commented Sep 19, 2023

andrewnester commented Aug 3, 2023 •

edited

Loading