bundle run fails for non unity-catalog workspace

[Eulenator]

I have deployed my pipelines with databricks bundle deploy, this worked without problems.

Now when i want to trigger a pipeline databricks bundle run then it fails with the following error message:

Library installation failed for library due to user error. Error messages: Library from /Workspace is not allowed on non Unity Catalog cluster. Please switch to DBR 13.1+ Shared cluster or 13.2+ Assigned cluster to use /Workspace libraries.

My problem is that we do not plan to use unity catalog soon.
So is there an option I have missed to use databricks asset bundles with non unity catalog workspaces?

[andrewnester]

@Eulenator what type of resources are you deploying and trying to run?

[Eulenator]

@andrewnester i have a spark-python project with the necessary entry points for my data pipelines. I build a wheel file and use a deployment.yaml (under /resources) like this to push it with asset bundles to databricks:

build:
  no_build: true
environments:
  dev:
    strict_path_adjustment_policy: true
    workflows:
    - access_control_list:
      - permission_level: CAN_MANAGE
        user_name: service-principal://some-sp
      job_clusters:
      - job_cluster_key: default
        new_cluster:
          custom_tags:
            ResourceClass: SingleNode
          node_type_id: Standard_F4s
          num_workers: 0
          spark_conf:
            spark.databricks.cluster.profile: singleNode
            spark.master: local[*, 4]
          spark_version: 13.3.x-scala2.12
      name: Ingest
      tasks:
      - job_cluster_key: default
        python_wheel_task:
          entry_point: ingest
          package_name: demo_data_pipelines
        task_key: ingest_landing
    - access_control_list:
      - permission_level: CAN_MANAGE
        user_name: service-principal://some-sp

My databricks.yaml for the bundle then looks like this:

bundle:
  name: demo_data_pipelines

include:
  - resources/*.yml

targets:
  dev:
    default: true
    workspace:
      host: DB_HOST

I already experimented with root_path or artifact_path to change the path to a dbfs one. So the cluster can access the files without unity catalog. But no luck so far.

[andrewnester]

@Eulenator thanks! Since you're using Python wheel tasks, you can enable the following

experimental:
  python_wheel_wrapper: true

It will allow to install wheel tasks on non UC clusters.
You can see more details here
#635
#797

[Eulenator]

Thanks @andrewnester this fixes one problem.

The other problem is that we want to trigger the pipeline with the jobs API with dynamic --python-params like this:

databricks jobs run-now $JOB_ID --json '{ "job_id": $JOB_ID, "python_params": [ "--conf-file", "/some/path/conf.yaml", "--dynamic-file-path", "/dynamic/file/path/test.parquet"] }'

But it looks like that the wrapping notebook to install the libraries overwrites the sys.argv arguments with the ones defined in the deployment.yaml:

import sys
sys.argv = ["..."]

Is there a chance to run the job with the cli client and still configure dynamic params?

[andrewnester]

Sorry for the delay in the reply. At the moment it's not possible to do so, but with when this PR lands #1037 you could use databricks bundle run <JOB_KEY> --python-params command which will work the same with or without python_wheel_wrapper on

[andrewnester]

The fix has been merged and will be released in the upcoming release next week. In the meantime you can try out snapshot version which already contains the fix https://github.com/databricks/cli/releases/tag/snapshot