-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Save Databricks init scripts in the workspace [databricks] #8961
Conversation
To close: NVIDIA#8916 Init scripts in DBFS will be deprecated sept 1 2023. Move Databricks init scripts in the workspace Signed-off-by: Tim Liu <timl@nvidia.com>
Signed-off-by: Tim Liu <timl@nvidia.com>
build |
Signed-off-by: Tim Liu <timl@nvidia.com>
build |
Oh, I'd like to move DB workspace create and clean up into the common functions, let me try it. |
@@ -129,7 +129,7 @@ String getDbType() { | |||
void databricksBuild() { | |||
def CLUSTER_ID = '' | |||
def SPARK_MAJOR = BASE_SPARK_VERSION_TO_INSTALL_DATABRICKS_JARS.replace('.', '') | |||
def dbfs_path = "$INIT_SCRIPTS_DIR-$DB_TYPE" | |||
def ws_path = "$INIT_SCRIPTS_DIR-$DB_TYPE" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
another way of doing this is to add spark-rapids github repo under /Repos. Then we can reference an initscript path /Repos/user@email/spark-rapids/...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we create a mirror spark-rapids GitHub repo on Databricks /Repos/user@email/spark-rapids dir, it is inflexible to point the init_script files at the PR for the pre-merge pipeline, or pointing to someone's private GitHub repo for testing pipelines?
Signed-off-by: Tim Liu <timl@nvidia.com>
… common functions Signed-off-by: Tim Liu <timl@nvidia.com>
089bee3
to
648d04d
Compare
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM to unblock usage after Sep. 1
If any follow-ups, lets handle in new PRs, thanks~
To close: #8916
Init scripts in DBFS will be deprecated sept 1 2023.
Move Databricks init scripts in the workspace
Signed-off-by: Tim Liu timl@nvidia.com