Addressing the Named Parameter change in Spark 4.0.0 [databricks]

[razajafri]

This PR addresses the change in Spark of renaming the named parameter shouldBroadcast to isDynamicPruning in this change.

We have taken the approach of overloading the naming of the variable and passing the parameter without naming it.

contributes to #9259

[razajafri]

build

[razajafri]

build

[razajafri]

341db instance failed to start

[2024-06-07T17:28:50.316Z] cluster response is {"cluster_id":"0607-172444-i09uaaxw","creator_user_name":"timl@nvidia.com","driver_healthy":false,"cluster_name":"CI-jenkins-rapids-databricks_premerge-github-799-3.4.1-1717781083","spark_version":"13.3.x-gpu-ml-scala2.12","aws_attributes":{"first_on_demand":1,"availability":"SPOT_WITH_FALLBACK","zone_id":"us-west-2c","spot_bid_price_percent":100,"ebs_volume_count":0},"node_type_id":"g5.4xlarge","driver_node_type_id":"g5.4xlarge","ssh_public_keys":["****"],"autotermination_minutes":390,"enable_elastic_disk":false,"disk_spec":{"disk_count":0},"cluster_source":"API","enable_local_disk_encryption":false,"instance_source":{"node_type_id":"g5.4xlarge"},"driver_instance_source":{"node_type_id":"g5.4xlarge"},"effective_spark_version":"13.3.x-gpu-ml-scala2.12","state":"TERMINATED","state_message":"Please reduce the number of instances in your request, or wait for additional capacity to become available. You can also try launching an instance by selecting different instance types (which you c...","start_time":1717781084273,"terminated_time":1717781302423,"last_state_loss_time":0,"last_activity_time":0,"last_restarted_time":1717781084273,"num_workers":0,"default_tags":{"Vendor":"Databricks","Creator":"timl@nvidia.com","ClusterName":"CI-jenkins-rapids-databricks_premerge-github-799-3.4.1-1717781083","ClusterId":"0607-172444-i09uaaxw"},"termination_reason":{"code":"AWS_INSUFFICIENT_INSTANCE_CAPACITY_FAILURE","type":"CLIENT_ERROR","parameters":{"aws_api_error_code":"InsufficientInstanceCapacity","aws_error_message":"We currently do not have sufficient g5.4xlarge capacity in the Availability Zone you requested (us-west-2c). Our system will be working on provisioning additional capacity. You can currently get g5.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-west-2a, us-west-2b."}},"init_scripts_safe_mode":false,"spec":{"cluster_name":"CI-jenkins-rapids-databricks_premerge-github-799-3.4.1-1717781083","spark_version":"13.3.x-gpu-ml-scala2.12","aws_attributes":{"first_on_demand":1,"availability":"SPOT_WITH_FALLBACK","zone_id":"us-west-2c","spot_bid_price_percent":100,"ebs_volume_count":0},"node_type_id":"g5.4xlarge","driver_node_type_id":"g5.4xlarge","ssh_public_keys":["****"],"autotermination_minutes":390,"enable_elastic_disk":false,"enable_local_disk_encryption":false,"num_workers":0}}

[2024-06-07T17:28:50.316Z] 0607-172444-i09uaaxw state:TERMINATED

[razajafri]

build

[gerashegalov]

LGTM but could be a way smaller change

[gerashegalov]

My preference is to minimize the change. I would leave the name as is, then we could have the whole patch a one-line change

[razajafri]

build