Consolidate Spark vendor shim dependency management [databricks]

[gerashegalov]

Replace numerous instance of duplicate dependency definitions for cloudera and databricks shims by aggregated definitions.

Verification along the lines :
buildall and unjar all jars in separate dirs

cd before
find . -path '*/target/*.jar' | grep -v 'dist/target/deps' | xargs -n 1 bash -c 'jar_dir=.jars/$(basename $1); mkdir -p $jar_dir; unzip -d $jar_dir $1 \*.class' _
...
diff -r before/spark-rapids/.jars after/spark-rapids/.jars
Only in before/spark-rapids/.jars/rapids-4-spark-integration-tests_2.12-23.10.0-SNAPSHOT-spark330db-jar-with-dependencies.jar/org/apache: arrow

The diff is because of the previous special-case compile-scope for arrow in integraion_tests just in the databricks prfoile. I think it may no longer be necessary. If a post-merge test breaks, will fix in a follow-up PR.

Signed-off-by: Gera Shegalov gera@apache.org

[gerashegalov]

build

[gerashegalov]

build

[tgravescs]

what does bom stand for?

[revans2]

Usually it is bill of materials.

https://www.investopedia.com/terms/b/bill-of-materials.asp

[gerashegalov]

I can rename if needed. My original intention was to use a pure BOM pattern I found recommended somewhere on stackoverflow. However, dependencyManagement does not support exclusions used in the CDH context, and even if it would, we still need to do some consolidation to avoid repeating <dependencies>. I found that the latter makes the former redundant. So the definition of a BOM in our project is not really the same what the Maven doc describes, but the same in spirit.

[tgravescs]

its fine to leave as it is

[revans2]

I assume that nothing needs to change with how we package, publish the resulting poms/jars?

[revans2]

build

[gerashegalov]

I assume that nothing needs to change with how we package, publish the resulting poms/jars?

at least that is the goal

[gerashegalov]

I ran another round of verification:

cd before
find . -path '*/target/*.jar' | grep -v 'dist/target/deps' | xargs -n 1 bash -c 'jar_dir=.jars/$(basename $1); mkdir -p $jar_dir; unzip -d $jar_dir $1 \*.class' _
...
diff -r before/spark-rapids/.jars after/spark-rapids/.jars
Only in before/spark-rapids/.jars/rapids-4-spark-integration-tests_2.12-23.10.0-SNAPSHOT-spark330db-jar-with-dependencies.jar/org/apache: arrow

revealing a difference in the presence of arrow classes in the before integration_tests assembly jar. Which is attributable to the databricks profile listing those artifacts as compile-scope

https://github.com/NVIDIA/spark-rapids/pull/9182/files#diff-4b412ae5512cbf72928bdd8bfe6aa562a04f709fcbb7d97b04b6781e338ab987L255-L272

It looks to me that the provided scope is correct but it looks it was intentionally set to compile https://github.com/NVIDIA/spark-rapids/pull/3335/files#diff-4b412ae5512cbf72928bdd8bfe6aa562a04f709fcbb7d97b04b6781e338ab987R213-R230
@tgravescs do you remember why we needed compile for arrow in db shims?

[tgravescs]

I do not, we have some arrow test classes for datasource v2 testing in the integration tests, maybe around that.

[gerashegalov]

I do not, we have some arrow test classes for datasource v2 testing in the integration tests, maybe around that.

thanks, merging as is, will put in an override if it breaks tests.