Transition to v2 shims [Databricks]

[razajafri]

This PR in a nutshell

• Moved all the SparkShims to a standard package com.nvidia.spark.rapids.shims.v2 so we don't have to dynamically look them up at runtime
• Combined duplicate classes to reduce code redundancy
• Removed the need to call the ShimLoader when accessing the Shims

fixes #4474

[tgravescs]

so one thing to be aware of here is that any of our partners that have shim layers are going to have to transition as well. which means more work on their side so lets try to get this done how we want in one release.

[razajafri]

build

[gerashegalov]

How did we end up solving incompatible ShuffleManager base across Apache Spark versions. I thought having a single facade class across shims required dynamic bytecode generation at Runtime depending on the shim because compile-time method signatures cannot be consistent for each Spark version?

I believe one of the issues in 30x vs 32x

where there is a reference to ShuffleBlockResolver https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala#L92

where in 31x we have a transitive reference to MergedBlockMeta https://github.com/apache/spark/blob/branch-3.2/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockResolver.scala#L56

However there is no MergedBlockMeta in branch-3.0. And we should get a class verification exception

[razajafri]

I tested everything after 3.1.1+ and it worked. I have reverted the change as it wasn't my original intention to remove the duplicate ShuffleManagers. The original intention was to get rid of the dynamic lookup of the shims at runtime.

[razajafri]

build

[sameerz]

build

[tgravescs]

build

[razajafri]

build

[razajafri]

build

[razajafri]

build

[razajafri]

build

[razajafri]

build

[razajafri]

build

[gerashegalov]

@razajafri there are more references to non-existent shims dir that will need updating

git grep '[^/]shims/'
CONTRIBUTING.md:There is a known issue with the shims/spark3xx submodules. After being enabled once, a module such as shims/spark312
docs/dev/shims.md:(i.e., in `sql-plugin/src/main/scala` as opposed to be duplicated either in `shims/spark3xx` submodules or over
jenkins/Jenkinsfile-blossom.premerge:                                                                      'sql-plugin/src/main/scala/,shims/spark311/src/main/scala/,' +
jenkins/Jenkinsfile-blossom.premerge:                                                                      'shims/spark301db/src/main/scala/,shims/spark301/src/main/scala/,' +
jenkins/Jenkinsfile-blossom.premerge:                                                                      'shims/spark302/src/main/scala/,shims/spark303/src/main/scala/,' +
jenkins/Jenkinsfile-blossom.premerge:                                                                      'shims/spark304/src/main/scala/,shims/spark312/src/main/scala/,' +
jenkins/Jenkinsfile-blossom.premerge:                                                                      'shims/spark313/src/main/scala/'

[pxLi]

we do exactly match for post-fix trigger [databricks] instead of [Databricks] for db build as https://github.com/NVIDIA/spark-rapids/blob/branch-22.04/CONTRIBUTING.md#blossom-ci to avoid accidental triggering

so this one was actually passed CI w/o db tests, created ticket here #4918

FYI also I decide to get the check case-insensitive to avoid cases like this #4919