Move the udf-examples module to the external repository spark-rapids-examples [databricks]

[res-life]

[BUG] udf-examples dependencies are incorrect #3971

Move the the whole udf-examples module to the external spark-rapids-examples repository.
The moved udf-examples depends on Spark Rapids repository and Spark Rapids repository should not depends on the external udf-examples. This will avoid circular dependency between the two repositories.
Done: Update the jenkens pipeline to depend on the external udf-examples for test purpose.

Background:

• mvn compile or mvn test for a clean build will fail, because of dist.jar is generated in the package phase and compile phase can't access the classes in the dist.jar. See issue #3966
• Dependency dist.jar is not the same with the published one. The published one always removes the rapids-4-spark-aggregator dependency, see issue #4253

Contributes to #3971

[res-life]

build

[jlowe]

I'm not a fan of removing some, but not all, of the udf examples. They should be bundled together, along with their tests, and built separately from the plugin build to ensure it is compiling and testing against the plugin dist artifact.

[res-life]

build

[res-life]

build

[res-life]

Move the whole udf-examples module to external.
The moved udf-examples depends on Spark Rapids repository and Spark Rapids repository should not depends on the external udf-examples. This will avoid circular dependency.

[res-life]

For the spark-rapids-examples code, refer to NVIDIA/spark-rapids-examples#94

[res-life]

Convert to draft because of the build is not passed.

[jlowe]

It's unfortunate there isn't a main branch in the spark-rapids-examples repository we can point to so we're always pointing to the latest stable release code. Otherwise we're going to forget about this and then point to possibly very stale examples, and the user would miss any new examples that were added in later versions of the spark-rapids-examples code. @GaryShen2008 should we be doing a merge to main on the spark-rapids-examples repository as we do for this one?

[GaryShen2008]

Sure, we can create a main branch for the latest release version.

[res-life]

Updated the code for the rapids_build-it-UDF-native weekly Jenkins job.
The Jenkins pipeline uses the following command to test native UDF test cases.
So although the Java and Scala code no longer depend on the external udf-examples, but the rapids_udf_test.py and run_pyspark_from_build.sh depends on the external udf-examples when running the Jenkins job.

LOCAL_JAR_PATH=/path-to/local/jars \
bash ./integration_tests/run_pyspark_from_build.sh \
-m rapids_udf_example_native --rapids_udf_example_native

After this PR merged, then should update the Jenkins job acorrdingly. @pxLi

[res-life]

build

[res-life]

build

[res-life]

build

[res-life]

build

[jlowe]

If we want the premerge and nightly tests to also build and run the UDF example tests, are the scripts going to be updated by this PR or done separately? If separately, we should file an issue to track.

[jlowe]

The extra whitespace ends up disconnecting this sentence when it's rendered.

Suggested change

[res-life]

Filed an issue to track the premerge and nightly tests: #4704

[res-life]

build

[res-life]

build

[jlowe]

Code looks OK, but it would be good to verify the nightly builds are ready for this before approving. cc: @pxLi

[res-life]

build

[res-life]

The nightly builds are not ready, convert to draft to prevent it from being accidentally merged.

[res-life]

Pipelines created by Tim are ready: Rapids-examples/examples-udf-examples-nightly, Rapids-examples/examples-udf-examples-native.
Pipeline issue is: #4704
@jlowe Please review

[jlowe]

I mentioned in this comment that we shouldn't lose premerge testing against at least some external UDFs for this repository. Having separate nightly tests is fine, I don't care much either way as long as we get nightly coverage of the UDF examples working against updated snapshot versions of the RAPIDS Accelerator, but as it is now I think we're losing premerge tests against truly external UDFs.

[jlowe]

Approving this for now since it looks like a bunch of nightly tests are failing because the udf example jars were removed. I still want to see premerge fixed to test truly external UDFs, but we can address that in a followup.

[jlowe]

This hasn't been built in a while, so getting a fresh build to verify it's still ready to go.

[jlowe]

build

[jlowe]

Filed #4834 to track covering the external UDF tests in the premerge build

[res-life]

build