[BUG] udf-examples module fails on mvn compile and mvn test

[firestarman]

Describe the bug
mvn compile and mvn test are failing due to the error as below

[INFO] --- scala-maven-plugin:4.3.0:compile (scala-compile-first) @ rapids-4-spark-udf-examples_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/liangcail/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.3.1-bin_2.12.8__52.0-1.3.1_20191012T045515.jar
[INFO] Compiling 2 Scala sources and 7 Java sources to /home/liangcail/work/projects/on_github/spark-rapids/udf-examples/target/classes ...
[ERROR] [Error] /home/liangcail/work/projects/on_github/spark-rapids/udf-examples/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLDecode.scala:22: object RapidsUDF is not a member of package com.nvidia.spark
[ERROR] [Error] /home/liangcail/work/projects/on_github/spark-rapids/udf-examples/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLDecode.scala:30: not found: type RapidsUDF
[ERROR] [Error] /home/liangcail/work/projects/on_github/spark-rapids/udf-examples/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLEncode.scala:22: object RapidsUDF is not a member of package com.nvidia.spark
[ERROR] [Error] /home/liangcail/work/projects/on_github/spark-rapids/udf-examples/src/main/scala/com/nvidia/spark/rapids/udf/scala/URLEncode.scala:30: not found: type RapidsUDF
[ERROR] four errors found
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Root Project 21.12.0-SNAPSHOT:
[INFO] 
[INFO] RAPIDS Accelerator for Apache Spark Root Project ... SUCCESS [  0.636 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... SUCCESS [ 26.145 s]
[INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SUCCESS [  4.883 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Shims SUCCESS [  0.037 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.1 Shim SUCCESS [  2.735 s]
[INFO] RAPIDS Accelerator for Apache Spark Scala UDF Plugin SUCCESS [  5.490 s]
[INFO] RAPIDS Accelerator for Apache Spark Aggregator ..... SUCCESS [  0.033 s]
[INFO] RAPIDS Accelerator for Apache Spark Distribution ... SUCCESS [  0.279 s]
[INFO] RAPIDS Accelerator for Apache Spark UDF Examples ... FAILURE [  1.341 s]
[INFO] RAPIDS Accelerator for Apache Spark Tests .......... SKIPPED
[INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
[INFO] rapids-4-spark-api-validation ...................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Steps/Code to reproduce bug
Fetch the latest commit and run mvn test under the project root path.

Expected behavior
mvn test and mvn compile should run successfully.

Additional context
mvn test -DwildcardSuites=XXX is quite useful during local development. Hope we can support it.

git log output:

commit bfaf73b2b96a156c4c321c2f0cbe36c6615574ad (NVIDIA/branch-21.12, branch-21.12)
Merge: 81186e9b 5733da23
Author: Jason Lowe <jlowe@nvidia.com>
Date:   Thu Oct 28 16:09:26 2021 -0500

    Merge pull request #3950 from gerashegalov/fix-merge
    
    Fix failed auto-merge #3939

commit 5733da23ca9c471daa42b2ff2e5421072e2ac0b5
Author: Gera Shegalov <gera@apache.org>
Date:   Thu Oct 28 11:13:31 2021 -0700

    revert doc update

commit 81186e9be1f30f1f8be7eddbcb3d163f54ee238f
Author: Robert (Bobby) Evans <bobby@apache.org>
Date:   Thu Oct 28 12:56:42 2021 -0500

    Document compatability of operations with side effects. (#3946)
    
    Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>

[tgravescs]

@jlowe likely related to #3945

[jlowe]

Yes, this is caused by #3945. We need the udf-examples project to depend only on the dist jar, as it is an example of how one is supposed to construct a RAPIDS Accelerated UDF. However the dist project itself doesn't provide anything until at least the package phase of Maven has been executed. When we run only a phase that is before that, like compile or test, then there's no dist jar to put on the classpath, and the dist project provides no target/classes to put on the classpath of downstream projects before the jar has been constructed.

Ideally udf-examples should be treated like an external project (or even moved to a separate repository!) that is built separately from the plugin, since that's what it is trying to model. I'll do some thinking on how this should work, but I suspect the cleanest way to solve this without abusing the udf-examples dependencies (and thus defeating the whole point of it being an example for users to follow) is to have udf-examples build in a separate Maven build from the dist jar. Building the udf-examples would first require the dist jar to be installed, i.e.: treating it like a separate project.

[gerashegalov]

I agree that udf-examples should be in a separate repo that will help test in a setup close to the end user experience.

If we don't revert #3945 I think a straightforward fix is to add a step that unpacks unshimmed classes to target/classes

A more involved fix is to just stop using target/parallel-world for package preparation, and use conventional target/classes instead. Then we can get rid of this kind of customizations https://github.com/NVIDIA/spark-rapids/pull/3923/files#diff-2681b51a40a6a26d3a6f7f839cd810422226193ef273ea02d1b0b93c3a87cfa6R895

                       <configuration>
                            <classesDirectory>${project.build.directory}/parallel-world</classesDirectory>
                        </configuration>

[jlowe]

If we don't revert #3945 I think a straightforward fix is to add a step that unpacks unshimmed classes to target/classes

I was thinking of a similar fix. After a discussion with @sameerz we agreed that udf-examples should be moved out to better emulate a true example environment, and this also has the bonus of testing the dependencies of our deployed distribution jars.

I'll revert #3945 to fix this and file a followup to migrate the udf-examples to a separate repository and fixup its dependencies in the process.