[BUG] mvn build FAILED with option -P noSnapshotsWithDatabricks

[NvTimLiu]

Describe the bug

To build the pre-release jars on branch-22.02 follow below steps:

1, change all the submodules' version to 22.02.0 from 22.02.0-SNAPSHOT

2, build release jars with the CLI mvn -P noSnapshotsWithDatabricks clean install

Build source code FAILED as below:

[INFO]
[INFO] --- jacoco-maven-plugin:0.8.5:prepare-agent (prepare-agent) @ rapids-4-spark_2.12 ---
[INFO] argLine set to -javaagent:.m2/repository/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=spark-rapids/dist/target/jacoco.exec,append=true,includes=ai.rapids.cudf.*:com.nvidia.spark.*:org.apache.spark.sql.rapids.*,excludes=spark301.com.nvidia.shaded.spark.*
[INFO]
[INFO] --- build-helper-maven-plugin:3.2.0:add-source (add-profile-src-30+) @ rapids-4-spark_2.12 ---
[INFO] Source directory: spark-rapids/dist/src/main/301+-nondb/scala added.
[INFO] Source directory: spark-rapids/dist/src/main/301until310-all/scala added.
[INFO] Source directory: spark-rapids/dist/src/main/301until310-nondb/scala added.
[INFO] Source directory: spark-rapids/dist/src/main/301until320-all/scala added.
[INFO] Source directory: spark-rapids/dist/src/main/301until320-nondb/scala added.
[INFO] Source directory: spark-rapids/dist/src/main/301until330-all/scala added.
[INFO] Source directory: spark-rapids/dist/src/main/pre320-treenode/scala added.
[INFO]
[INFO] --- maven-antrun-plugin:3.0.0:run (generate-build-info) @ rapids-4-spark_2.12 ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: spark-rapids/dist/target/extra-resources
[INFO]     [mkdir] Created dir: spark-rapids/dist/target/tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:3.0.0:run (create-parallel-world) @ rapids-4-spark_2.12 ---
[INFO] Executing tasks
[INFO]      [echo] Preparing parallel worlds with params:
[INFO]      [echo]             included_buildvers=301,
[INFO]      [echo]             302,
[INFO]      [echo]             303,
[INFO]      [echo]             311,
[INFO]      [echo]             311cdh,
[INFO]      [echo]             312,
[INFO]      [echo]             320,
[INFO]      [echo]                     301db,
[INFO]      [echo]             312db
[INFO]      [echo]             project.basedir=spark-rapids/dist
[INFO]      [echo]             project.build.directory=spark-rapids/dist/target
[INFO]      [echo]             spark.version.classifier=spark301
[INFO]      [echo]             URM_URL=https://****
[INFO]      [echo]             maven.repo.local=${maven.repo.local}
[INFO]      [echo]
[INFO]      [echo] copy spark301 aggregator from spark-rapids/dist/../aggregator/target
[INFO]      [copy] Copying 1 file to spark-rapids/dist/target/deps
[INFO]     [unzip] Expanding: spark-rapids/dist/target/deps/rapids-4-spark-aggregator_2.12-22.02.1-spark301.jar into spark-rapids/dist/target/parallel-world
[INFO]     [unzip] Expanding: spark-rapids/dist/target/deps/rapids-4-spark-aggregator_2.12-22.02.1-spark301.jar into spark-rapids/dist/target/parallel-world/spark301
[INFO]      [echo] dependency:get spark302 aggregator
[INFO]      [exec] [INFO] Scanning for projects...
[INFO]      [exec] [INFO]
[INFO]      [exec] [INFO] ------------------------------------------------------------------------
[INFO]      [exec] [INFO] Building RAPIDS Accelerator for Apache Spark Distribution 22.02.1
[INFO]      [exec] [INFO] ------------------------------------------------------------------------
[INFO]      [exec] [INFO]
[INFO]      [exec] [INFO] --- maven-dependency-plugin:2.8:get (default-cli) @ rapids-4-spark_2.12 ---
[INFO]      [exec] [INFO] Resolving com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark302:22.02.1
[INFO]      [exec] [INFO] Downloading: https://****/com/nvidia/rapids-4-spark-aggregator_2.12/22.02.1/rapids-4-spark-aggregator_2.12-22.02.1-spark302.jar
[INFO]      [exec] [INFO] ------------------------------------------------------------------------
[INFO]      [exec] [INFO] BUILD FAILURE
[INFO]      [exec] [INFO] ------------------------------------------------------------------------
[INFO]      [exec] [INFO] Total time: 1.766 s
[INFO]      [exec] [INFO] Finished at: 2021-12-30T13:53:28+00:00
[INFO]      [exec] [INFO] Final Memory: 28M/1566M
[INFO]      [exec] [INFO] ------------------------------------------------------------------------
[INFO]      [exec] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.8:get (default-cli) on project rapids-4-spark_2.12: Couldn't download artifact: Could not find artifact com.nvidia:rapids-4-spark-aggregator_2.12:jar:spark302:22.02.1 in apache-snapshots-repo (https://****)
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] Try downloading the file manually from the project website.
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] Then, install it using the command:
[INFO]      [exec] [ERROR] mvn install:install-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-aggregator_2.12 -Dversion=22.02.1 -Dclassifier=spark302 -Dpackaging=jar -Dfile=/path/to/file
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] Alternatively, if you host your own repository you can deploy the file there:
[INFO]      [exec] [ERROR] mvn deploy:deploy-file -DgroupId=com.nvidia -DartifactId=rapids-4-spark-aggregator_2.12 -Dversion=22.02.1 -Dclassifier=spark302 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] com.nvidia:rapids-4-spark-aggregator_2.12:jar:22.02.1
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] from the specified remote repositories:
[INFO]      [exec] [ERROR] apache-snapshots-repo (https://****, releases=true, snapshots=true),
[INFO]      [exec] [ERROR] apache.snapshots.https (https://****, releases=true, snapshots=true),
[INFO]      [exec] [ERROR] apache.snapshots (https://****, releases=true, snapshots=true),
[INFO]      [exec] [ERROR] central (https://****, releases=true, snapshots=false),
[INFO]      [exec] [ERROR] snapshots (https://****, releases=true, snapshots=true),
[INFO]      [exec] [ERROR] snapshots-repo (https://oss.sonatype.org/content/repositories/snapshots, releases=false, snapshots=true)
[INFO]      [exec] [ERROR] -> [Help 1]
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[INFO]      [exec] [ERROR] Re-run Maven using the -X switch to enable full debug logging.
[INFO]      [exec] [ERROR]
[INFO]      [exec] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[INFO]      [exec] [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] RAPIDS Accelerator for Apache Spark Root Project ... SUCCESS [ 10.380 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin ..... SUCCESS [01:34 min]
[INFO] RAPIDS Accelerator for Apache Spark Shuffle Plugin . SUCCESS [ 11.925 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Shims SUCCESS [  1.012 s]
[INFO] RAPIDS Accelerator for Apache Spark SQL Plugin Spark 3.0.1 Shim SUCCESS [  8.100 s]
[INFO] RAPIDS Accelerator for Apache Spark Scala UDF Plugin SUCCESS [ 55.146 s]
[INFO] RAPIDS Accelerator for Apache Spark Aggregator ..... SUCCESS [ 27.976 s]
[INFO] RAPIDS Accelerator for Apache Spark Distribution ... FAILURE [  4.727 s]
[INFO] RAPIDS Accelerator for Apache Spark UDF Examples ... SKIPPED
[INFO] rapids-4-spark-integration-tests_2.12 .............. SKIPPED
[INFO] RAPIDS Accelerator for Apache Spark Tests .......... SKIPPED
[INFO] rapids-4-spark-api-validation ...................... SKIPPED
[INFO] RAPIDS Accelerator for Apache Spark tools .......... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:33 min
[INFO] Finished at: 2021-12-30T13:53:28+00:00
[INFO] Final Memory: 189M/8543M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:3.0.0:run (create-parallel-world) on project rapids-4-spark_2.12: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] spark-rapids/dist/maven-antrun/build-parallel-worlds.xml:66: The following error occurred while executing this line:
[ERROR] spark-rapids/dist/maven-antrun/build-parallel-worlds.xml:91: exec returned: 1
[ERROR] around Ant part ...<ant antfile="spark-rapids/dist/maven-antrun/build-parallel-worlds.xml" target="build-parallel-worlds" />... @ 9:128 in spark-rapids/dist/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :rapids-4-spark_2.12

Steps/Code to reproduce bug

1, change all the submodules' version to 22.02.0 from 22.02.0-SNAPSHOT

2, build release jars with the CLI mvn -P noSnapshotsWithDatabricks clean install

Expected behavior
mvn CLI build successfully

Additional context
Build SNAPSHOT version PASS with the same CLI mvn -P noSnapshotsWithDatabricks clean install

This is because the nightly pipeline had uploaded the missing dependency SNAPSHOT jars to internal maven repo, via building spark-shims one by one, e.g. 301,302,303,....

So it should be FAILED if we run the build SNAPSHOT version for the first time, when no SNAPSHOT version of jars availablem, with the same CLI mvn -P noSnapshotsWithDatabricks clean install

[NvTimLiu]

Local build PASS, so we can close this issue.

./build/buildall -p=noSnapshots -P=8

[INFO] --- maven-jar-plugin:3.2.0:test-jar (default-test-jar) @ rapids-4-spark-tools_2.12 ---
[INFO] Building jar: /spark-rapids/tools/target/spark311/rapids-4-spark-tools_2.12-22.02.0-spark311tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for RAPIDS Accelerator for Apache Spark Distribution 22.02.0:
[INFO]
[INFO] RAPIDS Accelerator for Apache Spark Distribution ... SUCCESS [02:56 min]
[INFO] RAPIDS Accelerator for Apache Spark UDF Examples ... SUCCESS [ 5.548 s]
[INFO] rapids-4-spark-integration-tests_2.12 .............. SUCCESS [01:00 min]
[INFO] RAPIDS Accelerator for Apache Spark Tests .......... SUCCESS [ 25.149 s]
[INFO] rapids-4-spark-api-validation ...................... SUCCESS [ 3.567 s]
[INFO] RAPIDS Accelerator for Apache Spark tools .......... SUCCESS [ 57.777 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:29 min
[INFO] Finished at: 2022-01-05T04:43:18Z
[INFO] ------------------------------------------------------------------------

real 23m4.112s
user 82m19.052s
sys 1m26.713s

[tgravescs]

I'm still a bit worried about this, why did it fail in the first place if everything is right? Was there something else misconfigured in that? Where did it get the udf-compiler jar from locally? We want to make sure the release build will work.