Add support for Spark 2.x Explain Api

docs/get-started/getting-started-workload-qualification.md

@@ -63,13 +63,8 @@ the other is to modify your existing Spark application code to call a function d
 
 Please note that if using adaptive execution in Spark the explain output may not be perfect
 as the plan could have changed along the way in a way that we wouldn't see by looking at just
-the CPU plan.
-
-### Requirements
-
-- A Spark 3.x CPU cluster
-- The `rapids-4-spark` and `cudf` [jars](../download.md)
-- Ability to modify the existing Spark application code if using the function call directly
+the CPU plan. The same applies if you are using an older version of Spark. Spark planning
+may be slightly different if you go up to a newer version of Spark.
 
 ### Using the Configuration Flag for Explain Only Mode
 
@@ -78,6 +73,13 @@ This mode allows you to run on a CPU cluster and can help us understand the pote
 if there are any unsupported features. Basically it will log the output which is the same as
 the driver logs with `spark.rapids.sql.explain=all`.
 
+#### Requirements
+
+- A Spark 3.x CPU cluster
+- The `rapids-4-spark` and `cudf` [jars](../download.md)
+
+#### Usage
+
 1. In `spark-shell`, add the `rapids-4-spark` and `cudf` jars into --jars option or put them in the
    Spark classpath and enable the configs `spark.rapids.sql.mode=explainOnly` and
    `spark.plugins=com.nvidia.spark.SQLPlugin`.
@@ -125,20 +127,41 @@ pretty accurate.
 
 ### How to use the Function Call
 
-Starting with version 21.12 of the RAPIDS Accelerator, a new function named
-`explainPotentialGpuPlan` is added which can help us understand the potential GPU plan and if there
-are any unsupported features on a CPU cluster.  Basically it can return output which is the same as
-the driver logs with `spark.rapids.sql.explain=all`.
+A function named `explainPotentialGpuPlan` is available which can help us understand the potential
+GPU plan and if there are any unsupported features on a CPU cluster. Basically it can return output
+which is the same as the driver logs with `spark.rapids.sql.explain=all`.
 
-1. In `spark-shell`, add the `rapids-4-spark` and `cudf` jars into --jars option or put them in the
+#### Requirements with Spark 3.X
+
+- A Spark 3.X CPU cluster
+- The `rapids-4-spark` and `cudf` [jars](../download.md)
+- Ability to modify the existing Spark application code
+- RAPIDS Accelerator for Apache Spark version 21.12 or newer
+
+#### Requirements with Spark 2.4.X
+
+- A Spark 2.4.X CPU cluster
+- The `rapids-4-spark-sql-meta` [jar](../download.md)
+- Ability to modify the existing Spark application code
+- RAPIDS Accelerator for Apache Spark version 22.02 or newer
+
+#### Usage
+
+1. In `spark-shell`, add the necessary jars into --jars option or put them in the
    Spark classpath.
 
-   For example:
+   For example, on Spark 3.X:
 
    ```bash
    spark-shell --jars /PathTo/cudf-<version>.jar,/PathTo/rapids-4-spark_<version>.jar
    ```
 
+   For example, on Spark 2.4.X:
+
+   ```bash
+   spark-shell --jars /PathTo/rapids-4-spark-sql-meta-<version and classifier>.jar
+   ```
+
 2. Test if the class can be successfully loaded or not.
 
    ```scala
@@ -148,8 +171,8 @@ the driver logs with `spark.rapids.sql.explain=all`.
 3. Enable optional RAPIDS Accelerator related parameters based on your setup.
 
    Enabling optional parameters may allow more operations to run on the GPU but please understand
-   the meaning and risk of above parameters before enabling it. Please refer to [configs
-   doc](../configs.md) for details of RAPIDS Accelerator parameters.
+   the meaning and risk of above parameters before enabling it. Please refer to the
+   [configuration documentation](../configs.md) for details of RAPIDS Accelerator parameters.
 
    For example, if your jobs have `double`, `float` and `decimal` operators together with some Scala
    UDFs, you can set the following parameters:

jenkins/spark-nightly-build.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright (c) 2020-2021, NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2020-2022, NVIDIA CORPORATION. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -74,6 +74,14 @@ function distWithReducedPom {
         $mvnExtaFlags
 }
 
+# build the Spark 2.x explain jar
+mvn -B $MVN_URM_MIRROR -Dmaven.repo.local=$M2DIR -Dbuildver=24X clean install -DskipTests
+[[ $SKIP_DEPLOY != 'true' ]] && \
+    mvn -B deploy $MVN_URM_MIRROR \
+        -Dmaven.repo.local=$M2DIR \
+        -DskipTests \
+        -Dbuildver=24X
+
 # build, install, and deploy all the versions we support, but skip deploy of individual dist module since we
 # only want the combined jar to be pushed.
 # Note this does not run any integration tests

jenkins/spark-premerge-build.sh

@@ -35,6 +35,9 @@ mvn_verify() {
     # file size check for pull request. The size of a committed file should be less than 1.5MiB
     pre-commit run check-added-large-files --from-ref $BASE_REF --to-ref HEAD
 
+    # build the Spark 2.x explain jar
+    env -u SPARK_HOME mvn -B $MVN_URM_MIRROR -Dbuildver=24X clean install -DskipTests
+
     # build all the versions but only run unit tests on one 3.0.X version (base version covers this), one 3.1.X version, and one 3.2.X version.
     # All others shims test should be covered in nightly pipelines
     env -u SPARK_HOME mvn -U -B $MVN_URM_MIRROR -Dbuildver=302 clean install -Drat.skip=true -DskipTests -Dmaven.javadoc.skip=true -Dskip -Dmaven.scalastyle.skip=true -Dcuda.version=$CUDA_CLASSIFIER -pl aggregator -am