Merge qa test to integration test

[shotai]

This PR is for merging QA test into our integration test folder.
Test SQLs are from QA's test cases

All tests have the "qarun" marker. Can be disselected by adding -m 'not qarun' to command line options.

Update run script to disselect these tests by default

[revans2]

build

[revans2]

The build failed with an error of

07:31:18  ERROR: file not found: qarun'

My guess is that there is something wrong with the shell script. I am happy to try and help debug it if you want.

[shotai]

Sure, please help. Let me know which part need to change. My thought is the "not qarun" passed to the pytest.main as 2 arguments, so qarun become the file name, not part of expression.

[revans2]

I would suggest you apply the following patch. It makes it so the qa tests don't run on a normal developer build, but they run on the pre-merge build, where if it takes a longer time it is less important. It also allows developers to select tests through a maven command line option.

diff --git a/integration_tests/pom.xml b/integration_tests/pom.xml
index c40e47a5..365a32f9 100644
--- a/integration_tests/pom.xml
+++ b/integration_tests/pom.xml
@@ -152,6 +152,7 @@
                                 <SKIP_TESTS>${skipTests}</SKIP_TESTS>
                                 <TEST>${test}</TEST>
                                 <COVERAGE_SUBMIT_FLAGS>${argLine}</COVERAGE_SUBMIT_FLAGS>
+                                <TEST_TAGS>${pytest.TEST_TAGS}</TEST_TAGS>
                             </environmentVariables>
                         </configuration>
                     </execution>
diff --git a/integration_tests/run_pyspark_from_build.sh b/integration_tests/run_pyspark_from_build.sh
index a3c71140..bdd1e636 100755
--- a/integration_tests/run_pyspark_from_build.sh
+++ b/integration_tests/run_pyspark_from_build.sh
@@ -31,5 +31,9 @@ else
     then
         TEST_ARGS="-k $TEST"
     fi
-    "$SPARK_HOME"/bin/spark-submit --jars "${ALL_JARS// /,}" --conf "spark.driver.extraJavaOptions=-Duser.timezone=GMT $COVERAGE_SUBMIT_FLAGS" --conf 'spark.executor.extraJavaOptions=-Duser.timezone=GMT' --conf 'spark.sql.session.timeZone=UTC' --conf 'spark.sql.shuffle.partitions=12' $SPARK_SUBMIT_FLAGS ./runtests.py -v -rfExXs -m \'not qarun\' --std_input_path=./src/test/resources/ "$TEST_ARGS" $RUN_TEST_PARAMS "$@"
+    if [[ "${TEST_TAGS}" != "" ]];
+    then
+        TEST_TAGS="-m $TEST_TAGS"
+    fi
+    "$SPARK_HOME"/bin/spark-submit --jars "${ALL_JARS// /,}" --conf "spark.driver.extraJavaOptions=-Duser.timezone=GMT $COVERAGE_SUBMIT_FLAGS" --conf 'spark.executor.extraJavaOptions=-Duser.timezone=GMT' --conf 'spark.sql.session.timeZone=UTC' --conf 'spark.sql.shuffle.partitions=12' $SPARK_SUBMIT_FLAGS ./runtests.py -v -rfExXs "$TEST_TAGS" --std_input_path=./src/test/resources/ "$TEST_ARGS" $RUN_TEST_PARAMS "$@"
 fi
diff --git a/jenkins/spark-premerge-build.sh b/jenkins/spark-premerge-build.sh
index bca4ce1f..fa69c2d5 100755
--- a/jenkins/spark-premerge-build.sh
+++ b/jenkins/spark-premerge-build.sh
@@ -49,7 +49,7 @@ export PATH="$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH"
 tar zxf $SPARK_HOME.tar.gz -C $ARTF_ROOT && \
     rm -f $SPARK_HOME.tar.gz
 
-mvn -U -B "$@" clean verify
+mvn -U -B "$@" clean verify -Dpytest.TEST_TAGS=''
 
 # The jacoco coverage should have been collected, but because of how the shade plugin
 # works and jacoco we need to clean some things up so jacoco will only report for the
diff --git a/pom.xml b/pom.xml
index 30402690..5555ea94 100644
--- a/pom.xml
+++ b/pom.xml
@@ -64,6 +64,7 @@
         <rapids.shuffle.manager.override>false</rapids.shuffle.manager.override>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <project.reporting.sourceEncoding>UTF-8</project.reporting.sourceEncoding>
+        <pytest.TEST_TAGS>not qarun</pytest.TEST_TAGS>
     </properties>
 
     <dependencyManagement>

[shotai]

I've update with the patch above. Thanks

[revans2]

build