[BUILD] cleanup unused jenkins files and scripts

[tgravescs]

Is your feature request related to a problem? Please describe.
Directory:
https://github.com/NVIDIA/spark-rapids/tree/branch-0.4/jenkins

has jenkins files, docker files, and scripts. I believe many of those are not used anymore (like the databricks ones). We should remove any files from here that aren't used.

I'm not sure if one of the Docker files was an example for users. We do already have one for k8s here: https://github.com/NVIDIA/spark-rapids/tree/branch-0.4/docs/get-started. If we want it visible to users perhaps we should document and put it in similar location or create a separate directory for it.

[tgravescs]

@GaryShen2008

[NvTimLiu]

@tgravescs
1, The below 2 Dockerfiles are useless, we’ll remove them from GitHub/spark-rapids repo.

• Dockerfile.integration.ubuntu16
• Dockerfile.ubuntu16

2, Other Dockerfiles/Jenkinsfiles/scripts are necessary for spark-rapids pre-merge and integration jobs.
3, We put these Dockerfiles/Jenkinsfiles/scripts in GitHub, then dev team can update them conveniently.
--- For example, if someone needs to install a python module for IT, he can change the Dockerfile-blossom.integration.centos7 directly on the GitHub repo.

So after clean useless Dockerfiles in item 1, need we to move the whole Jenkins dir? or the subdirs[jenkins/databricks] or files out of spark-rapids Github and move them into internal Gitlab repo?

[tgravescs]

2. ok, I guess Jenkinsfile.databricks301nightly is still used here, but its inconsistent with the other nightly now so I would rather see it moved out to be consistent. I do see some of the other scripts are used from here.
3. The problem is that some of the scripts are here and some are in the other repo so Dev doesn't know where to update all the builds, which is confusing. For instance we wanted to add build argument, the scripts updated nightly but not all the IT tests. It would be have a single scripts for running the IT tests for instance that others could call with parameters if needed. I don't think it matters which repo this in as much as keeping it consistent and documenting where things are at.

[pxLi]

1. ok, I guess Jenkinsfile.databricks301nightly is still used here, but its inconsistent with the other nightly now so I would rather see it moved out to be consistent. I do see some of the other scripts are used from here.
2. The problem is that some of the scripts are here and some are in the other repo so Dev doesn't know where to update all the builds, which is confusing. For instance we wanted to add build argument, the scripts updated nightly but not all the IT tests. It would be have a single scripts for running the IT tests for instance that others could call with parameters if needed. I don't think it matters which repo this in as much as keeping it consistent and documenting where things are at.

Hi, I totally agree that put all those stuff in 2 different places could confuse people, but it is required by blossom and security team to move at least jenkins files to internal.

And yes, we may just keep jenkins files and scripts at the same place if possible,

1. To leave the integration test scripts at github/spark-rapids since developers could still benefit from them (I am assuming most of developers do care about test scripts but not jenkins files) and we put all CSP-related scripts to the internal repo
2. or to move ALL CICD-related stuff back to internal (people should remember to update test scripts in internal repo after they submit code change to the external repo),
   IMHO, I would prefer the first one, any thoughts?

[tgravescs]

the problem comes in the definition of we put all CSP related scripts... What does that include, right now that includes scripts for running tests, which I don't want them split. If we can split and commonize them such that the scripts for running tests are in spark-rapids and other Jenkinsfiles and CSP setup scripts are in private, I'm good with that.

[tgravescs]

also here, it seems we aren't running all of the tests. I'm fixing the integration test jar to include the resources directory, but all the test environment should be using the run_pyspark_from_build.sh where its appropriate.
We may need to modify it to support running against other clusters like EMR

[tgravescs]

note that this should go alone with #1640 and basically ideally we commonize to use the same script to run all the tests. see the comments in that issue.

[NvTimLiu]

MR on GitLab: 123

[NvTimLiu]

Todo

1. To move DB IT scripts into GitHub
2. Make it common to create cluster, setup pytests ... with python API

[tgravescs]

@NvTimLiu are you still working on commonizing some of the scripts for blossom?

[NvTimLiu]

@tgravescs I've done all the common scripts for all the cloud environments.

@jlowe Need we to add some tag similar to the [-Dpytest.TEST_TYPE="nightly"](https://github.com/NVIDIA/spark-rapids/blob/branch-0.5/jenkins/spark-nightly-build.sh#L25) for all the integration tests pipelines? If not, I suppose we are good to close this issue.

[jlowe]

Need we to add some tag similar to the -Dpytest.TEST_TYPE="nightly" for all the integration tests pipelines?

Yes, all of the CI pipelines should be specifying something for TEST_TYPE so we don't allow certain tests that are a bit tricky to setup to keep skipping, otherwise we risk inadvertently never running those tests before we release.

[NvTimLiu]

close per PR2059 merged