[BUG] Nightly build failed on Databricks with "pip: No such file or directory"

[NvTimLiu]

Describe the bug

Nightly build failed on Databricks with "pip: No such file or directory", due to the update of conda/mamba default to Python 3.10 from Python3.8

10:13:40  + PYSPARK_PYTHON=/databricks/conda/envs/cudf-udf/bin/python
10:13:40  ++ which pip
10:13:40  + sudo /databricks/conda/envs/cudf-udf/bin/pip install pytest sre_yield requests pandas pyarrow findspark pytest-xdist pytest-order
10:13:40  sudo: unable to execute /databricks/conda/envs/cudf-udf/bin/pip: No such file or directory

Steps/Code to reproduce bug
Run nightly build with init script "init_cudf_udf.sh"

[NvTimLiu]

This is due to the mismatch of python=3.8 vs pip-python=3.10, details on Databricks

$ ls /databricks/conda/envs/cudf-udf/bin/python*

lrwxrwxrwx 1 root root        9 Feb 28 10:26 /databricks/conda/envs/cudf-udf/bin/python -> python3.8
lrwxrwxrwx 1 root root        9 Feb 28 10:26 /databricks/conda/envs/cudf-udf/bin/python3 -> python3.8
-rwxrwxr-x 1 root root 22766568 Feb 28 10:26 /databricks/conda/envs/cudf-udf/bin/python3.8
-rwxrwxr-x 1 root root     3552 Feb 28 10:26 /databricks/conda/envs/cudf-udf/bin/python3.8-config
lrwxrwxrwx 1 root root       16 Feb 28 10:26 /databricks/conda/envs/cudf-udf/bin/python3-config -> python3.8-config

$ cat /databricks/conda/envs/cudf-udf/bin/pip

#!/databricks/conda/envs/cudf-udf/bin/python3.1
# -*- coding: utf-8 -*-
import re
import sys

from pip._internal.cli.main import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

When installing manba via conda [conda install -c conda-forge mamba](https://github.com/NVIDIA/spark-rapids/blob/branch-22.04/jenkins/databricks/init_cudf_udf.sh#L30) , the default python=3.10 alone with pip-python=3.10 are installed, logs as below:

## Package Plan ##

  environment location: /databricks/conda/envs/cudf-udf

  added / updated specs:
    - mamba


The following packages will be downloaded:


    pip-22.0.3                 |     pyhd8ed1ab_0         1.5 MB  conda-forge
    pybind11-abi-4             |       hd8ed1ab_3          10 KB  conda-forge
    pycosat-0.6.3              |py310h6acc77f_1009         236 KB  conda-forge
    pycparser-2.21             |     pyhd8ed1ab_0         100 KB  conda-forge
    pyopenssl-22.0.0           |     pyhd8ed1ab_0          49 KB  conda-forge
    pysocks-1.7.1              |  py310hff52083_4          28 KB  conda-forge
    python-3.10.2              |h85951f9_3_cpython        30.1 MB  conda-forge
    python_abi-3.10            |          2_cp310           4 KB  conda-forge

As our integration test runs with python=3.8. Then we tried to down-grade to [mamba install ... python=3.8 ](https://github.com/NVIDIA/spark-rapids/blob/branch-22.04/jenkins/databricks/init_cudf_udf.sh#L30), the mismatch occoured, logs as below:

$ mamba install -y python=3.8

The following packages will be downloaded:
 - python                           3.10.2  h85951f9_3_cpython              installed
  + python                           3.8.12  ha38a3c6_3_cpython              conda-forge/linux-64           28MB
  - python_abi                         3.10  2_cp310                         installed
  + python_abi                          3.8  2_cp38                          conda-forge/linux-64            4kB

There was no such issue before, as the default python was python =3.8 for conda/mamba package.