Install locales and generate en_US.UTF-8

[GaryShen2008]

Related to #5549, but not close it.
Install locales and generate en_US.UTF-8 in ubuntu docker image.
No need to update centos one because there's already en_US.UTF-8.

Signed-off-by: gashen gashen@nvidia.com

[GaryShen2008]

We may also need to update the integration test docker image.

[GaryShen2008]

build

[jlowe]

This fixes the unit test in the Docker container, but what about someone who runs the test outside of this container (i.e.: everybody other than CI) who doesn't happen to have LANG=en_US.UTF-8 setup in their environment?

[andygrove]

At a minimum, we should document this requirement for correct behavior for regexp handling of Unicode text. We could also consider falling back to CPU if LANG != en_US.UTF-8.

[jlowe]

I just want to make sure this issue isn't considered "fixed" because we finally got the tests to pass. Filed #5629 to track it.

[GaryShen2008]

Yes, updated the PR as my comment. It doesn't fix the issue, but prepare the condition to fix it in test scripts.

[pxLi]

I would prefer to put this in the https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/integration_tests/run_pyspark_from_build.sh (or the pytest file) directly instead of the dockerfile as its actually part of the test

locales installation should be fine to be put inside the dockerfile. And would be nice to doc this requirement under https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/integration_tests/README.md#prerequisities

[GaryShen2008]

build

[GaryShen2008]

Updated the PR to only install locales and generate the UTF-8, leave the LANG be set in test script.
I only updated ubuntu docker images, because centos7 docker has the en_US.UTF-8 locale, we can just set LANG=en_US.UTF-8 to switch locale.

[andygrove]

LGTM. Thanks @GaryShen2008

[andygrove]

I would prefer to put this in the https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/integration_tests/run_pyspark_from_build.sh (or the pytest file) directly instead of the dockerfile as its actually part of the test

locales installation should be fine to be put inside the dockerfile. And would be nice to doc this requirement under https://github.com/NVIDIA/spark-rapids/blob/branch-22.06/integration_tests/README.md#prerequisities

@pxLi We also need to set LANG when running the Scala unit tests and I'm not sure which script we would need to update to do that.

IMHO, if only one case/test requires that change, it would be better to be put in the scala test code directly.
Otherwise all other tests will have a non-default ENV w/ specific LANG, this is not expected.
or
if we want to make our default runtime using UTF8, then we would better to document this somewhere to make sure our developers and users know that our plugin requires UTF8 as default LANG in their ENV