[FEA] Run Scale tests regularly under a memory constrained situation

[revans2]

Is your feature request related to a problem? Please describe.
As a part of our efforts to have host memory limits we would like to run the scale/stress tests with host memory limited.

https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/integration_tests/ScaleTest.md

This is very similar to #9448, except that we really just want to run this under kubernetes or YARN with a configuration where the amount of host memory is limited. We can talk about how much host memory should be allowed, especially when we consider all of the other configs in Spark around memory usage and java heap size. Initially we would expect these tests to fail until we turn on strict host memory limits and add in retry for all of the operators that we are concerned about.