-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] GC/OOM on parquet_test.py::test_small_file_memory
#9135
Comments
I should point out that this is a pre-existing test (to stress-test coalesced reads in Parquet). Literally 3 years ago: 7ac919b. |
I'm wondering if this message has anything to do with the problem:
|
I've done some more digging. I think this might have to do with resetting the thread pool size or something. Here is an easy repro:
// Write corpus.
spark.conf.set("spark.rapids.sql.enabled", false)
(0 to 2048).toDF.repartition(2000).write.mode("overwrite").parquet("file:///tmp/myth_parq_ints")
// Read for boom.
spark.conf.set("spark.rapids.sql.enabled", true)
spark.read.parquet("file:///tmp/myth_parq_ints").show
I'm not yet convinced that this behaviour is specific to CDH. I wonder if this could be an artifact of having a large number of threads, maybe?
|
More experiments:
|
What is the executor core count set to on the cluster? Is the test running in local mode or in cluster mode? |
I think this is related to #9051. I suspect we were using the incorrect driver core count instead of the (I suspect larger) executor core count when setting up the multithreaded reader pool. Easiest fix is to give the executors more heap space on the cloudera cluster setup if that's possible. Long-term fix is budgeting host memory usage. This initiative is underway for off-heap memory, but this is a heap OOM. Would be interesting to get a heap dump on OOM to see what's taking up all the heap space. |
I've run it in local mode, for both CDH and Apache Spark. I have set I've captured a heap-dump on OOM. Analyzing it now. |
Heap dump indicates that there are 128 I see that the
This is with There might be a couple of ways around the OOM:
I'm inclined to try the latter. I'll update here with results. Edit: There might be a longer term solution here, to size the |
We may want to revisit the 8MB heap buffer that is being used. This was copied from some existing Parquet reading code and could make sense to produce large, chunky reads, especially for cloud applications, but asking for 8MB when there are hundreds of threads is...problematic. We may want to automatically scale that buffer size based on the number of threads or simply use a smaller value for the temporary buffer. |
Ah, it turns out that one can't simply set One option is to set |
Fixes NVIDIA#9135. (By workaround.) This change sets `spark.executor.cores` to `10`, if it is unset. This allows integration tests to work around the failure seen in `parquet_test.py:test_small_file_memory`, where the `COALESCING` Parquet reader's thread pool accidentally uses 128 threads with 8MB memory each, thus consuming the entire heap. Note that this is a bit of a workaround. A more robust solution would be to scale the Parquet reader's buffers based on the amount of available memory, and the number of threads. Signed-off-by: MithunR <mythrocks@gmail.com>
There is a tentative workaround to get this test to run on the large CDH nodes here: The more robust fix (to size the Parquet reader's buffers "appropriately") may be tackled later on. |
parquet_test.py::test_small_file_memory
on CDHparquet_test.py::test_small_file_memory
We also saw this case failed OOM constantly while testing in arm environment
|
It stands to reason that it fails. :/ High core count, with low heap allocation. |
The
parquet_test.py::test_small_file_memory
test runs out of memory on CDH (at least on the equivalent of Spark 3.3):The failing test takes about a minute to run on my workstation, but considerably longer on CDH, before it fails.
As indicated above, this test was run manually on CDH. As a control group, I was able to run the
window_function_test.py
on the same setup, without failures.Additionally,
parquet_test.py::test_small_file_memory
runs properly on Apache Spark 3.2.x:It appears that something is indeed up.
The text was updated successfully, but these errors were encountered: