Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] FAILED dpp_test.py::test_dpp_reuse_broadcast_exchange[false-0-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] #9817

Closed
andygrove opened this issue Nov 21, 2023 · 2 comments · Fixed by #9818
Closed

[BUG] FAILED dpp_test.py::test_dpp_reuse_broadcast_exchange[false-0-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] #9817

andygrove opened this issue Nov 21, 2023 · 2 comments · Fixed by #9818
Labels
bug Something isn't working

Comments

@andygrove
Copy link
Contributor

Describe the bug

I had a build fail in CI due to:

FAILED^[[0m ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-0-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER]
WARN GpuMergeAggregateIterator: Unable to merge reduction-only aggregated batches within target batch limit of 328, attempting to merge remaining 5 batches beyond limit
WARN GpuMergeAggregateIterator: Unable to merge reduction-only aggregated batches within target batch limit of 328, attempting to merge remaining 4 batches beyond limit

Steps/Code to reproduce bug

Expected behavior

Environment details (please complete the following information)
CI

Additional context
@andygrove andygrove added bug Something isn't working ? - Needs Triage Need team to review and classify labels Nov 21, 2023
@andygrove
Copy link
Contributor Author 

2023-11-21T14:19:00.5832447Z [2023-11-21T13:32:58.381Z] [gw2] [ 28%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-0-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:32:58 WARN GpuMergeAggregateIterator: Unable to merge reduction-only aggregated batches within target batch limit of 328, attempting to merge remaining 5 batches beyond limit
2023-11-21T14:19:00.5914984Z [2023-11-21T13:33:03.468Z] [gw2] [ 28%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-0-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:33:03 WARN GpuOverrides: 
2023-11-21T14:19:00.5985839Z [2023-11-21T13:33:09.057Z] [gw2] [ 28%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-1-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:33:08 WARN GpuMergeAggregateIterator: Unable to merge reduction-only aggregated batches within target batch limit of 328, attempting to merge remaining 6 batches beyond limit
2023-11-21T14:19:00.7121007Z [2023-11-21T13:33:13.193Z] [gw2] [ 28%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-1-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:01.0997119Z [2023-11-21T13:33:17.402Z] [gw2] [ 28%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-2-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:01.5772735Z [2023-11-21T13:33:21.346Z] [gw2] [ 28%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-2-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:01.5928053Z [2023-11-21T13:33:24.938Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-3-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:33:24 WARN GpuOverrides: 
2023-11-21T14:19:01.6287960Z [2023-11-21T13:33:28.806Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-3-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:33:28 WARN GpuOverrides: 
2023-11-21T14:19:01.6547921Z [2023-11-21T13:33:33.444Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-4-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:33:33 WARN GpuOverrides: 
2023-11-21T14:19:01.6716885Z [2023-11-21T13:33:37.561Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-4-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:33:37 WARN GpuOverrides: 
2023-11-21T14:19:01.7117770Z [2023-11-21T13:33:41.421Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-5-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:01.7596789Z [2023-11-21T13:33:45.271Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange[false-5-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:33:45 WARN ResolveSessionCatalog: A Hive serde table will be created as there is no table provider specified. You can set spark.sql.legacy.createHiveTableByDefault to false so that native data source table will be created instead.
2023-11-21T14:19:01.7975442Z [2023-11-21T13:33:48.875Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange_cpu_scan[DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER, ALLOW_NON_GPU(FileSourceScanExec)] 
2023-11-21T14:19:01.8431192Z [2023-11-21T13:33:52.987Z] [gw2] [ 29%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-0-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:33:52 WARN GpuOverrides: 
2023-11-21T14:19:01.8915777Z [2023-11-21T13:33:56.605Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-0-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:33:56 WARN GpuOverrides: 
2023-11-21T14:19:01.9179567Z [2023-11-21T13:34:00.725Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-1-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:01.9346085Z [2023-11-21T13:34:04.626Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-1-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:01.9535759Z [2023-11-21T13:34:08.474Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-2-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:01.9722659Z [2023-11-21T13:34:12.845Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-2-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:34:12 WARN GpuOverrides: 
2023-11-21T14:19:02.0100956Z [2023-11-21T13:34:16.441Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-3-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:34:16 WARN GpuOverrides: 
2023-11-21T14:19:02.0383261Z [2023-11-21T13:34:20.304Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-3-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:02.1054129Z [2023-11-21T13:34:23.913Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-4-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:34:23 WARN GpuOverrides: 
2023-11-21T14:19:02.2079578Z [2023-11-21T13:34:27.541Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-4-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:34:27 WARN GpuOverrides: 
2023-11-21T14:19:02.2895249Z [2023-11-21T13:34:31.666Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-5-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:02.3220724Z [2023-11-21T13:34:35.527Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_bypass[false-5-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:02.4109399Z [2023-11-21T13:34:39.964Z] [gw2] [ 30%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-0-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:34:39 WARN GpuOverrides: 
2023-11-21T14:19:02.5497072Z [2023-11-21T13:34:43.853Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-0-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:34:43 WARN GpuOverrides: 
2023-11-21T14:19:02.5969154Z [2023-11-21T13:34:48.046Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-1-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:34:47 WARN GpuOverrides: 
2023-11-21T14:19:02.6602354Z [2023-11-21T13:34:53.019Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-1-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:34:53 WARN GpuOverrides: 
2023-11-21T14:19:02.7792163Z [2023-11-21T13:34:58.621Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-2-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:03.0314308Z [2023-11-21T13:35:03.825Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-2-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:03.2310835Z [2023-11-21T13:35:07.212Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-3-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:35:07 WARN GpuOverrides: 
2023-11-21T14:19:03.4263000Z [2023-11-21T13:35:11.112Z] [gw2] [ 31%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-3-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:10 WARN GpuOverrides: 
2023-11-21T14:19:03.6345269Z [2023-11-21T13:35:14.760Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-4-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:35:14 WARN GpuOverrides: 
2023-11-21T14:19:03.8093343Z [2023-11-21T13:35:18.654Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-4-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:18 WARN GpuOverrides: 
2023-11-21T14:19:03.9265596Z [2023-11-21T13:35:22.566Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-5-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:22 WARN GpuOverrides: 
2023-11-21T14:19:04.0397333Z [2023-11-21T13:35:26.706Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_via_aggregate_subquery[false-5-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:35:26 WARN GpuOverrides: 
2023-11-21T14:19:04.0831635Z [2023-11-21T13:35:30.319Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-0-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:35:30 WARN GpuOverrides: 
2023-11-21T14:19:04.1376554Z [2023-11-21T13:35:34.190Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-0-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:04.3538599Z [2023-11-21T13:35:38.205Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-1-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:38 WARN GpuOverrides: 
2023-11-21T14:19:04.6830403Z [2023-11-21T13:35:42.430Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-1-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER] 23/11/21 13:35:42 WARN GpuOverrides: 
2023-11-21T14:19:04.9055511Z [2023-11-21T13:35:46.342Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-2-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:46 WARN GpuOverrides: 
2023-11-21T14:19:05.1202743Z [2023-11-21T13:35:49.984Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-2-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:49 WARN GpuOverrides: 
2023-11-21T14:19:05.1975472Z [2023-11-21T13:35:53.594Z] [gw2] [ 32%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-3-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:05.2266552Z [2023-11-21T13:35:57.448Z] [gw2] [ 33%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-3-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 23/11/21 13:35:57 WARN GpuOverrides: 
2023-11-21T14:19:05.3018190Z [2023-11-21T13:36:01.317Z] [gw2] [ 33%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-4-parquet][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:05.3187083Z [2023-11-21T13:36:05.476Z] [gw2] [ 33%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-4-orc][DATAGEN_SEED=1700572856, INJECT_OOM, IGNORE_ORDER] 
2023-11-21T14:19:05.3338430Z [2023-11-21T13:36:09.586Z] [gw2] [ 33%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-5-parquet][DATAGEN_SEED=1700572856, IGNORE_ORDER] 
2023-11-21T14:19:05.3461412Z [2023-11-21T13:36:13.489Z] [gw2] [ 33%] FAILED ../../src/main/python/dpp_test.py::test_dpp_skip[false-5-orc][DATAGEN_SEED=1700572856, IGNORE_ORDER]

@andygrove
Copy link
Contributor Author

I can reproduce locally.

FAILED ../../src/main/python/dpp_test.py::test_dpp_reuse_broadcast_exchange_cpu_scan[DATAGEN_SEED=1700580977, INJECT_OOM, IGNORE_ORDER, ALLOW_NON_GPU(FileSourceScanExec)] - pyspark.errors.exceptions.captured.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name `None` cannot be resolved. Did you mean one of the following? [`dim`.`key`, `fact`.`key`, `dim`.`skey`, `fact`.`skey`, `dim`.`value`].; line 6 pos 23;

@andygrove andygrove mentioned this issue Nov 21, 2023
Merged
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Nov 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Specify nullable=False when generating filter values in dpp tests andygrove/spark-rapids
2 participants
@andygrove @mattahrens