Avoid unapply on PromotePrecision [databricks]

[jlowe]

Fixes #4581. DB9.1 added an extra argument to PromotePrecision which is not critical to our usage, so this changes the code to avoid using unapply to side-step the incompatibility between DB9.1 and other Spark versions.

[jlowe]

build

[jlowe]

Looks like an abrupt connection loss on the Databricks instance. Rekicking.

[jlowe]

build

[revans2]

looks like the databricks test timed out in some way.

....compute.amazonaws.com closed by remote host.

[pxLi]

build

[pxLi]

increased timeout did not help. looks like the parallel test on DB 9.1 could stuck there for a long time

[2022-01-21T03:41:22.324Z] [gw1] [ 99%] PASSED ../../src/main/python/window_function_test.py::test_multi_types_window_aggs_for_rows[partAndOrderBy:Timestamp-String][IGNORE_ORDER({'local': True}), APPROXIMATE_FLOAT] 

[2022-01-21T03:41:23.248Z] ../../src/main/python/window_function_test.py::test_multi_types_window_aggs_for_rows[partAndOrderBy:Decimal(18,1)-String][IGNORE_ORDER({'local': True}), APPROXIMATE_FLOAT] 

[2022-01-21T03:41:23.249Z] [gw1] [ 99%] PASSED ../../src/main/python/window_function_test.py::test_multi_types_window_aggs_for_rows[partAndOrderBy:Decimal(18,1)-String][IGNORE_ORDER({'local': True}), APPROXIMATE_FLOAT] 

[2022-01-21T03:41:23.249Z] ../../src/main/python/window_function_test.py::test_window_aggs_lead_ignore_nulls_fallback[partBy:Long-orderBy:Long-orderBy:LongRange(not_null)-agg:Byte][IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(WindowExec,Alias,WindowExpression,Lead,Literal,WindowSpecDefinition,SpecifiedWindowFrame)] 

[2022-01-21T03:41:23.249Z] [gw1] [ 99%] SKIPPED ../../src/main/python/window_function_test.py::test_window_aggs_lead_ignore_nulls_fallback[partBy:Long-orderBy:Long-orderBy:LongRange(not_null)-agg:Byte][IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(WindowExec,Alias,WindowExpression,Lead,Literal,WindowSpecDefinition,SpecifiedWindowFrame)] 

[2022-01-21T03:41:23.249Z] ../../src/main/python/window_function_test.py::test_window_aggs_lead_ignore_nulls_fallback[partBy:Long-orderBy:Long-orderBy:LongRange(not_null)-agg:Short][IGNORE_ORDER({'local': True}), ALLOW_NON_GPU(WindowExec,Alias,WindowExpression,Lead,Literal,WindowSpecDefinition,SpecifiedWindowFrame)]

time now is 2022-01-21T04:25:02 comparing to 2022-01-21T03:41:23.249Z

this reminds me about an elder issue #1811, not sure what else did they change to 9.1 runtime except function signature

[pxLi]

for an internal test, per the test log

Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
...
...
executor-heartbeater WARN HangingTaskDetector: Doing a full thread dump to debug potential hanging tasks.

22/01/21 07:43:53.734 dispatcher-event-loop-3 WARN TaskSetManager: Stage 12306 contains a task of very large size (3000 KiB). The maximum recommended task size is 1000 KiB.
22/01/21 07:43:56.819 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:43:56.819 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:43:56.819 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:43:56.819 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:00.360 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:00.361 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:00.361 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:00.361 Thread-27 WARN RunningWindowFunctionExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.852 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.853 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.853 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.853 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.967 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.967 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.967 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:07.967 Thread-27 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
22/01/21 07:44:08.270 dispatcher-event-loop-0 WARN TaskSetManager: Stage 12308 contains a task of very large size (3000 KiB). The maximum recommended task size is 1000 KiB.
22/01/21 07:44:38.206 dispatcher-event-loop-1 WARN TaskSetManager: Stage 12310 contains a task of very large size (5012 KiB). The maximum recommended task size is 1000 KiB.
22/01/21 07:45:01.808 dispatcher-event-loop-3 WARN TaskSetManager: Stage 12312 contains a task of very large size (5012 KiB). The maximum recommended task size is 1000 KiB.
22/01/21 07:45:27.189 dispatcher-event-loop-2 WARN TaskSetManager: Stage 12314 contains a task of very large size (3411 KiB). The maximum recommended task size is 1000 KiB.
22/01/21 07:45:49.333 dispatcher-event-loop-2 WARN TaskSetManager: Stage 12316 contains a task of very large size (3411 KiB). The maximum recommended task size is 1000 KiB.
22/01/21 07:45:55.100 executor-heartbeater WARN HangingTaskDetector: Task 85937 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 7267, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:35:45 UTC 2022
22/01/21 07:45:55.101 executor-heartbeater WARN HangingTaskDetector: Doing a full thread dump to debug potential hanging tasks.

22/01/21 08:52:36.430 executor-heartbeater WARN HangingTaskDetector: Task 181296 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 2729, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:36:26 UTC 2022
22/01/21 08:52:36.430 executor-heartbeater WARN HangingTaskDetector: Task 181297 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 3387, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:36:26 UTC 2022
22/01/21 08:52:45.097 executor-heartbeater WARN HangingTaskDetector: Task 85937 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 7267, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:35:45 UTC 2022
22/01/21 08:52:45.097 executor-heartbeater WARN HangingTaskDetector: Task 85936 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 7231, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:35:45 UTC 2022
22/01/21 08:52:45.097 executor-heartbeater WARN HangingTaskDetector: Task 85939 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 6504, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:35:45 UTC 2022
22/01/21 08:52:45.097 executor-heartbeater WARN HangingTaskDetector: Task 85938 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 6823, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:35:45 UTC 2022
22/01/21 08:52:46.430 executor-heartbeater WARN HangingTaskDetector: Task 181298 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 3105, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:36:26 UTC 2022
22/01/21 08:52:46.430 executor-heartbeater WARN HangingTaskDetector: Task 181299 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 3196, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:36:26 UTC 2022
22/01/21 08:52:46.430 executor-heartbeater WARN HangingTaskDetector: Task 181296 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 2729, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:36:26 UTC 2022
22/01/21 08:52:46.430 executor-heartbeater WARN HangingTaskDetector: Task 181297 is probably not making progress because its metrics (Map(internal.metrics.shuffle.read.localBlocksFetched -> 4, internal.metrics.shuffle.read.remoteBytesReadToDisk -> 0, internal.metrics.shuffle.write.bytesWritten -> 0, internal.metrics.output.recordsWritten -> 0, internal.metrics.shuffle.write.recordsWritten -> 0, internal.metrics.memoryBytesSpilled -> 0, internal.metrics.shuffle.read.remoteBytesRead -> 0, internal.metrics.diskBytesSpilled -> 0, internal.metrics.shuffle.read.localBytesRead -> 3387, internal.metrics.shuffle.read.recordsRead -> 4, internal.metrics.output.bytesWritten -> 0, internal.metrics.input.bytesRead -> 0, internal.metrics.input.recordsRead -> 0, internal.metrics.shuffle.read.remoteBlocksFetched -> 0)) has not changed since Fri Jan 21 07:36:26 UTC 2022

To download the log: scala-test-detailed-output.log

[jlowe]

I think I found the tests that are hanging, see #4599. I'll skip these tests in this PR to unblock the Databricks build.

[jlowe]

build