You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
We have the init code in the executor that is supposed to prevent non-UTC default timezone on the executor side if the driver side is on UTC
it's does not deal with the fact that JVM's default timezone is mutable.
scala> spark.read.orc("integration_tests/src/test/resources/timestamp-date-test.orc").select($"time").take(1)
22/06/1522:27:49WARNGpuOverrides:!Exec <CollectLimitExec> cannot run on GPU because the ExecCollectLimitExec has been disabled, and is disabled by default because CollectLimit replacement can be slower on the GPU, if huge number of rows in a batch it could help by limiting the number of rows transferred from GPU to CPU. Set spark.rapids.sql.exec.CollectLimitExec to trueif you wish to enable it
@Partitioning <SinglePartition$> could run on GPU*Exec <FileSourceScanExec> will run on GPU
res45:Array[org.apache.spark.sql.Row] =Array([1900-05-0512:34:56.1])
gerashegalov
changed the title
[BUG] driver time zone check is brittle
[BUG] driver time zone check does not cover run-time default timezone changes
Jun 16, 2022
Describe the bug
We have the init code in the executor that is supposed to prevent non-UTC default timezone on the executor side if the driver side is on UTC
it's does not deal with the fact that JVM's default timezone is mutable.
Steps/Code to reproduce bug
Start driver & executor in GMT-8
to avoid the check
spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/Plugin.scala
Lines 219 to 225 in 6a4599d
Change the default timezone to UTC on the driver
Read the orc file from test_basic_reads on GPU
read on CPU
and observe an 8-hour difference between CPU and GPU
Expected behavior
Environment details (please complete the following information)
Additional context
Add any other context about the problem here.
Originally posted by @gerashegalov in #5767 (comment)
The text was updated successfully, but these errors were encountered: