[BUG] test_cast_decimal_to failing in databricks 7.3

[abellina]

Seeing the following in last night's build:

07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ByteType-Decimal(12,2)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ShortType-Decimal(12,2)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[IntegerType-Decimal(12,2)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[LongType-Decimal(12,2)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[FloatType-Decimal(12,2)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[DoubleType-Decimal(12,2)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ByteType-Decimal(7,-3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ShortType-Decimal(7,-3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[IntegerType-Decimal(7,-3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[LongType-Decimal(7,-3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[FloatType-Decimal(7,-3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[DoubleType-Decimal(7,-3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ByteType-Decimal(7,3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ShortType-Decimal(7,3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[IntegerType-Decimal(7,3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[LongType-Decimal(7,3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[FloatType-Decimal(7,3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[DoubleType-Decimal(7,3)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ByteType-Decimal(18,0)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ShortType-Decimal(18,0)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[IntegerType-Decimal(18,0)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ByteType-Decimal(7,7)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[LongType-Decimal(18,0)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[ShortType-Decimal(7,7)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[IntegerType-Decimal(7,7)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[FloatType-Decimal(18,0)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[LongType-Decimal(7,7)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[DoubleType-Decimal(18,0)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[FloatType-Decimal(7,7)][APPROXIMATE_FLOAT]
07:38:32  FAILED ../../src/main/python/cast_test.py::test_cast_decimal_to[DoubleType-Decimal(7,7)][APPROXIMATE_FLOAT]

[abellina]

@revans2 fyi

[abellina]

07:53:51  e = IllegalArgumentException('Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec\nProject [...:79)\n\tat py4j.GatewayConnection.run(GatewayConnection.java:251)\n\tat java.lang.Thread.run(Thread.java:748)\n', None)
07:53:51  
07:53:51  >   ???
07:53:51  E   pyspark.sql.utils.IllegalArgumentException: Part of the plan is not columnar class org.apache.spark.sql.execution.ProjectExec
07:53:51  E   Project [cast(a#73593 as int) AS a#73595, a#73593]
07:53:51  E   +- Scan ExistingRDD[a#73593]

[revans2]

I just ran it manually on databricks 7.3 and it all passed. Will try to look at the nightly build and see if something else is happening.

[revans2]

The logs don't show anything being built, so the only thing I can think of is that there must have been a mismatch between the jars that were uploaded and the tests that were checked out of git. The tests were newer but the jars were older so the tests failed.

[tgravescs]

lets see what todays build does, the builds were a bit messed up yesterday

[tgravescs]

the last 4 builds have worked fine since this, closing