[Spark 4.0] Address test failures in cast_test.py [databricks] #11559
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes NVIDIA#11009. This commit addresses the test failures in cast_test.py, on Spark 4.0. These generally have to do with changes in behaviour of Spark when ANSI mode is enabled. This commit specifically addresses `test_cast_date_integral_and_fp`, `test_cast_decimal_to`, and `test_cast_decimal_to_decimal`.
Signed-off-by: MithunR <mithunr@nvidia.com>
mythrocks
force-pushed
the
spark4-cast_test
branch
from
41ffc5d
to
b7879b6
Compare
gerashegalov
reviewed
Oct 4, 2024
mythrocks
changed the title
|
Build |
|
Build |
|
Build |
gerashegalov
approved these changes
Oct 8, 2024
| """ | ||
| This tests that STRING->DATE conversion is run on CPU, via a fallback. | ||
| The point of this test is to exercise the fallback, and not to examine any errors in casting. | ||
| There is no change in behaviour between Apache Spark and the plugin, since they're both |
There was a problem hiding this comment.
nitpicking but what if the fallback does not happen in the ANSI-enabled mode?
There was a problem hiding this comment.
Once the plugin adds support for string->date casting, we will have to modify this test. At a minimum, there won't be a fallback. I don't foresee a case where we might support ANSI=on and not ANSI=off, but that's hard to predict.
I'd postpone any further futzing with this until the plugin's behaviour changes.
|
I've merged this change. Thank you for the review, @gerashegalov. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11009 and #11530.
This commit addresses the test failures in cast_test.py, on Spark 4.0.
These generally have to do with changes in behaviour of Spark when
ANSI mode is enabled. In these cases, the tests have been split out into ANSI=on and ANSI=off.
The bugs uncovered from the tests have been spun into their own issues; fixing all of them was beyond the scope of this change.