fix flaky array_item test failures #11054
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
|
build |
thirtiseven
reviewed
Jun 13, 2024
Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
|
build |
thirtiseven
reviewed
Jun 13, 2024
| @@ -224,7 +224,11 @@ def test_all_null_int96(spark_tmp_path): | |||
| class AllNullTimestampGen(TimestampGen): | |||
| def start(self, rand): | |||
| self._start(rand, lambda : None) | |||
| data_path = spark_tmp_path + '/PARQUET_DATA' | |||
|
|
|||
| def _cache_repr(self): | |||
There was a problem hiding this comment.
So we need to add a _cache_repr, even if it will only be used once, to prevent a key conflict in the cache. Didn't realize that, nice catch.
thirtiseven
previously approved these changes
Jun 13, 2024
Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
|
build |
revans2
approved these changes
Jun 13, 2024
SurajAralihalli
pushed a commit
to SurajAralihalli/spark-rapids
that referenced
this pull request
Jul 12, 2024
* fix flaky array_item test failures Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * fix indent Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> * fix whitespace Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org> --------- Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR closes #8652
Currently, the cache key for ArrayGen (as well as some other Gens) is incomplete, which will lead to unexpected input being reused.
For the test case described in the issue, it's veriying GetArrayElement(array, index) will throw an exception when index is a invalid value(like negative values). However, if this test case is preceded by another test case which will populate cache with an array with all nulls, GetArrayElement will return null regardless of the index value (Thus not throwing exceptions).
The solution is to refine cache key for all of the XXGen.