Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Override the seed for test_map_scalars_supported_key_types for version of Spark before 3.4.0 [Databricks] #9949

Merged
merged 4 commits into from
Dec 5, 2023
Merged

Override the seed for test_map_scalars_supported_key_types for version of Spark before 3.4.0 [Databricks] #9949

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ea540d7
Override the seed for version of Spark before 3.4.0
razajafri Dec 4, 2023
0d6f92d
Signing off
razajafri Dec 4, 2023
3487774
addressed review comments
razajafri Dec 5, 2023
a32a6b3
made seed param mandatory
razajafri Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion integration_tests/src/main/python/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,9 @@ def pytest_runtest_setup(item):
_inject_oom = item.get_closest_marker('inject_oom')
datagen_overrides = item.get_closest_marker('datagen_overrides')
if datagen_overrides:
_test_datagen_random_seed = datagen_overrides.kwargs.get('seed', _test_datagen_random_seed)
override_seed = datagen_overrides.kwargs.get('condition', True)
if override_seed:
_test_datagen_random_seed = datagen_overrides.kwargs.get('seed', _test_datagen_random_seed)
jlowe marked this conversation as resolved.
Show resolved Hide resolved

order = item.get_closest_marker('ignore_order')
if order:
Expand Down
4 changes: 3 additions & 1 deletion integration_tests/src/main/python/map_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,9 @@ def query_map_scalar(spark):


@allow_non_gpu('WindowLocalExec')
@datagen_overrides(seed=0, reason='https://github.com/NVIDIA/spark-rapids/issues/9683')
@datagen_overrides(condition=is_before_spark_314()
or (not is_before_spark_320() and is_before_spark_323())
or (not is_before_spark_330() and is_before_spark_331()), seed=0, reason="https://issues.apache.org/jira/browse/SPARK-40089")
@pytest.mark.parametrize('data_gen', supported_key_map_gens, ids=idfn)
@pytest.mark.xfail(condition = is_not_utc(), reason = 'xfail non-UTC time zone tests because of https://github.com/NVIDIA/spark-rapids/issues/9653')
def test_map_scalars_supported_key_types(data_gen):
Expand Down
6 changes: 6 additions & 0 deletions integration_tests/src/main/python/spark_session.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,18 @@ def is_before_spark_312():
def is_before_spark_313():
return spark_version() < "3.1.3"

def is_before_spark_314():
return spark_version() < "3.1.4"

def is_before_spark_320():
return spark_version() < "3.2.0"

def is_before_spark_322():
return spark_version() < "3.2.2"

def is_before_spark_323():
return spark_version() < "3.2.3"

def is_before_spark_330():
return spark_version() < "3.3.0"

Expand Down