-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update first/last tests to avoid non-determinisim and ordering differences #933
Conversation
…ences Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
build |
@@ -134,6 +146,8 @@ def idfn(val): | |||
'spark.rapids.sql.hasNans': 'false', | |||
'spark.rapids.sql.castStringToFloat.enabled': 'true', | |||
'spark.rapids.sql.castFloatToString.enabled': 'true', | |||
# some of the first/last tests need a single partition to work reliably when run on a large cluster. | |||
'spark.sql.shuffle.partitions': '1' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would it be better to split the first/last test to use different confs so we can test others with multiple partitions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure simple enough to do.
build |
…ences (NVIDIA#933) Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
…ences (NVIDIA#933) Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
…ences (NVIDIA#933) Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>
…IDIA#933) Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
This fixes the majority of #812 but it does not fix all of the issues. I'll try to address them in separate PRs as I debug them further.