Removed limit from the test #5033
Conversation
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
|
build |
| @@ -122,7 +122,7 @@ def test_cache_reverse_order(enable_vectorized_conf): | |||
| col1 = StructGen([['child0', byte_gen]]) | |||
| def partial_return(): | |||
| def partial_return_cache(spark): | |||
| return two_col_df(spark, col0, col1).select(f.col("a"), f.col("b")).cache().limit(50).select(f.col("b"), f.col("a")) | |||
| return two_col_df(spark, col0, col1).select(f.col("a"), f.col("b")).cache().select(f.col("b"), f.col("a")) | |||
There was a problem hiding this comment.
Is the order really guaranteed without limit? Adding @ignore_order test marker looks like a more intuitive solution to me
There was a problem hiding this comment.
That will only sort the collected list AFAIK which by the time it comes back to the driver is too late, no?
There was a problem hiding this comment.
Or are you asking to add ignore_order in addition to the change? I can do that
There was a problem hiding this comment.
You are right about the local sort, but you can also say @ignore_order(local=False)
There was a problem hiding this comment.
and you are right about removing limit, since this ends up just sorting a non-deterministic subset instead of taking the head of a sorted list.
There was a problem hiding this comment.
So, I would expect that to work but setting local=False still applies sort after the limit resulting in the test to fail.
As discussed offline, I will remove the limit and add ignore_order
| @@ -178,7 +178,7 @@ else | |||
| export PYSP_TEST_spark_driver_extraClassPath="${ALL_JARS// /:}" | |||
| export PYSP_TEST_spark_executor_extraClassPath="${ALL_JARS// /:}" | |||
| export PYSP_TEST_spark_driver_extraJavaOptions="-ea -Duser.timezone=UTC $COVERAGE_SUBMIT_FLAGS" | |||
| export PYSP_TEST_spark_executor_extraJavaOptions='-ea -Duser.timezone=UTC' | |||
| export PYSP_TEST_spark_executor_extraJavaOptions="-ea -Duser.timezone=UTC $COVERAGE_SUBMIT_FLAGS_EXEC" | |||
There was a problem hiding this comment.
We have an issue for this: #4948 . It should be a separate PR.
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
|
build |
The order isn't guaranteed when using limit which causes the test to fail intermittently
ignore_orderfixes #5021
Signed-off-by: Raza Jafri rjafri@nvidia.com