Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Fix Spark 3.1.0 integration tests #416

Closed
tgravescs opened this issue Jul 23, 2020 · 10 comments · Fixed by #580
Closed

[BUG] Fix Spark 3.1.0 integration tests #416

tgravescs opened this issue Jul 23, 2020 · 10 comments · Fixed by #580
Assignees
@tgravescs
Labels
bug Something isn't working P0 Must have for release
Milestone
Aug 17 - Aug 28

Comments

@tgravescs
Copy link
Collaborator

Depends on #414

There are some integration tests failing against 3.1.0 and we just need to investigate and fix those.
@tgravescs tgravescs added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jul 23, 2020
@tgravescs tgravescs mentioned this issue Jul 23, 2020
Merged
@tgravescs
Copy link
Collaborator Author

see failures https://ci.ngcc.nvidia.com/job/spark/job/rapids_it-3.1.0-SNAPSHOT-0.2-github/2/console

@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jul 28, 2020
@sameerz sameerz added the P0 Must have for release label Jul 28, 2020
@sameerz
Copy link
Collaborator

sameerz commented Jul 28, 2020

Issue #382 is one of the tests that is failing.

@sameerz sameerz added this to the Aug 3 - Aug 14 milestone Aug 4, 2020
@tgravescs
Copy link
Collaborator Author

tests currently failing:

FAILED integration_tests/src/main/python/arithmetic_ops_test.py::test_log[Double][APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/arithmetic_ops_test.py::test_log1p[Double][APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/arithmetic_ops_test.py::test_log2[Double][APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/arithmetic_ops_test.py::test_log10[Double][APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(-584, 1563)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(1943, 1101)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(2693, 2167)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(2729, 0)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(44, 1534)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(2635, 3319)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(1885, -2828)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(0, 2463)]
FAILED integration_tests/src/main/python/date_time_test.py::test_timesub[(932, 2286)]
FAILED integration_tests/src/main/python/join_test.py::test_broadcast_join_mixed[FullOuter][IGNORE_ORDER]
FAILED integration_tests/src/main/python/orc_test.py::test_input_meta - py4j....
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.byteF, test_table1.byteF FULL JOIN test_table1 ON test_table.byteF=test_table1.byteF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.shortF, test_table1.shortF FULL JOIN test_table1 ON test_table.shortF=test_table1.shortF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.intF, test_table1.intF FULL JOIN test_table1 ON test_table.intF=test_table1.intF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.longF, test_table1.longF FULL JOIN test_table1 ON test_table.longF=test_table1.longF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.floatF, test_table1.floatF FULL JOIN test_table1 ON test_table.floatF=test_table1.floatF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.doubleF, test_table1.doubleF FULL JOIN test_table1 ON test_table.doubleF=test_table1.doubleF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.booleanF, test_table1.booleanF FULL JOIN test_table1 ON test_table.booleanF=test_table1.booleanF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.strF, test_table1.strF FULL JOIN test_table1 ON test_table.strF=test_table1.strF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.dateF, test_table1.dateF FULL JOIN test_table1 ON test_table.dateF=test_table1.dateF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
FAILED integration_tests/src/main/python/qa_nightly_select_test.py::test_select_join[test_table.timestampF, test_table1.timestampF FULL JOIN test_table1 ON test_table.timestampF=test_table1.timestampF][IGNORE_ORDER({'local': True}), INCOMPAT, APPROXIMATE_FLOAT]
= 25 failed, 2211 passed, 168 skipped, 49 xfailed, 1 xpassed, 10 warnings in 2780.07s (0:46:20) =

@tgravescs
Copy link
Collaborator Author

https://issues.apache.org/jira/browse/SPARK-32640 filed for arithmetic ops

@tgravescs
Copy link
Collaborator Author

#574 filed for time sub functions

@tgravescs
Copy link
Collaborator Author

join test failure: #575

@tgravescs
Copy link
Collaborator Author

orc test failure: #576

@tgravescs
Copy link
Collaborator Author

#577 for arithmetic test failures

@tgravescs
Copy link
Collaborator Author

#578 is for the full outer join in qa test

@tgravescs
Copy link
Collaborator Author

so with this I'm xfailing all the failing tests above and we have specific issues to followup with each.

@tgravescs tgravescs mentioned this issue Aug 18, 2020
Merged
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@pxLi
Init 22.10.0-SNAPSHOT (NVIDIA#416)
830ddb6 
* Init 22.10.0-SNAPSHOT

Signed-off-by: Peixin Li <pxli@nyu.edu>

* fix spark-rapids build url
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tgravescs tgravescs

Labels
bug Something isn't working P0 Must have for release
Projects
None yet
Milestone
Aug 17 - Aug 28
Development

Successfully merging a pull request may close this issue.

xfail the Spark 3.1.0 integration tests that fail tgravescs/spark-rapids
2 participants
@tgravescs @sameerz