Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for Spark 2.x Explain Api #4529

Merged
merged 96 commits into from
Jan 20, 2022
Merged

Add support for Spark 2.x Explain Api #4529

merged 96 commits into from
Jan 20, 2022

Commits on Dec 1, 2021

  1. Add an explain only mode configuration to run plugin on CPU to get wh… 

    …at would have run on GPU

Signed-off-by: Thomas Graves <tgraves@apache.org>
    @tgravescs
    tgravescs committed Dec 1, 2021
    Configuration menu
    Copy the full SHA
    8eb81f6 View commit details
    Browse the repository at this point in the history

  2. Updates

    @tgravescs
    tgravescs committed Dec 1, 2021
    Configuration menu
    Copy the full SHA
    1fca5ef View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2021

  1. Don't allocate gpu or enable shuffle for explain only mode

    @tgravescs
    tgravescs committed Dec 2, 2021
    Configuration menu
    Copy the full SHA
    f043f86 View commit details
    Browse the repository at this point in the history

  2. explain only mode check for rapids shuffle internal manager

    @tgravescs
    tgravescs committed Dec 2, 2021
    Configuration menu
    Copy the full SHA
    431426c View commit details
    Browse the repository at this point in the history

  3. update doc

    @tgravescs
    tgravescs committed Dec 2, 2021
    Configuration menu
    Copy the full SHA
    3541832 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2021

  1. Change how we check explain with sql enabled

    @tgravescs
    tgravescs committed Dec 3, 2021
    Configuration menu
    Copy the full SHA
    6ec51d0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'explainonlymode' of github.com:tgravescs/spark-rapids i… 

    …nto explainonlymode
    @tgravescs
    tgravescs committed Dec 3, 2021
    Configuration menu
    Copy the full SHA
    21a79ca View commit details
    Browse the repository at this point in the history

  3. update not work message:

    @tgravescs
    tgravescs committed Dec 3, 2021
    Configuration menu
    Copy the full SHA
    b8e9e26 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/branch-22.02' into explainonlymode

    @tgravescs
    tgravescs committed Dec 3, 2021
    Configuration menu
    Copy the full SHA
    c15b786 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2021

  1. fix spacing

    @tgravescs
    tgravescs committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    ba3b355 View commit details
    Browse the repository at this point in the history

  2. Update doc adding explain option

    @tgravescs
    tgravescs committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    42aa80f View commit details
    Browse the repository at this point in the history

  3. update docs

    @tgravescs
    tgravescs committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    278d96c View commit details
    Browse the repository at this point in the history

  4. Add explian only mode test to make sure runs on cpu

    @tgravescs
    tgravescs committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    2a3b228 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2021

  1. add note about adaptive

    @tgravescs
    tgravescs committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    3bdf4f2 View commit details
    Browse the repository at this point in the history

  2. get rest of broadcast shims

    @tgravescs
    tgravescs committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    4b88718 View commit details
    Browse the repository at this point in the history

  3. Update logging of enabled and explain only mode

    @tgravescs
    tgravescs committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    8f9301a View commit details
    Browse the repository at this point in the history

  4. update config docs

    @tgravescs
    tgravescs committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    d79c9a4 View commit details
    Browse the repository at this point in the history

Commits on Jan 10, 2022

  1. Add spark2 module that supports the Explain only api 

    Signed-off-by: Thomas Graves <tgraves@nvidia.com>
    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    ca880f1 View commit details
    Browse the repository at this point in the history

  2. copyrights

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    c961f12 View commit details
    Browse the repository at this point in the history

  3. imports

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    1e5b1ab View commit details
    Browse the repository at this point in the history

  4. Revert "imports" 

    This reverts commit 1e5b1ab.
    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    b0f815c View commit details
    Browse the repository at this point in the history

  5. imports

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    25b1a0a View commit details
    Browse the repository at this point in the history

  6. cleanup imports

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    7f5e4bb View commit details
    Browse the repository at this point in the history

  7. cleanup imports datasource

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    5cc57dc View commit details
    Browse the repository at this point in the history

  8. remove cudf as dependency

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    d981107 View commit details
    Browse the repository at this point in the history

  9. upmerge joins to latest

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    645c39b View commit details
    Browse the repository at this point in the history

  10. more upmerge and copyrights

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    879c005 View commit details
    Browse the repository at this point in the history

  11. upmerge and diffed overrides

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    aed2d13 View commit details
    Browse the repository at this point in the history

  12. shim Sequence not in 2.3:

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    352c090 View commit details
    Browse the repository at this point in the history

  13. update copyrights and update RapidsConf

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    6e5ea9a View commit details
    Browse the repository at this point in the history

  14. finish copyright updates

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    bc4eede View commit details
    Browse the repository at this point in the history

  15. building 248 and 232

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    eda55f2 View commit details
    Browse the repository at this point in the history

  16. add pom file

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    ddddf35 View commit details
    Browse the repository at this point in the history

  17. fix style issues

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    c0346e8 View commit details
    Browse the repository at this point in the history

  18. fix style

    @tgravescs
    tgravescs committed Jan 10, 2022
    Configuration menu
    Copy the full SHA
    36396f9 View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2022

  1. remove shim layers

    @tgravescs
    tgravescs committed Jan 11, 2022
    Configuration menu
    Copy the full SHA
    7282c0c View commit details
    Browse the repository at this point in the history

  2. update pom description

    @tgravescs
    tgravescs committed Jan 11, 2022
    Configuration menu
    Copy the full SHA
    03610be View commit details
    Browse the repository at this point in the history

  3. Add CheckOverflow

    @tgravescs
    tgravescs committed Jan 11, 2022
    Configuration menu
    Copy the full SHA
    84fd72f View commit details
    Browse the repository at this point in the history

  4. fix line length

    @tgravescs
    tgravescs committed Jan 11, 2022
    Configuration menu
    Copy the full SHA
    70ff0bf View commit details
    Browse the repository at this point in the history

  5. fix copyright

    @tgravescs
    tgravescs committed Jan 11, 2022
    Configuration menu
    Copy the full SHA
    233b259 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2022

  1. Merge remote-tracking branch 'origin/branch-22.02' into explainonlymode

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    6402631 View commit details
    Browse the repository at this point in the history

  2. cleanup

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    13d7ee1 View commit details
    Browse the repository at this point in the history

  3. Update to use the spark.rapids.sql.mode config 

    Signed-off-by: Thomas Graves <tgraves@nvidia.com>
    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    ab49521 View commit details
    Browse the repository at this point in the history

  4. update config doc and formatting

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    ce38223 View commit details
    Browse the repository at this point in the history

  5. update docs

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    b1dd70e View commit details
    Browse the repository at this point in the history

  6. fix typo

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    c94d732 View commit details
    Browse the repository at this point in the history

  7. update configs.md

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    0003e4e View commit details
    Browse the repository at this point in the history

  8. change pom to 24x

    @tgravescs
    tgravescs committed Jan 12, 2022
    Configuration menu
    Copy the full SHA
    241f62c View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2022

  1. Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se… 

    …p-module-upmerge
    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    6ee3c0c View commit details
    Browse the repository at this point in the history

  2. Upmerge some files 

    Signed-off-by: Thomas Graves <tgraves@nvidia.com>
    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    bb87652 View commit details
    Browse the repository at this point in the history

  3. change name of jar

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    1340d9a View commit details
    Browse the repository at this point in the history

  4. Change to check for isSqlEnabled and the mode separately because we may 

    want to initialize stuff on startup with the plugin disabled and
dynamically enable it afterwards
    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    77c82a2 View commit details
    Browse the repository at this point in the history

  5. fix spacing

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    c02dae5 View commit details
    Browse the repository at this point in the history

  6. update auto generated configs doc

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    2afeb2b View commit details
    Browse the repository at this point in the history

  7. Add ExplainPlan

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    4c98c6e View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'tgravescs/explainonlymode' into spark2s… 

    …him-sep-module-upmerge
    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    f5dc619 View commit details
    Browse the repository at this point in the history

  9. Update docs for 2.x explain api

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    1eca823 View commit details
    Browse the repository at this point in the history

  10. cleanup

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    ac9049a View commit details
    Browse the repository at this point in the history

  11. update build scripts to build 24x

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    6c06b31 View commit details
    Browse the repository at this point in the history

  12. remove unneeded tag

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    1dc83df View commit details
    Browse the repository at this point in the history

  13. start diff scripts

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    48ae9fd View commit details
    Browse the repository at this point in the history

  14. checkpoint

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    85088fa View commit details
    Browse the repository at this point in the history

  15. checkpoint

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    935c25e View commit details
    Browse the repository at this point in the history

  16. checkpoint

    @tgravescs
    tgravescs committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    e7c7a0b View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2022

  1. checkpoint

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    cf55dc6 View commit details
    Browse the repository at this point in the history

  2. checkpoint

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    359c306 View commit details
    Browse the repository at this point in the history

  3. checkpoint

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    5b1d40a View commit details
    Browse the repository at this point in the history

  4. add more

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    cbf62a5 View commit details
    Browse the repository at this point in the history

  5. checkpoint

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    126919a View commit details
    Browse the repository at this point in the history

  6. more diffs done

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    75edb2d View commit details
    Browse the repository at this point in the history

  7. more diffs

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    0db3a3d View commit details
    Browse the repository at this point in the history

  8. diff files

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    8bb8291 View commit details
    Browse the repository at this point in the history

  9. split apart some functions for easier diff

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    b7aa8c7 View commit details
    Browse the repository at this point in the history

  10. Overrides diffs and split apart for easier diffing

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    2dc5251 View commit details
    Browse the repository at this point in the history

  11. more diffs and move

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    8b5d720 View commit details
    Browse the repository at this point in the history

  12. update script to tmp dir

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    8907fac View commit details
    Browse the repository at this point in the history

  13. builds

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    abd12b0 View commit details
    Browse the repository at this point in the history

  14. Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se… 

    …p-module-upmerge-compare
    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    7437f30 View commit details
    Browse the repository at this point in the history

  15. Upmerge pull changes

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    8b40694 View commit details
    Browse the repository at this point in the history

  16. update diff file

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    de98f8b View commit details
    Browse the repository at this point in the history

  17. Finish diffing the rest of execs, split one out to another file for 

    easier diffing
    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    9933cdd View commit details
    Browse the repository at this point in the history

  18. Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se… 

    …p-module-upmerge-compare
    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    82d5818 View commit details
    Browse the repository at this point in the history

  19. Update spark2 RapidsConf with changes in sql-plugin

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    b912fc6 View commit details
    Browse the repository at this point in the history

  20. Update RapidsConf to deal with toBytes different 2.x

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    9116768 View commit details
    Browse the repository at this point in the history

  21. cleanup documentation

    @tgravescs
    tgravescs committed Jan 14, 2022
    Configuration menu
    Copy the full SHA
    52447c8 View commit details
    Browse the repository at this point in the history

Commits on Jan 18, 2022

  1. Update docs/get-started/getting-started-workload-qualification.md 

    Co-authored-by: Jason Lowe <jlowe@nvidia.com>
    @tgravescs @jlowe
    tgravescs and jlowe authored Jan 18, 2022
    Configuration menu
    Copy the full SHA
    f3c2636 View commit details
    Browse the repository at this point in the history

  2. copyright, remove 2.3 from DateUtils, fix up some comments and remove 

    commented out code and some uneeded AQE checks
    @tgravescs
    tgravescs committed Jan 18, 2022
    Configuration menu
    Copy the full SHA
    f1c55d2 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'spark2shim-sep-module-upmerge' of github.com:tgravescs/… 

    …spark-rapids into spark2shim-sep-module-upmerge
    @tgravescs
    tgravescs committed Jan 18, 2022
    Configuration menu
    Copy the full SHA
    3bfde90 View commit details
    Browse the repository at this point in the history

  4. Update RapidsMeta comment and diff

    @tgravescs
    tgravescs committed Jan 18, 2022
    Configuration menu
    Copy the full SHA
    6f190a6 View commit details
    Browse the repository at this point in the history

  5. Update 2.x to sql-plugin diffs

    @tgravescs
    tgravescs committed Jan 18, 2022
    Configuration menu
    Copy the full SHA
    08b2490 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2022

  1. add deploy of new jar for nightly

    @tgravescs
    tgravescs committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    b724245 View commit details
    Browse the repository at this point in the history

  2. Remove unneeded dependencies from pom file, update NOTICE pulled, remove 

    extra imports, don't build tests jar if empty
    @tgravescs
    tgravescs committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    33da75b View commit details
    Browse the repository at this point in the history

  3. remove python files

    @tgravescs
    tgravescs committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    0bad0cc View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/branch-22.02' into spark2shim-se… 

    …p-module-upmerge
    @tgravescs
    tgravescs committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    f532d81 View commit details
    Browse the repository at this point in the history

  5. Upmerged to latest sql-plugin code

    @tgravescs
    tgravescs committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    008d88b View commit details
    Browse the repository at this point in the history

  6. update NOTICE copy and remove transient

    @tgravescs
    tgravescs committed Jan 19, 2022
    Configuration menu
    Copy the full SHA
    cc97619 View commit details
    Browse the repository at this point in the history