Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark list hashing #11292

Merged
merged 48 commits into from
Jul 28, 2022
Merged

Spark list hashing #11292

merged 48 commits into from
Jul 28, 2022

Commits on Jun 28, 2022

  1. Add test for Spark list-of-list hashing.

    @bdice
    bdice committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    ebf079a View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2022

  1. Improve test.

    @bdice
    bdice committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    5632e7c View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/branch-22.08' into spark-list-… 

    …hashing
    @bdice
    bdice committed Jul 7, 2022
    Configuration menu
    Copy the full SHA
    d721597 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2022

  1. Merge remote-tracking branch 'upstream/branch-22.08' into spark-list-… 

    …hashing
    @bdice
    bdice committed Jul 18, 2022
    Configuration menu
    Copy the full SHA
    aee8fe0 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2022

  1. Copy experimental row hasher for modification.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    6e55198 View commit details
    Browse the repository at this point in the history

  2. Make preprocessed table methods public.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    95f25cb View commit details
    Browse the repository at this point in the history

  3. Use structured binding to make names map more clearly to the objects.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    ac21705 View commit details
    Browse the repository at this point in the history

  4. Add note about hash types, fix tests.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    4106aef View commit details
    Browse the repository at this point in the history

  5. Align hashing behavior closer to Spark. Some tests passing, but not all.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    ca8557d View commit details
    Browse the repository at this point in the history

  6. Remove commented code, fix dispatch for decimal32 and decimal128.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    703956f View commit details
    Browse the repository at this point in the history

  7. Reorder members to match constructor signature.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    a98c130 View commit details
    Browse the repository at this point in the history

  8. Update nested type hashing to act more like the previous serial hash … 

    …code.
    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    659bca9 View commit details
    Browse the repository at this point in the history

  9. Use seed 42 for consistency with Spark.

    @bdice
    bdice committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    fcca18a View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2022

  1. Clean up.

    @bdice
    bdice committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    189f3bf View commit details
    Browse the repository at this point in the history

  2. Fix bug in test (wrong index).

    @bdice
    bdice committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    ee754a7 View commit details
    Browse the repository at this point in the history

  3. Template row_hasher on device_row_hasher class.

    @bdice
    bdice committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    b163b49 View commit details
    Browse the repository at this point in the history

  4. Fix up friends / private methods.

    @bdice
    bdice committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    387390a View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2022

  1. Merge remote-tracking branch 'pub/branch-22.08' into bdice/list_hashing

    @rwlee
    rwlee committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    931bd33 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2022

  1. Enable list types in spark32BitMurmurHash3.

    @bdice
    bdice authored Jul 22, 2022
    Configuration menu
    Copy the full SHA
    2218338 View commit details
    Browse the repository at this point in the history

  2. Remove template, always use Spark hash functor.

    @bdice
    bdice authored Jul 22, 2022
    Configuration menu
    Copy the full SHA
    bad0b5e View commit details
    Browse the repository at this point in the history

  3. Add summary of differences in Spark hashing.

    @bdice
    bdice authored Jul 22, 2022
    Configuration menu
    Copy the full SHA
    2d30d2c View commit details
    Browse the repository at this point in the history

  4. clang-format

    @bdice
    bdice committed Jul 22, 2022
    Configuration menu
    Copy the full SHA
    4fee0f8 View commit details
    Browse the repository at this point in the history

  5. Switch back to template parameter so the device row hasher matches th… 

    …e template template parameter needed upstream.
    @bdice
    bdice committed Jul 22, 2022
    Configuration menu
    Copy the full SHA
    1194723 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2022

  1. Merge remote-tracking branch 'pub/pull-request/11292' into bdice/list… 

    …_hashing
    @rwlee
    rwlee committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    bec00fa View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/branch-22.08' into spark-list-… 

    …hashing
    @bdice
    bdice committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    ee6cfda View commit details
    Browse the repository at this point in the history

  3. Use has_nested_nulls instead of has_nulls.

    @bdice
    bdice committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    873452c View commit details
    Browse the repository at this point in the history

  4. Rename to spark_device_row_hasher.

    @bdice
    bdice committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    2826655 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2022

  1. Fix copyright. 

    Co-authored-by: Vyas Ramasubramani <vyas.ramasubramani@gmail.com>
    @bdice @vyasr
    bdice and vyasr authored Jul 26, 2022
    Configuration menu
    Copy the full SHA
    3e70ac5 View commit details
    Browse the repository at this point in the history

  2. Add test for structs of lists.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    c4586f3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'spark-list-hashing' of github.com:bdice/cudf into spark… 

    …-list-hashing
    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    481ac82 View commit details
    Browse the repository at this point in the history

  4. Add failing test for lists of structs.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    1f75acf View commit details
    Browse the repository at this point in the history

  5. Fail if a table with lists of structs is passed.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    e179a4c View commit details
    Browse the repository at this point in the history

  6. Use spark_hash_value_type.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    f057bc6 View commit details
    Browse the repository at this point in the history

  7. Change template name to DeviceRowHasher.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    929c589 View commit details
    Browse the repository at this point in the history

  8. Merge remote-tracking branch 'pub/pull-request/11292' into bdice/list… 

    …_hashing
    @rwlee
    rwlee committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    64727c9 View commit details
    Browse the repository at this point in the history

  9. Improve comment.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    248cfaa View commit details
    Browse the repository at this point in the history

  10. Rename to spark_murmur_device_row_hasher.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    f90af13 View commit details
    Browse the repository at this point in the history

  11. Use spark_hash_value_type more consistently, and enforce that null ha… 

    …sh values are always the seed.
    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    4a80876 View commit details
    Browse the repository at this point in the history

  12. Added developer documentation.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    5be3ea0 View commit details
    Browse the repository at this point in the history

  13. Use list initialization.

    @bdice
    bdice committed Jul 26, 2022
    Configuration menu
    Copy the full SHA
    cf6f6cd View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2022

  1. add list test

    @rwlee
    rwlee committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    6475517 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'pub/pull-request/11292' into bdice/list… 

    …_hashing
    @rwlee
    rwlee committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    0c0a1fb View commit details
    Browse the repository at this point in the history

  3. nested struct java test

    @rwlee
    rwlee committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    028b0bc View commit details
    Browse the repository at this point in the history

  4. Remove deleted constructor.

    @bdice
    bdice committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    5d05984 View commit details
    Browse the repository at this point in the history

  5. Require SparkMurmurHash3_32 in constructor.

    @bdice
    bdice committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    9281aee View commit details
    Browse the repository at this point in the history

  6. Remove deleted public constructor because the constructor is declared… 

    … to be private.
    @bdice
    bdice committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    fa50d2c View commit details
    Browse the repository at this point in the history

  7. Template only the device_hasher method.

    @bdice
    bdice committed Jul 27, 2022
    Configuration menu
    Copy the full SHA
    ef0ad3b View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2022

  1. Temporarily remove JNI tests due to an outstanding allocation bug.

    @bdice
    bdice authored Jul 28, 2022
    Configuration menu
    Copy the full SHA
    aaadac2 View commit details
    Browse the repository at this point in the history