Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fully support date/time legacy rebase for nested input [databricks] #9660

Merged
merged 78 commits into from
Nov 16, 2023
Merged

Fully support date/time legacy rebase for nested input [databricks] #9660

merged 78 commits into from
Nov 16, 2023

Commits on Aug 28, 2023

  1. Add check for nested types

    @ttnghia
    ttnghia committed Aug 28, 2023
    Configuration menu
    Copy the full SHA
    c578a64 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. Add check for nested types

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    e368aa6 View commit details
    Browse the repository at this point in the history

  2. Recursively check for rebasing

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    7da416b View commit details
    Browse the repository at this point in the history

  3. Extract common code

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    df8f861 View commit details
    Browse the repository at this point in the history

  4. Allow nested type in rebase check

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    95d19ee View commit details
    Browse the repository at this point in the history

  5. Enable nested timestamp in roundtrip test

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    b426610 View commit details
    Browse the repository at this point in the history

  6. Fix another test 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    7343b17 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'check_rebase_nested' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    0d48f57 View commit details
    Browse the repository at this point in the history

  8. Enable LEGACY rebase in read

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    024e6c9 View commit details
    Browse the repository at this point in the history

  9. Remove comment

    @ttnghia
    ttnghia committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    9a39628 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2023

  1. Change function/class signatures

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    e686bb0 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-23.12' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    b49963e View commit details
    Browse the repository at this point in the history

  3. Complete modification

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    2c232f8 View commit details
    Browse the repository at this point in the history

  4. Misc 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    ac0f3e4 View commit details
    Browse the repository at this point in the history

  5. Add explicit type 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    c773794 View commit details
    Browse the repository at this point in the history

  6. Rename file and add some stuff in DateTimeRebaseHelpers.scala

    @ttnghia
    ttnghia committed Nov 3, 2023
    Configuration menu
    Copy the full SHA
    29df7cd View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2023

  1. Move file and rename class

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    1b5112d View commit details
    Browse the repository at this point in the history

  2. Adopt new enum type 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    63342a9 View commit details
    Browse the repository at this point in the history

  3. Add name for the enum classes

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    6b2d795 View commit details
    Browse the repository at this point in the history

  4. Change exception messages

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    37aa40b View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-23.12' into refactor_parquet_scan

    @ttnghia
    ttnghia committed Nov 4, 2023
    Configuration menu
    Copy the full SHA
    d4cdc1b View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2023

  1. Does not yet support legacy rebase in read 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    03f681e View commit details
    Browse the repository at this point in the history

  2. Change legacy to corrected mode 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    14f230f View commit details
    Browse the repository at this point in the history

  3. Extract common code 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    1b464ec View commit details
    Browse the repository at this point in the history

  4. Rename functions 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    0d26d97 View commit details
    Browse the repository at this point in the history

  5. Reformat 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    c2504fd View commit details
    Browse the repository at this point in the history

  6. Make classes serializable 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 5, 2023
    Configuration menu
    Copy the full SHA
    edb6c81 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2023

  1. Revert "Support rebase checking for nested dates and timestamps (NVID… 

    …IA#9617)"

This reverts commit 401d0d8.

Signed-off-by: Nghia Truong <nghiat@nvidia.com>

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    ea86e8f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'refactor_parquet_scan' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    b14463f View commit details
    Browse the repository at this point in the history

  3. Implement date time rebase

    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    adc8ae2 View commit details
    Browse the repository at this point in the history

  4. Optimize rebase op

    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    791573c View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-23.12' into refactor_parquet_scan 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/RebaseHelper.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    54e959f View commit details
    Browse the repository at this point in the history

  6. Change comment 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    3f01690 View commit details
    Browse the repository at this point in the history

  7. Merge branch 'refactor_parquet_scan' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    6d9c20b View commit details
    Browse the repository at this point in the history

  8. Move tests

    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    8c63273 View commit details
    Browse the repository at this point in the history

  9. Add test for datatime rebase 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    1b1fdc3 View commit details
    Browse the repository at this point in the history

  10. Various changes 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    e6559ce View commit details
    Browse the repository at this point in the history

  11. Various changes 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>

# Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    74fe84a View commit details
    Browse the repository at this point in the history

  12. Fix compile errors 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    a455a90 View commit details
    Browse the repository at this point in the history

  13. Fix comments 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    b87493c View commit details
    Browse the repository at this point in the history

  14. Fix indentations 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    321e516 View commit details
    Browse the repository at this point in the history

  15. Merge branch 'refactor_parquet_scan' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/datetimeRebaseUtils.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    4bc33be View commit details
    Browse the repository at this point in the history

  16. Change comments and indentations 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    4aab36b View commit details
    Browse the repository at this point in the history

  17. Merge branch 'rebase_datatime' into rebase_nested_timestamp 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/datetimeRebaseUtils.scala
    @ttnghia
    ttnghia committed Nov 6, 2023
    Configuration menu
    Copy the full SHA
    1b4744a View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2023

  1. Allow nested check for rebase

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    70310db View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-23.12' into rebase_datatime 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    c615925 View commit details
    Browse the repository at this point in the history

  3. Write different timestamp types in test

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    be92368 View commit details
    Browse the repository at this point in the history

  4. Fix conversion if timestamp is not micros

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    b09c61f View commit details
    Browse the repository at this point in the history

  5. Rename var

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    00d96e4 View commit details
    Browse the repository at this point in the history

  6. Dont have to down cast after up cast 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    7d81311 View commit details
    Browse the repository at this point in the history

  7. Change comment 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    116bf3e View commit details
    Browse the repository at this point in the history

  8. Still cast timestamp to the old type after rebasing 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    273b2c4 View commit details
    Browse the repository at this point in the history

  9. Rename test 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    996d9d4 View commit details
    Browse the repository at this point in the history

  10. Should not transform non-datetime types 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    5fd6ef5 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'rebase_datatime' into rebase_nested_timestamp 

    # Conflicts:
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    d53ecfa View commit details
    Browse the repository at this point in the history

  12. Fix test

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    4144655 View commit details
    Browse the repository at this point in the history

  13. Update tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    5a8b44c View commit details
    Browse the repository at this point in the history

  14. Merge branch 'rebase_datatime' into rebase_nested_timestamp 

    # Conflicts:
#	integration_tests/src/main/python/parquet_test.py
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    a33bfd6 View commit details
    Browse the repository at this point in the history

  15. Enable int96 rebase in write

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    e366e5a View commit details
    Browse the repository at this point in the history

  16. Change tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    247f47f View commit details
    Browse the repository at this point in the history

  17. Complete tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    8eba053 View commit details
    Browse the repository at this point in the history

  18. Revert unrelated changes 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    bda59ef View commit details
    Browse the repository at this point in the history

  19. Merge branch 'branch-23.12' into int96_rebase_write

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    bbcd9d9 View commit details
    Browse the repository at this point in the history

  20. Merge branch 'branch-23.12' into rebase_datatime

    @ttnghia
    ttnghia committed Nov 7, 2023
    Configuration menu
    Copy the full SHA
    fbe37d7 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2023

  1. Change configs 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    4a92d54 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'rebase_datatime' into rebase_nested_timestamp 

    # Conflicts:
#	integration_tests/src/main/python/parquet_test.py
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    54c53d3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'int96_rebase_write' into rebase_nested_timestamp 

    # Conflicts:
#	integration_tests/src/main/python/parquet_test.py
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    2f30ce9 View commit details
    Browse the repository at this point in the history

  4. Merge tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    af817de View commit details
    Browse the repository at this point in the history

  5. Simplify test data 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    13242f4 View commit details
    Browse the repository at this point in the history

  6. Add a new write test 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    e1d9f74 View commit details
    Browse the repository at this point in the history

  7. Add a mixed rebase test 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    82012b6 View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. Merge branch 'branch-23.12' into rebase_nested_timestamp 

    # Conflicts:
#	integration_tests/src/main/python/parquet_test.py
#	integration_tests/src/main/python/parquet_write_test.py
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuParquetScan.scala
#	sql-plugin/src/main/scala/com/nvidia/spark/rapids/datetimeRebaseUtils.scala
    @ttnghia
    ttnghia committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    76694ad View commit details
    Browse the repository at this point in the history

  2. Change tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    cbef912 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-23.12' into rebase_nested_timestamp 

    # Conflicts:
#	integration_tests/src/main/python/parquet_write_test.py
    @ttnghia
    ttnghia committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    1474dda View commit details
    Browse the repository at this point in the history

  4. Fix seed in tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    14487bf View commit details
    Browse the repository at this point in the history

  5. Rename tests 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    0fff5e6 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. Merge branch 'branch-23.12' into rebase_nested_timestamp

    @ttnghia
    ttnghia committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    8bfca59 View commit details
    Browse the repository at this point in the history

  2. Add default seed 

    Signed-off-by: Nghia Truong <nghiat@nvidia.com>
    @ttnghia
    ttnghia committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    61d7d3d View commit details
    Browse the repository at this point in the history