Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] POC add data total size information to Parquet file metadata #12974

Closed
wants to merge 58 commits into from
Closed

[WIP] POC add data total size information to Parquet file metadata #12974

wants to merge 58 commits into from

Commits on Mar 10, 2023

  1. add raw data size info to column chunk metadata

    @etseidl
    etseidl committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    ffcf963 View commit details
    Browse the repository at this point in the history

  2. remove prints, calculate page sizes too

    @etseidl
    etseidl committed Mar 10, 2023
    Configuration menu
    Copy the full SHA
    54249e5 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 10, 2023
    Configuration menu
    Copy the full SHA
    3212720 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 12, 2023
    Configuration menu
    Copy the full SHA
    66e420e View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 14, 2023
    Configuration menu
    Copy the full SHA
    066a0da View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 15, 2023
    Configuration menu
    Copy the full SHA
    ccee4f6 View commit details
    Browse the repository at this point in the history

  2. add serialized structure to footer

    @etseidl
    etseidl committed Mar 15, 2023
    Configuration menu
    Copy the full SHA
    7024534 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2023

  1. read back page indexes and size info and save in metadata

    @etseidl
    etseidl committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    d75d53c View commit details
    Browse the repository at this point in the history

  2. don't read indexes if not doing chunking...update copyright

    @etseidl
    etseidl committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    a424425 View commit details
    Browse the repository at this point in the history

  3. calculate splits with metadata

    @etseidl
    etseidl committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    372529b View commit details
    Browse the repository at this point in the history

  4. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 16, 2023
    Configuration menu
    Copy the full SHA
    c59013b View commit details
    Browse the repository at this point in the history

  5. check for empty offset index

    @etseidl
    etseidl committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    be492fa View commit details
    Browse the repository at this point in the history

  6. do not clear chunk_read_limit if size stats are not present

    @etseidl
    etseidl committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    ab61586 View commit details
    Browse the repository at this point in the history

  7. fudge sizes for now

    @etseidl
    etseidl committed Mar 16, 2023
    Configuration menu
    Copy the full SHA
    501a209 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 17, 2023
    Configuration menu
    Copy the full SHA
    b3741c5 View commit details
    Browse the repository at this point in the history

  2. get rid of sizes with overhead

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    5e3608c View commit details
    Browse the repository at this point in the history

  3. add writer for vector<int64_t>

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    2b77f22 View commit details
    Browse the repository at this point in the history

  4. only do size metadata if also doing column indexes

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    5c7480e View commit details
    Browse the repository at this point in the history

  5. get rid of host_pages vector, add some TODOs

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    cdf8abe View commit details
    Browse the repository at this point in the history

  6. use constexpr for KV keys

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    3ea748b View commit details
    Browse the repository at this point in the history

  7. checkpoint before refactor

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    9f0e9ec View commit details
    Browse the repository at this point in the history

  8. refactor to reuse compute_splits

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    5fc7ace View commit details
    Browse the repository at this point in the history

  9. cleanup

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    0096169 View commit details
    Browse the repository at this point in the history

  10. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 17, 2023
    Configuration menu
    Copy the full SHA
    81981ca View commit details
    Browse the repository at this point in the history

  11. post refactor cleanup

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    a9bf41b View commit details
    Browse the repository at this point in the history

  12. fix writer::impl::close

    @etseidl
    etseidl committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    8eca6f4 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 20, 2023
    Configuration menu
    Copy the full SHA
    6de2552 View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 21, 2023
    Configuration menu
    Copy the full SHA
    79b4319 View commit details
    Browse the repository at this point in the history

  2. clean up copy paste cruft

    @etseidl
    etseidl committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    3d47934 View commit details
    Browse the repository at this point in the history

  3. need to pass memory resource to compute_splits

    @etseidl
    etseidl committed Mar 21, 2023
    Configuration menu
    Copy the full SHA
    e9a4312 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 23, 2023
    Configuration menu
    Copy the full SHA
    3e17232 View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2023

  1. Merge branch 'rapidsai:branch-23.04' into feature/chunk_stats

    @etseidl
    etseidl authored Mar 24, 2023
    Configuration menu
    Copy the full SHA
    2a034e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. Merge remote-tracking branch 'origin/branch-23.06' into feature/chunk… 

    …_stats
    @etseidl
    etseidl committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    2e2aa07 View commit details
    Browse the repository at this point in the history

  2. post merge cleanup

    @etseidl
    etseidl committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    7ff4033 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. Merge remote-tracking branch 'origin/branch-23.08' into feature/chunk… 

    …_stats
    @etseidl
    etseidl committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    39f5e8c View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 16, 2023
    Configuration menu
    Copy the full SHA
    f3e96f2 View commit details
    Browse the repository at this point in the history

  2. restore more stuff lost in merge

    @etseidl
    etseidl committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    5f41d24 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'feature/chunk_stats' of github.com:etseidl/cudf into fe… 

    …ature/chunk_stats
    @etseidl
    etseidl committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    4391c93 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 20, 2023
    Configuration menu
    Copy the full SHA
    5d8b54e View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 23, 2023
    Configuration menu
    Copy the full SHA
    a057a0b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 23, 2023
    Configuration menu
    Copy the full SHA
    21c7414 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 23, 2023
    Configuration menu
    Copy the full SHA
    307e8a6 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 26, 2023
    Configuration menu
    Copy the full SHA
    62f55af View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 30, 2023
    Configuration menu
    Copy the full SHA
    a87432b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jun 30, 2023
    Configuration menu
    Copy the full SHA
    e6432c5 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jul 7, 2023
    Configuration menu
    Copy the full SHA
    e35c002 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. Merge branch 'rapidsai:branch-23.08' into feature/chunk_stats

    @etseidl
    etseidl authored Jul 14, 2023
    Configuration menu
    Copy the full SHA
    54e7a08 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. Merge branch 'branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Jul 26, 2023
    Configuration menu
    Copy the full SHA
    bc1d084 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. Merge remote-tracking branch 'origin/branch-23.10' into feature/chunk… 

    …_stats
    @etseidl
    etseidl committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    8d4afec View commit details
    Browse the repository at this point in the history

  2. finish merge

    @etseidl
    etseidl committed Jul 28, 2023
    Configuration menu
    Copy the full SHA
    0fa655b View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Jul 31, 2023
    Configuration menu
    Copy the full SHA
    88035b8 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 1, 2023
    Configuration menu
    Copy the full SHA
    5389423 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 2, 2023
    Configuration menu
    Copy the full SHA
    13e5b08 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 3, 2023
    Configuration menu
    Copy the full SHA
    62ebeb6 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 4, 2023
    Configuration menu
    Copy the full SHA
    d0c3d94 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 9, 2023
    Configuration menu
    Copy the full SHA
    41800e4 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 14, 2023
    Configuration menu
    Copy the full SHA
    e2bdd57 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2023

  1. Merge branch 'rapidsai:branch-23.10' into feature/chunk_stats

    @etseidl
    etseidl authored Aug 17, 2023
    Configuration menu
    Copy the full SHA
    c22ce08 View commit details
    Browse the repository at this point in the history