Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use cuFile for Parquet IO when available #7444

Merged
merged 74 commits into from
Mar 17, 2021
Merged

Use cuFile for Parquet IO when available #7444

merged 74 commits into from
Mar 17, 2021

Commits on Sep 17, 2020

  1. add owning_buffer to datasource; add device_read path to parquet reader

    sft-managed committed Sep 17, 2020
    Configuration menu
    Copy the full SHA
    1dfd27d View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2020

  1. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    f88569c View commit details
    Browse the repository at this point in the history

  2. name fix

    sft-managed committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    5cfc08f View commit details
    Browse the repository at this point in the history

  3. CMake change to find and link to cufile lib

    sft-managed committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    eb0a075 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    b494b43 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    582dbac View commit details
    Browse the repository at this point in the history

  6. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 18, 2020
    Configuration menu
    Copy the full SHA
    a7873f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2020

  1. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 19, 2020
    Configuration menu
    Copy the full SHA
    460af52 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2020

  1. link to cufile

    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    3d00911 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    8adf068 View commit details
    Browse the repository at this point in the history

  3. basic device_read

    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    c834cdc View commit details
    Browse the repository at this point in the history

  4. extract gds file into a separate source file

    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    585d3a2 View commit details
    Browse the repository at this point in the history

  5. missing file from previous commit

    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    be84443 View commit details
    Browse the repository at this point in the history

  6. gds sink

    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    62135c6 View commit details
    Browse the repository at this point in the history

  7. refactor cufile RAII into a separate struct

    sft-managed committed Sep 22, 2020
    Configuration menu
    Copy the full SHA
    f70974e View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2020

  1. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 25, 2020
    Configuration menu
    Copy the full SHA
    da3207b View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2020

  1. Merge branch 'branch-1.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Sep 30, 2020
    Configuration menu
    Copy the full SHA
    9220769 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2020

  1. refactor gds threshold logic

    sft-managed committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    2acfeca View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Oct 2, 2020
    Configuration menu
    Copy the full SHA
    37cbb35 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2020

  1. override the cufile config file to always enable compatibility mode

    sft-managed committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    73092fc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-0.16' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    f8017e9 View commit details
    Browse the repository at this point in the history

  3. fix missed rename

    sft-managed committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    2aa7da7 View commit details
    Browse the repository at this point in the history

  4. add docs; rename data members

    sft-managed committed Oct 7, 2020
    Configuration menu
    Copy the full SHA
    689a4d2 View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2020

  1. Merge branch 'branch-0.17' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    sft-managed committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    674023b View commit details
    Browse the repository at this point in the history

  2. fix merge

    sft-managed committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    5807bfd View commit details
    Browse the repository at this point in the history

  3. add missing EOF newlines

    sft-managed committed Oct 30, 2020
    Configuration menu
    Copy the full SHA
    5284591 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2020

  1. fall back to host read/write if cufile fails

    @vuule
    vuule committed Nov 20, 2020
    Configuration menu
    Copy the full SHA
    7000d55 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2021

  1. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Feb 22, 2021
    Configuration menu
    Copy the full SHA
    8548871 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2021

  1. update to 0.19 stuff

    @vuule
    vuule committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    5896b7e View commit details
    Browse the repository at this point in the history

  2. don't enforce compatibility mode through config file

    @vuule
    vuule committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    e5071ab View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2021

  1. add stream parameter to device_read

    @vuule
    vuule committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    10a1eee View commit details
    Browse the repository at this point in the history

  2. compile-time disable cufile code if the library is not installed

    @vuule
    vuule committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    3014923 View commit details
    Browse the repository at this point in the history

  3. style fix

    @vuule
    vuule committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    f388265 View commit details
    Browse the repository at this point in the history

  4. fix building without cufile

    @vuule
    vuule committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    f433757 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2021

  1. CMake fix

    @vuule
    vuule committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    fb6ea8b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    1189cf1 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    2cef6f3 View commit details
    Browse the repository at this point in the history

  4. cmake clean up

    @vuule
    vuule committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    ec52e85 View commit details
    Browse the repository at this point in the history

  5. add missing null check

    @vuule
    vuule committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    d181fec View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2021

  1. avoid repeated failed attempts to initialize cuFile

    @vuule
    vuule committed Feb 26, 2021
    Configuration menu
    Copy the full SHA
    1b6070e View commit details
    Browse the repository at this point in the history

  2. link to libcufile at runtime

    @vuule
    vuule committed Feb 26, 2021
    Configuration menu
    Copy the full SHA
    349973c View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2021

  1. remove newline 

    Co-authored-by: Keith Kraus <keith.j.kraus@gmail.com>
    @vuule @kkraus14
    vuule and kkraus14 authored Mar 2, 2021
    Configuration menu
    Copy the full SHA
    3560949 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    d1a4c26 View commit details
    Browse the repository at this point in the history

  3. add file path to error messages

    @vuule
    vuule committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    378023b View commit details
    Browse the repository at this point in the history

  4. Merge branch 'fea-gds-parquet' of https://github.com/vuule/cudf into … 

    …fea-gds-parquet
    @vuule
    vuule committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    43a39da View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    9c3e8fc View commit details
    Browse the repository at this point in the history

  6. use FindcuFile

    @vuule
    vuule committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    5ed479b View commit details
    Browse the repository at this point in the history

  7. add new file to cmake list

    @vuule
    vuule committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    1e2727e View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2021

  1. avoid using unique_ptr to store the singleton

    @vuule
    vuule committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    a2f4019 View commit details
    Browse the repository at this point in the history

  2. add is_valid to shim

    @vuule
    vuule committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    fb94b56 View commit details
    Browse the repository at this point in the history

  3. add config class

    @vuule
    vuule committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    b2771ad View commit details
    Browse the repository at this point in the history

  4. read env var to enable GDS

    @vuule
    vuule committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    3077bd4 View commit details
    Browse the repository at this point in the history

  5. rename file_util.hpp

    @vuule
    vuule committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    9d350db View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2021

  1. control compatiblity mode

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    09730f4 View commit details
    Browse the repository at this point in the history

  2. style fix

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    d5c67f8 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    2e4d33c View commit details
    Browse the repository at this point in the history

  4. update device_write condition in CSV writer

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    8ac417e View commit details
    Browse the repository at this point in the history

  5. fall back to host if cufile io initialization fails

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    273962c View commit details
    Browse the repository at this point in the history

  6. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    0ef4e31 View commit details
    Browse the repository at this point in the history

  7. docs and such

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    071c619 View commit details
    Browse the repository at this point in the history

  8. (c) year

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c1010a9 View commit details
    Browse the repository at this point in the history

  9. one more (c) year

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    13b3c6a View commit details
    Browse the repository at this point in the history

  10. namespace

    @vuule
    vuule committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    67d901b View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2021

  1. docs, comments; disable fallback w/ compat mode

    @vuule
    vuule committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    851c0bd View commit details
    Browse the repository at this point in the history

  2. moar docs

    @vuule
    vuule committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    b853f97 View commit details
    Browse the repository at this point in the history

  3. remove includes

    @vuule
    vuule committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    750460f View commit details
    Browse the repository at this point in the history

  4. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    6815b38 View commit details
    Browse the repository at this point in the history

  5. CMake improvement

    @vuule
    vuule committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    96c5b53 View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2021

  1. address review feedback

    @vuule
    vuule committed Mar 9, 2021
    Configuration menu
    Copy the full SHA
    6be526e View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2021

  1. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    19d6ff4 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2021

  1. fix a CMake error

    @vuule
    vuule committed Mar 13, 2021
    Configuration menu
    Copy the full SHA
    0ad1418 View commit details
    Browse the repository at this point in the history

  2. allocate the pinned buffer only if used

    @vuule
    vuule committed Mar 13, 2021
    Configuration menu
    Copy the full SHA
    380287e View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2021

  1. Merge branch 'branch-0.19' of https://github.com/rapidsai/cudf into f… 

    …ea-gds-parquet
    @vuule
    vuule committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    d374f49 View commit details
    Browse the repository at this point in the history

  2. fix up CMake for cufile header only builds

    @vuule
    vuule committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    e5f12b9 View commit details
    Browse the repository at this point in the history