Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote IO: http support #464

Merged
merged 73 commits into from
Oct 8, 2024
Merged

Remote IO: http support #464

merged 73 commits into from
Oct 8, 2024

Conversation

madsbk
Copy link
Member

@madsbk madsbk commented Sep 18, 2024

Support read directly from a http server like:

import kvikio
import cupy

with kvikio.RemoteFile.from_http_url("http://127.0.0.1:9000/myfile") as f:
    ary = cupy.empty(f.nbytes, dtype="uint8")
    f.read(ary)

This PR is the first step to support S3 using libcurl instead of aws-s3-sdk, which has some pros and cons:

  • Pros
  • Cons
    • Hard to support the AWS configuration file. We will require the user to either specify the options programmatically or through environment variables like AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY .

@madsbk madsbk added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Sep 18, 2024
Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C++, not python yet

cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
cpp/include/kvikio/remote_handle.hpp Outdated Show resolved Hide resolved
dependencies.yaml Outdated Show resolved Hide resolved
@madsbk madsbk mentioned this pull request Sep 24, 2024
3 tasks
@madsbk madsbk force-pushed the remote-io branch 2 times, most recently from 701f603 to b886419 Compare September 30, 2024 08:47
@madsbk madsbk requested review from wence- and bdice October 3, 2024 10:59
cpp/CMakeLists.txt Outdated Show resolved Hide resolved
python/kvikio/kvikio/_lib/CMakeLists.txt Outdated Show resolved Hide resolved
Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing more from me, looks good.

cpp/include/kvikio/remote_handle.hpp Show resolved Hide resolved
cpp/include/kvikio/shim/libcurl.hpp Show resolved Hide resolved
@madsbk
Copy link
Member Author

madsbk commented Oct 8, 2024

/merge

@rapids-bot rapids-bot bot merged commit dc536af into rapidsai:branch-24.12 Oct 8, 2024
57 checks passed
@madsbk madsbk deleted the remote-io branch October 8, 2024 11:39
rapids-bot bot pushed a commit to rapidsai/cudf that referenced this pull request Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improves an existing functionality non-breaking Introduces a non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants