Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use nvcomp defaults for algo options. #450

Merged
merged 4 commits into from
Sep 3, 2024

Conversation

bdice
Copy link
Contributor

@bdice bdice commented Aug 29, 2024

This PR updates the nvcomp bindings to use nvcomp's defaults rather than hardcode the default options. The defaults changed in nvcomp 4.0.1, so this is needed for #449.

@bdice bdice requested a review from a team as a code owner August 29, 2024 22:44
@bdice bdice added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Aug 29, 2024
@bdice bdice self-assigned this Aug 29, 2024
@bdice bdice requested a review from vuule August 29, 2024 22:45
@vuule vuule added the python Affects the Python API of KvikIO label Aug 29, 2024
@bdice bdice mentioned this pull request Aug 29, 2024
@jakirkham
Copy link
Member

@madsbk @Alexey-Kamenev @thomcom could you please look over these changes when you have a moment? 🙂

Copy link
Contributor

@vuule vuule left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semantically the change looks good to me, not qualified to evaluate the (Python) code quality.

Copy link
Contributor

@Alexey-Kamenev Alexey-Kamenev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with minor comment.

python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
Copy link
Member

@jakirkham jakirkham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After Alexey observed the f-string typo. Saw a few more. Flagged them below

python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
@jakirkham
Copy link
Member

As it appears we do tests constructing compressors with default arguments, this should already be covered in the existing test suite

@pytest.mark.parametrize(
"inputs",
[
{},
{"chunk_size": 1 << 16, "algo": 0, "device_id": 0},
{
"chunk_size": 1 << 16,
},
{
"algo": 0,
},
{
"device_id": 0,
},
],
)
def test_gdeflate_inputs(inputs):
size = 10000
dtype = inputs.get("data_type") if inputs.get("data_type") else np.int8
data = cupy.array(np.arange(0, size // dtype(0).itemsize, dtype=dtype))
compressor = libnvcomp.GdeflateManager(**inputs)
final = compressor.compress(data)
assert_compression_size(len(final), LEN["Gdeflate"])

Though please let me know if I've missed anything here

Copy link
Member

@madsbk madsbk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I only have some type hint suggestions

python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Show resolved Hide resolved
Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nits

python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved

def _get_comp_temp_size(
self,
size_t batch_size,
size_t max_uncompressed_chunk_bytes,
) -> (nvcompStatus_t, size_t):
) -> tuple[nvcompStatus_t, size_t]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, how did this ever work? I guess cython is less strict with parsing.

Copy link
Member

@jakirkham jakirkham Aug 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are valid, the type format we had before is called a ctuple and has been around in Cython longer

Support for typing.Tuple is relatively new

Here's an example in the docs (please see 2nd entry)

Copy link
Member

@jakirkham jakirkham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cython already checks whether the value is an int or None based on the function signature. So the exceptions can be dropped. If we wanted to be extra careful, we could add an assert. Though likely that is unneeded

Example of this behavior demonstrated below:

In [1]: %load_ext cython

In [2]: %%cython
   ...: 
   ...: from typing import Optional
   ...: 
   ...: class Class:
   ...:     def __init__(self, v: Optional[int] = None):
   ...:         print(f"v = {v}")
   ...:         self.v = v
   ...: 

In [3]: Class()
v = None
Out[3]: <_cython_magic_96fdfa8e6c04d6dbb7c0c7398f039f6decfcbe5b.Class at 0x10524ad10>

In [4]: Class(1)
v = 1
Out[4]: <_cython_magic_96fdfa8e6c04d6dbb7c0c7398f039f6decfcbe5b.Class at 0x104f4c610>

In [5]: Class(3.14)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[5], line 1
----> 1 Class(3.14)

TypeError: Argument 'v' has incorrect type (expected int, got float)

python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
python/kvikio/kvikio/_lib/libnvcomp_ll.pyx Outdated Show resolved Hide resolved
Co-authored-by: jakirkham <jakirkham@gmail.com>
@vuule
Copy link
Contributor

vuule commented Aug 30, 2024

Thanks everyone for the reviews! I'll wait another work day to make sure folks are happy with the way the comments like #450 (comment) have been resolved.

Copy link
Contributor Author

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the reviews and fixes. I am happy with this.

@bdice
Copy link
Contributor Author

bdice commented Sep 3, 2024

/merge

@rapids-bot rapids-bot bot merged commit 0e9ebcb into rapidsai:branch-24.10 Sep 3, 2024
48 checks passed
rapids-bot bot pushed a commit that referenced this pull request Sep 5, 2024
Update the nvCOMP version used for compression/decompression to 4.0.1.

See also:

rapidsai/rapids-cmake#633
rapidsai/cudf#16076

Depends on #450.

Authors:
  - Vukasin Milovanovic (https://github.com/vuule)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Mads R. B. Kristensen (https://github.com/madsbk)
  - Bradley Dice (https://github.com/bdice)

URL: #449
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improves an existing functionality non-breaking Introduces a non-breaking change python Affects the Python API of KvikIO
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants