Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable building C speedups on PyPy #1157

Closed
wants to merge 1 commit into from

Conversation

mgorny
Copy link
Contributor

@mgorny mgorny commented Apr 7, 2022

Disable building the C speedups on PyPy since it does not implement
"creating contiguous readonly buffer from non-contiguous". As a result,
the respective tests fail with e.g.:

======================================================================
ERROR: test_apply_mask_non_contiguous_memoryview (tests.test_utils.SpeedupsTests) (data_in=<memory at 0x00007f452c3758f8>, mask=b'4321'
)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/portage/dev-python/websockets-10.2/work/websockets-10.2/tests/test_utils.py", line 61, in test_apply_mask_non_contiguous_memoryview
    result = self.apply_mask(data_in, mask)
  File "/tmp/portage/dev-python/websockets-10.2/work/websockets-10.2/tests/test_utils.py", line 91, in apply_mask
    return c_apply_mask(*args, **kwargs)
NotImplementedError: creating contiguous readonly buffer from non-contiguous not implemented yet

However, I suspect this may also affect real life use.

Disable building the C speedups on PyPy since it does not implement
"creating contiguous readonly buffer from non-contiguous".  As a result,
the respective tests fail with e.g.:

    ======================================================================
    ERROR: test_apply_mask_non_contiguous_memoryview (tests.test_utils.SpeedupsTests) (data_in=<memory at 0x00007f452c3758f8>, mask=b'4321'
    )
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/tmp/portage/dev-python/websockets-10.2/work/websockets-10.2/tests/test_utils.py", line 61, in test_apply_mask_non_contiguous_memoryview
        result = self.apply_mask(data_in, mask)
      File "/tmp/portage/dev-python/websockets-10.2/work/websockets-10.2/tests/test_utils.py", line 91, in apply_mask
        return c_apply_mask(*args, **kwargs)
    NotImplementedError: creating contiguous readonly buffer from non-contiguous not implemented yet

However, I suspect this may also affect real life use.
@aaugustin
Copy link
Member

This change would disable a generally useful optimization in order to support an extremely weird edge case: what's the practical use case for sending a non-contiguous memoryview as a WebSocket message?

This code and test exists mostly because I wanted edge cases of memoryviews, not because of real-life use cases. In the extremely unlikely event that someone hits this error on PyPy, they'll just convert the non-contiguous memoryview to bytes before passing it to websockets.

If PyPy provides same or better performance than speedups, then you could make an argument for disabling speedups. While PyPy is amazing, I am skeptical that it beats this.

I would consider skipping that particular test on PyPy if having a failing test is a problem for you.

@aaugustin
Copy link
Member

FWIW a PyPy maintainer had pretty much the same reaction — is there any real life use case for this?

I wonder if that codepath is hit when using the library in a project.

@mgorny
Copy link
Contributor Author

mgorny commented Apr 8, 2022

To be honest, I don't really know this package and how it's supposed to be used — I'm merely maintaining it in Gentoo as a dependency. My reasoning was basically that: there is some edge case that probably breaks when speedups are used but works when Python code path is used. If you don't think this is likely to happen with real code, I'm fine with skipping the tests.

Another interesting option would be to catch NonImplementedError and fall back to the Python version but I don't really know how to do that.

As for performance, I'd refrain from making any claims without benchmarking it first. Firstly, because even if the code itself is faster, PyPy might suffer some overhead from entering C code in the first place. Secondly, because I've seen optimized SSE2 code being slower on some AMD processors than non-optimized code in the past — I think it was in BLAKE2 or SHA3. But again, I don't really know the details here, just mentioning as a possible curiosity.

@aaugustin
Copy link
Member

This is vanishingly unlikely to happen in real life use cases. I will just skip the test.

@mgorny
Copy link
Contributor Author

mgorny commented Apr 8, 2022

Ok, thanks.

@aaugustin
Copy link
Member

Filed #1158 to keep track of this.

mgorny added a commit to mgorny/websockets that referenced this pull request Apr 8, 2022
PyPy3.9 7.3.9 does not implement creating contiguous buffers from
non-contiguous, causing the respective tests to fail due to
NotImplementedError.  Catch it and skip the tests appropriately
as discussed in python-websockets#1157.
mgorny added a commit to mgorny/websockets that referenced this pull request Apr 8, 2022
PyPy3.9 7.3.9 does not implement creating contiguous buffers from
non-contiguous, causing the respective tests to fail due to
NotImplementedError.  Catch it and skip the tests appropriately
as discussed in python-websockets#1157.

Fixes python-websockets#1158
aaugustin pushed a commit that referenced this pull request Apr 17, 2022
PyPy3.9 7.3.9 does not implement creating contiguous buffers from
non-contiguous, causing the respective tests to fail due to
NotImplementedError.  Catch it and skip the tests appropriately
as discussed in #1157.

Fixes #1158
aaugustin pushed a commit that referenced this pull request Apr 17, 2022
PyPy3.9 7.3.9 does not implement creating contiguous buffers from
non-contiguous, causing the respective tests to fail due to
NotImplementedError.  Catch it and skip the tests appropriately
as discussed in #1157.

Fixes #1158
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants