feat(rust, python): impl hex and base64 for binary

[ozgrakkurt]

mostly replicated the implementation for string

[ritchie46]

Thanks for the PR. I think we should finish extending this the same way we do for Utf8. That means that we need a Binary namespace on expressions that is also exposed to python Expr/Series similar to what we do for str, cat and arr.

[ozgrakkurt]

@ritchie46 added namespace as far as I can understand

[ritchie46]

@ritchie46 added namespace as far as I can understand

Thanks. Can you expose it to python and add tests on the python side as well? That's our ultimate integration test. Let me know if you need any pointers.

[ozgrakkurt]

how can I expose it to python? where should I add the code?

[alexander-beedie]

how can I expose it to python? where should I add the code?

Expression support:

• Add the new functions to PyExpr, in py-polars/src/lazy/dsl.rs (probably with a bin prefix; eg: pub fn bin_xyz(&self, ...) -> PyExpr {, consistent with their str-prefixed counterparts).
• Create ExprBinaryNameSpace in py-polars/polars/internals/expr/binary.py, patterned after the other Expr*NameSpace objects, with _accessor = "bin"; this is the object on which your new binary expression functions will be exposed.
• Import the new namespace and make it available on the python-side Expr by adding to py-polars/polars/internals/expr/expr.py (just look for @accessor to see how/where).

Series support (trivial - will redirect to the expression code above):

• Create BinaryNameSpace in py-polars/polars/internals/series/binary.py, patterned after the other Series namespaces. Create the function definition/docstrings, as per the Expr one above, but omit the function body (it'll automatically call the expression implementation via @expr_dispatch black magic ;)
• Register the namespace in py-polars/polars/internals/series/series.py (via @accessor, as above).

[ozgrakkurt]

@alexander-beedie Thanks!

what would be the type for binary to use in py-polars/polars/internals/series/binary.py? I see it uses str in string file

[ritchie46]

@alexander-beedie Thanks!

what would be the type for binary to use in py-polars/polars/internals/series/binary.py? I see it uses str in string file

Do you mean the accessor name? I think this should be bin.

[ozgrakkurt]

I mean str in def ends_with(self, sub: str) -> pli.Series:

[ozgrakkurt]

Also shouldn't the decode functions on string return binary?

[alexander-beedie]

Ahh, gotcha; the python binary type is bytes.

[ozgrakkurt]

does this look good for exposing to python part?

[alexander-beedie]

does this look good for exposing to python part?

Almost there; change the binary accessor/namespace shortcut to bin, and then add some python-side unit test coverage for the new functions to help validate it all 👍

(Looks like there is already a test_binary.py, so would suggest adding the tests there).

[ozgrakkurt]

I added a test file but didn't register it anywhere. does it look good?

[ozgrakkurt]

should be good to merge

[ritchie46]

I think the base_64 encode must return a Utf8Chunked. It is free to cast from Utf8Chunked to BinaryChunked but not in the other direction as we need to do utf8 checking.

In base64 we already have this utf8 guarantee, so I think we should encode this in the type we return.

[ozgrakkurt]

I think the base_64 encode must return a Utf8Chunked. It is free to cast from Utf8Chunked to BinaryChunked but not in the other direction as we need to do utf8 checking.

In base64 we already have this utf8 guarantee, so I think we should encode this in the type we return.

yes makes sense to me but also decode should return binary because there is no guarantee the returned value is valid string, it can be arbitrary bytes

[ozgrakkurt]

Is there a way to do something like cast_and_apply on BinaryChunked and Utf8Chunked?

[ozgrakkurt]

@ritchie46 made the changes but had to convert encode functions to return Series and not Utf8Chunked in BinaryChunked encode functions. Not sure the implementation is efficient. Changed Utf8Chunked decode functions to cast it to binary first and then do the decode and also return BinaryChunked

[ritchie46]

@ritchie46 made the changes but had to convert encode functions to return Series and not Utf8Chunked in BinaryChunked encode functions. Not sure the implementation is efficient. Changed Utf8Chunked decode functions to cast it to binary first and then do the decode and also return BinaryChunked

Thank you @ozgrakkurt. I still have two comments on the decoding of utf8 data via hex/base64.

[ozgrakkurt]

@ritchie46 can you check again? I cleaned up the decode functions, also is it possible to cast while applying for string and binary? I see it is possible for integers

[ozgrakkurt]

closes #5118

[ritchie46]

Thank you @ozgrakkurt. And merry christmas! :)

[ozgrakkurt]

Thank you @ozgrakkurt. And merry christmas! :)

thanks, merry Christmas to you too!