Fix writing raw messages to pubsub #32342
Conversation
Signed-off-by: Jeffrey Kinard <jeff@thekinards.com>
| @@ -259,7 +259,7 @@ def _create_formatter( | |||
| field_names = [field.name for field in beam_schema.fields] | |||
| if len(field_names) != 1: | |||
| raise ValueError(f'Expecting exactly one field, found {field_names}') | |||
| return lambda row: getattr(row, field_names[0]) | |||
| return lambda row: getattr(row, field_names[0]).encode('utf-8') | |||
There was a problem hiding this comment.
Should we only do this for string types? (E.g. if it's already bytes, should we leave it alone?)
Should we provide a good error for non-string-bytes fields? (Ideally checked at construction time when possible?)
There was a problem hiding this comment.
Good point, let me add those cases
There was a problem hiding this comment.
@robertwb I went a different direction and instead try to cast object to string, failing if it cannot be done.
Not sure how to best check input type - and what would be considered not compatible? Should we fail a pipeline that is trying to write an integer which can easily be cast to string?
There was a problem hiding this comment.
We can check the input type by querying beam_schema.fields just like we do to ensure there's exactly one field and we know its name (though we should ensure we properly handle the Any (aka "I don't know") type.
I can see that it could be convenient in some cases to turn things like integers into decimal stringified bytes, but that's not always desirable and feels a bit untypesafe. I'd prefer to err on the side of caution and reject non-string-or-bytes (it'd be backwards compatible to allow that in the future if we need, but not the other way around).
Signed-off-by: Jeffrey Kinard <jeff@thekinards.com>
|
Assigning reviewers. If you would like to opt out of this review, comment R: @shunping for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
Reminder, please take a look at this pr: @shunping |
|
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment R: @damccorm for label python. Available commands:
|
Writing raw messages to pubsub in Beam YAML writes a string object instead of encoded bytes (which is the expected type) - this PR fixes raw messages to instead be converted to bytes format
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.