New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update from_json to use null as line delimiter #11499

Closed

revans2 wants to merge 1 commit into NVIDIA:branch-24.10 from revans2:null_delim

Collaborator

revans2 commented Sep 25, 2024

I am leaving this as draft for two reasons.

it does not fix having \r in the JSON. See [BUG] \n is not considered whitespace when tokenizing JSON rapidsai/cudf#16915
There is a very large performance hit going to a regexp for stripping the characters from the input. I am a bit conflicted here because we do need a fix for this at some point even without trying to support \r and \n in the data because \t is not being treated as an empty line and will fail.


          Update from_json to use null as line delimiter

2a5bc80

Signed-off-by: Robert (Bobby) Evans <bobby@apache.org>

revans2 mentioned this pull request

Adopt JSONUtils.concatenateJsonStrings for concatenating JSON strings #11549

Merged

Collaborator Author

revans2 commented Oct 9, 2024

We are going to do this a different way.

revans2 closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet