Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support casting of LIST type to STRING in JSON #15278

Closed
Tracked by #9
revans2 opened this issue Mar 12, 2024 · 2 comments · Fixed by #16472
Closed
Tracked by #9

[FEA] Support casting of LIST type to STRING in JSON #15278

revans2 opened this issue Mar 12, 2024 · 2 comments · Fixed by #16472
Labels
feature request New feature or request Spark Functionality that helps Spark RAPIDS

Comments

@revans2
Copy link
Contributor

revans2 commented Mar 12, 2024

Is your feature request related to a problem? Please describe.

This is a follow on issue for #14936 (comment)

Really we just want to be able to request that a column or child column, be returned as a string and it works, even if the data is a nested type.

@GregoryKimball
Copy link
Contributor

Hello @revans2, would you please share a bit about the case where a list type is coerced to string? I understand the map type case, but why would you want this for the top-level list rather than only for its child values?

@revans2
Copy link
Contributor Author

revans2 commented Mar 14, 2024

@GregoryKimball Spark allows anything to be coerced into a string, if the user asks for it. Also Spark has requirements to treat quoted and unquoted strings differently for essentially all data types, so the schema we pass to the JSON parser has all strings at the leaf nodes. So this can end up being a very common problem, and we have no way to detect it and fall back to the CPU because the issue is in the data, not in anything we have access to at planning time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request Spark Functionality that helps Spark RAPIDS
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants