Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Clarify whether we support LEGACY timeParserPolicy when parsing CSV files #2931

Open
Tracked by #2063
andygrove opened this issue Jul 14, 2021 · 1 comment
Open
Tracked by #2063
Labels
feature request New feature or request

Comments

@andygrove
Copy link
Contributor

Is your feature request related to a problem? Please describe.
The LEGACY timeParserPolicy setting affects how Spark parses timestamps from CSV files. We currently have no tests for parsing CSV files containing timestamps on the GPU with LEGACY mode enabled, so we don't know what our current behavior is.

Describe the solution you'd like
Add tests so that we understand the current behavior and then update our documentation to clarify our behavior and potentially file issues for follow on work to fix any bugs found.

Describe alternatives you've considered
None

Additional context
None

@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jul 14, 2021
@andygrove andygrove added this to the July 5 - July 16 milestone Jul 14, 2021
@andygrove andygrove self-assigned this Jul 14, 2021
@andygrove
Copy link
Contributor Author

CSV parsing uses a different code path to our CAST logic and does not have special handling for things like ansi mode or timeParserPolicy. Also, parsing from all non-string types is disabled by default and the documentation points out the reasons for this.

If we did want to support ansi/legacy in CSV parsing than we could consider a different approach where we just read strings from the csv file and then wrap in a projection that casts columns from string to another type.

@Salonijain27 Salonijain27 removed the ? - Needs Triage Need team to review and classify label Jul 20, 2021
@Salonijain27 Salonijain27 removed this from the July 19 - July 30 milestone Jul 20, 2021
@andygrove andygrove removed their assignment Aug 23, 2021
@revans2 revans2 mentioned this issue Oct 27, 2022
38 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants