Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support GpuHour expression for timezones other than UTC #6834

Closed
revans2 opened this issue Oct 18, 2022 · 0 comments · Fixed by #10024
Closed

[FEA] Support GpuHour expression for timezones other than UTC #6834

revans2 opened this issue Oct 18, 2022 · 0 comments · Fixed by #10024
Assignees
Labels
feature request New feature or request

Comments

@revans2
Copy link
Collaborator

revans2 commented Oct 18, 2022

Is your feature request related to a problem? Please describe.
This depends on https://github.com/NVIDIA/spark-rapids/issues/6831and #6832.

First we need to verify that we can use the existing hour cudf code for timestamps that are in different time zones and we get the correct answer. It should work because they are all relative to an epoch in the given time zone and offsets should have been taken into account properly so there should not be values that actually exist in the gaps when daylight savings time happens. GpuHour should be updated so if the ZoneId is set and it is not UTC, then we would first convert the timestamps to the desired time zone, and then extract the hour from it.

The hardest part will be in testing this. We should have a list of zone ids that are supported and ones that are not in the tests. Then we can parameterize the tests to include these and set spark.sql.session.timeZone accordingly as the test runs. We may want/need to include some outliers for really old and really new dates that we know python cannot handle, but Spark can.

This is separate from many of the other operators because it is one of the first ones and we want to be sure that it works properly before we plan out and execute a lot of other operators.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants