Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] support ‘yyyyMMdd’ format in from_unixtime function #8807

Closed
nvliyuan opened this issue Jul 26, 2023 · 3 comments · Fixed by #10023
Closed

[FEA] support ‘yyyyMMdd’ format in from_unixtime function #8807

nvliyuan opened this issue Jul 26, 2023 · 3 comments · Fixed by #10023
Labels
feature request New feature or request

Comments

@nvliyuan
Copy link
Collaborator

It would be nice if we support 'yyyyMMdd' format for from_unixtime function. I know we could avoid the fallback by setting spark.rapids.sql.incompatibleDateFormats.enabled=true, but it would simplify the configs for customers if we support more LEGACY format.
For example:

create a hive table and insert some rows:
CREATE TABLE `xxx_timestamp`(
  `id` int,
  `name` string,
  `created_at` timestamp)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'

run below SQL:
spark.sql("select from_unixtime(unix_timestamp(created_at, 'yyyy-MM-dd HH:mm:ss'), 'yyyyMMdd') from xxx_timestamp ").show

fallback info:

!Expression <FromUnixTime> from_unixtime(unix_timestamp(created_at#27, yyyy-MM-dd HH:mm:ss, Some(UTC), false), yyyyMMdd, Some(UTC)) cannot run on GPU because CORRECTED format 'yyyyMMdd' on the GPU is not guaranteed to produce the same results as Spark on CPU. Set spark.rapids.sql.incompatibleDateFormats.enabled=true to force onto GPU.
@nvliyuan nvliyuan added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jul 26, 2023
@nvliyuan nvliyuan changed the title [FEA] [FEA] support more LEGACY format in from_unixtime function Jul 26, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Aug 8, 2023
@nvliyuan
Copy link
Collaborator Author

nvliyuan commented Nov 2, 2023

Low priority for customers.

@nvliyuan
Copy link
Collaborator Author

A similar fallback:
!Expression <FromUnixTime> from_unixtime(report_create_time#174L, yyyyMMdd, Some(Etc/UTC)) cannot run on GPU because LEGACY format 'yyyyMMdd' is not supported on the GPU.

@nvliyuan
Copy link
Collaborator Author

nvliyuan commented Dec 12, 2023

CC @firestarman @winningsix

@nvliyuan nvliyuan changed the title [FEA] support more LEGACY format in from_unixtime function [FEA] support ‘yyyyMMdd’ format in from_unixtime function Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants