Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More formats support by additional conversions for FromUnixTime #7

Merged
merged 1 commit into from
Dec 5, 2023

Conversation

firestarman
Copy link
Owner

@firestarman firestarman commented Dec 4, 2023

Support format yyyyMMdd for GpuFromUnixTime

scala> Seq(System.currentTimeMillis()/1000).toDF("dt").repartition(1).selectExpr("*", "from_unixtime(dt, 'yyyyMMdd')").show
23/12/04 12:51:28 WARN GpuOverrides: 
*Exec <ProjectExec> will run on GPU
  *Expression <Alias> cast(dt#263L as string) AS dt#270 will run on GPU
    *Expression <Cast> cast(dt#263L as string) will run on GPU
  *Expression <Alias> from_unixtime(dt#263L, yyyyMMdd, Some(Etc/UTC)) AS from_unixtime(dt, yyyyMMdd)#271 will run on GPU
    *Expression <FromUnixTime> from_unixtime(dt#263L, yyyyMMdd, Some(Etc/UTC)) will run on GPU
  *Exec <ShuffleExchangeExec> will run on GPU
    *Partitioning <SinglePartition$> will run on GPU
    ! <LocalTableScanExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.LocalTableScanExec
      @Expression <AttributeReference> dt#263L could run on GPU

+----------+---------------------------+
|        dt|from_unixtime(dt, yyyyMMdd)|
+----------+---------------------------+
|1701694288|                   20231204|
+----------+---------------------------+

Signed-off-by: Firestarman <firestarmanllc@gmail.com>
@firestarman firestarman merged commit 09e7541 into branch-23.12 Dec 5, 2023
35 of 37 checks passed
@firestarman firestarman deleted the branch-hotfix branch December 5, 2023 02:30
@firestarman firestarman restored the branch-hotfix branch December 5, 2023 02:30
@firestarman firestarman deleted the branch-hotfix branch December 5, 2023 02:30
firestarman pushed a commit that referenced this pull request May 13, 2024
* A hacky approach for regexpr rewrite

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* Use contains instead for that case

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* add config to switch

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* Rewrite some rlike expression to StartsWith/EndsWith/Contains

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* clean up

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* wip

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* wip

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* add tests and config

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

* support range filter

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>

---------

Signed-off-by: Haoyang Li <haoyangl@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant