Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Add rebase mode notes for databricks doc [skip ci] #5408

Merged
merged 4 commits into from
May 2, 2022

Conversation

viadea
Copy link
Collaborator

@viadea viadea commented Apr 29, 2022

Add rebase mode notes for databricks doc.

By default, those 2 parameters are LEGACY in Databricks(at least 9.1ML and 10.4ML).

 spark.sql.legacy.parquet.datetimeRebaseModeInWrite
 spark.sql.legacy.parquet.int96RebaseModeInWrite

If we are writing a parquet file with date/timestamp/int96, below fallback will happen:

!Output <InsertIntoHadoopFsRelationCommand> cannot run on GPU because LEGACY rebase mode for dates and timestamps is not supported; LEGACY rebase mode for int96 timestamps is not supported

Minimum repro is:

import scala.collection.Seq
import java.sql.Date
Seq(java.sql.Timestamp.valueOf("1500-01-01 00:00:00")).toDF("ts").write.format("parquet").mode("overwrite").save("/tmp/testparquet_legacy")

We need to manually set them back to "EXCEPTION" which is default value in Apache Spark.

Signed-off-by: Hao Zhu <hazhu@nvidia.com>
@viadea viadea added the documentation Improvements or additions to documentation label Apr 29, 2022
@viadea viadea requested review from jlowe and sameerz April 29, 2022 21:48
@sameerz sameerz added this to the May 2 - May 20 milestone Apr 29, 2022
viadea and others added 3 commits April 29, 2022 17:50
Co-authored-by: Jason Lowe <jlowe@nvidia.com>
Co-authored-by: Jason Lowe <jlowe@nvidia.com>
Signed-off-by: Hao Zhu <hazhu@nvidia.com>
@jlowe jlowe changed the title [Doc]Add rebase mode notes for databricks doc[skip ci] [DOC] Add rebase mode notes for databricks doc [skip ci] May 2, 2022
@jlowe
Copy link
Member

jlowe commented May 2, 2022

build

@jlowe jlowe merged commit 3f2abe8 into NVIDIA:branch-22.06 May 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants