Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Add Spark 3.5.0 shim layer #8121

Closed
tgravescs opened this issue Apr 14, 2023 · 1 comment · Fixed by #9252
Closed

[FEA] Add Spark 3.5.0 shim layer #8121

tgravescs opened this issue Apr 14, 2023 · 1 comment · Fixed by #9252
Assignees
@andygrove
Labels
feature request New feature or request Spark 3.5+ Spark 3.5+ issues

Comments

@tgravescs
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Add Spark 3.5.0 shim layer.

We may want to hold off on this until 3.4 stable and decide when its useful.
@tgravescs tgravescs added feature request New feature or request ? - Needs Triage Need team to review and classify labels Apr 14, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Apr 18, 2023
@andygrove
Copy link
Contributor

andygrove commented Aug 2, 2023 •
edited
Loading

I started looking into this. There are some small refactors in 3.5.0 that affect us:

  • LegacyBehaviorPolicy has moved out of SQLConf into a top-level class (Move LegacyBehaviorPolicy references to shim layer [databricks] #8915)
  • StructType.fromAttributes moved to DataTypeUtils
  • BatchScanExec refactor to move some attributes to StoragePartitionJoinParams
  • org.apache.spark.sql.execution.PartitionedFileUtil#splitFiles has extra param filePath

There are also some potentially more significant refactors that may be more work, but I haven't looked too closely yet:

  • Hive UDF refactor
  • Python UDF refactor and Arrow support

@andygrove andygrove mentioned this issue Aug 2, 2023
Merged
@sameerz sameerz added the Spark 3.5+ Spark 3.5+ issues label Aug 5, 2023
This was referenced Aug 7, 2023
Merged
Merged
Merged
@andygrove andygrove mentioned this issue Sep 18, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andygrove andygrove

Labels
feature request New feature or request Spark 3.5+ Spark 3.5+ issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Spark 3.5.0 to list of supported Spark versions [skip ci] andygrove/spark-rapids
4 participants
@andygrove @tgravescs @mattahrens @sameerz