Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fall back to CPU for Delta Lake metadata queries [databricks] #5912

Merged
merged 9 commits into from
Jul 8, 2022
Merged

Fall back to CPU for Delta Lake metadata queries [databricks] #5912

merged 9 commits into from
Jul 8, 2022

Conversation

andygrove
Copy link
Contributor

@andygrove andygrove commented Jun 24, 2022 •
edited
Loading

Closes #5624

This PR adds a very specific "is this a Delta Lake metadata query?" check and then falls back to CPU in this case.

The integration test currently only runs on Databricks. I filed #5966 to expand Delta Lake testing to other Spark versions.

@andygrove andygrove added this to the Jun 20 - Jul 8 milestone Jun 24, 2022
@andygrove andygrove self-assigned this Jun 24, 2022
@abellina
Copy link
Collaborator

Should this fallback be under a config that can be disabled? In case someone does have a table named _delta_log, but it is not a delta lake table.

@andygrove
Copy link
Contributor Author

Should this fallback be under a config that can be disabled? In case someone does have a table named _delta_log, but it is not a delta lake table.

Yes, that makes sense. I will add that.

@andygrove andygrove changed the title WIP: Fall back to CPU for Delta Lake metadata queries Fall back to CPU for Delta Lake metadata queries Jul 6, 2022
@andygrove andygrove marked this pull request as ready for review July 6, 2022 20:45
@andygrove andygrove changed the title Fall back to CPU for Delta Lake metadata queries WIP: Fall back to CPU for Delta Lake metadata queries Jul 6, 2022
@andygrove andygrove marked this pull request as draft July 6, 2022 21:18
@andygrove andygrove changed the title WIP: Fall back to CPU for Delta Lake metadata queries WIP: Fall back to CPU for Delta Lake metadata queries [databricks] Jul 7, 2022
@andygrove
Copy link
Contributor Author

build

@andygrove andygrove changed the title WIP: Fall back to CPU for Delta Lake metadata queries [databricks] Fall back to CPU for Delta Lake metadata queries [databricks] Jul 7, 2022
@andygrove andygrove marked this pull request as ready for review July 7, 2022 17:07
This was referenced Jul 7, 2022
Closed
Open
@andygrove andygrove merged commit 6ee8d0f into NVIDIA:branch-22.08 Jul 8, 2022
@andygrove andygrove deleted the delta-lake-meta-fallback branch July 8, 2022 21:13
@sameerz sameerz added the task Work required that improves the product but is not user facing label Jul 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tgravescs tgravescs tgravescs approved these changes

@jlowe jlowe Awaiting requested review from jlowe jlowe is a code owner

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@zhanga5 zhanga5 Awaiting requested review from zhanga5

Assignees

@andygrove andygrove

Labels
task Work required that improves the product but is not user facing
Projects
None yet
Milestone
Jun 20 - Jul 8
Development

Successfully merging this pull request may close these issues.

[FEA] Let CPU handle Delta table's metadata related queries
4 participants
@andygrove @abellina @tgravescs @sameerz