Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] [Delta Lake] Add support for deletion vectors in OSS Delta Lake #8554

Open
andygrove opened this issue Jun 12, 2023 · 0 comments
Open
Labels
feature request New feature or request

Comments

@andygrove
Copy link
Contributor

Is your feature request related to a problem? Please describe.

Deletion vectors are a storage optimization feature that can be enabled on Delta Lake tables. By default, when a single row in a data file is deleted, the entire Parquet file containing the record must be rewritten. With deletion vectors enabled for the table, some Delta operations use deletion vectors to mark existing rows as removed without rewriting the Parquet file. Subsequent reads on the table resolve current table state by applying the deletions noted by deletion vectors to the most recent table version.

Describe the solution you'd like
We need to ensure that we respect deletion vectors in Delta Lake 2.3.0 and later when performing reads. We should also add support for writing deletion vectors when performing writes.

Describe alternatives you've considered
None

Additional context
https://docs.delta.io/2.4.0/delta-deletion-vectors.html

@andygrove andygrove added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jun 12, 2023
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Jun 13, 2023
@andygrove andygrove changed the title [FEA] [Delta Lake] Add support for deletion vectors [FEA] [Delta Lake] Add support for deletion vectors in OSS Delta Lake Jul 3, 2023
@andygrove andygrove removed their assignment Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants