Deduplication support in RewriteDataFilesAction

[skandasa23]

RewriteDataFilesAction helps to compact the smaller files into bigger files by configuring the targetSizeInBytes. If we take the example of compacting all the files post the day closure, it is possible that duplicate events could have arrived at different point of time over the day duration. Compaction with deduplication will be useful.
Deduplicate could be an additional method in RewriteDataFilesAction with default behaviour turned off.

[skandasa23]

@jerryshao @rdblue @aokolnychyi Can you please share your thoughts?

[RussellSpitzer]

In the new rewriteDataFIles, we could add this as a new strategy ...

[skandasa23]

Thanks @RussellSpitzer for getting back on this. Can you please point me to any doc/pr describing about this new rewriteDataFiles? Thanks!

[RussellSpitzer]

#2501 - #2585 - #2591 - #2379

[rdblue]

I think that we would need to be careful about this. Deduplication makes sense, but it is an overwrite and not a replace operation because it modifies the data in the table. Compaction is a pure replace that does not modify the table data. We should keep the two separate because replace operations can be safely ignored when looking for change to the table data.

[skandasa23]

[Deleted my previous comment, hadn't tried out overwrite earlier]
Thank you @rdblue for the comment, makes sense to mark the operation as overwrite.
I'd a look at the code here, for incremental downstream consumers only "append" snapshots are being considered , delete is ignored and overwrite results in exception.
Is there a plan to propagate the mutations of delete/overwrite operations as well in future? @rdsr thoughts?

[rdblue]

Bulk operations are difficult to propagate as mutations. Most of the early incremental consumption focuses on the easy case, appends, for that reason. But there are ways to make it work. For example, you could read all the deleted files and all the added files in an overwrite and use a full outer join to label each row deleted, added, or kept and then feed those rows into incremental processing. That join is expensive, though.

Another strategy is to read just the added files and consider all of the changes sort of an upsert operation. As long as you know that all of the data coming in replaces 0 or 1 rows, then you may not need to know what the previous row was. The problem is that this makes assumptions about the operation that happened and doesn't apply to all cases. For your deduplication case, it wouldn't tell you when duplicate rows are removed.

I think the easier solution is turning a row-delta commit into changes because you can read the deleted rows and added rows directly without needing to process (and usually discard) the kept rows.

[skandasa23]

Thank you for the detailed comments @rdblue , issue 2782 seems to be tracking the delete/overwrite mutations already, wasn't aware of it earlier.
row-delta commit - guess you are talking about merge on read semantics?
I'm newer to Iceberg and catching up on newer rewrite commits. Will check those and get back on the approach for deduplication support while compaction.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'