Add interfaces for Action RemoveExpiredFiles #10643
Conversation
Co-authored-by: Yufei Gu <yufei@apache.org>
| @@ -70,4 +70,10 @@ default RewritePositionDeleteFiles rewritePositionDeletes(Table table) { | |||
| throw new UnsupportedOperationException( | |||
| this.getClass().getName() + " does not implement rewritePositionDeletes"); | |||
| } | |||
|
|
|||
| /** Instantiates an action to remove expired files. */ | |||
| default RemoveExpiredFiles removeExpiredFiles(Table table) { | |||
There was a problem hiding this comment.
@huaxingao: Can you please provide more background about why this interface is needed and what spark action we are planning to build on top of it? As per me, expireSnapshots action itself can clean the expired data and metadata files. So, I am wondering why this is required?
There was a problem hiding this comment.
This is part of earlier PR #4705. I think this PR was separated out to get consensus on the interface for this action.
There was a problem hiding this comment.
Yes, this PR was separated out from #4705 to get consensus on the interface for this action.
The ExpireSnapshots action does delete the files associated with expired snapshots. However, the RemoveExpiredFiles action serves a different purpose. While ExpireSnapshots removes the expired snapshots from the top-level metadata, some files that should be deleted might still remain due to actions like table copying. Essentially, RemoveExpiredFiles allows the target table to simulate the ExpireSnapshots behavior in the source table, ensuring that the files are cleaned up appropriately.
There was a problem hiding this comment.
a) Let me go through the table copying PR and the changes looks like it is specifically needed only for table copy.
b) Can't the existing remove orphan files clean those left over files after table copy?
c) RemoveExpiredFiles, I definitely want the naming to be different. It is already confusing. Maybe as RemoveTableCopyOrphanFiles
There was a problem hiding this comment.
Yes, the existing remove orphan files can clean those left over files, but it is very expensive, especially for S3.
api/src/main/java/org/apache/iceberg/actions/RemoveExpiredFiles.java
Outdated
Show resolved
Hide resolved
Co-authored-by: Yufei Gu yufei@apache.org
Co-authored-by: Huaxin Gao huaxin_gao@apple.com
This PR adds interfaces for Action RemoveExpiredFiles. We will have another PR to implement the interfaces in Spark 3.4 and Spark 3.5.