-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[HUDI-5823] Partition ttl management #9723
Conversation
59df678
to
8081b29
Compare
b81e894
to
2c1e5f6
Compare
Will version 1.0 support this feature? This feature is greatly needed. |
2c1e5f6
to
2d82832
Compare
9ef8e0c
to
6550cc1
Compare
0aa63bd
to
67ee450
Compare
throw new IllegalArgumentException("No PartitionTTLStrategyType found for class name: " + className); | ||
} | ||
|
||
public static List<String> getNames() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename to getPartitionTTLStrategyNames?
@@ -583,6 +594,12 @@ protected void runTableServicesInline(HoodieTable table, HoodieCommitMetadata me | |||
metadata.addMetadata(HoodieClusteringConfig.SCHEDULE_INLINE_CLUSTERING.key(), "true"); | |||
inlineScheduleClustering(extraMetadata); | |||
} | |||
|
|||
// Do an inline partition ttl management if enabled | |||
if (config.isAutoPartitionTTL()) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
isInlinePartitionTTLEnabled would be better?
79c2b8d
to
8712b4d
Compare
/** | ||
* Strategy for ttl management. | ||
*/ | ||
public interface TTLStrategy { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this interface is empty with no methods?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it's reserved for row-level ttl management.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please also update the docs to show how to use the partition TTL.
* @param referenceTime last commit time or creation time for partition | ||
*/ | ||
protected boolean isPartitionExpired(String referenceTime) { | ||
String expiredTime = instantTimePlusMillis(fixInstantTimeCompatibility(referenceTime), ttlInMilis); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, fixInstantTimeCompatibility is called twice.
hudi/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java
Lines 97 to 106 in f8e559f
public static String instantTimePlusMillis(String timestamp, long milliseconds) { | |
try { | |
String timestampInMillis = fixInstantTimeCompatibility(timestamp); | |
LocalDateTime dt = LocalDateTime.parse(timestampInMillis, MILLIS_INSTANT_TIME_FORMATTER); | |
ZoneId zoneId = HoodieTimelineTimeZone.UTC.equals(commitTimeZone) ? ZoneId.of("UTC") : ZoneId.systemDefault(); | |
return MILLIS_INSTANT_TIME_FORMATTER.format(dt.atZone(zoneId).toInstant().plusMillis(milliseconds).atZone(zoneId).toLocalDateTime()); | |
} catch (DateTimeParseException e) { | |
throw new HoodieException(e); | |
} | |
} |
Change Logs
Poc code for Partition ttl management
Impact
NONE
Risk level (write none, low medium or high below)
NONE
Documentation Update
NONE
Contributor's checklist