You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We test the merge of iceberg v2 table.
step 1.start a flink job writing iceberg v2 table
step 2.start a scheduler job to merge and expire snapshot for flink
MergingSnapshotProducer#validateDataFilesExist method throw ValidException after the merging and expiring snapshot operation of iceberg v2 table remain only one snapshot.
2021-07-01 08:47:58,954 WARN org.apache.flink.runtime.taskmanager.Task [] - IcebergFilesCommitter -> Sink: IcebergSink iceberg-specific-catalog.rt_fdm.rt_fdm_tb_poi_pool (1/1)#739 (10834a66d2fc68e08c7c8430e8d97d5d) switched from RUNNING to FAILED.
org.apache.iceberg.exceptions.ValidationException: Cannot determine history between starting snapshot null and current 5038879599156612903
at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:46) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.MergingSnapshotProducer.validateDataFilesExist(MergingSnapshotProducer.java:317) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.BaseRowDelta.validate(BaseRowDelta.java:95) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:172) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:293) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:213) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:197) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:292) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:376) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:351) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:269) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.iceberg.flink.sink.IcebergFilesCommitter.initializeState(IcebergFilesCommitter.java:175) ~[iceberg-flink-runtime-flink_12_v1-19-gc19f858.dirty.jar-1624956123202.jar:?]
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:111) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:290) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:427) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:545) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:535) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:575) ~[flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:758) [flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:573) [flink-dist_2.11-1.12-auto1.5.jar:1.12-auto1.5]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92]
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
We test the merge of iceberg v2 table.
step 1.start a flink job writing iceberg v2 table
step 2.start a scheduler job to merge and expire snapshot for flink
MergingSnapshotProducer#validateDataFilesExist method throw ValidException after the merging and expiring snapshot operation of iceberg v2 table remain only one snapshot.
and the metadata file
The text was updated successfully, but these errors were encountered: