Core: Add KLL Datasketch and Hive ColumnStatisticsObj as standard blo…

[simhadri-g]

…b types to puffin file

Issue:
#8198

Hi Everyone,

Hive now supports writing column statistics to puffin files.

The statistics calculated by Hive include histograms, NDV (Number of Distinct Values), Min and Max values, the number of nulls, the number of true values, column name, and column type. You can find the full list of supported stats here: Link to GitHub.

These statistics are stored as a Hive columnStatistics object, which is serialized and saved as a blob in puffin. You can refer to the code here for more information: Link to GitHub.

Currently, this object is supported by Hive and partially by Impala as well: Link to GitHub. We also plan to incorporate the KLL datasketch for histograms.

As a result, we are looking to add columnStatistics object and KLL datasketch as standard blob types for the puffin file. Link to GitHub

Any feedback would be greatly appreciated.

Thanks!

[simhadri-g]

Thanks a lot for the review! :)

[simhadri-g]

@findepi I have addressed the review comments.

Can you please have a look at the PR when you are free?
Thanks in advance! :)

[simhadri-g]

Hello @findepi @nastra ,
I would greatly appreciate it if you could find some time to review the pull request.
Thanks!

[simhadri-g]

Thanks for the review :)
I will update the PR and get back.

[simhadri-g]

@nastra , I have updated the PR and moved the iceberg-docs(apache/iceberg-docs#269) changes to the main repo.

Please have a look when you are free.

Thanks again for the review!

[nastra]

LGTM, @findepi could you review this as well please?

[simhadri-g]

thanks @nastra for the review! :)

@findepi Could you please have a look as well?
Thanks!

[ZacBlanco]

Any progress update here? It would be great to get the blessing from the Iceberg community for these as supported puffin blob types

[bitsondatadev]

@findepi @danielcweeks could one of you PTAL this? Don't want this effort to get lost.

[bitsondatadev]

@simhadri-g update I'm following up on this today.

[simhadri-g]

thanks @bitsondatadev !

[zhangbutao]

Any update? hope we can continue to push this review forward.
Thanks.

[tdcmeehan]

Is what's referenced here the Thrift ColumnStatisticsObj in the Hive IDL? https://github.com/apache/hive/blob/ffb1165f59defa66b31b4fd9cb6367b71050071b/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L583

If so, I'd recommend correcting the name, and linking to the Thrift IDL, and explicitly calling out that this is Thrift-serialized.

I'm also wondering if we need to think about versioning. If this is based on the Thrift IDL, I am not sure if those are intended to be persisted. At the very least, I am concerned if Hive decides to introduce a backwards-incompatible field to this struct, some engines begin to serialize with this newly introduced backwards incompatible field, and other engines begin to attempt to deserialize it with an older IDL, then it will break in the Iceberg library.

Please let me know if I'm misunderstanding anything.

[simhadri-g]

Hi,
In hive , we writes the statistics to HMS in addition to puffin files.
So the Thrift ColumnStatisticsObj is used to write the statistics to HMS.

hive-column-statistics-obj referred here is a serialized using the org.apache.commons.lang3.SerializationUtils and stored into puffin file.

[simhadri-g]

Hi everyone,
I would be most grateful if we could get help with reviewing this.
Thanks!

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.