Use Theta sketch compression

[AlexanderSaydakov]

Theta sketch compression is available for quite some time in the Apache DataSketches library. I would suggest enabling it in Druid. The simplest way would be to start serializing Theta sketches in compressed format. Deserialization automatically detects and supports that format starting from datasketches-java-4.0.0 and datasketches-cpp-4.1.0 (May 2023).
There is some overhead in converting sketches to bytes, but in an I/O bound system usually this is a reasonable CPU vs I/O tradeoff. In other words, compression reduces I/O (and storage cost) by spending more CPU, which is likely to yield overall benefit.

[AlexanderSaydakov]

Alternative to compression by default would be making it configurable by the user per column or per table or per installation or some other way. I am not sure this extra complexity is needed.

[abhishekagarwal87]

How much of a CPU overhead does the compression come with?

[AlexanderSaydakov]

Sorry I could not find measurements in Java. I will run them again, but that takes quite a while. Here are measurements in C++ just to have some idea.

[AlexanderSaydakov]

This time is just to convert sketches to bytes.