Query Failures (InvalidRoaringFormat)

[forzamehlano]

When running a query, we're seeing it fail due to an error on some of the historical nodes around the InvalidRoaringFormat

Affected Version

26.0.0

Description

Our historicals are running with the following config:

runtime.properties: | druid.server.tier=hot druid.service=druid/historical/hot druid.plaintextPort=8083 druid.server.http.numThreads=100 druid.processing.buffer.sizeBytes=2000MiB druid.processing.numMergeBuffers=5 druid.processing.numThreads=19 druid.processing.tmpDir=var/druid/processing

And we're seeing the following error on a handful of historicals:

2023-08-07T09:07:31,673 ERROR [processing-35] org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences! org.roaringbitmap.InvalidRoaringFormat: I failed to find one of the right cookies. 75124725 at org.roaringbitmap.buffer.ImmutableRoaringArray.<init>(ImmutableRoaringArray.java:48) ~[RoaringBitmap-0.9.0.jar:?] at org.roaringbitmap.buffer.ImmutableRoaringBitmap.<init>(ImmutableRoaringBitmap.java:1057) ~[RoaringBitmap-0.9.0.jar:?] at org.apache.druid.segment.data.RoaringBitmapSerdeFactory$ImmutableRoaringBitmapObjectStrategy.fromByteBuffer(RoaringBitmapSerdeFactory.java:80) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.data.RoaringBitmapSerdeFactory$ImmutableRoaringBitmapObjectStrategy.fromByteBuffer(RoaringBitmapSerdeFactory.java:64) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.data.GenericIndexed$BufferIndexed.get(GenericIndexed.java:497) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex.getBitmap(IndexedUtf8ValueSetIndex.java:128) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex.access$100(IndexedUtf8ValueSetIndex.java:40) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex$3$1.next(IndexedUtf8ValueSetIndex.java:230) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex$3$1.next(IndexedUtf8ValueSetIndex.java:205) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.collections.bitmap.RoaringBitmapFactory$1$1.next(RoaringBitmapFactory.java:84) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.collections.bitmap.RoaringBitmapFactory$1$1.next(RoaringBitmapFactory.java:68) ~[druid-processing-26.0.0.jar:26.0.0] at org.roaringbitmap.buffer.BufferFastAggregation.naive_or(BufferFastAggregation.java:620) ~[RoaringBitmap-0.9.0.jar:?] at org.roaringbitmap.buffer.BufferFastAggregation.or(BufferFastAggregation.java:713) ~[RoaringBitmap-0.9.0.jar:?] at org.roaringbitmap.buffer.ImmutableRoaringBitmap.or(ImmutableRoaringBitmap.java:875) ~[RoaringBitmap-0.9.0.jar:?] at org.apache.druid.collections.bitmap.RoaringBitmapFactory.union(RoaringBitmapFactory.java:142) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.DefaultBitmapResultFactory.unionDimensionValueBitmaps(DefaultBitmapResultFactory.java:73) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.DefaultBitmapResultFactory.unionDimensionValueBitmaps(DefaultBitmapResultFactory.java:25) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.column.SimpleImmutableBitmapIterableIndex.computeBitmapResult(SimpleImmutableBitmapIterableIndex.java:40) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.filter.AndFilter.getBitmapIndex(AndFilter.java:83) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.FilterAnalysis.analyzeFilter(FilterAnalysis.java:107) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.QueryableIndexCursorSequenceBuilder.buildVectorized(QueryableIndexCursorSequenceBuilder.java:213) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.segment.QueryableIndexStorageAdapter.makeVectorCursor(QueryableIndexStorageAdapter.java:236) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$1.make(VectorGroupByEngine.java:148) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$1.make(VectorGroupByEngine.java:144) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:98) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:185) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:165) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:252) ~[druid-processing-26.0.0.jar:26.0.0] at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:239) ~[druid-processing-26.0.0.jar:26.0.0] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:251) ~[druid-processing-26.0.0.jar:26.0.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?]

The query itself can be summarised as :
select column1 from (select distinct column1, column2 from table where __time >= CURRENT TIMESTAMP - INTERVAL '3' DAY AND column1 IN ('list_of_values')) GROUP BY column1

Memory/CPU doesn't appear to be an issue. We have tried wiping all historicals and loading the data fresh from deep storage but the error persists.

Not really sure where to go from here...

[forzamehlano]

We're seeing this on 27.0.0 as well.

Any ideas?

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.