Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect comparator usage in FinalizingPostAggregators #16554

Open
kgyrtkirk opened this issue Jun 5, 2024 · 0 comments · May be fixed by #16555
Open

Incorrect comparator usage in FinalizingPostAggregators #16554

kgyrtkirk opened this issue Jun 5, 2024 · 0 comments · May be fixed by #16555

Comments

@kgyrtkirk
Copy link
Member

kgyrtkirk commented Jun 5, 2024

For aggregators which have a non-trivial finalization AggregatorFactory#getResultType() != AggregatorFactory#getIntermediateType() (so essentialy those which will most likely need some work to be done).

The comparator is derived for the underlying aggregator for the postAggregator here - so a comparision of finalized values might be tried with the aggregator's comparator.

This could cause an exception in case the postAggregators output is being used to order the rows by it.

Example query:

select dim1,e
from (SELECT dim1, EARLIEST(f1) e FROM druid.numfoo GROUP BY 1 ORDER BY 2 LIMIT 10) t
order by 1 limit 2
This causes a ClassCastException
java.lang.ClassCastException: class java.lang.Float cannot be cast to class org.apache.druid.collections.SerializablePair (java.lang.Float is in module java.base of loader 'bootstrap'; org.apache.druid.collections.SerializablePair is in unnamed module of loader 'app')
	at java.base/java.util.Collections$ReverseComparator2.compare(Collections.java:5278)
	at java.base/java.util.Comparators$NullComparator.compare(Comparators.java:83)
	at org.apache.druid.query.topn.TopNNumericResultBuilder.lambda$0(TopNNumericResultBuilder.java:75)
	at java.base/java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:675)
	at java.base/java.util.PriorityQueue.siftUp(PriorityQueue.java:652)
	at java.base/java.util.PriorityQueue.offer(PriorityQueue.java:345)
	at java.base/java.util.PriorityQueue.add(PriorityQueue.java:326)
	at org.apache.druid.query.topn.TopNNumericResultBuilder.addEntry(TopNNumericResultBuilder.java:157)
	at org.apache.druid.query.topn.TopNNumericResultBuilder.addEntry(TopNNumericResultBuilder.java:1)
	at org.apache.druid.query.topn.PooledTopNAlgorithm.updateResults(PooledTopNAlgorithm.java:765)
	at org.apache.druid.query.topn.PooledTopNAlgorithm.updateResults(PooledTopNAlgorithm.java:1)
	at org.apache.druid.query.topn.BaseTopNAlgorithm.runWithCardinalityKnown(BaseTopNAlgorithm.java:121)
	at org.apache.druid.query.topn.BaseTopNAlgorithm.run(BaseTopNAlgorithm.java:82)
	at org.apache.druid.query.topn.TopNMapFn.apply(TopNMapFn.java:70)
	at org.apache.druid.query.topn.TopNQueryEngine.lambda$0(TopNQueryEngine.java:100)
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
	at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41)
	at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45)
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
	at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45)
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:98)
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:185)
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$1(SpecificSegmentQueryRunner.java:181)
	at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:165)
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
	at org.apache.druid.java.util.common.guava.Sequence.toList(Sequence.java:87)
	at org.apache.druid.query.ChainedExecutionQueryRunner$1$3.call(ChainedExecutionQueryRunner.java:112)
	at org.apache.druid.query.ChainedExecutionQueryRunner$1$3.call(ChainedExecutionQueryRunner.java:1)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:79)
	at org.apache.druid.query.ForwardingQueryProcessingPool.submitRunnerTask(ForwardingQueryProcessingPool.java:45)
	at org.apache.druid.query.ChainedExecutionQueryRunner$1.lambda$0(ChainedExecutionQueryRunner.java:100)
	at com.google.common.collect.Iterators$6.transform(Iterators.java:828)
	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:52)
	at com.google.common.collect.Iterators.addAll(Iterators.java:366)
	at com.google.common.collect.Lists.newArrayList(Lists.java:146)
	at com.google.common.collect.Lists.newArrayList(Lists.java:132)
	at org.apache.druid.query.ChainedExecutionQueryRunner$1.make(ChainedExecutionQueryRunner.java:92)
	at org.apache.druid.java.util.common.guava.BaseSequence.toYielder(BaseSequence.java:66)
	at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78)
	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
	at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78)
	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
	at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49)
	at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46)
	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88)
	at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:1)
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
	at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83)
	at org.apache.druid.java.util.common.guava.MergeSequence.lambda$1(MergeSequence.java:66)
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
	at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:63)
	at org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:133)
	at org.apache.druid.java.util.common.guava.YieldingSequenceBase.accumulate(YieldingSequenceBase.java:35)
	at org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62)
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77)
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
	at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:42)
	at org.apache.druid.server.ClientQuerySegmentWalker.materializeResultsAsArray(ClientQuerySegmentWalker.java:810)
	at org.apache.druid.server.ClientQuerySegmentWalker.toInlineDataSource(ClientQuerySegmentWalker.java:680)
	at org.apache.druid.server.ClientQuerySegmentWalker.inlineIfNecessary(ClientQuerySegmentWalker.java:440)
	at org.apache.druid.server.ClientQuerySegmentWalker.getQueryRunnerForIntervals(ClientQuerySegmentWalker.java:219)
	at org.apache.druid.server.SpecificSegmentsQuerySegmentWalker.getQueryRunnerForIntervals(SpecificSegmentsQuerySegmentWalker.java:214)
	at org.apache.druid.query.spec.MultipleIntervalSegmentSpec.lookup(MultipleIntervalSegmentSpec.java:57)
	at org.apache.druid.query.BaseQuery.getRunner(BaseQuery.java:119)
	at org.apache.druid.query.QueryPlus.run(QueryPlus.java:149)
	at org.apache.druid.server.QueryLifecycle.execute(QueryLifecycle.java:316)
	at org.apache.druid.server.QueryLifecycle.runSimple(QueryLifecycle.java:163)
	at org.apache.druid.sql.calcite.run.NativeQueryMaker.execute(NativeQueryMaker.java:193)
	at org.apache.druid.sql.calcite.run.NativeQueryMaker.runQuery(NativeQueryMaker.java:151)
	at org.apache.druid.sql.calcite.rel.DruidRel.runQuery(DruidRel.java:60)
	at org.apache.druid.sql.calcite.planner.QueryHandler.lambda$8(QueryHandler.java:619)
	at org.apache.druid.sql.calcite.planner.PlannerResult.run(PlannerResult.java:62)
	at org.apache.druid.sql.DirectStatement$ResultSet.run(DirectStatement.java:109)
	at org.apache.druid.sql.DirectStatement.execute(DirectStatement.java:184)
	at org.apache.druid.sql.calcite.QueryTestRunner$ExecuteQuery.lambda$0(QueryTestRunner.java:308)
	at org.apache.druid.sql.calcite.util.QueryLogHook.logQueriesFor(QueryLogHook.java:67)
	at org.apache.druid.sql.calcite.QueryTestRunner$ExecuteQuery.runQuery(QueryTestRunner.java:306)
	at org.apache.druid.sql.calcite.QueryTestRunner$ExecuteQuery.run(QueryTestRunner.java:286)
	at org.apache.druid.sql.calcite.QueryTestRunner.run(QueryTestRunner.java:768)
	at org.apache.druid.sql.calcite.QueryTestBuilder.run(QueryTestBuilder.java:303)
	at org.apache.druid.sql.calcite.BaseCalciteQueryTest.testQuery(BaseCalciteQueryTest.java:712)
	at org.apache.druid.sql.calcite.CalciteQueryTest.testOrderByEarliestFloat(CalciteQueryTest.java:1772)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:95)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:91)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:60)
	at org.eclipse.jdt.internal.junit5.runner.JUnit5TestReference.run(JUnit5TestReference.java:98)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:40)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:529)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:756)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:452)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:210)
Testcase for CalciteQueryTest
  @Test
  public void testOrderByEarliestFloatSubQuery()
  {
    testBuilder()
        .sql(
            "select dim1,e from (SELECT dim1, EARLIEST(f1) e FROM druid.numfoo GROUP BY 1 ORDER BY 2 LIMIT 10) t order by 1 limit 2"
        )
        .run();
  }
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant