-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
query slow about near real time data #9918
Comments
Hi @cxlRay, |
some issue here. |
@yuanlihan thanks for your answer, minimise intermediatePersistPeriod will create much more small file |
@cxlRay that's true. But the temporary persist files will be cleaned up when the hourly tasks finished. And the persisted incremental files/indexes(with extra indexes to speed up query processing) will be more efficient than the in-memory |
timeseries is very slow with filter. (10-20s) |
@exherb say more about your question, if your cluster run on ssd, there will be better |
|
just like yuanlihan say,minimise intermediatePersistPeriod or maxRowsInMemory , add query parmeter vectorize |
intermediatePersistPeriod: P10M
still slow. |
rows in no-rollup incremental index are stored as
which seemed not easy to be fast, imho. Anyway, can you try with coarser granularity, like 15 minutes or something? |
we enabled rollup to 1minutes, segment granularity by hour. Are you suggestion change segment granularity to 15 minutes? |
@exherb how do you know the slow part is peon query,the query process: client -> broker -> indexing service(Peon), use monitor metric or analysis code? |
by druid metrics: |
@exherb I mean |
same issue. |
when send batch query to Realtime processes tasks, the performance is too bad. TPS only have 80, response time is more than one second,
max response time can be 30 second and 99% response time can be 15 second.
what I confused is the data of Realtime processes are in mem, why response time of query is so long?
Affected Version
v druid-0.16.1-incubating.
Description
Cluster size
coordinator and overlord: 2
historical: 7
middleManager: 7
broker: 5
The testing tool is jmeter
testing result
thread num: 500
TPS: 107.80
average response time: 4372ms
99% response time: 15559ms
max response time: 30150ms
min response time: 373
the Flame chart of Realtime task code when test
the configuration of middleManager
the configuration of Realtime tasks
about segments
my query
The text was updated successfully, but these errors were encountered: