You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are running a very large Druid cluster and we have 8 autocompaction tasks ("compact" type) running in parallel to consolidate a kafka-based datasource.
Although the compaction tasks don't have the same duration, we have noticed that they are always in sync, meaning that there is no new compact task started if there's at least one still running. This is technically ok, but it's a massively inefficient use of CPU resources.
For clarity, the compaction tasks duration is roughly 4-5 hours, and druid.coordinator.period.indexingPeriod=PT10M.
It doesn't look like there is any way to tune this, and the behaviour seems undocumented as far as I can read, so as it's not expected, I'm filling it as a PR.
The text was updated successfully, but these errors were encountered:
gl
changed the title
Auto-Compaction tasks not started if any are running
Auto-Compaction parent tasks not started if any are running
Sep 13, 2023
gl
changed the title
Auto-Compaction parent tasks not started if any are running
Auto-Compaction parent tasks not started if any are still running
Sep 13, 2023
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
Affected Version
Druid 27
Description
We are running a very large Druid cluster and we have 8 autocompaction tasks ("compact" type) running in parallel to consolidate a kafka-based datasource.
Although the compaction tasks don't have the same duration, we have noticed that they are always in sync, meaning that there is no new compact task started if there's at least one still running. This is technically ok, but it's a massively inefficient use of CPU resources.
For clarity, the compaction tasks duration is roughly 4-5 hours, and druid.coordinator.period.indexingPeriod=PT10M.
It doesn't look like there is any way to tune this, and the behaviour seems undocumented as far as I can read, so as it's not expected, I'm filling it as a PR.
The text was updated successfully, but these errors were encountered: