Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kill task using exclusive locks currently and no way to change it #16361

Closed
IgorBerman opened this issue Apr 30, 2024 · 0 comments · Fixed by #16362
Closed

Kill task using exclusive locks currently and no way to change it #16361

IgorBerman opened this issue Apr 30, 2024 · 0 comments · Fixed by #16362

Comments

@IgorBerman
Copy link
Contributor

Description

Currently kill tasks take exclusive locks only(hardcoded). I believe there will be benefit to permit it in some scenarios to run without lock at all or append lock(TBD)

Motivation

In concurrent append and replace lock system we can run kafka ingestors in parallel to compaction tasks. However multiple compaction will create more unused segments, so to keep it controllable it will be good to clean every segment after compaction task finished. However currently kafka ingestors lock intervals with append lock, so kill task can't cleanup unused segments.
So my proposition is to permit kill task to run with replace lock(same as of compaction task) or no lock at all if we not marking segments as used(for any reason).
This should permit kill , compaction and kafka ingestors tasks to run concurrently, thus achieving best parallelism

@AmatyaAvadhanula cc

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant