You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently kill tasks take exclusive locks only(hardcoded). I believe there will be benefit to permit it in some scenarios to run without lock at all or append lock(TBD)
Motivation
In concurrent append and replace lock system we can run kafka ingestors in parallel to compaction tasks. However multiple compaction will create more unused segments, so to keep it controllable it will be good to clean every segment after compaction task finished. However currently kafka ingestors lock intervals with append lock, so kill task can't cleanup unused segments.
So my proposition is to permit kill task to run with replace lock(same as of compaction task) or no lock at all if we not marking segments as used(for any reason).
This should permit kill , compaction and kafka ingestors tasks to run concurrently, thus achieving best parallelism
Description
Currently kill tasks take exclusive locks only(hardcoded). I believe there will be benefit to permit it in some scenarios to run without lock at all or append lock(TBD)
Motivation
In concurrent append and replace lock system we can run kafka ingestors in parallel to compaction tasks. However multiple compaction will create more unused segments, so to keep it controllable it will be good to clean every segment after compaction task finished. However currently kafka ingestors lock intervals with append lock, so kill task can't cleanup unused segments.
So my proposition is to permit kill task to run with replace lock(same as of compaction task) or no lock at all if we not marking segments as used(for any reason).
This should permit kill , compaction and kafka ingestors tasks to run concurrently, thus achieving best parallelism
@AmatyaAvadhanula cc
The text was updated successfully, but these errors were encountered: