Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compactor: Support --min-time/--max-time filter. #4884

Closed
hanjm opened this issue Nov 20, 2021 · 4 comments
Closed

Compactor: Support --min-time/--max-time filter. #4884

hanjm opened this issue Nov 20, 2021 · 4 comments

Comments

@hanjm
Copy link
Member

hanjm commented Nov 20, 2021

Is your proposal related to a problem?

For my case, the compactor halt or broken for some reason. the compact progress will very be lag.
Then i restart compactor, beacuse the bucket has many blocks, I prefer it could process the more fresh blocks first. but currently, the compactor will start process the oldest block.

Describe the solution you'd like

So, Could we add a time filter like the thanos-store componet does?

Describe alternatives you've considered

Additional context

@yeya24
Copy link
Contributor

yeya24 commented Nov 20, 2021

The request is valid. But configuring min/max time for compaction seems much more complex than setting it for the store gateway.
Another way to do this is to do compaction planning from the freshest blocks. Right now all blocks are planned in time increasing order so fresh blocks are processed last. Maybe this can be done via a flag in the planner side.

@yeya24
Copy link
Contributor

yeya24 commented Nov 22, 2021

After 1:1 with @metonymic-smokey, we think reversing the planning order would work, but it is not as easy as it looks. So it is good to start adding the min/max time filter first.
But for users, they need to know the required time range for compaction levels to set these flags correctly.

@jordy1024
Copy link
Contributor

Work in progress.

jordy1024 added a commit to jordy1024/thanos that referenced this issue Nov 27, 2021
jordy1024 added a commit to jordy1024/thanos that referenced this issue Nov 27, 2021
Signed-off-by: wenmaoba <jordy1024@163.com>
jordy1024 added a commit to jordy1024/thanos that referenced this issue Dec 7, 2021
Signed-off-by: wenmaoba <jordy1024@163.com>
jordy1024 pushed a commit to jordy1024/thanos that referenced this issue Dec 7, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>
jordy1024 pushed a commit to jordy1024/thanos that referenced this issue Dec 10, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>
jordy1024 pushed a commit to jordy1024/thanos that referenced this issue Dec 12, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>
jordy1024 pushed a commit to jordy1024/thanos that referenced this issue Dec 12, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>
jordy1024 pushed a commit to jordy1024/thanos that referenced this issue Dec 12, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>
jordy1024 pushed a commit to jordy1024/thanos that referenced this issue Dec 12, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>
yeya24 pushed a commit that referenced this issue Dec 12, 2021
Signed-off-by: wenmaoba <wenmaoba@tencent.com>

Co-authored-by: wenmaoba <wenmaoba@tencent.com>
@yeya24
Copy link
Contributor

yeya24 commented Dec 12, 2021

Addressed by #4909

@yeya24 yeya24 closed this as completed Dec 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants