Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query: add query metrics to calls going through the Store API #5741

Merged
merged 37 commits into from
Oct 18, 2022

Conversation

douglascamata
Copy link
Contributor

@douglascamata douglascamata commented Sep 29, 2022

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Overall, this pull request's intent is to implement the first step in the proposal More granular query performance metrics.

Verification

  • Built the containers and ran it locally.
  • Added some tests, including e2e tests.

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
…ent-reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
@douglascamata douglascamata requested review from jessicalins and matej-g and removed request for matej-g and jessicalins October 10, 2022 12:44
@douglascamata
Copy link
Contributor Author

I wanted to re-request review from the 3 reviewers, but Github is drunk. Sorry for the possible notification spam.

image

jessicalins
jessicalins previously approved these changes Oct 13, 2022
Copy link
Contributor

@jessicalins jessicalins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

…etrics

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks 💪🏽 (:

pkg/store/telemetry/telemetry.go Outdated Show resolved Hide resolved
Comment on lines 59 to 60
"series_le": strconv.Itoa(int(seriesLeBucket)),
"samples_le": strconv.Itoa(int(samplesLeBucket)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concerns is still this: Is such "weird" metric usable in PromQL? (two dimensional histogram)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed in #5741 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it's actually len(series_le) * len(samples_le) number of histograms.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I am fine then, thanks for explanation (:

bwplotka
bwplotka previously approved these changes Oct 17, 2022
Comment on lines 59 to 60
"series_le": strconv.Itoa(int(seriesLeBucket)),
"samples_le": strconv.Itoa(int(samplesLeBucket)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it's actually len(series_le) * len(samples_le) number of histograms.

Comment on lines 59 to 60
"series_le": strconv.Itoa(int(seriesLeBucket)),
"samples_le": strconv.Itoa(int(samplesLeBucket)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I am fine then, thanks for explanation (:

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
matej-g
matej-g previously approved these changes Oct 17, 2022
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Copy link
Collaborator

@matej-g matej-g left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(previously approved, approving after merging main and merging this PR now)

@matej-g matej-g merged commit be95bb1 into thanos-io:main Oct 18, 2022
utukJ pushed a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
…-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>
utukJ added a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
utukJ added a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
…thanos-io#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <utukphd@gmail.com>
utukJ pushed a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
…-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>
utukJ added a commit to utukJ/thanos that referenced this pull request Oct 18, 2022
…thanos-io#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <utukphd@gmail.com>
GiedriusS added a commit that referenced this pull request Oct 27, 2022
…specify metric labels to collect (#5785)

* added unit tests for long labels and no external labels

Signed-off-by: utukj <utukphd@gmail.com>

* trimmed too long external labels

Signed-off-by: utukj <utukphd@gmail.com>

* added optional label selection

Signed-off-by: utukj <utukphd@gmail.com>

* added cmd flag for choosing metric labels

Signed-off-by: utukj <utukphd@gmail.com>

* updated docs

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset.go

full sentence fix

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* minor fixes from code review

Signed-off-by: utukj <utukphd@gmail.com>

* fixed code comments

Signed-off-by: utukj <utukphd@gmail.com>

* used enum for labels

Signed-off-by: utukj <utukphd@gmail.com>

* updated query docs

Signed-off-by: utukj <utukphd@gmail.com>

* cleaned up tests

Signed-off-by: utukj <utukphd@gmail.com>

* Updates busybox SHA (#5793)

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Receive: Reload tenant limit configuration on file change (#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run goimports

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update lastReload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run tests in parallel

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use absolute path to config file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of parallel test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless sleep

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused fucntion

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Query: add query metrics to calls going through the Store API (#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* docs: mark me as shepherd for next release (#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Revert "docs: mark me as shepherd for next release (#5797)"

This reverts commit ea646a6.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Query: add query metrics to calls going through the Store API (#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Receive: Reload tenant limit configuration on file change (#5673)"

This reverts commit 24e1cc0.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Updates busybox SHA (#5793)"

This reverts commit 9474c00.

Signed-off-by: utukj <utukphd@gmail.com>

* Updates busybox SHA (#5793)

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Receive: Reload tenant limit configuration on file change (#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run goimports

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update lastReload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run tests in parallel

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use absolute path to config file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of parallel test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless sleep

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused fucntion

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Query: add query metrics to calls going through the Store API (#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* docs: mark me as shepherd for next release (#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Revert "docs: mark me as shepherd for next release (#5797)"

This reverts commit c509c0e.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Updates busybox SHA (#5793)"

This reverts commit ad11a03.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Query: add query metrics to calls going through the Store API (#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Receive: Reload tenant limit configuration on file change (#5673)"

This reverts commit 32ca327.

Signed-off-by: utukj <utukphd@gmail.com>

* fixed lint issue

Signed-off-by: utukj <utukphd@gmail.com>

* added unit test for truncate and clean up

Signed-off-by: utukj <utukphd@gmail.com>

* fixed truncate label func and added more tests

Signed-off-by: utukj <utukphd@gmail.com>

* removed name from truncate test

Signed-off-by: utukj <utukphd@gmail.com>

* reorganized test cases and removed redundant comments

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* fixed failing checks

Signed-off-by: utukj <utukphd@gmail.com>

* e2e: Adding test for querier with two stores loadbalancing across them.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* dumped long expected output in unittest

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "e2e: Adding test for querier with two stores loadbalancing across them."

This reverts commit 96b1545.

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* moved label definition to endpointset

Signed-off-by: utukj <utukphd@gmail.com>

Signed-off-by: utukj <utukphd@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>
Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
GiedriusS added a commit to vinted/thanos that referenced this pull request Oct 31, 2022
…specify metric labels to collect (thanos-io#5785)

* added unit tests for long labels and no external labels

Signed-off-by: utukj <utukphd@gmail.com>

* trimmed too long external labels

Signed-off-by: utukj <utukphd@gmail.com>

* added optional label selection

Signed-off-by: utukj <utukphd@gmail.com>

* added cmd flag for choosing metric labels

Signed-off-by: utukj <utukphd@gmail.com>

* updated docs

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset.go

full sentence fix

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* minor fixes from code review

Signed-off-by: utukj <utukphd@gmail.com>

* fixed code comments

Signed-off-by: utukj <utukphd@gmail.com>

* used enum for labels

Signed-off-by: utukj <utukphd@gmail.com>

* updated query docs

Signed-off-by: utukj <utukphd@gmail.com>

* cleaned up tests

Signed-off-by: utukj <utukphd@gmail.com>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run goimports

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update lastReload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run tests in parallel

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use absolute path to config file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of parallel test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless sleep

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused fucntion

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit ea646a6.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 24e1cc0.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit 9474c00.

Signed-off-by: utukj <utukphd@gmail.com>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run goimports

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update lastReload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run tests in parallel

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use absolute path to config file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of parallel test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless sleep

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused fucntion

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit c509c0e.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit ad11a03.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 32ca327.

Signed-off-by: utukj <utukphd@gmail.com>

* fixed lint issue

Signed-off-by: utukj <utukphd@gmail.com>

* added unit test for truncate and clean up

Signed-off-by: utukj <utukphd@gmail.com>

* fixed truncate label func and added more tests

Signed-off-by: utukj <utukphd@gmail.com>

* removed name from truncate test

Signed-off-by: utukj <utukphd@gmail.com>

* reorganized test cases and removed redundant comments

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* fixed failing checks

Signed-off-by: utukj <utukphd@gmail.com>

* e2e: Adding test for querier with two stores loadbalancing across them.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* dumped long expected output in unittest

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "e2e: Adding test for querier with two stores loadbalancing across them."

This reverts commit 96b1545.

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* moved label definition to endpointset

Signed-off-by: utukj <utukphd@gmail.com>

Signed-off-by: utukj <utukphd@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>
Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
ngraham20 pushed a commit to ngraham20/thanos that referenced this pull request May 18, 2023
…specify metric labels to collect (thanos-io#5785)

* added unit tests for long labels and no external labels

Signed-off-by: utukj <utukphd@gmail.com>

* trimmed too long external labels

Signed-off-by: utukj <utukphd@gmail.com>

* added optional label selection

Signed-off-by: utukj <utukphd@gmail.com>

* added cmd flag for choosing metric labels

Signed-off-by: utukj <utukphd@gmail.com>

* updated docs

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset.go

full sentence fix

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* minor fixes from code review

Signed-off-by: utukj <utukphd@gmail.com>

* fixed code comments

Signed-off-by: utukj <utukphd@gmail.com>

* used enum for labels

Signed-off-by: utukj <utukphd@gmail.com>

* updated query docs

Signed-off-by: utukj <utukphd@gmail.com>

* cleaned up tests

Signed-off-by: utukj <utukphd@gmail.com>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run goimports

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update lastReload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run tests in parallel

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use absolute path to config file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of parallel test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless sleep

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused fucntion

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit ea646a6.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit eec4fd0.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 24e1cc0.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit 9474c00.

Signed-off-by: utukj <utukphd@gmail.com>

* Updates busybox SHA (thanos-io#5793)

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Receive: Reload tenant limit configuration on file change (thanos-io#5673)

* Create a PathOrContent reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add docs to staticPathContent.Rewrite

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run goimports

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Properly cancel the context in the test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Watch parent directory of file

This helps handling deletes and other situations.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless ctx.Done()

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add a debounce timer to config reload

It helps managing situations where a create event is followed by a write or when a big file write is sent by the fsnotify backend as many write events.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix event.Op bitmask check

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update lastReload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix debouncer for path content reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Improve documentation of the PathContentRealoder

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Dain reload timer before resetting

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Run tests in parallel

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify debouncing logic

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add more tests to file reloader

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Simplify condition for triggering reload

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use absolute path to config file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of parallel test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back 2s wait between fs operations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless sleep

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Stop reloadTimer when context cancelled

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove unused fucntion

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright to test file

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Auto-reload tenant limit config on file changes

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Wrap error when reloading config

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Move limiter config reloader and update logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of useless types and allocations

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove errorChan from config reload starter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Use UnRegisterer in the Limiter

To ensure that limit reloads will be able to re-register their metrics.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Better guard against nil registerer in the limiter

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove wrong nil guard

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Query: add query metrics to calls going through the Store API (thanos-io#5741)

* Implement granular query performance metrics for Thanos Query

These are grabbed from the data returned by multiple Store APIs after execution of a query.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix some linter warnings

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless logs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix long function definition (newQuerier)

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove TODO comment

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix query tests

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Reformat query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove useless return

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Put back old query docs

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update query docs again

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix e2e env name

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Add missing copyright notice.

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Bump wait time to twice scrape interval

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Attempt to fix randomly failing test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Checking more metrics to ensure the store is ready

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Clean up test

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Do not record store api metrics when didn't touch series or samples

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Retrigger CI

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Also skip store api metrics on zero chunks touched

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Update changelog

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Fix broken changelog after merge

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Remove extra empty line

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Refactor names and (un)exported types and fields

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Start listing metrics exported by Thanos Query

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Rename pkg/store/metrics -> pkg/store/telemetry

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

* Get rid of the pkg/store/telemetry package

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>

Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: utukj <utukphd@gmail.com>

* docs: mark me as shepherd for next release (thanos-io#5797)

Let's release the RC on Friday.

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: utukj <utukphd@gmail.com>

* Revert "docs: mark me as shepherd for next release (thanos-io#5797)"

This reverts commit c509c0e.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Updates busybox SHA (thanos-io#5793)"

This reverts commit ad11a03.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Query: add query metrics to calls going through the Store API (thanos-io#5741)"

This reverts commit 7a77769.

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "Receive: Reload tenant limit configuration on file change (thanos-io#5673)"

This reverts commit 32ca327.

Signed-off-by: utukj <utukphd@gmail.com>

* fixed lint issue

Signed-off-by: utukj <utukphd@gmail.com>

* added unit test for truncate and clean up

Signed-off-by: utukj <utukphd@gmail.com>

* fixed truncate label func and added more tests

Signed-off-by: utukj <utukphd@gmail.com>

* removed name from truncate test

Signed-off-by: utukj <utukphd@gmail.com>

* reorganized test cases and removed redundant comments

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* fixed failing checks

Signed-off-by: utukj <utukphd@gmail.com>

* e2e: Adding test for querier with two stores loadbalancing across them.

Signed-off-by: bwplotka <bwplotka@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* dumped long expected output in unittest

Signed-off-by: utukj <utukphd@gmail.com>

* Revert "e2e: Adding test for querier with two stores loadbalancing across them."

This reverts commit 96b1545.

Signed-off-by: utukj <utukphd@gmail.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* Update pkg/query/endpointset_test.go

Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>

* moved label definition to endpointset

Signed-off-by: utukj <utukphd@gmail.com>

Signed-off-by: utukj <utukphd@gmail.com>
Signed-off-by: Uwakmfon Utuk <41128987+utukJ@users.noreply.github.com>
Signed-off-by: GitHub <noreply@github.com>
Signed-off-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Signed-off-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Signed-off-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>
Signed-off-by: bwplotka <bwplotka@gmail.com>
Co-authored-by: Bartlomiej Plotka <bwplotka@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: yeya24 <yeya24@users.noreply.github.com>
Co-authored-by: Douglas Camata <159076+douglascamata@users.noreply.github.com>
Co-authored-by: Matej Gera <38492574+matej-g@users.noreply.github.com>
Co-authored-by: Giedrius Statkevičius <giedrius.statkevicius@vinted.com>

// Aggregate is an aggregator for merging `storepb.SeriesStatsCounter` for each incoming fanned out query.
func (s *seriesStatsAggregator) Aggregate(stats storepb.SeriesStatsCounter) {
s.seriesStats.Series += stats.Series

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't number of series overlap too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does, but we are (or were back then) not tracking the what individual series was touched. It would be quite expensive to do the tracking.

@@ -168,6 +169,7 @@ func (g *GRPCAPI) QueryRange(request *querypb.QueryRangeRequest, srv querypb.Que
request.EnableQueryPushdown,
false,
request.ShardInfo,
query.NoopSeriesStatsReporter,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this set to no-op here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because this is an internally initiated request, not something started by a user.

@christopherzli
Copy link

christopherzli commented May 16, 2024

just curious if we thought about expose these stats with the query response as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants