Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(static/metrics/instance): fix duplicate metrics registration panic when recreating the instance #6608

Merged
merged 3 commits into from
Mar 5, 2024
Merged

fix(static/metrics/instance): fix duplicate metrics registration panic when recreating the instance #6608

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0349eb5
fix(static/metrics/instance): fix duplicate metrics registration pani…
hainenber Mar 5, 2024
679ed0f
Merge branch 'main' into fix-write-handler-duplicate-metrics-collecto…
hainenber Mar 5, 2024
56d1bf8
chore(static/metrics/instance): re-add step to recreate instance but …
hainenber Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
fix(static/metrics/instance): fix duplicate metrics registration pani… 
…c when recreating the instance

Signed-off-by: hainenber <dotronghai96@gmail.com>
  • Loading branch information
@hainenber
hainenber committed Mar 5, 2024
commit 0349eb5366fb78efb8c2b9e332c687d075ef7d42
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ Main (unreleased)

- Fix an issue where Loki could reject a batch of logs when structured metadata feature is used. (@thampiotr)

- Fix a duplicate metrics registration panic when recreating static
mode metric instance's write handler. (@rfratto, @hainenber)

v0.40.1 (2024-02-27)
--------------------

Expand Down
2 changes: 1 addition & 1 deletion internal/static/metrics/instance/instance.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,7 @@ func (i *Instance) initialize(ctx context.Context, reg prometheus.Registerer, cf
return fmt.Errorf("error creating WAL: %w", err)
}

i.writeHandler = remote.NewWriteHandler(i.logger, i.reg, i.wal)
i.writeHandler = remote.NewWriteHandler(i.logger, reg, i.wal)

i.discovery, err = i.newDiscoveryManager(ctx, cfg)
if err != nil {
Expand Down
1 change: 0 additions & 1 deletion internal/static/metrics/instance/instance_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,6 @@ func TestInstance_Recreate(t *testing.T) {

// Recreate the instance, no panic should happen.
require.NotPanics(t, func() {
inst, err := New(prometheus.NewRegistry(), cfg, walDir, logger)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually recreates a blank new Prom registry and the children write handler won't have any duplicate registration, thus no panic.

Removing it allows for reproduction of the panic

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should remove this line (the comment doesn't recreate the instance anymore), but rather pass in the same registry on both line 261 and here. Can we add it back and make that change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-added :D

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I can confirm that without the change in instance.go, that also reproduces the panic.

require.NoError(t, err)
runInstance(t, inst)

Expand Down
Loading