Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query Frontend: fix downsampled middleware returning duplicate samples #4017

Merged

Conversation

krya-kryak
Copy link
Contributor

@krya-kryak krya-kryak commented Apr 4, 2021

Downsampled middleware incorrectly determined minimum response timestamp
using only one data point, which sometimes resulted in superflous requests
and duplicate samples in response. Those duplicate samples in turn broke
graph rendering in grafana in Thanos UI.

Signed-off-by: Vladimir Kononov krya-kryak@users.noreply.github.com

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Verification

Unit tests added for minResponseTime

Downsampled middleware incorrectly determined minimum response timestamp
using only one data point, which resulted in superflous requests
and duplicate samples in response. Those duplicate samples in turn broke
graph rendering in grafana in Thanos UI.

Signed-off-by: Vladimir Kononov <krya-kryak@users.noreply.github.com>
@krya-kryak krya-kryak force-pushed the fix-query-frontend-downsampling-iterator branch from 92e0439 to 6aaf4c2 Compare April 4, 2021 18:48
@krya-kryak krya-kryak changed the title Query Frontend: fix downsampling iterator returning duplicate samples Query Frontend: fix downsampled middleware returning duplicate samples Apr 4, 2021
@krya-kryak
Copy link
Contributor Author

I wonder if this calls for 0.19.1.

Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. CC @bwplotka

@krya-kryak
Copy link
Contributor Author

krya-kryak commented Apr 5, 2021

For the record: this is how broken graphs looked like in thanos UI:
image

The response contained duplicate points for label status="timeout" up to and including 1617264240:

        "result": [
            {
                "metric": {
                    "status": "error"
                },
                "values": [
                    [1617264270, "0.05340622092250431"],
                    [1617264300, "0.042536326980866526"]
                ]
            },
            {
                "metric": {
                    "status": "timeout"
                },
                "values": [
                    [1617264180, "0.31111111111111106"],
                    [1617264210, "0.37779284253030704"],
                    [1617264240, "0.38888888888888884"],
                    [1617264180, "0.31111111111111106"],
                    [1617264210, "0.37779284253030704"],
                    [1617264240, "0.38888888888888884"],
                    [1617264270, "0.811111111111111"],
                    [1617264300, "0.6666522236665222"]
                ]
            }
        ]
    }
}

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! Solid work, LGTM 🤗

@bwplotka bwplotka merged commit 81b1591 into thanos-io:main Apr 6, 2021
@krya-kryak krya-kryak deleted the fix-query-frontend-downsampling-iterator branch July 7, 2021 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants