Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mimir query engine: add tool to find disabled but supported test cases #9018

Merged
merged 5 commits into from
Aug 16, 2024

Conversation

charleskorn
Copy link
Contributor

@charleskorn charleskorn commented Aug 16, 2024

What this PR does

This PR adds a tool to find upstream test cases that are currently disabled but are now supported by MQE, and enables those test cases that are supported.

Which issue(s) this PR fixes or relates to

(none)

Checklist

  • Tests updated.
  • [n/a] Documentation added.
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
  • [n/a] about-versioning.md updated with experimental features.

@charleskorn charleskorn marked this pull request as ready for review August 16, 2024 05:03
@charleskorn charleskorn requested a review from a team as a code owner August 16, 2024 05:03
Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we perhaps at a point where we can just enable all the tests but not fail on ones that emit NotSupportedError?

Granted it's possible that we might miss something if we're inadvertently returning NotSupportedError but if we only check that on upstream test cases hopefully we're not marking anything as not supported incorrectly and it would be detected by our test cases if that were to happen.

I feel like the risk here is low enough, and we add a TODO to remove the disabled test once we reach parity. Or once we're very close to parity we check for a specific disabled reason.

@charleskorn
Copy link
Contributor Author

Are we perhaps at a point where we can just enable all the tests but not fail on ones that emit NotSupportedError?

Unfortunately promqltest doesn't support filtering for expected errors like these, and I'm not sure Prometheus would be keen on adding support for this given it's not relevant for Prometheus, and won't be relevant for us once we reach feature parity.

I also prefer the explicitness of enabling and disabling tests to ensure we don't accidentally start rejecting queries as unsupported.

It also allows us to disable tests that are supported but are expected to return incorrect results for features under development.

Copy link
Contributor

@jhesketh jhesketh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. This is already done, no point messing around with other options (eg exporting test from prometheus so we can write our own runTest).

@charleskorn charleskorn merged commit ec2b221 into main Aug 16, 2024
29 checks passed
@charleskorn charleskorn deleted the charleskorn/find-supported-test-cases branch August 16, 2024 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants