Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for hysteresis thresholds for benchmarks #826

Open
chevdor opened this issue May 12, 2022 · 1 comment
Open

Add support for hysteresis thresholds for benchmarks #826

chevdor opened this issue May 12, 2022 · 1 comment
Labels
I5-enhancement An additional feature request.

Comments

@chevdor
Copy link
Contributor

chevdor commented May 12, 2022

Currently, considering calls for which the code does not change, we can see weights oscillating.
This is expected due to the nature of the benchmarks.

For instance 6_605_0007_054_0006_856_000.

We could implement configurable thresholds for the weight differences and handle them with an hysteresis to limit the noise.

If we set the threshold(s) to 0, we would get the current behavior where any change in weight is considered.

However, setting this threshold to 10% for instance, would not update the weights unless the change is over the threshold. As a result, we would greatly limit the noise in the weight updates.

We can use different thresholds for increases and decreasses.

This would make the diff and json smaller, and the reviews easier/faster as we could focus more on the changes that do matter.

@chevdor
Copy link
Contributor Author

chevdor commented May 12, 2022

For some context, here is how a diff looks like today.

@Sophia-Gold Sophia-Gold transferred this issue from paritytech/polkadot Aug 24, 2023
@the-right-joyce the-right-joyce added I5-enhancement An additional feature request. and removed J0-enhancement labels Aug 25, 2023
claravanstaden pushed a commit to Snowfork/polkadot-sdk that referenced this issue Dec 8, 2023
* Cleanups for CommonPrefix review

* misc renaming
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
I5-enhancement An additional feature request.
Projects
None yet
Development

No branches or pull requests

3 participants
@chevdor @the-right-joyce and others