Baseline weights VM runners

[mateo-moon]

Baseline weights generated on VM runners.

https://forum.parity.io/t/investigating-new-benchmarking-hardware/687/134

[ggwpez]

@ggwpez This is what i've told you about) I will run them separately.

Dont tell me, tell @mordamax 😆

[mateo-moon]

bot bench-vm -v UPSTREAM_MERGE=n $ all

[bkchr]

Did we already solve the benchmarking noise on these "machines"?

We didn't, because there wasn't one in the first place)

Could you give a Tldr?

[mateo-moon]

Did we already solve the benchmarking noise on these "machines"?

We didn't, because there wasn't one in the first place)

Could you give a Tldr?

During the test runs at firsts by @bakhtin then in my tests and @ggwpez ones we didn't observe any cases of 'noise neighbors'. Also i've only heard about this but never saw real proves. More info could be found in this thread https://forum.parity.io/t/investigating-new-benchmarking-hardware/687/134

[ggwpez]

Did we already solve the benchmarking noise on these "machines"?

The only noise we still see is in 3-5 benchmarks (two of them in pallet-contracts) consistently, but that is also present on bm3. So I assume that these benches are unstable themselves, and not a hardware problem.
@oleg-plakida can you maybe post some SWC links from the past VM+BM runs to demonstrate?

[mateo-moon]

Did we already solve the benchmarking noise on these "machines"?

The only noise we still see is in 3-5 benchmarks (two of them in pallet-contracts) consistently, but that is also present on bm3. So I assume that these benches are unstable themselves, and not a hardware problem. @oleg-plakida can you maybe post some SWC links from the past VM+BM runs to demonstrate?

Here is one for BM with some fluctuations.
Here is the vm one with similar results.

[bkchr]

The only noise we still see is in 3-5 benchmarks (two of them in pallet-contracts) consistently, but that is also present on bm3. So I assume that these benches are unstable themselves, and not a hardware problem.

What happened to paritytech/polkadot-sdk#379?

[ggwpez]

The only noise we still see is in 3-5 benchmarks (two of them in pallet-contracts) consistently, but that is also present on bm3. So I assume that these benches are unstable themselves, and not a hardware problem.

What happened to paritytech/polkadot-sdk#379?

Nothing so far. Problem is that even with very high repetitions (2000 instead of 50), some calls show consistent outlier behaviour.
For example batch. I ran it on bm2 on the last 10 master commits (this was a month back or so) and 10 times on each commit with 2000 reps. WASM shows consistent outliers - although nothing in the code of batch changed. Dont know if it is some WASM memory alignment or other oddities that screw it.

This lead me to de-prioritize the dynamic reps, as it wont solve that fluctuation.

Native looks a bit better:

[ggwpez]

After this is merged, please make another MR and then check that the bench-bot uses the new VM runners.

[mateo-moon]

After this is merged, please make another MR and then check that the bench-bot uses the new VM runners.

Needs one more approve from @paritytech/core-devs.

[koute]

Problem is that even with very high repetitions (2000 instead of 50), some calls show consistent outlier behaviour.

@ggwpez For the benches which are noisy, maybe you could try running them with this script of mine and see if it helps? https://gist.github.com/koute/83fb86c0b80213949d2985903a03a7f8 (Make sure ruby, setarch and schedtool are installed, download the script, chmod +x it, and run ./perftest.rb $command.)

It applies a bunch of system-level tweaks to make benchmarking more consistent; I find that on my machine it significantly helps in CPU bound benchmarks. (Most of which we honestly should most likely permanently preapply on our benchmarking machines.)

[mateo-moon]

bot bench-vm $ all