Add execution overhead benchmarking #10977

ggwpez · 2022-03-03T22:04:46Z

Adds the benchmark-overhead sub command; tracked in paritytech/polkadot-sdk#393.
Builds one empty and one full block and measures their per-block and per-extrinsic execution overhead.

Generated weight files (with bogus values): block_weights.rs.txt, extrinsic_weights.rs.txt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

This reverts commit 9ce7f32.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

bin/node-template/node/src/command.rs

utils/frame/benchmarking-cli/src/storage/record.rs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

shawntabrizi · 2022-03-10T01:16:49Z

utils/frame/benchmarking-cli/src/storage/record.rs

@@ -107,6 +107,7 @@ impl BenchRecord {
 	}
 }

+// TODO refactor once https://doc.rust-lang.org/test/stats/trait.Stats.html is stabilized.


for TODOs, please open an issue, and put the issue number into the commented code

Looking at this again: it says This is a nightly-only experimental API. (test).
Does this mean that it is only available for testing? Then I cannot use it...

utils/frame/benchmarking-cli/src/block/bench.rs

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez · 2022-03-16T16:26:24Z

bin/node/cli/src/command.rs

@@ -95,13 +98,21 @@ pub fn run() -> Result<()> {
 				You can enable it with `--features runtime-benchmarks`."
 					.into())
 			},
-		Some(Subcommand::BenchmarkStorage(cmd)) => {
-			if !cfg!(feature = "runtime-benchmarks") {


I kicked this feature check out since it is not needed.

ggwpez · 2022-03-16T16:28:08Z

I addressed all your comments and added the Polkadot side: paritytech/polkadot#5137
@bkchr

ggwpez · 2022-03-17T10:40:22Z

bot merge

* Add benchmark-block Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove first approach This reverts commit 9ce7f32. * Add block and extrinsic benchmarks * Doc Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix template Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Check for non-empty chain Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add tests for Stats Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Apply suggestions from code review Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Push first version again Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Push first version again Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Update utils/frame/benchmarking-cli/src/overhead/template.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Doc + Template fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Comment fix Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add test Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Pust merge fixup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fixup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Move code to better place Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Add benchmark-block

9ce7f32

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

ggwpez added the A3-in_progress Pull request is in progress. No review needed at this stage. label Mar 3, 2022

ggwpez added 3 commits March 7, 2022 15:59

Remove first approach

f07941f

This reverts commit 9ce7f32.

Add block and extrinsic benchmarks

60d2ebd

Merge remote-tracking branch 'origin/master' into oty-block-bench

5fb334b

ggwpez marked this pull request as ready for review March 7, 2022 15:14

github-actions bot added A0-please_review Pull request needs code review. and removed A3-in_progress Pull request is in progress. No review needed at this stage. labels Mar 7, 2022

ggwpez added A3-in_progress Pull request is in progress. No review needed at this stage. and removed A0-please_review Pull request needs code review. labels Mar 7, 2022

ggwpez changed the title ~~WIP: Add block benchmarking~~ Add block benchmarking Mar 7, 2022

ggwpez added 4 commits March 7, 2022 16:29

Doc

8a3a8a8

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Fix template

8928d9f

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Beauty fixes

6631602

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Check for non-empty chain

e683edb

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

shawntabrizi reviewed Mar 8, 2022

View reviewed changes

bin/node-template/node/src/command.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Mar 8, 2022

View reviewed changes

utils/frame/benchmarking-cli/src/storage/record.rs Outdated Show resolved Hide resolved

ggwpez mentioned this pull request Mar 9, 2022

Runtime dependent weights paritytech/polkadot#5064

Merged

ggwpez added 3 commits March 9, 2022 17:34

Add tests for Stats

3f2eaf4

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Review fixes

a4ec657

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Review fixes

a73644b

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

shawntabrizi reviewed Mar 10, 2022

View reviewed changes

utils/frame/benchmarking-cli/src/block/bench.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Mar 10, 2022

View reviewed changes

utils/frame/benchmarking-cli/src/block/bench.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Mar 10, 2022

View reviewed changes

utils/frame/benchmarking-cli/src/block/bench.rs Outdated Show resolved Hide resolved

ggwpez and others added 10 commits March 15, 2022 12:43

Update utils/frame/benchmarking-cli/src/overhead/template.rs

a440311

Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>

Review fixes

0878563

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Doc + Template fixes

6076428

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Review fixes

59f478e

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Comment fix

533667c

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Add test

a2c788a

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Merge remote-tracking branch 'origin/master' into oty-block-bench

fc9f675

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Pust merge fixup

7c6b2c0

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Fixup

2bca7bb

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

Move code to better place

cf2e763

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

shawntabrizi mentioned this pull request Mar 16, 2022

Integrate benchmark-overhead command paritytech/polkadot#5137

Merged

shawntabrizi approved these changes Mar 16, 2022

View reviewed changes

ggwpez commented Mar 16, 2022

View reviewed changes

paritytech-processbot bot merged commit 89682c1 into master Mar 17, 2022

paritytech-processbot bot deleted the oty-block-bench branch March 17, 2022 10:40

ggwpez mentioned this pull request Mar 22, 2022

Add benchmark-block command #11091

Merged

github-actions bot mentioned this pull request Apr 20, 2022

Update substrate/polkadot/cumulus from v0.9.18 to v0.9.19 moonbeam-foundation/moonbeam#1428

Closed

Dengjianping mentioned this pull request Jul 7, 2022

Bump deps to v0.9.22 Manta-Network/Manta#571

Merged

10 tasks

github-actions bot mentioned this pull request Jul 18, 2022

Update substrate/polkadot/cumulus from v0.9.18 to v0.9.19 imstar15/moonbeam#1

Open

ghzlatarev mentioned this pull request Aug 17, 2022

Integrate benchmark-overhead command Manta-Network/Manta#760

Open

ggwpez mentioned this pull request Sep 28, 2022

Member Request polkadot-fellows/seeding#21

Merged

MRamanenkau mentioned this pull request Jun 22, 2023

Feature/update to polkadot v0 9 27 Cerebellum-Network/blockchain-node#38

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add execution overhead benchmarking #10977

Add execution overhead benchmarking #10977

ggwpez commented Mar 3, 2022 •

edited

Loading

shawntabrizi Mar 10, 2022

ggwpez Mar 10, 2022

ggwpez Mar 16, 2022

ggwpez commented Mar 16, 2022

ggwpez commented Mar 17, 2022

Add execution overhead benchmarking #10977

Add execution overhead benchmarking #10977

Conversation

ggwpez commented Mar 3, 2022 • edited Loading

shawntabrizi Mar 10, 2022

Choose a reason for hiding this comment

ggwpez Mar 10, 2022

Choose a reason for hiding this comment

ggwpez Mar 16, 2022

Choose a reason for hiding this comment

ggwpez commented Mar 16, 2022

ggwpez commented Mar 17, 2022

ggwpez commented Mar 3, 2022 •

edited

Loading