seal: Add automated weights for contract API calls #7017
Conversation
Benchmarks should only measure the overhead of the API calls itself. For that reason we want to run them without instrumentation.
Data used in events has storage implications for archive nodes. Those need to keep the events in storage forever. For that reason we want to limit the amount of storage that can be used inside events.
athei
added
A0-please_review
kianenigma
changed the title
Co-authored-by: Sergei Shulepov <sergei@parity.io>
|
On ice until the bench bot is available. I want to merge this PR with the proper weights. |
|
/benchmark runtime pallet pallet_contracts |
|
Finished benchmark for branch: at-seal-api-benchmarks Benchmark: Runtime Benchmarks Pallet cargo run --release --features runtime-benchmarks --manifest-path bin/node/cli/Cargo.toml -- benchmark --chain dev --steps 50 --repeat 20 --extrinsic "*" --execution=wasm --wasm-execution=compiled --output ./bin/node/runtime/src/weights --header ./HEADER --pallet pallet_contracts ResultsCompiling node-runtime v2.0.0 (/home/shawntabrizi/bench-bot/git/substrate/bin/node/runtime) ==================== Version: 2.0.0-9d1c66b23-x86_64-linux-gnu 0: sp_panic_handler::set::{{closure}} Thread 'main' panicked at 'Failed to allocate memory: "Error in allocator: Allocator ran out of space"', primitives/io/src/lib.rs:981 This is a bug. Please report it at: Error: Input("Error executing runtime benchmark: Other("Wasm execution trapped: Failed to allocate memory: \"Error in allocator: Allocator ran out of space\"\nwasm backtrace:\n 0: 0x2e3d79 - !sp_io::allocator::extern_host_function_impls::malloc::h4dd6ba4b078e8d4f\n 1: 0x1c69 - !__rg_alloc\n 2: 0x1c62 - !__rust_alloc\n 3: 0x4783b - !<core::iter::adapters::ResultShunt<I,E> as core::iter::traits::iterator::Iterator>::next::h4c8aef765a5a05a8\n 4: 0x4771a - !<alloc::vec::Vec as alloc::vec::SpecFromIter<T,I>>::from_iter::h481478c6e7f463b4\n 5: 0x21792d - !<pallet_contracts::benchmarking::SelectedBenchmark as frame_benchmarking::utils::BenchmarkingSetup>::instance::h4b9a4b65c9a780a3\n 6: 0x2109ba - !pallet_contracts::benchmarking::<impl frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults> for pallet_contracts::Module>::run_benchmark::{{closure}}::h173d3ea795cfac6a\n 7: 0x1e1111 - !pallet_contracts::benchmarking::<impl frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults> for pallet_contracts::Module>::run_benchmark::hd1c2a62c9e5ec34c\n 8: 0x1d80ed - !<node_runtime::Runtime as frame_benchmarking::utils::runtime_decl_for_Benchmark::Benchmark<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<<pallet_indices::Module<node_runtime::Runtime> as sp_runtime::traits::StaticLookup>::Source,node_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<node_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<node_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<node_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<node_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<node_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<node_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<node_runtime::Runtime>)>>>>::dispatch_benchmark::h30ab2a74fd1ee1d4\n 9: 0x2ad738 - !Benchmark_dispatch_benchmark\n")") |
|
/benchmark runtime custom --chain dev --steps 50 --repeat 20 --extrinsic "*" --execution=wasm --wasm-execution=compiled --output ./bin/node/runtime/src/weights --header ./HEADER --pallet pallet_contracts --heap-pages 4096 |
|
Finished benchmark for branch: at-seal-api-benchmarks Benchmark: Runtime Benchmarks Custom cargo run --release --features runtime-benchmarks --manifest-path bin/node/cli/Cargo.toml -- benchmark --chain dev --steps 50 --repeat 20 --extrinsic "*" --execution=wasm --wasm-execution=compiled --output ./bin/node/runtime/src/weights --header ./HEADER --pallet pallet_contracts --heap-pages 4096 ResultsPallet: "pallet_contracts", Extrinsic: "update_schedule", Lowest values: [], Highest values: [], Steps: [50], Repeat: 20
|
…/node/cli/Cargo.toml -- benchmark --chain dev --steps 50 --repeat 20 --extrinsic * --execution=wasm --wasm-execution=compiled --output ./bin/node/runtime/src/weights --header ./HEADER --pallet pallet_contracts --heap-pages 4096
athei
added
A0-please_review
|
bot merge |
|
Trying merge. |
ghost
merged commit 
…up-updates * master: Async keystore + Authority-Discovery async/await (#7000) Fixes logging of target names with dashes (#7281) seal: Add automated weights for contract API calls (#7017) add ss58 id for nodle (#7279) Refactor CurrencyToVote (#6896) bump-allocator: document & poison (#7277) Reset flaming fir network (#7274) reschedule (#6860) Drop system cache for trie benchmarks (#7242) client: improve log formatting (#7272) Rework `InspectState` (#7271) SystemOrigin trait (#7226) Update ss58 registry for Dock network (#7263) .maintain/monitoring: Add alert when continuous task ends (#7250) Rename `TRANSACTION_VERSION` to `EXTRINSIC_VERSION` (#7258) Split block announce processing into two parts (#6958) Fix offchain election to respect the weight (#7215)
When a contract calls into the supervisor using an API call like
seal_gas_leftthe execution of the contract itself is interrupted and the processing continues inside the supervisor's code (the chains' runtime). The code executed there is unmetered. For that reason, we need to measure the weight for all of those entry points into the supervisor and charge the weight from the gas meter on entry.This PR adds one or multiple benchmarks per API call and uses them to bootstrap a new cost
Schedule. The costScheduleis a list of all costs (in weight) a contract can incur. It is kept in storage to allow for live adjustments without deploying a new runtime. Lastly, the code is adjusted to make use of the newSchedule.The commits of this PR can be reviewed individually.
@jacogr The in-storage struct
pallet_contracts::Schedulechanged and should be carried over to javascript.@shawntabrizi Currently, the substrate node uses the
WeightInfoimplementation on()which I generated on my local computer. Once this PR is merged I need you to run the benchmarks on the reference machine for the substrate runtime. Expect this benchmark suite to run for ~ 20 mins with-r 23. Also, you need to supply--heap-pages 4096to prevent it from going out of memory.closes #6486
Benchmarking Strategy
The
rcomponentThe main challenge here is that we are not benchmarking individual dispatchables as a whole but pieces that are executed as part of that dispatchable: We execute the
pallet_contracts:: calldispatchable in order to execute a contract that contains the API call we desire to benchmark. However, by doing that we also record all the overhead that is necessary to start executing the contract. The solution we came up with is introducing a benchmark componentrthat determines the number of repetitions of the API inside the contract. The linear regression can then determine the amount of time eachrcontributes. The weight of an API call can then be constructed (this happens onSchedulebootstrapping) by calculating:Assuming the only component is
rand that the API call in question isseal_gas_left.T::WeightInfois a trait containing one function per benchmark and whose implementation is generated automatically by executing the benchmarks.Batching
Sometimes the contribution of a single repetition
rcan be too small for the linear regression to properly recognize. For that reason, we batch thercomponent. This means that eachradds not one but a specified amount of repetitions (API_BENCHMARK_BATCH_SIZE) each. When bootstrapping theSchedulewe divide the weight by that number. The other batching strategy is to define each component that measures data length in kilobytes and not bytes to increase the influence of that component when compared to the extrinsic overhead.Multiple benchmarks per API call
When an API call has no static weight but depends on its inputs for weight determination we need to add a component to measure the influence for each input parameter. However, we cannot add those to the benchmark where we are using the
rcomponent because then those components become dependent on each other. For that reason, we have multiple benchmarks per API call whenever they have components of their own.Future Work
With this PR all weights are automated (and hence secure for production) except the weights for individual wasm instructions. Those still use preliminary numbers. The next step will be to automate those using the benchmarking strategies worked out during the development of this PR.