[NPoS] Implements dynamic number of nominators (#12970)

* Implements dynamic nominations per nominator * Adds SnapshotBounds and ElectionSizeTracker * Changes the ElectionDataProvider interface to receive ElectionBounds as input * Implements get_npos_voters with ElectionBounds * Implements get_npos_targets with ElectionBounds * Adds comments * tests * Truncates nomninations that exceed nominations quota; Old tests passing * Uses DataProviderBounds and ElectionBounds (to continue) * Finishes conversions - tests passing * Refactor staking in babe mocks * Replaces MaxElectableTargets and MaxElectingVoters with ElectionBounds; Adds more tests * Fixes nits; node compiling * bechmarks * removes nomination_quota extrinsic to request the nomination quota * Lazy quota check, ie. at nominate time only * remove non-working test (for now) * tests lazy nominations quota when quota is lower than current number of nominated targets * Adds runtime API and custom RPC call for clients to query the nominations quota for a given balance * removes old rpc * Cosmetic touches * All mocks working * Fixes benchmarking mocks * nits * more tests * renames trait methods * nit * ".git/.scripts/commands/fmt/fmt.sh" * Fix V2 PoV benchmarking (#13485) * Bump default 'additional_trie_layers' to two The default here only works for extremely small runtimes, which have no more than 16 storage prefices. This is changed to a "sane" default of 2, which is save for runtimes with up to 4096 storage prefices (eg StorageValue). Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update tests and test weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix PoV weights Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_balances * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_message_queue * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton * ".git/.scripts/commands/bench/bench.sh" pallet dev pallet_glutton * Fix sanity check >0 would also do as a check, but let's try this. Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: command-bot <> * Move BEEFY code to consensus (#13484) * Move beefy primitives to consensus dir * Move beefy gadget to client consensus folder * Rename beefy crates * chore: move genesis block builder to chain-spec crate. (#13427) * chore: move genesis block builder to block builder crate. * add missing file * chore: move genesis block builder to sc-chain-spec * Update client/chain-spec/src/genesis.rs Co-authored-by: Bastian Köcher <git@kchr.de> * Update test-utils/runtime/src/genesismap.rs Co-authored-by: Bastian Köcher <git@kchr.de> * Update test-utils/runtime/client/src/lib.rs * fix warnings * fix warnings --------- Co-authored-by: Bastian Köcher <git@kchr.de> * Speed up storage iteration from within the runtime (#13479) * Speed up storage iteration from within the runtime * Move the cached iterator into an `Option` * Use `RefCell` in no_std * Simplify the code slightly * Use `Option::replace` * Update doc comment for `next_storage_key_slow` * Make unbounded channels size warning exact (part 1) (#13490) * Replace `futures-channel` with `async-channel` in `out_events` * Apply suggestions from code review Co-authored-by: Koute <koute@users.noreply.github.com> * Also print the backtrace of `send()` call * Switch from `backtrace` crate to `std::backtrace` * Remove outdated `backtrace` dependency * Remove `backtrace` from `Cargo.lock` --------- Co-authored-by: Koute <koute@users.noreply.github.com> * Removal of Prometheus alerting rules deployment in cloud-infra (#13499) * sp-consensus: remove unused error variants (#13495) * Expose `ChargedAmount` (#13488) * Expose `ChargedAmount` * Fix imports * sc-consensus-beefy: fix metrics: use correct names (#13494) Signed-off-by: acatangiu <adrian@parity.io> * clippy fix * removes NominationsQuotaExceeded event * Update frame/staking/src/lib.rs Co-authored-by: Ross Bulat <ross@parity.io> * adds back the npos_max_iter * remove duplicate imports added after merge * fmt * Adds comment in public struct; Refactors CountBound and SizeCount to struct * addresses various pr comments * PR comment reviews * Fixes on-chain election bounds and related code * EPM checks the size of the voter list returned by the data provider * cosmetic changes * updates e2e tests mock * Adds more tests for size tracker and refactors code * Adds back only_iterates_max_2_times_max_allowed_len test * Refactor * removes unecessary dependency * empty commit -- restart all stuck CI jobs * restarts ci jobs * Renames ElectionBounds -> Bounds in benchmarking mocks et al * updates mocks * Update frame/election-provider-support/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/staking/src/pallet/impls.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/election-provider-support/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/staking/src/tests.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * more checks in api_nominations_quota in tests * Improves docs * fixes e2e tests * Uses size_hint rather than mem::size_of in size tracker; Refactor size tracker to own module * nits from reviews * Refactors bounds to own module; improves docs * More tests and docs * fixes docs * Fixes benchmarks * Fixes rust docs * fixes bags-list remote-ext-tests * Simplify bound checks in create_snapshot_external * Adds target size check in get_npos_targets * ".git/.scripts/commands/fmt/fmt.sh" * restart ci * rust doc fixes and cosmetic nits * rollback upgrade on parity-scale-codec version (unecessary) * reset cargo lock, no need to update it --------- Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Signed-off-by: acatangiu <adrian@parity.io> Co-authored-by: command-bot <> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: yjh <yjh465402634@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Koute <koute@users.noreply.github.com> Co-authored-by: Dmitry Markin <dmitry@markin.tech> Co-authored-by: Anthony Lazam <lazam@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Piotr Mikołajczyk <piomiko41@gmail.com> Co-authored-by: Adrian Catangiu <adrian@parity.io> Co-authored-by: Ross Bulat <ross@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com>
paritytech · Aug 10, 2023 · 57d2e53 · 57d2e53
1 parent 19eb56a
commit 57d2e53
Show file tree

Hide file tree

Showing 30 changed files with 1,415 additions and 307 deletions.
diff --git a/bin/node/runtime/src/lib.rs b/bin/node/runtime/src/lib.rs
@@ -24,6 +24,7 @@
 
 use codec::{Decode, Encode, MaxEncodedLen};
 use frame_election_provider_support::{
+	bounds::{ElectionBounds, ElectionBoundsBuilder},
 	onchain, BalancingConfig, ElectionDataProvider, SequentialPhragmen, VoteWeight,
 };
 use frame_support::{
@@ -562,14 +563,16 @@ parameter_types! {
 	pub HistoryDepth: u32 = 84;
 }
 
+/// Upper limit on the number of NPOS nominations.
+const MAX_QUOTA_NOMINATIONS: u32 = 16;
+
 pub struct StakingBenchmarkingConfig;
 impl pallet_staking::BenchmarkingConfig for StakingBenchmarkingConfig {
 	type MaxNominators = ConstU32<1000>;
 	type MaxValidators = ConstU32<1000>;
 }
 
 impl pallet_staking::Config for Runtime {
-	type MaxNominations = MaxNominations;
 	type Currency = Balances;
 	type CurrencyBalance = Balance;
 	type UnixTime = Timestamp;
@@ -594,6 +597,7 @@ impl pallet_staking::Config for Runtime {
 	type ElectionProvider = ElectionProviderMultiPhase;
 	type GenesisElectionProvider = onchain::OnChainExecution<OnChainSeqPhragmen>;
 	type VoterList = VoterList;
+	type NominationsQuota = pallet_staking::FixedNominationsQuota<MAX_QUOTA_NOMINATIONS>;
 	// This a placeholder, to be introduced in the next PR as an instance of bags-list
 	type TargetList = pallet_staking::UseValidatorsMap<Self>;
 	type MaxUnlockingChunks = ConstU32<32>;
@@ -647,17 +651,20 @@ frame_election_provider_support::generate_solution_type!(
 		VoterIndex = u32,
 		TargetIndex = u16,
 		Accuracy = sp_runtime::PerU16,
-		MaxVoters = MaxElectingVoters,
+		MaxVoters = MaxElectingVotersSolution,
 	>(16)
 );
 
 parameter_types! {
+	// Note: the EPM in this runtime runs the election on-chain. The election bounds must be
+	// carefully set so that an election round fits in one block.
+	pub ElectionBoundsMultiPhase: ElectionBounds = ElectionBoundsBuilder::default()
+		.voters_count(10_000.into()).targets_count(1_500.into()).build();
+	pub ElectionBoundsOnChain: ElectionBounds = ElectionBoundsBuilder::default()
+		.voters_count(5_000.into()).targets_count(1_250.into()).build();
+
 	pub MaxNominations: u32 = <NposSolution16 as frame_election_provider_support::NposSolution>::LIMIT as u32;
-	pub MaxElectingVoters: u32 = 40_000;
-	pub MaxElectableTargets: u16 = 10_000;
-	// OnChain values are lower.
-	pub MaxOnChainElectingVoters: u32 = 5000;
-	pub MaxOnChainElectableTargets: u16 = 1250;
+	pub MaxElectingVotersSolution: u32 = 40_000;
 	// The maximum winners that can be elected by the Election pallet which is equivalent to the
 	// maximum active validators the staking pallet can have.
 	pub MaxActiveValidators: u32 = 1000;
@@ -712,8 +719,7 @@ impl onchain::Config for OnChainSeqPhragmen {
 	type DataProvider = <Runtime as pallet_election_provider_multi_phase::Config>::DataProvider;
 	type WeightInfo = frame_election_provider_support::weights::SubstrateWeight<Runtime>;
 	type MaxWinners = <Runtime as pallet_election_provider_multi_phase::Config>::MaxWinners;
-	type VotersBound = MaxOnChainElectingVoters;
-	type TargetsBound = MaxOnChainElectableTargets;
+	type Bounds = ElectionBoundsOnChain;
 }
 
 impl pallet_election_provider_multi_phase::MinerConfig for Runtime {
@@ -761,9 +767,8 @@ impl pallet_election_provider_multi_phase::Config for Runtime {
 	type GovernanceFallback = onchain::OnChainExecution<OnChainSeqPhragmen>;
 	type Solver = SequentialPhragmen<AccountId, SolutionAccuracyOf<Self>, OffchainRandomBalancing>;
 	type ForceOrigin = EnsureRootOrHalfCouncil;
-	type MaxElectableTargets = MaxElectableTargets;
 	type MaxWinners = MaxActiveValidators;
-	type MaxElectingVoters = MaxElectingVoters;
+	type ElectionBounds = ElectionBoundsMultiPhase;
 	type BenchmarkingConfig = ElectionProviderBenchmarkConfig;
 	type WeightInfo = pallet_election_provider_multi_phase::weights::SubstrateWeight<Self>;
 }

diff --git a/frame/babe/src/mock.rs b/frame/babe/src/mock.rs
@@ -19,12 +19,16 @@
 
 use crate::{self as pallet_babe, Config, CurrentSlot};
 use codec::Encode;
-use frame_election_provider_support::{onchain, SequentialPhragmen};
+use frame_election_provider_support::{
+	bounds::{ElectionBounds, ElectionBoundsBuilder},
+	onchain, SequentialPhragmen,
+};
 use frame_support::{
 	parameter_types,
 	traits::{ConstU128, ConstU32, ConstU64, KeyOwnerProofSystem, OnInitialize},
 };
 use pallet_session::historical as pallet_session_historical;
+use pallet_staking::FixedNominationsQuota;
 use sp_consensus_babe::{AuthorityId, AuthorityPair, Randomness, Slot, VrfSignature};
 use sp_core::{
 	crypto::{KeyTypeId, Pair, VrfSecret},
@@ -161,6 +165,7 @@ parameter_types! {
 	pub const SlashDeferDuration: EraIndex = 0;
 	pub const RewardCurve: &'static PiecewiseLinear<'static> = &REWARD_CURVE;
 	pub const OffendingValidatorsThreshold: Perbill = Perbill::from_percent(16);
+	pub static ElectionsBounds: ElectionBounds = ElectionBoundsBuilder::default().build();
 }
 
 pub struct OnChainSeqPhragmen;
@@ -170,12 +175,10 @@ impl onchain::Config for OnChainSeqPhragmen {
 	type DataProvider = Staking;
 	type WeightInfo = ();
 	type MaxWinners = ConstU32<100>;
-	type VotersBound = ConstU32<{ u32::MAX }>;
-	type TargetsBound = ConstU32<{ u32::MAX }>;
+	type Bounds = ElectionsBounds;
 }
 
 impl pallet_staking::Config for Test {
-	type MaxNominations = ConstU32<16>;
 	type RewardRemainder = ();
 	type CurrencyToVote = ();
 	type RuntimeEvent = RuntimeEvent;
@@ -197,6 +200,7 @@ impl pallet_staking::Config for Test {
 	type GenesisElectionProvider = Self::ElectionProvider;
 	type VoterList = pallet_staking::UseNominatorsAndValidatorsMap<Self>;
 	type TargetList = pallet_staking::UseValidatorsMap<Self>;
+	type NominationsQuota = FixedNominationsQuota<16>;
 	type MaxUnlockingChunks = ConstU32<32>;
 	type HistoryDepth = ConstU32<84>;
 	type EventListeners = ();

diff --git a/frame/bags-list/remote-tests/src/snapshot.rs b/frame/bags-list/remote-tests/src/snapshot.rs
@@ -16,7 +16,10 @@
 
 //! Test to execute the snapshot using the voter bag.
 
-use frame_election_provider_support::SortedListProvider;
+use frame_election_provider_support::{
+	bounds::{CountBound, DataProviderBounds},
+	SortedListProvider,
+};
 use frame_support::traits::PalletInfoAccess;
 use remote_externalities::{Builder, Mode, OnlineConfig};
 use sp_runtime::{traits::Block as BlockT, DeserializeOwned};
@@ -62,8 +65,13 @@ where
 			<Runtime as pallet_staking::Config>::VoterList::count(),
 		);
 
+		let bounds = match voter_limit {
+			None => DataProviderBounds::default(),
+			Some(v) => DataProviderBounds { count: Some(CountBound(v as u32)), size: None },
+		};
+
 		let voters =
-			<pallet_staking::Pallet<Runtime> as ElectionDataProvider>::electing_voters(voter_limit)
+			<pallet_staking::Pallet<Runtime> as ElectionDataProvider>::electing_voters(bounds)
 				.unwrap();
 
 		let mut voters_nominator_only = voters

diff --git a/frame/beefy/src/mock.rs b/frame/beefy/src/mock.rs
@@ -17,7 +17,10 @@
 
 use std::vec;
 
-use frame_election_provider_support::{onchain, SequentialPhragmen};
+use frame_election_provider_support::{
+	bounds::{ElectionBounds, ElectionBoundsBuilder},
+	onchain, SequentialPhragmen,
+};
 use frame_support::{
 	construct_runtime, parameter_types,
 	sp_io::TestExternalities,
@@ -184,6 +187,7 @@ parameter_types! {
 	pub const BondingDuration: EraIndex = 3;
 	pub const RewardCurve: &'static PiecewiseLinear<'static> = &REWARD_CURVE;
 	pub const OffendingValidatorsThreshold: Perbill = Perbill::from_percent(17);
+	pub static ElectionsBoundsOnChain: ElectionBounds = ElectionBoundsBuilder::default().build();
 }
 
 pub struct OnChainSeqPhragmen;
@@ -193,12 +197,10 @@ impl onchain::Config for OnChainSeqPhragmen {
 	type DataProvider = Staking;
 	type WeightInfo = ();
 	type MaxWinners = ConstU32<100>;
-	type VotersBound = ConstU32<{ u32::MAX }>;
-	type TargetsBound = ConstU32<{ u32::MAX }>;
+	type Bounds = ElectionsBoundsOnChain;
 }
 
 impl pallet_staking::Config for Test {
-	type MaxNominations = ConstU32<16>;
 	type RewardRemainder = ();
 	type CurrencyToVote = ();
 	type RuntimeEvent = RuntimeEvent;
@@ -220,6 +222,7 @@ impl pallet_staking::Config for Test {
 	type GenesisElectionProvider = Self::ElectionProvider;
 	type VoterList = pallet_staking::UseNominatorsAndValidatorsMap<Self>;
 	type TargetList = pallet_staking::UseValidatorsMap<Self>;
+	type NominationsQuota = pallet_staking::FixedNominationsQuota<16>;
 	type MaxUnlockingChunks = ConstU32<32>;
 	type HistoryDepth = ConstU32<84>;
 	type EventListeners = ();

diff --git a/frame/election-provider-multi-phase/src/benchmarking.rs b/frame/election-provider-multi-phase/src/benchmarking.rs
@@ -20,6 +20,7 @@
 use super::*;
 use crate::{unsigned::IndexAssignmentOf, Pallet as MultiPhase};
 use frame_benchmarking::account;
+use frame_election_provider_support::bounds::DataProviderBounds;
 use frame_support::{
 	assert_ok,
 	traits::{Hooks, TryCollect},
@@ -270,8 +271,9 @@ frame_benchmarking::benchmarks! {
 
 		// we don't directly need the data-provider to be populated, but it is just easy to use it.
 		set_up_data_provider::<T>(v, t);
-		let targets = T::DataProvider::electable_targets(None)?;
-		let voters = T::DataProvider::electing_voters(None)?;
+		// default bounds are unbounded.
+		let targets = T::DataProvider::electable_targets(DataProviderBounds::default())?;
+		let voters = T::DataProvider::electing_voters(DataProviderBounds::default())?;
 		let desired_targets = T::DataProvider::desired_targets()?;
 		assert!(<MultiPhase<T>>::snapshot().is_none());
 	}: {