Skip to content

Commit

Permalink
pallet-migrations: try-runtime support, and manage author mapping bla…
Browse files Browse the repository at this point in the history 
…ke2 migration (#796)

* Add initial migrations pallet sketch

* Sketch out Migration impls

* Make it build

* Squelch warnings

* Sketch out process_runtime_upgrades

* Leave note for reviewers

* Add &self to Migrations trait fns

* Make it compile

* Refactor migrations design to use stepping instead of one-shot

* Fix typo/bug

* Track overall migration doneness

* Optimize when progress remains unchanged

* Resolve compiler warnings

* Incremental progress on mock

* Mock is getting close

* Make mock build

* Plumb genesis building in mock

* Baby's first tests

* Fix events

* Use Vec<u8> instead of String

* Make MigrationsList part of pallet config; plumb through Moonbase runtime

* Appease the compiler

* Fix up CommonMigrations list

* Remove comment

* Cargo fmt

* Per-test MigrationsList

* Attempt at a glue

* Fix FIXME

* Getting close

* Sort out lifetimes

* Simplify FnMut arguments/storage

* Clean up, fix FIXMEs

* It works

* Implement Migrations::on_initialize

* Resolve compilation warnings, add comments about how mock glue works

* Move migration event impl

* Let tests manage ExtBuilder ... execute_with()

* Test that migrations are only run once

* Remove TODO/comment; events are not cheap and should be used conservatively

* Post merge-master fixes

* Remove cruft

* Track some db reads and writes and charge accordingly

* cargo fmt

* Add failing test about one-migration-at-a-time

* Don't start next migration until current is done

* Add notes from meeting

* Allow multi-block migrations to be disabled

* Add failing test about overweight migrations

* Explicitly embrace allowing overweight migrations

* cargo fmt

* Clean up / add comments

* Derive block weight from Config (still needs improvement)

* cargo fmt

* Configure all runtimes to include pallet-migrations

* Add pallet-migrations genesis to specs

* Update pallets/migrations/src/lib.rs

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>

* Update pallets/migrations/src/lib.rs

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>

* First pass at ripping out multi-block migration support

* Incremental work @ removing multi-block migration support

* Make migration tests compile (not passing yet)

* Clean up runtime to reflect removal of multi-block migrations

* You know your tests are good when they catch a critical refactor mistake

* Fix test logic to reflect no multi-block migrations

* cargo fmt

* Remove phantomdata field from pallet_migrations::GenesisConfig (#701)

* remove phantomdata from pallet migrations genesis config struct

* skip migration if no weight available for step

* revert

* Better log statement

Co-authored-by: Amar Singh <asinghchrony@protonmail.com>

* Use ValueQuery instead of OptionQuery

* Update Cargo.lock

* Manually add back version = 3

* Make some deps dev-dependencies

* Fix branch

* Use hotfix branch in Migrations

* Clean up from merge

* cargo fmt

* Remove prior hack in test

* Initial concept of try-runtime support for pallet-migrations

* Impl for post_upgrade()

* Extend tests to work with pre_ and post_upgrade hooks

* Explicitly invoke try-runtime hooks in tests

* Leave hint about which migrations are performed for post_upgrade

* cargo fmt

* Add pallet-migration's try-runtime feature to all runtimes

* fix warnings

* Move autho mapping migration to dedicated type

* brain dump

* Make author hasher migration compatible with pallet migrations.

* Condense code

Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>

* Only run try-runtime tests with --features=try-runtime

* Add missing import

* Add try-runtime-only imports

* Hook AuthorMapping migration up to common migrations

* Actually hook it up... also, trait bounds

* cargo fmt

* Remove accidental .cargo/config.toml addition

* better logging

* bump spec version for testing purposes

* cleanups

* line length

* std feature

Co-authored-by: Alexander Popiak <alexander.popiak@gmail.com>
Co-authored-by: Amar Singh <asinghchrony@protonmail.com>
Co-authored-by: Joshy Orndorff <admin@joshyorndorff.com>
Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com>
  • Loading branch information
@notlesh @apopiak
@4meta5 @JoshOrndorff
5 people authored Oct 3, 2021
1 parent f7380e8 commit feab214
Show file tree
Hide file tree
Showing 15 changed files with 392 additions and 162 deletions.
3 changes: 1 addition & 2 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

119 changes: 3 additions & 116 deletions pallets/author-mapping/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ mod mock;
#[cfg(test)]
mod tests;

pub mod migrations;

#[pallet]
pub mod pallet {
use crate::WeightInfo;
Expand Down Expand Up @@ -237,7 +239,7 @@ pub mod pallet {
#[pallet::getter(fn account_and_deposit_of)]
/// We maintain a mapping from the AuthorIds used in the consensus layer
/// to the AccountIds runtime (including this staking pallet).
type MappingWithDeposit<T: Config> = StorageMap<
pub type MappingWithDeposit<T: Config> = StorageMap<
_,
Blake2_128Concat,
T::AuthorId,
Expand Down Expand Up @@ -283,119 +285,4 @@ pub mod pallet {
Self::account_and_deposit_of(author_id).map(|info| info.account)
}
}

#[pallet::hooks]
impl<T: Config> Hooks<BlockNumberFor<T>> for Pallet<T> {
fn on_runtime_upgrade() -> Weight {
use frame_support::storage::migration::{remove_storage_prefix, storage_key_iter};
use sp_std::{convert::TryInto, vec::Vec};

let pallet_prefix: &[u8] = b"AuthorMapping";
let storage_item_prefix: &[u8] = b"MappingWithDeposit";

// Read all the data into memory.
// https://crates.parity.io/frame_support/storage/migration/fn.storage_key_iter.html
let stored_data: Vec<_> = storage_key_iter::<
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
Twox64Concat,
>(pallet_prefix, storage_item_prefix)
.collect();

let migrated_count: Weight = stored_data
.len()
.try_into()
.expect("There are between 0 and 2**64 mappings stored.");

// Now remove the old storage
// https://crates.parity.io/frame_support/storage/migration/fn.remove_storage_prefix.html
remove_storage_prefix(pallet_prefix, storage_item_prefix, &[]);

// Assert that old storage is empty
assert!(storage_key_iter::<
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
Twox64Concat,
>(pallet_prefix, storage_item_prefix)
.next()
.is_none());

// Write the mappings back to storage with the new secure hasher
for (author_id, account_id) in stored_data {
MappingWithDeposit::<T>::insert(author_id, account_id);
}

// Return the weight used. For each migrated mapping there is a red to get it into
// memory, a write to clear the old stored value, and a write to re-store it.
let db_weights = T::DbWeight::get();
migrated_count.saturating_mul(2 * db_weights.write + db_weights.read)
}

#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<(), &'static str> {
use frame_support::storage::migration::{storage_iter, storage_key_iter};
use frame_support::traits::OnRuntimeUpgradeHelpersExt;

let pallet_prefix: &[u8] = b"AuthorMapping";
let storage_item_prefix: &[u8] = b"MappingWithDeposit";

// We want to test that:
// There are no entries in the new storage beforehand
// The same number of mappings exist before and after
// As long as there are some mappings stored, one representative key maps to the
// same value after the migration.
// There are no entries in the old storage afterward

// Assert new storage is empty
// Because the pallet and item prefixes are the same, the old storage is still at this
// key. However, the values can't be decoded so the assertion passes.
assert!(MappingWithDeposit::<T>::iter().next().is_none());

// Check number of entries, and set it aside in temp storage
let mapping_count = storage_iter::<RegistrationInfo<T::AccountId, BalanceOf<T>>>(
pallet_prefix,
storage_item_prefix,
)
.count() as u64;
Self::set_temp_storage(mapping_count, "mapping_count");

// Read an example pair from old storage and set it aside in temp storage
if mapping_count > 0 {
let example_pair = storage_key_iter::<
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
Twox64Concat,
>(pallet_prefix, storage_item_prefix)
.next()
.expect("We already confirmed that there was at least one item stored");

Self::set_temp_storage(example_pair, "example_pair");
}

Ok(())
}

#[cfg(feature = "try-runtime")]
fn post_upgrade() -> Result<(), &'static str> {
use frame_support::traits::OnRuntimeUpgradeHelpersExt;

// Check number of entries matches what was set aside in pre_upgrade
let old_mapping_count: u64 = Self::get_temp_storage("mapping_count")
.expect("We stored a mapping count; it should be there; qed");
let new_mapping_count = MappingWithDeposit::<T>::iter().count() as u64;
assert_eq!(old_mapping_count, new_mapping_count);

// Check that our example pair is still well-mapped after the migration
if new_mapping_count > 0 {
let (account, original_info): (
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
) = Self::get_temp_storage("example_pair").expect("qed");
let migrated_info = MappingWithDeposit::<T>::get(account).expect("qed");
assert_eq!(original_info, migrated_info);
}

Ok(())
}
}
}
144 changes: 144 additions & 0 deletions pallets/author-mapping/src/migrations.rs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
// Copyright 2019-2021 PureStake Inc.
// This file is part of Moonbeam.

// Moonbeam is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.

// Moonbeam is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.

// You should have received a copy of the GNU General Public License
// along with Moonbeam. If not, see <http://www.gnu.org/licenses/>.

use crate::{BalanceOf, Config, MappingWithDeposit, RegistrationInfo};
use frame_support::{
pallet_prelude::PhantomData,
storage::migration::{remove_storage_prefix, storage_key_iter},
traits::{Get, OnRuntimeUpgrade},
weights::Weight,
Twox64Concat,
};

use sp_std::convert::TryInto;
//TODO sometimes this is unused, sometimes its necessary
use sp_std::vec::Vec;

/// Migrates the AuthorMapping's storage map fro mthe insecure Twox64 hasher to the secure
/// BlakeTwo hasher.
pub struct TwoXToBlake<T>(PhantomData<T>);
impl<T: Config> OnRuntimeUpgrade for TwoXToBlake<T> {
fn on_runtime_upgrade() -> Weight {
log::info!(target: "TwoXToBlake", "actually running it");
let pallet_prefix: &[u8] = b"AuthorMapping";
let storage_item_prefix: &[u8] = b"MappingWithDeposit";

// Read all the data into memory.
// https://crates.parity.io/frame_support/storage/migration/fn.storage_key_iter.html
let stored_data: Vec<_> = storage_key_iter::<
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
Twox64Concat,
>(pallet_prefix, storage_item_prefix)
.collect();

let migrated_count: Weight = stored_data
.len()
.try_into()
.expect("There are between 0 and 2**64 mappings stored.");

// Now remove the old storage
// https://crates.parity.io/frame_support/storage/migration/fn.remove_storage_prefix.html
remove_storage_prefix(pallet_prefix, storage_item_prefix, &[]);

// Assert that old storage is empty
assert!(storage_key_iter::<
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
Twox64Concat,
>(pallet_prefix, storage_item_prefix)
.next()
.is_none());

// Write the mappings back to storage with the new secure hasher
for (author_id, account_id) in stored_data {
MappingWithDeposit::<T>::insert(author_id, account_id);
}

log::info!(target: "TwoXToBlake", "almost done");

// Return the weight used. For each migrated mapping there is a red to get it into
// memory, a write to clear the old stored value, and a write to re-store it.
let db_weights = T::DbWeight::get();
migrated_count.saturating_mul(2 * db_weights.write + db_weights.read)
}

#[cfg(feature = "try-runtime")]
fn pre_upgrade() -> Result<(), &'static str> {
use frame_support::{storage::migration::storage_iter, traits::OnRuntimeUpgradeHelpersExt};

let pallet_prefix: &[u8] = b"AuthorMapping";
let storage_item_prefix: &[u8] = b"MappingWithDeposit";

// We want to test that:
// There are no entries in the new storage beforehand
// The same number of mappings exist before and after
// As long as there are some mappings stored, one representative key maps to the
// same value after the migration.
// There are no entries in the old storage afterward

// Assert new storage is empty
// Because the pallet and item prefixes are the same, the old storage is still at this
// key. However, the values can't be decoded so the assertion passes.
assert!(MappingWithDeposit::<T>::iter().next().is_none());

// Check number of entries, and set it aside in temp storage
let mapping_count = storage_iter::<RegistrationInfo<T::AccountId, BalanceOf<T>>>(
pallet_prefix,
storage_item_prefix,
)
.count() as u64;
Self::set_temp_storage(mapping_count, "mapping_count");

// Read an example pair from old storage and set it aside in temp storage
if mapping_count > 0 {
let example_pair = storage_key_iter::<
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
Twox64Concat,
>(pallet_prefix, storage_item_prefix)
.next()
.expect("We already confirmed that there was at least one item stored");

Self::set_temp_storage(example_pair, "example_pair");
}

Ok(())
}

#[cfg(feature = "try-runtime")]
fn post_upgrade() -> Result<(), &'static str> {
use frame_support::traits::OnRuntimeUpgradeHelpersExt;

// Check number of entries matches what was set aside in pre_upgrade
let old_mapping_count: u64 = Self::get_temp_storage("mapping_count")
.expect("We stored a mapping count; it should be there; qed");
let new_mapping_count = MappingWithDeposit::<T>::iter().count() as u64;
assert_eq!(old_mapping_count, new_mapping_count);

// Check that our example pair is still well-mapped after the migration
if new_mapping_count > 0 {
let (account, original_info): (
T::AuthorId,
RegistrationInfo<T::AccountId, BalanceOf<T>>,
) = Self::get_temp_storage("example_pair").expect("qed");
let migrated_info = MappingWithDeposit::<T>::get(account).expect("qed");
assert_eq!(original_info, migrated_info);
}

Ok(())
}
}
3 changes: 3 additions & 0 deletions pallets/migrations/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,6 @@ std = [
"sp-std/std",
"sp-runtime/std",
]
try-runtime = [
"frame-support/try-runtime",
]
Loading

0 comments on commit feab214

Please sign in to comment.