Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Split block announce processing into two parts #6958

Merged
merged 126 commits into from
Oct 2, 2020
Merged
Show file tree
Hide file tree
Changes from 125 commits
Commits
Show all changes
126 commits
Select commit Hold shift + click to select a range
b336104
Split block announce processing into two parts
bkchr Aug 25, 2020
95c1420
Apply suggestions from code review
bkchr Sep 7, 2020
0416a0b
Update client/network/src/protocol/sync.rs
bkchr Sep 15, 2020
f974fe7
client/authority-discovery: Append PeerId to Multiaddr at most once (…
mxinden Aug 24, 2020
7afdc67
expose Deposit (#6943)
xlc Aug 24, 2020
67d811b
Add a `LightSyncState` field to the chain spec (#6894)
expenses Aug 24, 2020
e714641
Dynamically generate CHT roots on a full client (#6944)
expenses Aug 24, 2020
508dfcd
Enable verification logic when executing benchmarks (#6929)
shawntabrizi Aug 24, 2020
43cd1d0
grandpa: always create and send justification if there are any subscr…
andresilva Aug 24, 2020
0de5451
.maintain/monitoring/alerting-rules: Add fd alert (#6946)
mxinden Aug 24, 2020
20d4ca4
Fix benchmark read/write key tracker for keys in child storages. (#6905)
gui1117 Aug 24, 2020
4b3197c
client/authority-discovery: Limit number of addresses per authority (…
mxinden Aug 24, 2020
9f590df
⛓ ✨Add ShiftNrg Network SS58 address type (#6942)
mswezey23 Aug 24, 2020
761ec7c
update tracing attribute (#6950)
gui1117 Aug 24, 2020
f7791d0
Fix unwraps and other issues with benchmarks (#6957)
shawntabrizi Aug 25, 2020
127673f
Remove implementation of `Randomness for ()` (#6959)
bkchr Aug 26, 2020
144192c
Fix staking fuzzer. (#6954)
kianenigma Aug 26, 2020
1f33b6e
Enforce that ProtocolId is a string (#6953)
tomaka Aug 26, 2020
ebd6d5c
Support Staking Payout to Any Account (#6832)
shawntabrizi Aug 26, 2020
ce89cc9
Better prime election. (#6939)
kianenigma Aug 26, 2020
117bb45
babe: fix report_equivocation weight (#6936)
andresilva Aug 26, 2020
4c0a477
fix bench db wipe (#6965)
gui1117 Aug 26, 2020
3bddf6c
Implement request-responses protocols (#6634)
tomaka Aug 27, 2020
e4178cc
add generated weight info for pallet-collective (#6789)
apopiak Aug 28, 2020
8f28655
client/*: Treat protocol name as str and not [u8] (#6967)
mxinden Aug 28, 2020
6b2f7ea
update kvdb-rocksdb to 0.9.1 and rocksdb to 6.11.4 (#6963)
ordian Aug 29, 2020
7d53c94
Use AsyncReadExt::read_exact, not just read (#6977)
tomaka Aug 31, 2020
539f4d2
client/cli/src/config: Warn on low file descriptor limit (#6956)
mxinden Aug 31, 2020
8182870
Update substrate bip39 version. (#6955)
cheme Aug 31, 2020
b622015
Inverting events set and changed in nicks pallet (#6989)
g2udevelopment Aug 31, 2020
dbbaef1
Silence the error about non-registered protocols (#6987)
tomaka Aug 31, 2020
7972673
Change browser-demo build.sh to use python 3 again (#6992)
expenses Aug 31, 2020
9231166
fix pallet-evm features (#6995)
xlc Sep 1, 2020
9007d28
Move subcommands from sc-cli to nodes (#6948)
expenses Sep 1, 2020
8502300
ci: deploy alerting rules: fix run on changes (#6998)
gabreal Sep 1, 2020
f723458
*: Update to Prometheus v0.10.0 (#6964)
mxinden Sep 1, 2020
0f302b9
Ensure that handshake is sent back even in case of back-pressure (#6979)
tomaka Sep 2, 2020
3541fa8
frame/authority-discovery: Have authorities() return both current and…
mxinden Sep 2, 2020
f8189fc
Stop sending messages on legacy substream altogether (#6975)
tomaka Sep 2, 2020
658c389
manual seal is now consensus agnostic (#7010)
seunlanlege Sep 3, 2020
b309241
grandpa: report metrics on prevotes and precommits cast (#6970)
andresilva Sep 4, 2020
130b7d4
Fix compact npos solution edge count calculation (#7021)
kianenigma Sep 4, 2020
29c993a
Refactor & detach network metrics. (#6986)
romanb Sep 6, 2020
4dee242
Node template complete import pipeline (#7014)
JoshOrndorff Sep 7, 2020
68bdfec
client/authority-discovery: Throttle DHT requests (#7018)
mxinden Sep 7, 2020
8c92258
Update Nicks docs to clarify that it is not production-ready (#6990)
danforbes Sep 7, 2020
6deea30
Ignore wasm_gc for debug build. (#6962)
cheme Sep 8, 2020
8f4db26
Make `--file` optional for `generate-node-key` (#7043)
bkchr Sep 8, 2020
2bc6943
Downgrade wabt = 0.9.1 (#7042)
gnunicorn Sep 8, 2020
52ae515
Add metadata shadows to multisig pallet (#7029)
Sep 8, 2020
27aaf0a
Fix broken link to democracy pallet. (#7026)
humb1t Sep 8, 2020
49191fd
Revert "Fix broken link to democracy pallet. (#7026)" (#7047)
gavofyork Sep 8, 2020
79cc67d
Update the service tasks Grafana dashboard (#7038)
tomaka Sep 8, 2020
a22edf0
babe, grandpa: waive fees on valid equivocation report (#6981)
andresilva Sep 8, 2020
b4fbdda
Clarify Nicks docs (#7049)
danforbes Sep 8, 2020
67c778b
Improves EVM gas price check (#7051)
crystalin Sep 8, 2020
39d8f4d
Change wabt to wat (#7050)
pepyakin Sep 8, 2020
5fa219d
Add Dock network id for address generation (#6714)
lovesh Sep 8, 2020
0a850ae
Partial fix for transaction priority (#7034)
kianenigma Sep 9, 2020
29b42c3
What happens if we remove wat? (#7056)
pepyakin Sep 9, 2020
384f29f
Make SlashingSpans Public (#6961)
kianenigma Sep 9, 2020
d123792
client/authority-discovery/src/service: Improve docs (#7059)
mxinden Sep 9, 2020
d35044c
Decrease poll interval (#7063)
s3krit Sep 9, 2020
de59f04
Remove unused code (#7027)
Sep 9, 2020
f96646a
Disambiguate `BlockNumber` type in `decl_module` (#7061)
shawntabrizi Sep 9, 2020
89e6d66
Implement `FromStr` for `Ss58AddressFormat` (#7068)
bkchr Sep 10, 2020
bd44423
Set reserved nodes with offchain worker. (#6996)
kaichaosun Sep 10, 2020
0f61352
Rename `TRIGGER_WASM_BUILD` to `FORCE_WASM_BUILD` (#7080)
bkchr Sep 10, 2020
0fd0d95
Make decoding of `compact<perthing>` saturating instead of invalid (#…
gui1117 Sep 11, 2020
006f3f0
state_machine no_std witness externalities (#6934)
cheme Sep 11, 2020
cc3040d
Support hex encoded secret key for `--node-key` (#7052)
bkchr Sep 11, 2020
6db5fe3
Add a `build-sync-spec` subcommand and remove the CHT roots from the …
expenses Sep 11, 2020
bb7052d
Fix build sync spec (#7086)
expenses Sep 11, 2020
a9a3be3
Fail docs on warnings (#5923)
TriplEight Sep 11, 2020
4b471dd
Fix `storage::read` (#7084)
bkchr Sep 12, 2020
6c89d07
Add fuzzer for the compact custom codec implementation from PR #6720 …
viniul Sep 13, 2020
8b82790
add instantiable support for treasury pallet (#7058)
pfcoder Sep 14, 2020
a5977f5
grandpa-rpc don't share subscription manager, only executor (#7039)
octol Sep 14, 2020
caab538
pallet-collective: allow customized default vote (#6984)
sorpaas Sep 14, 2020
5f6987a
Upgrade to libp2p-0.28. (#7077)
romanb Sep 14, 2020
c071d06
pow: support uniform tie breaking in fork choice (#7073)
sorpaas Sep 14, 2020
72ea91f
Allow remotes to not open a legacy substream (#7075)
tomaka Sep 14, 2020
f5e0c63
Use diener for Polkadot companion prs (#7102)
bkchr Sep 14, 2020
b2b0db5
Update ui tests for rust 1.46.0 (#7106)
bkchr Sep 15, 2020
9d9cf8a
client/network: Expose number of entries per Kademlia bucket (#7104)
mxinden Sep 15, 2020
c2ef92d
Improve error output of wasm-builder when wasm ins't installed (#7105)
bkchr Sep 15, 2020
b805faf
Add ss58 address for Dark network (#6982)
Sep 15, 2020
c867bc2
Frame-support storage: make iterations and translate consistent (#5470)
gui1117 Sep 15, 2020
111a110
fix js dependancy alert, bumping bl version (#7110)
HarryHong Sep 15, 2020
f406f49
Make `transactional` attribute less scope dependent (#7112)
bkchr Sep 15, 2020
d2c7c1f
Add SS58 Registry (#7020)
joepetrowski Sep 16, 2020
0ddcd66
Move Staking Weights to T::WeightInfo (#7007)
kianenigma Sep 16, 2020
4e4c321
Send import notification always for re-orgs (#7118)
bkchr Sep 16, 2020
9167fe0
WeightInfo for Vesting Pallet (#7103)
shawntabrizi Sep 16, 2020
59b4668
Add benchmarking pipeline to node-template (#7122)
shawntabrizi Sep 17, 2020
2c65036
Use tracing-based subscriber logging (#6825)
sorpaas Sep 17, 2020
86026fc
Typo in error text (#7126)
Swader Sep 17, 2020
da02c0b
WeightInfo for ImOnline (#7128)
shawntabrizi Sep 17, 2020
f1380be
fix the new staking weight in substrate-node (#7131)
kianenigma Sep 17, 2020
585629a
Remove warning about deprecated PeerIds (#7132)
tomaka Sep 17, 2020
9bf2f2f
Fix db initialization for light client (#7130)
arkpar Sep 17, 2020
fa2fbe7
WeightInfo for Identity Pallet (#7107)
shawntabrizi Sep 17, 2020
fbf617e
Make sure we update the `Cargo.lock` in the polkadot companion (#7135)
bkchr Sep 17, 2020
5a2c400
Tracing for wasm with bridging to native (#6916)
gnunicorn Sep 18, 2020
b7a3b2b
Pallet Indices (#7137)
shawntabrizi Sep 18, 2020
f8ee0e1
pow: replace the thread-base mining loop with a future-based mining w…
sorpaas Sep 18, 2020
ecaf240
Bounties (#5715)
xlc Sep 18, 2020
6b1d600
Update SS58 configuration for Bifrost (#7142)
Dengjianping Sep 18, 2020
7cc397f
Prometheus metrics for RPC calls (#7088)
grbIzl Sep 18, 2020
98a6a8c
WeightInfo for Scheduler (#7138)
shawntabrizi Sep 18, 2020
f75c540
grandpa-rpc: use FinalityProofProvider to check finality for rpc (#6215)
octol Sep 18, 2020
3a0cb2a
Make it compile
bkchr Sep 18, 2020
a837031
Rework the implementation and decrease the peer reputation on invalid
bkchr Sep 18, 2020
a0f3789
Implement limits for block announce validation
bkchr Sep 24, 2020
20c00ea
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 24, 2020
3f977ab
Remove accidentally added file
bkchr Sep 25, 2020
e352e41
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 25, 2020
f7f0873
Apply suggestions from code review
bkchr Sep 29, 2020
4f252be
Rename `BlockAnnounceResult` to `PollBlockAnnounceValidation`
bkchr Sep 29, 2020
e08cbcf
Always return result using the internal future
bkchr Sep 29, 2020
41aa638
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 30, 2020
2b18234
Move the polling and make sure all validation futures are registered
bkchr Sep 30, 2020
047f80c
Ignore the future in the legacy stuff
bkchr Sep 30, 2020
9cbfc19
Merge remote-tracking branch 'origin/master' into bkchr-async-sync
bkchr Sep 30, 2020
8ea6c88
Remove leftover stuff
bkchr Sep 30, 2020
8e5a00a
Update client/network/src/protocol/sync.rs
bkchr Sep 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 83 additions & 29 deletions client/network/src/protocol.rs
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,8 @@ mod rep {
pub const BAD_ROLE: Rep = Rep::new_fatal("Unsupported role");
/// Peer response data does not have requested bits.
pub const BAD_RESPONSE: Rep = Rep::new(-(1 << 12), "Incomplete response");
/// Peer send us a block announcement that failed at validation.
pub const BAD_BLOCK_ANNOUNCEMENT: Rep = Rep::new(-(1 << 12), "Bad block announcement");
}

struct Metrics {
Expand Down Expand Up @@ -542,7 +544,9 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
pub fn update_chain(&mut self) {
let info = self.context_data.chain.info();
self.sync.update_chain_info(&info.best_hash, info.best_number);
self.behaviour.set_legacy_handshake_message(build_status_message(&self.config, &self.context_data.chain));
self.behaviour.set_legacy_handshake_message(
build_status_message(&self.config, &self.context_data.chain),
);
self.behaviour.set_notif_protocol_handshake(
&self.block_announces_protocol,
BlockAnnouncesHandshake::build(&self.config, &self.context_data.chain).encode()
Expand All @@ -568,16 +572,21 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
self.context_data.peers.iter().map(|(id, peer)| (id, &peer.info))
}

pub fn on_custom_message(
fn on_custom_message(
&mut self,
who: PeerId,
data: BytesMut,
) -> CustomMessageOutcome<B> {

let message = match <Message<B> as Decode>::decode(&mut &data[..]) {
Ok(message) => message,
Err(err) => {
debug!(target: "sync", "Couldn't decode packet sent by {}: {:?}: {}", who, data, err.what());
debug!(
target: "sync",
"Couldn't decode packet sent by {}: {:?}: {}",
who,
data,
err.what(),
);
self.peerset_handle.report_peer(who, rep::BAD_MESSAGE);
return CustomMessageOutcome::None;
}
Expand All @@ -590,11 +599,8 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
match message {
GenericMessage::Status(_) =>
debug!(target: "sub-libp2p", "Received unexpected Status"),
GenericMessage::BlockAnnounce(announce) => {
let outcome = self.on_block_announce(who.clone(), announce);
self.update_peer_info(&who);
return outcome;
},
GenericMessage::BlockAnnounce(announce) =>
self.push_block_announce_validation(who.clone(), announce),
GenericMessage::Transactions(m) =>
self.on_transactions(who, m),
GenericMessage::BlockResponse(_) =>
Expand Down Expand Up @@ -1156,51 +1162,82 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
}
}

fn on_block_announce(
/// Push a block announce validation.
///
/// It is required that [`ChainSync::poll_block_announce_validation`] is
/// called later to check for finished validations. The result of the validation
/// needs to be passed to [`Protocol::process_block_announce_validation_result`]
/// to finish the processing.
///
/// # Note
///
/// This will internally create a future, but this future will not be registered
/// in the task before being polled once. So, it is required to call
/// [`ChainSync::poll_block_announce_validation`] to ensure that the future is
/// registered properly and will wake up the task when being ready.
fn push_block_announce_validation(
&mut self,
who: PeerId,
announce: BlockAnnounce<B::Header>,
) -> CustomMessageOutcome<B> {
) {
let hash = announce.header.hash();
let number = *announce.header.number();

if let Some(ref mut peer) = self.context_data.peers.get_mut(&who) {
peer.known_blocks.insert(hash.clone());
}

let is_their_best = match announce.state.unwrap_or(message::BlockState::Best) {
let is_best = match announce.state.unwrap_or(message::BlockState::Best) {
message::BlockState::Best => true,
message::BlockState::Normal => false,
};

match self.sync.on_block_announce(&who, hash, &announce, is_their_best) {
sync::OnBlockAnnounce::Nothing => {
self.sync.push_block_announce_validation(who, hash, announce, is_best);
}

/// Process the result of the block announce validation.
fn process_block_announce_validation_result(
&mut self,
validation_result: sync::PollBlockAnnounceValidation<B::Header>,
) -> CustomMessageOutcome<B> {
let (header, is_best, who) = match validation_result {
sync::PollBlockAnnounceValidation::Nothing { is_best, who, header } => {
self.update_peer_info(&who);

// `on_block_announce` returns `OnBlockAnnounce::ImportHeader`
// when we have all data required to import the block
// in the BlockAnnounce message. This is only when:
// 1) we're on light client;
// AND
// 2) parent block is already imported and not pruned.
if is_their_best {
return CustomMessageOutcome::PeerNewBest(who, number);
if is_best {
return CustomMessageOutcome::PeerNewBest(who, *header.number())
} else {
return CustomMessageOutcome::None;
return CustomMessageOutcome::None
}
}
sync::OnBlockAnnounce::ImportHeader => () // We proceed with the import.
}
sync::PollBlockAnnounceValidation::ImportHeader { header, is_best, who } => {
self.update_peer_info(&who);
(header, is_best, who)
}
sync::PollBlockAnnounceValidation::Failure { who } => {
self.report_peer(who, rep::BAD_BLOCK_ANNOUNCEMENT);
return CustomMessageOutcome::None
}
};

let number = *header.number();

// to import header from announced block let's construct response to request that normally would have
// been sent over network (but it is not in our case)
// to import header from announced block let's construct response to request that normally
// would have been sent over network (but it is not in our case)
let blocks_to_import = self.sync.on_block_data(
&who,
None,
message::generic::BlockResponse {
id: 0,
blocks: vec![
message::generic::BlockData {
hash: hash,
header: Some(announce.header),
hash: header.hash(),
header: Some(header),
body: None,
receipt: None,
message_queue: None,
Expand All @@ -1210,8 +1247,10 @@ impl<B: BlockT, H: ExHashT> Protocol<B, H> {
},
);

if is_their_best {
self.pending_messages.push_back(CustomMessageOutcome::PeerNewBest(who, number));
if is_best {
self.pending_messages.push_back(
CustomMessageOutcome::PeerNewBest(who, number),
);
}

match blocks_to_import {
Expand Down Expand Up @@ -1549,6 +1588,15 @@ impl<B: BlockT, H: ExHashT> NetworkBehaviour for Protocol<B, H> {
warn!(target: "sub-libp2p", "Inconsistent state, no peers for pending transaction!");
}
}

// Check if there is any block announcement validation finished.
while let Poll::Ready(result) = self.sync.poll_block_announce_validation(cx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

match self.process_block_announce_validation_result(result) {
CustomMessageOutcome::None => {},
outcome => self.pending_messages.push_back(outcome),
}
}

if let Some(message) = self.pending_messages.pop_front() {
return Poll::Ready(NetworkBehaviourAction::GenerateEvent(message));
}
Expand Down Expand Up @@ -1643,9 +1691,15 @@ impl<B: BlockT, H: ExHashT> NetworkBehaviour for Protocol<B, H> {
}
Some(Fallback::BlockAnnounce) => {
if let Ok(announce) = message::BlockAnnounce::decode(&mut message.as_ref()) {
let outcome = self.on_block_announce(peer_id.clone(), announce);
self.update_peer_info(&peer_id);
outcome
self.push_block_announce_validation(peer_id, announce);

// Make sure that the newly added block announce validation future was
// polled once to be registered in the task.
if let Poll::Ready(res) = self.sync.poll_block_announce_validation(cx) {
self.process_block_announce_validation_result(res)
} else {
CustomMessageOutcome::None
}
} else {
warn!(target: "sub-libp2p", "Failed to decode block announce");
CustomMessageOutcome::None
Expand Down
Loading