Mixnet integration #14207

zdave · 2023-05-23T20:19:04Z

This adds all the necessary mixnet components, and puts them together in the "kitchen-sink" node/runtime. The components added are:

A pallet (frame/mixnet). This is responsible for determining the current mixnet session and phase, and the mixnodes to use in each session. It includes an offchain worker which attempts to automatically register validator nodes as mixnodes. The logic of this pallet is very similar to that of the im-online pallet.
A service (client/mixnet). This implements the core mixnet logic, building on the mixnet crate. The service communicates with other nodes using notifications sent over the "mixnet" protocol.
An RPC interface. This currently only supports sending transactions over the mixnet.

zdave · 2023-05-24T11:28:33Z

Ideally the new CLI options would only be added to the kitchen-sink node, but the best way to achieve this is not obvious to me.

cheme

Had a few question from my first reading, at this point I did not yet get the session even odd logic, but will look at it soon.

cheme · 2023-05-24T11:49:07Z

client/api/src/execution_extensions.rs

@@ -165,6 +168,7 @@ impl<T: offchain::DbExternalities + Clone + Sync + Send + 'static> DbExternaliti
 pub struct ExecutionExtensions<Block: BlockT> {
 	strategies: ExecutionStrategies,
 	keystore: Option<KeystorePtr>,
+	mixnet_kx_public_store: Option<Arc<MixnetKxPublicStore>>,


Thinking out loud (no idea if better), could this have been part of existing keystore?

I didn't think the requirements overlapped enough for this to make sense. The existing keystore is (AFAICT) designed for long-lived keys that persist across node restarts. The mixnet key-exchange keys are only used for a single session (a few hours), and should be destroyed once the session ends. They intentionally do not persist across node restarts, as this would be non-trivial to implement properly:

When destroying keys you would have to be careful to properly erase the key data from disk.

The corresponding replay filters would also have to persist, or you would be vulnerable to replay attacks after a node restart.

I opened a GitHub issue about this a while back (#12595), there wasn't really any conclusion though.

client/cli/src/params/mixnet_params.rs

client/mixnet/src/packet_dispatcher.rs

client/api/src/execution_extensions.rs

client/mixnet/src/extrinsic_queue.rs

cheme · 2023-05-24T13:32:11Z

frame/mixnet/src/lib.rs

+
+impl<MaxPeerIdSize, MaxExternalAddressSize, MaxExternalAddresses> Into<OpaqueMixnode>
+	for BoundedOpaqueMixnode<
+		BoundedVec<u8, MaxPeerIdSize>,


Not sure if doable but type aliasing BoundedVec<u8, MaxPeerIdSize> to PeerIdRuntime or a better name would be good.
Also for BoundedVec<u8, MaxExternalAddressSize>.

I would like to write something like impl<T: Config> Into<OpaqueMixnode> for BoundedOpaqueMixnodeFor<T> here, but the Rust compiler doesn't like this:

error[E0207]: the type parameter `T` is not constrained by the impl trait, self type, or predicates --> frame/mixnet/src/lib.rs:84:6 | 84 | impl<T: Config> Into<OpaqueMixnode> for BoundedOpaqueMixnodeFor<T> | ^ unconstrained type parameter

I can do something like this, but not sure it's better:

pub type BoundedPeerId<MaxSize> = BoundedVec<u8, MaxSize>; pub type BoundedExternalAddress<MaxSize> = BoundedVec<u8, MaxSize>; pub type BoundedExternalAddresses<MaxSizeEach, Max> = BoundedVec<BoundedExternalAddress<MaxSizeEach>, Max>; impl<MaxPeerIdSize, MaxExternalAddressSize, MaxExternalAddresses> Into<OpaqueMixnode> for BoundedOpaqueMixnode< BoundedPeerId<MaxPeerIdSize>, BoundedExternalAddresses<MaxExternalAddressSize, MaxExternalAddresses>, >

frame/mixnet/src/lib.rs

cheme · 2023-05-24T13:51:24Z

frame/mixnet/src/lib.rs

+
+	#[pallet::call]
+	impl<T: Config> Pallet<T> {
+		/// Register a mixnode for the following session.


is there a way to register a change of external address for the current session?

No. Doesn't feel worthwhile to implement this to me; sessions should only be a few hours?

if a few hour offline is acceptable (probably is initially).

frame/mixnet/src/lib.rs

tomaka · 2023-05-25T11:19:57Z

Is there (or is there going to be) some kind of documentation for how this works at the protocol level somewhere? (source code doesn't count, since it is extremely hard to decipher for non-maintainers)

cheme · 2023-05-26T07:16:13Z

Is there (or is there going to be) some kind of documentation for how this works at the protocol level somewhere? (source code doesn't count, since it is extremely hard to decipher for non-maintainers)

That is a really good question, the design did change a lot since the first prototyping, and a high level documentation would be probably good too.
Not sure when is the right time for the effort, and the correct medium.
I mean probably what I would want is a good technical blog post to give me reading direction on the project (with frame description and more general network behaviour (delays, routings...)).
Could also wonder if the protocol could be in a Psp?

arkpar · 2023-05-26T07:30:02Z

Here's a tracking issue for the documentaiton:
paritytech/mixnet#13

zdave · 2023-05-26T11:19:40Z

Is there (or is there going to be) some kind of documentation for how this works at the protocol level somewhere? (source code doesn't count, since it is extremely hard to decipher for non-maintainers)

There isn't really anything yet. I'll try to put something together next week. If you're interested in something to read now, the packet format is very close to what is described in the Sphinx paper.

client/mixnet/src/request.rs

arkpar · 2023-05-30T10:43:31Z

From what I can see, after the initial sync the node will wait for the registration phase of the next session before registering. How does it it handle traffic in the meantime and what key will be used? Would it make sense to register immediatelly after the sync is completed?

primitives/io/src/lib.rs

zdave · 2023-05-30T13:53:46Z

From what I can see, after the initial sync the node will wait for the registration phase of the next session before registering. How does it it handle traffic in the meantime and what key will be used? Would it make sense to register immediatelly after the sync is completed?

If a node isn't registered as a mixnode it should only receive reply and loop cover traffic, which it will generate a random key pair for. The public key doesn't need to be shared with any other nodes.

If a node is registered as a mixnode and then restarts, it will no longer have the private key corresponding to the public key that was registered, so it will only be able to operate as a non-mixnode until the next session.

I haven't really thought about allowing registrations after the start of a session. Maybe this could be made to work, but it seems like a bunch of extra complexity for little gain. Sessions are only a few hours long, and if a node is an active validator in a session, it should really stay online for the whole session.

cla-bot-2021 · 2023-06-01T13:40:48Z

User @zdave-parity, please sign the CLA here.

zdave · 2023-06-05T23:28:06Z

Ideally the new CLI options would only be added to the kitchen-sink node, but the best way to achieve this is not obvious to me.

Done this now. And got rid of the client/service changes; all that stuff is just in bin/node now.

zdave · 2023-07-17T19:39:27Z

Been on holiday for 2 weeks, just getting back to this.

Re moving the registration stuff from an offchain worker to a separate call into the runtime: I can do this, but it will involve duplicating some of the stuff in client/offchain, or making it public. Currently I'm calling sp_io::offchain::random_seed(), sp_io::offchain::is_validator(), and sp_io::offchain::network_state() from the offchain worker. I'll have to copy across some of the Externalities implementation from client/offchain, or I guess pass this stuff in as arguments.

It's not clear to me why the offchain stuff is handled differently to the other execution extensions -- why is the Externalities implementation attached to ExecutionContext rather than ExecutionExtensions? This would be easier if the latter was the case.

Responsible for determining the mixnode set for each session.

zdave · 2023-07-27T18:44:03Z

I've reworked things to get rid of the offchain worker. The mixnet service now just calls the MixnetApi::maybe_register runtime function every block instead, which does basically the same thing.

arkpar · 2023-07-31T09:54:17Z

@bkchr Could you please take another look?

client/cli/src/params/mixnet_params.rs

bkchr · 2023-08-18T09:17:54Z

client/cli/src/params/mixnet_params.rs

@@ -0,0 +1,67 @@
+// This file is part of Substrate.


It would also be really nice to move this into sc-mixnet.

Hmm I'm actually thinking the kitchen-sink node would be the right place for this? It makes assumptions that are not made by sc-mixnet. For example, sc-mixnet does not assume that non-authorities cannot be mixnodes (sc-mixnet doesn't care whether nodes are authorities or not). Similarly, sc-mixnet doesn't really know anything about RPC (so the comments about RPC would be out of place).

If you agree, I'll move it there instead.

client/mixnet/src/api.rs

client/mixnet/src/config.rs

frame/mixnet/src/lib.rs

frame/mixnet/Cargo.toml

frame/mixnet/src/lib.rs

Not necessary; the mixnet pallet can track the session index itself.

To avoid node depending on pallet_mixnet.

In particular, avoid referring to "the register phase". Just refer to it as the DisconnectFromPrev phase, which is the name of the enum variant.

Note that registrations are now accepted in all phases, and the previous mixnode set is kept for the whole session.

paritytech-cicd-pr · 2023-08-22T15:17:13Z

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-check-benches
Logs: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/3426351

zdave requested review from cheme, arkpar and a team May 23, 2023 20:19

arkpar requested a review from bkchr May 23, 2023 22:57

cheme reviewed May 24, 2023

View reviewed changes

arkpar reviewed May 30, 2023

View reviewed changes

client/mixnet/src/request.rs Outdated Show resolved Hide resolved

arkpar reviewed May 30, 2023

View reviewed changes

primitives/io/src/lib.rs Outdated Show resolved Hide resolved

zdave requested a review from a team June 1, 2023 13:40

zdave requested a review from a team as a code owner June 1, 2023 13:40

arkpar approved these changes Jun 6, 2023

View reviewed changes

sergejparity approved these changes Jun 7, 2023

View reviewed changes

paritytech-ci requested a review from a team June 7, 2023 11:11

davxy mentioned this pull request Aug 24, 2023

Sassafras Protocol paritytech/polkadot-sdk#41

Open

32 tasks

zdave added 2 commits July 27, 2023 00:58

Disconnect peer on unreserving if no free slot for

60a7b74

Mixnet pallet

a7581f2

Responsible for determining the mixnode set for each session.

paritytech-ci requested a review from a team July 27, 2023 18:36

zdave and others added 5 commits July 27, 2023 19:58

Mixnet service

f77fcd6

Mixnet RPC interface

b48e4d3

Basic mixnet CLI options

38fae40

Add mixnet to kitchen sink node/runtime

b068a9e

Update Cargo.lock

8aa640b

arkpar mentioned this pull request Jul 31, 2023

Is this repo based on libp2p? paritytech/mixnet#16

Closed

zdave-parity added 2 commits July 31, 2023 17:47

Fix mixnode registration

3ae4dba

Update spec URL

6e43842

youngjoon-lee mentioned this pull request Aug 4, 2023

Mixnet PoC logos-co/nomos-node#273

Closed

bkchr reviewed Aug 18, 2023

View reviewed changes

zdave added 11 commits August 21, 2023 15:39

Drop ValidatorSet from pallet_mixnet::Config

692dbee

Not necessary; the mixnet pallet can track the session index itself.

Move session key types from pallet_mixnet to sp_mixnet

3d9af21

To avoid node depending on pallet_mixnet.

Drop years from copyright headers

f28ea4b

Improve Api::submit_extrinsic doc

128b31b

Doc formatting

ab11fc8

Some doc comments

cae93a8

Clarify some log messages

ba17d24

Avoid arrayref dependency

f4cf5ba

Improve mixnet pallet comments

da77d6b

In particular, avoid referring to "the register phase". Just refer to it as the DisconnectFromPrev phase, which is the name of the enum variant.

Improve maybe_register doc comment

dbe4d92

Simplify mixnode storage in mixnet pallet

a6194b1

Note that registrations are now accepted in all phases, and the previous mixnode set is kept for the whole session.

zdave closed this by deleting the head repository Aug 31, 2023

zdave-parity mentioned this pull request Aug 31, 2023

Mixnet integration paritytech/polkadot-sdk#1346

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mixnet integration #14207

Mixnet integration #14207

zdave commented May 23, 2023

zdave commented May 24, 2023

cheme left a comment

cheme May 24, 2023

zdave May 26, 2023

cheme May 24, 2023

zdave May 30, 2023

cheme May 24, 2023

zdave May 24, 2023

cheme May 25, 2023

tomaka commented May 25, 2023 •

edited

Loading

cheme commented May 26, 2023

arkpar commented May 26, 2023

zdave commented May 26, 2023

arkpar commented May 30, 2023

zdave commented May 30, 2023

cla-bot-2021 bot commented Jun 1, 2023

zdave commented Jun 5, 2023

zdave commented Jul 17, 2023

zdave commented Jul 27, 2023

arkpar commented Jul 31, 2023

bkchr Aug 18, 2023

zdave Aug 21, 2023 •

edited

Loading

paritytech-cicd-pr commented Aug 22, 2023

Mixnet integration #14207

Mixnet integration #14207

Conversation

zdave commented May 23, 2023

zdave commented May 24, 2023

cheme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomaka commented May 25, 2023 • edited Loading

cheme commented May 26, 2023

arkpar commented May 26, 2023

zdave commented May 26, 2023

arkpar commented May 30, 2023

zdave commented May 30, 2023

cla-bot-2021 bot commented Jun 1, 2023

zdave commented Jun 5, 2023

zdave commented Jul 17, 2023

zdave commented Jul 27, 2023

arkpar commented Jul 31, 2023

Choose a reason for hiding this comment

zdave Aug 21, 2023 • edited Loading

Choose a reason for hiding this comment

paritytech-cicd-pr commented Aug 22, 2023

tomaka commented May 25, 2023 •

edited

Loading

zdave Aug 21, 2023 •

edited

Loading