feat(erasure_coding): introduce erasure coding for PoV Distributor #3281

edwardmack · 2023-05-25T16:41:10Z

Changes

This PR introduces functions to implement erasure coding for use by PoV Distributor for sharding PoV bytes.

ObtainChunks splits the provided data into validatorsQty number of chunks plus additional parity chunks for reconsturction
Reconstruct reconstructs decodable data from a set of chunks.

This uses Reed-Solomon Erasure Coding go library created by Backblaze, see blog post for details.

Currently these function work with []byte data, future functions will construct chunks from PoV structs.

Tests

go test github.com/ChainSafe/gossamer/lib/erasure-coding -v

Issues

Regarding issue #3275

Primary Reviewer

@kishansagathiya
@timwu20

CLAassistant · 2023-05-25T16:41:15Z

All committers have signed the CLA.

EclesioMeloJunior

LGTM

kishansagathiya

Really cool stuff. I have a few comments. It should be good to go after that!

lib/erasure/erasure.go

lib/erasure/erasure_test.go

lib/erasure/erasure.go

kishansagathiya · 2023-05-31T15:56:40Z

Now that there is a feature branch for parachains feat/parachain, we should target this PR to that branch.

kishansagathiya

Epic!!

kishansagathiya · 2023-06-01T05:18:27Z

dot/network/utils.go

@@ -186,17 +183,12 @@ func readLEB128ToUint64(r io.Reader, buf []byte) (uint64, int, error) {
 }

 // readStream reads from the stream into the given buffer, returning the number of bytes read
-func readStream(stream libp2pnetwork.Stream, bufPointer *[]byte, maxSize uint64) (int, error) {
+func readStream(stream libp2pnetwork.Stream, bufPointer *[]byte, maxSize uint64) (tot int, err error) {


opinion/preference: better to not use named returns to avoid returning unexpectedly by mistake

That makes sense, these changes were from @EclesioMeloJunior (I'm not sure why they're showing as changes on this PR, I think I may have done something wrong when rebasing this branch onto feat/parachain.)

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

…hainSafe#3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

…butor (#3281)" This reverts commit ab33e13.

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack requested review from timwu20, EclesioMeloJunior, jimjbrettj, kishansagathiya, dimartiro and kanishkatn as code owners May 25, 2023 16:41

edwardmack force-pushed the ed/erasure_coding branch 3 times, most recently from 3fb226c to 76b4323 Compare May 25, 2023 17:44

EclesioMeloJunior approved these changes May 29, 2023

View reviewed changes

edwardmack force-pushed the ed/erasure_coding branch 2 times, most recently from fd5543b to 76ff2bf Compare May 30, 2023 21:46

kishansagathiya reviewed May 31, 2023

View reviewed changes

lib/erasure/erasure.go Outdated Show resolved Hide resolved

lib/erasure/erasure.go Show resolved Hide resolved

lib/erasure/erasure.go Show resolved Hide resolved

lib/erasure/erasure_test.go Show resolved Hide resolved

lib/erasure/erasure.go Outdated Show resolved Hide resolved

edwardmack changed the base branch from development to feat/parachain May 31, 2023 21:36

EclesioMeloJunior and others added 11 commits May 31, 2023 17:46

fix(dot/network): remove maxReads limitation to read stream (#3287)

70a99a6

test erasure encoding

a0a8b2a

add erasure coding to lib, add tests

07c65bd

implement Reconstruct, add tests

618f02a

add comments

fbbef69

add copyright notice

6d6ae8a

address lint issues

fb8e0f9

rename package

3acd29f

go imported

ea9a9a6

add return data to reconstruct

c559266

address lint issues

4ed66eb

edwardmack force-pushed the ed/erasure_coding branch from b0ac8d0 to 4ed66eb Compare May 31, 2023 21:47

kishansagathiya approved these changes Jun 1, 2023

View reviewed changes

edwardmack merged commit 1782a1b into feat/parachain Jun 8, 2023

edwardmack deleted the ed/erasure_coding branch June 8, 2023 14:30

edwardmack added a commit that referenced this pull request Jun 8, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

0359eb4

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Jun 13, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

1ac5a56

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Jun 14, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

0412aff

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jun 20, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

b1273f3

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kanishkatn pushed a commit that referenced this pull request Jul 5, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

ab33e13

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kanishkatn added a commit that referenced this pull request Jul 5, 2023

Revert "feat(erasure_coding): introduce erasure coding for PoV Distri…

fac53cd

…butor (#3281)" This reverts commit ab33e13.

kishansagathiya pushed a commit that referenced this pull request Jul 12, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

467af59

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jul 12, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

318a214

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jul 14, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

5657715

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jul 27, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

e9ec1de

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Jul 27, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

5515f79

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Aug 4, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

39e8c83

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Aug 22, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

7e36ff3

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Aug 22, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

e580bfe

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Aug 30, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

8dfb2b0

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Aug 31, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

168c41f

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Aug 31, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

7dff387

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Sep 14, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

f001854

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jan 23, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

ba7eb83

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jan 24, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

4eaa722

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

timwu20 pushed a commit that referenced this pull request Jun 13, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

1fe734d

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

timwu20 pushed a commit that referenced this pull request Jun 17, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

4921493

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

timwu20 pushed a commit that referenced this pull request Jun 20, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

864c4ab

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Jun 26, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

a72b63b

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

edwardmack added a commit that referenced this pull request Jul 12, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

654af6a

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jul 15, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

97de7a1

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

kishansagathiya pushed a commit that referenced this pull request Jul 15, 2024

feat(erasure_coding): introduce erasure coding for PoV Distributor (#…

fa708e4

…3281) Co-authored-by: Eclésio Junior <eclesiomelo.1@gmail.com> Co-authored-by: Edward Mack <emack@pop-os.localdomain>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(erasure_coding): introduce erasure coding for PoV Distributor #3281

feat(erasure_coding): introduce erasure coding for PoV Distributor #3281

edwardmack commented May 25, 2023

CLAassistant commented May 25, 2023 •

edited

Loading

EclesioMeloJunior left a comment

kishansagathiya left a comment

kishansagathiya commented May 31, 2023

kishansagathiya left a comment

kishansagathiya Jun 1, 2023

edwardmack Jun 2, 2023

feat(erasure_coding): introduce erasure coding for PoV Distributor #3281

feat(erasure_coding): introduce erasure coding for PoV Distributor #3281

Conversation

edwardmack commented May 25, 2023

Changes

Tests

Issues

Primary Reviewer

CLAassistant commented May 25, 2023 • edited Loading

EclesioMeloJunior left a comment

Choose a reason for hiding this comment

kishansagathiya left a comment

Choose a reason for hiding this comment

kishansagathiya commented May 31, 2023

kishansagathiya left a comment

Choose a reason for hiding this comment

kishansagathiya Jun 1, 2023

Choose a reason for hiding this comment

edwardmack Jun 2, 2023

Choose a reason for hiding this comment

CLAassistant commented May 25, 2023 •

edited

Loading