Shwap #2675

Wondertan · 2023-09-07T20:33:04Z

Prototype of the new storage for edses - edsfile and optimized roundtrip less ipldv2 protocol

TODO:

edsfile
- Blockstore over EDSFile
- Support for Col sample reads
- ODS only mode
  - ~~Support for Axis, Share and ShareWithProof~~
- Measure size diff
- Delete EDS mode
- GetSharesByNamespace (assigned to @walldiss)
- File streaming (Write / Read)
- Concurrency safety for file
edsStore
- Base implementation
- Caching
- Migrate shrex server to new storage
- Store getter
ipldv2
- Define Sample type and refactor the protocol and the Hasher
- Migrate to proto
- Row Sampling
- ~~Versioning and backwards compatibility with old protocol~~
- Use heights instead of DataHash to identify block
- Hash requested Axis roots to shorten requests
- Retriever over v2 (IPLDv2 reconstruction getter) (assigned to @walldiss)
- Metrics
- IPLDv2 row based getter
  - Getter should support requesting batch of shares by idx at once
  - Store only samples by default
  - Bitswap Sessions
- ~~IPLDv2 Server over LN storage (blockservice)~~ Will work as is
- Migrate actual sampling logic over GetShares share/light: LightAvailability to use GetShares #2984

Stretch Goals:

Range of shares Multihash(by extending SampleID with additional uint16 that defines the end idx of range)
Blob requests. BlobID{DataID, Comm} -> Blob{BlobShares, ProofsToSubroot}
Requesting proofs only for every request
Reconsider DHT discovery.

derrandz

I love the direction this is taking, this is so good to look at.

I think this would better answer to the needs specified in #2618 and would deprecate the gymnastics of reading from the car file done in #2638

(commenting for reference)

Wondertan · 2023-09-08T14:08:58Z

would deprecate the gymnastics of reading from the car file done in #2638

@derrandz, very true

codecov-commenter · 2023-09-18T15:34:01Z

Codecov Report

Merging #2675 (2baea7d) into main (303db05) will increase coverage by 0.22%.
Report is 9 commits behind head on main.
The diff coverage is 54.48%.

@@            Coverage Diff             @@
##             main    #2675      +/-   ##
==========================================
+ Coverage   51.79%   52.02%   +0.22%     
==========================================
  Files         163      173      +10     
  Lines       10854    11596     +742     
==========================================
+ Hits         5622     6033     +411     
- Misses       4738     4997     +259     
- Partials      494      566      +72

Files	Coverage Δ
share/eds/file_store.go	`0.00% <0.00%> (ø)`
share/ipldv2/axis_sample_hasher.go	`56.00% <56.00%> (ø)`
share/ipldv2/ipldv2.go	`60.71% <60.71%> (ø)`
share/ipldv2/share_sample_hasher.go	`56.00% <56.00%> (ø)`
share/eds/file_header.go	`51.16% <51.16%> (ø)`
share/ipldv2/axis_sample_id.go	`63.07% <63.07%> (ø)`
share/ipldv2/share_sample_id.go	`64.17% <64.17%> (ø)`
share/ipldv2/share_sample.go	`71.17% <71.17%> (ø)`
share/eds/ods_file.go	`15.38% <15.38%> (ø)`
share/ipldv2/axis_sample.go	`62.36% <62.36%> (ø)`
... and 2 more

... and 15 files with indirect coverage changes

derrandz · 2023-09-19T07:38:59Z

share/eds/file.go

+	return shr, nil
+}
+
+func (f *File) ShareWithProof(idx int, axis rsmt2d.Axis) (share.Share, nmt.Proof, error) {


[feature request]: Having AxisWithProof will be super useful for search operations to retrieve all rows with their proofs and roots to be able to filter out for specific shares and their proofs by roots

When you get the whole axis, there is no proof.

I mean, if you look at ShareWithProof, to generate a proof for that share, you have to get the whole axis, right? Retrieving an entire axis and recomputing its proof would be sufficient for the purposes I mentioned.

derrandz

a small feature request to make #2618 easier to implement

Wondertan · 2023-09-22T15:21:05Z

Initial file size measurements

EDS 256:
	NEW: ods_file: 8388640(-9.36x)
	NEW: eds_file: 33554464(-2.34x)
	OLD: car_file: 78618134

EDS 128
	NEW: ods_file: 2097184(-9.36x)
	NEW: eds_file: 8388640(-2.34x)
	OLD: car_file: 19631894

…ation method for ids

* Axis -> Row * It's not allowed anymore to request Columns * Delete AxisType from SampleID * Now server decides which Row or Column comittment to choose, not client * New SampleProofType field on Sample * Server's decision on the Sample proof response * Delete AxisHash from Row(Axis) * We break compatibility with Bitswap stateless Content Cerification and now verification has to be done with DAH

fl0rek · 2023-12-28T11:58:43Z

share/shwap/pb/shwap_pb.proto

@@ -0,0 +1,26 @@
+syntax = "proto3";
+


Suggested change

package share.p2p.shwap;

Or was package specifier ommited for some particular reason?

It's not necessary from my understanding

I don't think it's necessary, but other proto files specify the namespace so I expected this one too

Ok. Also, note that the name of the pkg is more like an implementation detail. The formal spec won't contain the pkg name in its proto file.

fl0rek · 2024-01-01T18:52:54Z

I don't think it's necessary but other protobuf files specify the namespace,so we expected this one too

…

On Sun, 31 Dec 2023, 2:26 am Hlib Kanunnikov, ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In share/shwap/pb/shwap_pb.proto <#2675 (comment)> : > @@ -0,0 +1,26 @@ +syntax = "proto3"; + It's not necessary from my understanding — Reply to this email directly, view it on GitHub <#2675 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACF3U6BLASXRHTIRAXGG4KTYMC5LPAVCNFSM6AAAAAA4PNTBF2VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTOOJZGYYDCMJRGY> . You are receiving this because you commented.Message ID: ***@***.***>

Wondertan self-assigned this Sep 7, 2023

Wondertan force-pushed the hlib/v2-prototype branch from 42ad52e to 2f99b52 Compare September 8, 2023 06:30

derrandz reviewed Sep 8, 2023

View reviewed changes

Wondertan force-pushed the hlib/v2-prototype branch from 8e0f0c2 to f1de60a Compare September 18, 2023 16:25

derrandz reviewed Sep 19, 2023

View reviewed changes

Wondertan force-pushed the hlib/v2-prototype branch from e804d2c to ba163cd Compare September 21, 2023 11:19

Wondertan mentioned this pull request Sep 21, 2023

Research Bitswap extension to avoid the top-level index in share.EDStore #1122

Closed

Wondertan force-pushed the hlib/v2-prototype branch 2 times, most recently from 98d475b to e7ebbb1 Compare September 22, 2023 00:14

zvolin mentioned this pull request Nov 13, 2023

shwap protocol eigerco/lumina#133

Closed

Wondertan added 16 commits November 29, 2023 16:29

chore(moddas): speed up sampling for LN

d953117

prototype

f122a71

feat(modp2p): listen on WebTransport by default

7237ad7

lint

ab6bec8

now test verifies all the share proofs

2f08bdd

refactor sampling protocol and use proto for serialization

b33515e

docs and tests for ipldv2

59aa730

add support for col proofs sampling

1b3d881

blockstore impl and various cleanups and improvements

b53769b

initial support for ODS Mode

ce31854

implement axis sampling

3ab6b37

introduce File interface and decouple ipldv2 tests from on disk file

041ed3d

use height as block id

1601460

chore: extract proto helper

600d186

successful experiment with request size shortening for axis sampling

6673564

docs fix

830860d

Wondertan added 7 commits December 2, 2023 18:01

request size optimization for share sample

9ffb284

refactor AxisID away and many more improvements

52f3ab9

remove serialization ambigiouty and ensure there is only one serializ…

21bd2fc

…ation method for ids

cleanup proto field names

cdbd694

namespace mh

c3e8450

namespace mh but finished and tested

8a66fd5

lol

bbcd956

Wondertan force-pushed the hlib/v2-prototype branch from 4b26aef to bbcd956 Compare December 2, 2023 17:49

Wondertan added 4 commits December 3, 2023 13:14

pass by value and cid must constructors

583481b

fix data id test

a28cfef

blockservice constructor

f6db8f9

implement Getter and tests for it

9a8b5ed

Wondertan mentioned this pull request Dec 5, 2023

Shwap workstream tracking issue #2971

Open

71 tasks

Wondertan added 4 commits December 6, 2023 01:18

rename to shwap

0d4dd27

ensure only shares a cached in blockstore

c96579a

add sessions

bb034b3

fl0rek reviewed Dec 28, 2023

View reviewed changes

Wondertan changed the title ~~ipldv2 and eds-file~~ Shwap Jan 8, 2024

walldiss mentioned this pull request Feb 12, 2024

WIP: Shwap prototype #3184

Draft

remove ids

a358713

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shwap #2675

Shwap #2675

Wondertan commented Sep 7, 2023 •

edited

Loading

derrandz left a comment •

edited

Loading

Wondertan commented Sep 8, 2023 •

edited

Loading

codecov-commenter commented Sep 18, 2023 •

edited

Loading

derrandz Sep 19, 2023 •

edited

Loading

Wondertan Sep 19, 2023

derrandz Sep 19, 2023

derrandz left a comment

Wondertan commented Sep 22, 2023 •

edited

Loading

fl0rek Dec 28, 2023 •

edited

Loading

Wondertan Dec 31, 2023

fl0rek Jan 7, 2024

Wondertan Jan 8, 2024

fl0rek commented Jan 1, 2024 via email

Shwap #2675

Are you sure you want to change the base?

Shwap #2675

Conversation

Wondertan commented Sep 7, 2023 • edited Loading

derrandz left a comment • edited Loading

Choose a reason for hiding this comment

Wondertan commented Sep 8, 2023 • edited Loading

codecov-commenter commented Sep 18, 2023 • edited Loading

Codecov Report

derrandz Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Wondertan Sep 19, 2023

Choose a reason for hiding this comment

derrandz Sep 19, 2023

Choose a reason for hiding this comment

derrandz left a comment

Choose a reason for hiding this comment

Wondertan commented Sep 22, 2023 • edited Loading

fl0rek Dec 28, 2023 • edited Loading

Choose a reason for hiding this comment

Wondertan Dec 31, 2023

Choose a reason for hiding this comment

fl0rek Jan 7, 2024

Choose a reason for hiding this comment

Wondertan Jan 8, 2024

Choose a reason for hiding this comment

fl0rek commented Jan 1, 2024 via email

Wondertan commented Sep 7, 2023 •

edited

Loading

derrandz left a comment •

edited

Loading

Wondertan commented Sep 8, 2023 •

edited

Loading

codecov-commenter commented Sep 18, 2023 •

edited

Loading

derrandz Sep 19, 2023 •

edited

Loading

Wondertan commented Sep 22, 2023 •

edited

Loading

fl0rek Dec 28, 2023 •

edited

Loading