feat: add blob protocol to upload-client #1425

joaosa · 2024-04-30T17:13:45Z

Replaces the store/add call with blob/add on the upload-client;
Add blob/remove and blob/list;
Ports store-related tests to create a blob protocol counterpart;
Fixes associated tests.

packages/upload-client/test/helpers/utils.js

Gozala

Added a comment

Gozala · 2024-05-06T21:43:16Z

@joaosa I suspect that w3up-client tests are failing because the upload-api in-memory service isn't wired with new capabilities, I'm not exactly sure why though

Actually looks like failing tests also use mocks as opposed to actual upload-api service hence the problem https://github.com/w3s-project/w3up/blob/21065624c20108b208dc106193a60d307fd7391b/packages/w3up-client/test/client.test.js#L37-L88

joaosa · 2024-05-06T23:28:31Z

Something is eluding me as suddenly the CI decided to start firing prettier errors with no changes from my side 🤔

alanshaw

Looks great, I've left some minor feedback but this should be good to merge after it's addressed IMO.

🚀

alanshaw · 2024-05-10T13:52:14Z

packages/upload-api/src/blob/add.js

@@ -161,6 +161,11 @@ async function allocate({ context, blob, space, cause }) {

  // 3. if not already allocated (or expired) execute `blob/allocate`
  if (!blobAllocateReceipt) {
+    // Create allocation task and save it
+    const saveTask = await context.tasksStorage.put(task)


Ok I think this is the quirk that @Gozala is going to fix today 🤞. I think it's ok to do here, but not necessary in production.

The code that stores invocations and receipts runs in w3infra. So this should be done automatically for you in allocate.execute(...) below.

In the tests here, there's no code that does this automatically, which is why in tests you see this: https://github.com/w3s-project/w3up/blob/f8132ca1fced72a4addc7e9f0a2162e823c1ea5f/packages/upload-api/test/helpers/blob.js#L113-L115

This is actually part of @Gozala's PR into my branch, but good to get more context 👍

Yeah I have added this before I realized what was going on, I have branch that I'm actively trying to finish that should fix this and will remove these lines also.

packages/upload-client/src/blob.js

alanshaw · 2024-05-10T14:11:45Z

packages/upload-client/src/types.ts

+  }
+  blob: {
+    add: ServiceMethod<BlobAdd, BlobAddSuccess, BlobAddFailure>
+    remove: ServiceMethod<BlobRemove, BlobRemoveSuccess, BlobRemoveFailure>


Argh, I completely forgot, we're also going to need blob/get for the console. Could you send a follow up PR to add? So sorry.

I don't really see this capability in the spec. How should we go about this?

Ah yes it'll need a specs PR also.

Essentially the same as store/get - it allows the caller to get the size of the blob (and potentially other stuff in the future). The allocations storage already has a method, you just need to create a handler, call it and return it: https://github.com/w3s-project/w3up/blob/9982d12b0a1f9a6da3f0d4264b9a35348e189dfb/packages/upload-api/src/types/blob.ts#L20-L23

alanshaw · 2024-05-10T14:14:45Z

packages/upload-client/test/blob.test.js

+  it('stores a DAG with the service', async () => {
+    const space = await Signer.generate()
+    const agent = await Signer.generate()
+    const car = await randomCAR(128)


I'd use just random bytes here - there's no need to associate this with CARs. Same for all of these tests.

packages/w3up-client/src/capability/store.js

alanshaw · 2024-05-10T14:19:38Z

packages/w3up-client/test/capability/blob.test.js

+import * as Test from '../test.js'
+
+export const BlobClient = Test.withContext({
+  'should store a CAR file': async (


Again, just reference blobs here!

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

Gozala · 2024-05-10T23:29:56Z

packages/upload-client/src/blob.js

+    })
+  }
+
+  return multihash


I would expect blob/add not to return until task corresponding to result.out.ok.site is done, which I realize from the current state of the backend implementation have no good way of doing. Never the less I'll try to make a PR that will add missing pieces. I don't think we should block this in the meantime.

🤖 I have created a release *beep* *boop* --- ## [17.1.0](capabilities-v17.0.0...capabilities-v17.1.0) (2024-05-14) ### Features * add "plan/create-admin-session" capability ([#1411](#1411)) ([50eeeb5](50eeeb5)) * add blob protocol to upload-client ([#1425](#1425)) ([49aef56](49aef56)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

🤖 I have created a release *beep* *boop* --- ## [18.4.0](access-v18.3.2...access-v18.4.0) (2024-05-14) ### Features * add "plan/create-admin-session" capability ([#1411](#1411)) ([50eeeb5](50eeeb5)) * add blob protocol to upload-client ([#1425](#1425)) ([49aef56](49aef56)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

🤖 I have created a release *beep* *boop* --- ## [14.1.0](upload-client-v14.0.0...upload-client-v14.1.0) (2024-05-14) ### Features * add blob protocol to upload-client ([#1425](#1425)) ([49aef56](49aef56)) ### Fixes * cid for filecoin offer must be raw ([#1450](#1450)) ([c3dd7b5](c3dd7b5)) ### Other Changes * appease linter ([782c6d0](782c6d0)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

🤖 I have created a release *beep* *boop* --- ## [13.1.0](w3up-client-v13.0.1...w3up-client-v13.1.0) (2024-05-14) ### Features * add "plan/create-admin-session" capability ([#1411](#1411)) ([50eeeb5](50eeeb5)) * add blob protocol to upload-client ([#1425](#1425)) ([49aef56](49aef56)) ### Fixes * test against actual api ([#1438](#1438)) ([f8132ca](f8132ca)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

🤖 I have created a release *beep* *boop* --- ## [14.0.0](upload-api-v13.0.2...upload-api-v14.0.0) (2024-05-14) ### ⚠ BREAKING CHANGES * deprecate issuer ([#1344](#1344)) ### Features * add "plan/create-admin-session" capability ([#1411](#1411)) ([50eeeb5](50eeeb5)) * add blob protocol to upload-client ([#1425](#1425)) ([49aef56](49aef56)) * deprecate issuer ([#1344](#1344)) ([afbbde3](afbbde3)) * move blob index logic from upload-api to blob-index lib ([#1434](#1434)) ([797f628](797f628)) * remove issuer row ([#1345](#1345)) ([cf5b0db](cf5b0db)) ### Fixes * `encodeURIComponent` on bucket origin ([#1448](#1448)) ([5618644](5618644)) * add format specifier to blob location claim URL ([#1445](#1445)) ([9982d12](9982d12)) * test against actual api ([#1438](#1438)) ([f8132ca](f8132ca)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Travis Vachon <travis.vachon@protocol.ai>

🤖 I have created a release *beep* *boop* --- ## [7.0.0](filecoin-api-v6.0.1...filecoin-api-v7.0.0) (2024-05-23) ### ⚠ BREAKING CHANGES * **upload-api:** integrate agent store for idempotence & invocation/receipt persistence ([#1444](#1444)) ### Features * add blob protocol to upload-client ([#1425](#1425)) ([49aef56](49aef56)) * **upload-api:** integrate agent store for idempotence & invocation/receipt persistence ([#1444](#1444)) ([c9bf33e](c9bf33e)) ### Fixes * check service did in w3filecoin ([#1476](#1476)) ([11b00bf](11b00bf)) ### Other Changes * appease linter ([782c6d0](782c6d0)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

- Verifies that `blob/accept` has succeeded before concluding `blob/add` as per #1425 (comment). A location commitment is returned instead of the blob multihash. --------- Co-authored-by: Vasco Santos <santos.vasco10@gmail.com> Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

joaosa changed the title ~~feat: add blob/add to upload-cli~~ feat: add blob/add to upload-client Apr 30, 2024

joaosa force-pushed the feat/blob-add-cli branch 3 times, most recently from d8e3391 to 28cbbac Compare April 30, 2024 18:40

joaosa marked this pull request as draft April 30, 2024 18:49

joaosa force-pushed the feat/blob-add-cli branch 2 times, most recently from bc77447 to b6e213f Compare May 2, 2024 15:44

Gozala reviewed May 6, 2024

View reviewed changes

packages/upload-client/test/helpers/utils.js Show resolved Hide resolved

Gozala reviewed May 6, 2024

View reviewed changes

joaosa marked this pull request as ready for review May 6, 2024 19:25

joaosa requested a review from alanshaw May 6, 2024 19:25

joaosa mentioned this pull request May 6, 2024

client work for blob protocol storacha/project-tracking#47

Closed

joaosa force-pushed the feat/blob-add-cli branch 5 times, most recently from 0a92134 to aaf2a42 Compare May 6, 2024 23:01

joaosa force-pushed the feat/blob-add-cli branch 9 times, most recently from 2dba4d4 to 3d28eb0 Compare May 7, 2024 19:06

Gozala mentioned this pull request May 8, 2024

fix: test against actual api #1438

Merged

joaosa force-pushed the feat/blob-add-cli branch from f3dc204 to 2cef37d Compare May 8, 2024 15:18

joaosa added 3 commits May 9, 2024 17:04

fix: use allocations storate for w3up blob client test

6ab2ffc

fix: address typo

6cf26eb

chore: remove unneeded boilerplate

e5916a8

alanshaw requested changes May 10, 2024

View reviewed changes

joaosa and others added 7 commits May 10, 2024 15:38

chore: remove blank comment line

9bb3d11

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

chore: fix blob/add param description

e9a7edb

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

chore: fix blob/list description

6ae74f8

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

chore: fix typo

4f71107

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

fix: use the store/list capability

6704748

Co-authored-by: Alan Shaw <alan.shaw@protocol.ai>

chore: fix blob/add description

42bd997

chore: replace test CARs with bytes

b9eaeea

joaosa requested a review from alanshaw May 10, 2024 15:25

Gozala reviewed May 10, 2024

View reviewed changes

alanshaw approved these changes May 13, 2024

View reviewed changes

joaosa merged commit 49aef56 into main May 13, 2024
14 checks passed

joaosa deleted the feat/blob-add-cli branch May 13, 2024 15:17

joaosa mentioned this pull request May 15, 2024

fix: check for blob/accept receipts before blob/add is concluded #1459

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add blob protocol to upload-client #1425

feat: add blob protocol to upload-client #1425

joaosa commented Apr 30, 2024 •

edited

Loading

Gozala left a comment

Gozala commented May 6, 2024 •

edited

Loading

joaosa commented May 6, 2024

alanshaw left a comment

alanshaw May 10, 2024

joaosa May 10, 2024

Gozala May 10, 2024

alanshaw May 10, 2024

joaosa May 13, 2024

alanshaw May 13, 2024 •

edited

Loading

alanshaw May 10, 2024

joaosa May 10, 2024

alanshaw May 10, 2024

joaosa May 10, 2024

Gozala May 10, 2024

feat: add blob protocol to upload-client #1425

feat: add blob protocol to upload-client #1425

Conversation

joaosa commented Apr 30, 2024 • edited Loading

Gozala left a comment

Choose a reason for hiding this comment

Gozala commented May 6, 2024 • edited Loading

joaosa commented May 6, 2024

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw May 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joaosa commented Apr 30, 2024 •

edited

Loading

Gozala commented May 6, 2024 •

edited

Loading

alanshaw May 13, 2024 •

edited

Loading