Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add blob protocol to upload-client #1425

Merged
merged 43 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
9aee035
feat: add blob add to upload-cli
joaosa Apr 30, 2024
48cde40
feat: use blob add on upload-cli
joaosa Apr 30, 2024
d5672c8
feat: modify tests to cover blob add on upload-client
joaosa Apr 30, 2024
0f1e592
chore: cleanup code and add current status
joaosa Apr 30, 2024
1ccb3c4
feat: revamp implementation
joaosa May 2, 2024
dc40432
chore: cleanup blob/add implementation
joaosa May 3, 2024
48a641e
chore: fix typo
joaosa May 3, 2024
99c1878
feat: expose blob add result types
joaosa May 3, 2024
8ae5f69
fix: have all test return types match the blob add response
joaosa May 3, 2024
2ff3748
fix: pass upload CAR and file tests
joaosa May 6, 2024
8fb6720
fix: pass upload-client tests
joaosa May 6, 2024
5642e01
feat: add client upload progress
joaosa May 6, 2024
1f600c3
chore: relock deps
joaosa May 6, 2024
2c73702
fix: blob add upload progress implementation
joaosa May 6, 2024
50929e6
chore: address linter errors
joaosa May 6, 2024
9391a86
fix: test blob upload on w3up-client tests
joaosa May 6, 2024
45507f2
chore: cleanup upload-client test code
joaosa May 6, 2024
0896614
fix: pass w3up-client upload tests
joaosa May 6, 2024
7552e9b
fix: missing import
joaosa May 6, 2024
4eee857
chore: appease prettier
joaosa May 6, 2024
24a99e9
fix: address w3up upload test expectations
joaosa May 7, 2024
c6aaa04
feat: implement blob/remove
joaosa May 8, 2024
65dfee1
feat: implement blob/list
joaosa May 8, 2024
21e2973
fix: pass blob protocol upload-client tests
joaosa May 8, 2024
f9c4043
chore: address coverage for upload-client blob
joaosa May 8, 2024
5196f9c
feat: add w3up-client upload-client blob boilerplate
joaosa May 8, 2024
5bad45f
fix: blob add (#1442)
Gozala May 9, 2024
e4ab876
fix: replace blob protocol CAR params with blobs
joaosa May 9, 2024
3825838
fix: use the bytes field of a multihash
joaosa May 9, 2024
f31e2f0
fix: replace CAR usage with blobs
joaosa May 9, 2024
722fec0
fix: cleanup blob interfaces
joaosa May 9, 2024
1294cf9
fix: cover blob client tests
joaosa May 9, 2024
726f2d7
fix: move c8 ignore block
joaosa May 9, 2024
6ab2ffc
fix: use allocations storate for w3up blob client test
joaosa May 9, 2024
6cf26eb
fix: address typo
joaosa May 10, 2024
e5916a8
chore: remove unneeded boilerplate
joaosa May 10, 2024
9bb3d11
chore: remove blank comment line
joaosa May 10, 2024
e9a7edb
chore: fix blob/add param description
joaosa May 10, 2024
6ae74f8
chore: fix blob/list description
joaosa May 10, 2024
4f71107
chore: fix typo
joaosa May 10, 2024
6704748
fix: use the store/list capability
joaosa May 10, 2024
42bd997
chore: fix blob/add description
joaosa May 10, 2024
b9eaeea
chore: replace test CARs with bytes
joaosa May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/access-client/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ import type {
UCANRevoke,
UCANRevokeSuccess,
UCANRevokeFailure,
UCANConclude,
UCANConcludeSuccess,
UCANConcludeFailure,
AccountDID,
ProviderDID,
SpaceDID,
Expand Down Expand Up @@ -132,6 +135,11 @@ export interface Service {
}
ucan: {
revoke: ServiceMethod<UCANRevoke, UCANRevokeSuccess, UCANRevokeFailure>
conclude: ServiceMethod<
UCANConclude,
UCANConcludeSuccess,
UCANConcludeFailure
>
}
plan: {
get: ServiceMethod<PlanGet, PlanGetSuccess, PlanGetFailure>
Expand Down
2 changes: 1 addition & 1 deletion packages/capabilities/src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -571,7 +571,7 @@ export type BlobAllocateFailure = NotEnoughStorageCapacity | Ucanto.Failure

// Blob accept
export interface BlobAcceptSuccess {
// A Link for a delegation with site commiment for the added blob.
// A Link for a delegation with site commitment for the added blob.
site: Link
}

Expand Down
4 changes: 4 additions & 0 deletions packages/filecoin-api/src/errors.js
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ export class RecordNotFound extends Server.Failure {
return this.message
}

describe() {
return `Record not found`
}

get name() {
return RecordNotFoundErrorName
}
Expand Down
5 changes: 5 additions & 0 deletions packages/upload-api/src/blob/add.js
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,11 @@ async function allocate({ context, blob, space, cause }) {

// 3. if not already allocated (or expired) execute `blob/allocate`
if (!blobAllocateReceipt) {
// Create allocation task and save it
const saveTask = await context.tasksStorage.put(task)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I think this is the quirk that @Gozala is going to fix today 🤞. I think it's ok to do here, but not necessary in production.

The code that stores invocations and receipts runs in w3infra. So this should be done automatically for you in allocate.execute(...) below.

In the tests here, there's no code that does this automatically, which is why in tests you see this: https://github.com/w3s-project/w3up/blob/f8132ca1fced72a4addc7e9f0a2162e823c1ea5f/packages/upload-api/test/helpers/blob.js#L113-L115

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually part of @Gozala's PR into my branch, but good to get more context 👍

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I have added this before I realized what was going on, I have branch that I'm actively trying to finish that should fix this and will remove these lines also.

if (!saveTask.ok) {
return saveTask
}
// Execute allocate invocation
const allocateRes = await allocate.execute(context.getServiceConnection())
if (allocateRes.out.error) {
Expand Down
4 changes: 4 additions & 0 deletions packages/upload-api/src/errors.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ export class RecordNotFound extends Server.Failure {
return this.message
}

describe() {
return `Record not found`
}

get name() {
return RecordNotFoundErrorName
}
Expand Down
2 changes: 1 addition & 1 deletion packages/upload-api/test/storage/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ export async function getServiceStorageImplementations(options) {
const dudewhereBucket = new DudewhereBucket()
const revocationsStorage = new RevocationsStorage()
const plansStorage = new PlansStorage()
const usageStorage = new UsageStorage(storeTable)
const usageStorage = new UsageStorage(storeTable, allocationsStorage)
const provisionsStorage = new ProvisionsStorage(options.providers)
const subscriptionsStorage = new SubscriptionsStorage(provisionsStorage)
const delegationsStorage = new DelegationsStorage()
Expand Down
27 changes: 22 additions & 5 deletions packages/upload-api/test/storage/usage-storage.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,26 @@

/** @implements {UsageStore} */
export class UsageStorage {
/** @param {import('./store-table.js').StoreTable} storeTable */
constructor(storeTable) {
/**
* @param {import('./store-table.js').StoreTable} storeTable
* @param {import('./allocations-storage.js').AllocationsStorage} allocationsStorage
*/
constructor(storeTable, allocationsStorage) {
this.storeTable = storeTable
this.allocationsStorage = allocationsStorage
}

get items() {
return [
...this.storeTable.items.map((item) => ({
...item,
cause: item.invocation,
})),
...this.allocationsStorage.items.map((item) => ({
...item,
size: item.blob.size,
})),
]
}

/**
Expand All @@ -13,11 +30,11 @@ export class UsageStorage {
* @param {{ from: Date, to: Date }} period
*/
async report(provider, space, period) {
const before = this.storeTable.items.filter((item) => {
const before = this.items.filter((item) => {
const insertTime = new Date(item.insertedAt).getTime()
return item.space === space && insertTime < period.from.getTime()
})
const during = this.storeTable.items.filter((item) => {
const during = this.items.filter((item) => {
const insertTime = new Date(item.insertedAt).getTime()
return (
item.space === space &&
Expand All @@ -39,7 +56,7 @@ export class UsageStorage {
size: { initial, final },
events: during.map((item) => {
return {
cause: item.invocation.link(),
cause: /** @type {import('../types.js').Link} */ (item.cause),
delta: item.size,
receiptAt: item.insertedAt,
}
Expand Down
1 change: 1 addition & 0 deletions packages/upload-client/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@
"@ipld/dag-cbor": "^9.0.6",
"@ipld/dag-ucan": "^3.4.0",
"@ipld/unixfs": "^2.1.1",
"@ucanto/core": "^10.0.1",
"@ucanto/client": "^9.0.1",
"@ucanto/interface": "^10.0.1",
"@ucanto/transport": "^9.1.1",
Expand Down
Loading
Loading