Skip to content
This repository has been archived by the owner on Jun 26, 2023. It is now read-only.

Datastore based pinner #4

Merged
merged 29 commits into from
Nov 30, 2020
Merged

Datastore based pinner #4

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
9fc54f6
start restructuring for move to datastore pinner
aschmahmann Sep 30, 2020
be03825
Store pins in datastore instead of in mdag
gammazero Oct 26, 2020
77c9a1d
Import and Export functions. Cid stored as bytes. Revise indexer inte…
gammazero Oct 27, 2020
5398bb2
add name index
gammazero Oct 27, 2020
922fab7
add benchmarks
gammazero Oct 27, 2020
fdf37b1
Use dirty flag to determine when to rebuild indexes
gammazero Oct 28, 2020
a2720f1
Fix benchmarks
gammazero Oct 28, 2020
37293b6
Do not keep pinned CID sets in memory (no-cache implementation)
gammazero Oct 29, 2020
0244724
Add comments and unit test
gammazero Oct 29, 2020
22a61da
Speed up pinning by avoining 2nd recursive check if no changes
gammazero Oct 29, 2020
1662bb8
correct log level
gammazero Oct 29, 2020
cad8378
improve import/export unit test
gammazero Oct 29, 2020
d1a44d7
Update returns error if from CID is not pinned, even when from and to…
gammazero Nov 16, 2020
9bb7a0c
test update of same pin
gammazero Nov 16, 2020
eb32271
Cleanup and better test coverage
gammazero Nov 17, 2020
22254b2
Change requested in review
gammazero Nov 19, 2020
d787682
Additional changes from review
gammazero Nov 19, 2020
0435ac4
Removed New in favor of only having LoadPinner
gammazero Nov 19, 2020
dde41e9
Indexer encodes index and key to allow arbitrary strings
gammazero Nov 19, 2020
0a5737f
Use int64 for dirty count and remove unused const
gammazero Nov 19, 2020
fb419e7
use base64 encoding
gammazero Nov 20, 2020
00764f0
Encode using multibase
gammazero Nov 20, 2020
a6d812c
Changes from review
gammazero Nov 24, 2020
86f36c2
Rename LoadPinner to New for both pinners
gammazero Nov 24, 2020
7a128c6
Check context when loading pinner and during iterative operations
gammazero Nov 24, 2020
cd2065d
indexer.New takes ds.Key
gammazero Nov 24, 2020
9c335cd
Change pin encoding. Add unit test
gammazero Nov 24, 2020
9fafc51
switch to atlas pin encoding
aschmahmann Nov 30, 2020
2586c60
removed type annotations from pin struct
aschmahmann Nov 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions dsindex/indexer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ package dsindex

import (
"context"
"encoding/base64"
"fmt"
"path"

ds "github.com/ipfs/go-datastore"
query "github.com/ipfs/go-datastore/query"
"github.com/multiformats/go-base36"
)

// Indexer maintains a secondary index. Each value of the secondary index maps
Expand Down Expand Up @@ -267,11 +267,11 @@ func encode(data string) string {
if data == "" {
return ""
}
return base36.EncodeToStringUc([]byte(data))
return base64.URLEncoding.WithPadding(base64.NoPadding).EncodeToString([]byte(data))
}

func decode(data string) (string, error) {
b, err := base36.DecodeString(data)
b, err := base64.URLEncoding.WithPadding(base64.NoPadding).DecodeString(data)
aschmahmann marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return "", err
}
Expand Down
1 change: 0 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,5 @@ require (
github.com/ipfs/go-ipld-format v0.0.2
github.com/ipfs/go-log v0.0.1
github.com/ipfs/go-merkledag v0.3.0
github.com/multiformats/go-base36 v0.1.0
github.com/polydawn/refmt v0.0.0-20190221155625-df39d6c2d992
)
2 changes: 0 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,8 +255,6 @@ github.com/mr-tron/base58 v1.1.2 h1:ZEw4I2EgPKDJ2iEw0cNmLB3ROrEmkOtXIkaG7wZg+78=
github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
github.com/multiformats/go-base32 v0.0.3 h1:tw5+NhuwaOjJCC5Pp82QuXbrmLzWg7uxlMFp8Nq/kkI=
github.com/multiformats/go-base32 v0.0.3/go.mod h1:pLiuGC8y0QR3Ue4Zug5UzK9LjgbkL8NSQj0zQ5Nz/AA=
github.com/multiformats/go-base36 v0.1.0 h1:JR6TyF7JjGd3m6FbLU2cOxhC0Li8z8dLNGQ89tUg4F4=
github.com/multiformats/go-base36 v0.1.0/go.mod h1:kFGE83c6s80PklsHO9sRn2NCoffoRdUUOENyW/Vv6sM=
github.com/multiformats/go-multiaddr v0.0.1/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44=
github.com/multiformats/go-multiaddr v0.0.2/go.mod h1:xKVEak1K9cS1VdmPZW3LSIb6lgmoS58qz/pzqmAxV44=
github.com/multiformats/go-multiaddr v0.0.4 h1:WgMSI84/eRLdbptXMkMWDXPjPq7SPLIgGUVm2eroyU4=
Expand Down