Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete the Jest DI with crypto #234

Merged
merged 26 commits into from
May 27, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e6311df
Use ipfs-core instead of ipfs for a lighter dependency
matheus23 May 18, 2021
88d6097
Make yarn node --version be 15.x in nix-shell
matheus23 May 18, 2021
21e43ce
Implement the complete crypto DI for node
matheus23 May 18, 2021
dbc3b3d
Fix cbor encode/decode errors on node
matheus23 May 10, 2021
d391663
ESLint fixes
matheus23 May 19, 2021
3cc7931
Merge branch 'main' into matheus23/complete-jest-di
matheus23 May 19, 2021
71a0796
Fix eslint errors in index.ts
matheus23 May 19, 2021
376a3d6
Revert "Fix cbor encode/decode errors on node"
matheus23 May 19, 2021
d08d0ce
Fix type
matheus23 May 19, 2021
b8ffa04
Test cbor en/decoding
matheus23 May 19, 2021
f9b594c
Implement helper for totally in-memory ipfs
matheus23 May 20, 2021
830ae07
Add integration test & fix filesystem running in node
matheus23 May 20, 2021
08d6833
Remove logging
matheus23 May 20, 2021
4ad48db
Use types added to the workaround repo
matheus23 May 20, 2021
feed976
Fix CBOR decoding errors of encrypted data in node
matheus23 May 21, 2021
9409e0e
Switch back to borc
matheus23 May 21, 2021
39a6695
Update jest & puppeteer & base58-universal
matheus23 May 21, 2021
20de6bd
Add jest puppeteer types
matheus23 May 21, 2021
7015861
Make it possible to run multiple ipfs in-memory
matheus23 May 21, 2021
63bdbcc
Actually switch back from cborg to borc
matheus23 May 21, 2021
409f287
Remove unneeded Buffer.from call
matheus23 May 21, 2021
eab6bc8
Make jest properly exit after all tests
matheus23 May 26, 2021
5cbfd03
wtfnode
matheus23 May 26, 2021
c5d97ce
Revert "wtfnode"
matheus23 May 26, 2021
796acae
Force jest to exit, even if ipfs holds resources
matheus23 May 26, 2021
d475e48
Revert "Make jest properly exit after all tests"
matheus23 May 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"prebuild": "rimraf dist && node scripts/gen-version.js",
"build": "tsc && rollup -c rollup.config.ts",
"start": "tsc -w",
"test": "jest",
"test": "jest --forceExit",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hrm. This smells like there's something wrong with the tests... async operations not resolving?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. I tried to track it down for a while, but gave up. From my testing it seems like in some edge-cases it's possible that ipfs leaves a child process and some sockets hanging after await ipfs.stop(). Not sure what's up with that, but after fighting with it for a while I just gave up.
The IPFS we're using in the tests is basically in-memory only. It doesn't write to disk. So killing it shouldn't be an issue.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool. yeah, something we can track down and fix as a follow up later, imo.

"test:watch": "jest --watch",
"test:prod": "yarn run build && yarn run lint && yarn run test -- --no-cache",
"test:types": "cp -RT tests/types/ dist/ && yarn run tsd",
Expand All @@ -61,12 +61,14 @@
"@babel/core": "^7.12.10",
"@babel/preset-env": "^7.12.7",
"@babel/preset-typescript": "^7.12.7",
"@ipld/car": "https://github.com/matheus23/ipld-car-jest-fix#28fa48a8d0e701ddaf6e4785d0d9f08735b67bc8",
"@rollup/plugin-babel": "^5.2.2",
"@rollup/plugin-commonjs": "^17.0.0",
"@rollup/plugin-inject": "^4.0.2",
"@rollup/plugin-json": "^4.1.0",
"@rollup/plugin-node-resolve": "^11.1.0",
"@types/jest": "^26.0.20",
"@types/jest-environment-puppeteer": "^4.4.1",
"@types/node": "^13.7.4",
"@types/throttle-debounce": "^2.1.0",
"@typescript-eslint/eslint-plugin": "^4.24.0",
Expand All @@ -75,15 +77,16 @@
"braces": "^3.0.2",
"eslint": "^7.26.0",
"fast-check": "^2.14.0",
"ipfs": "^0.54.4",
"interface-datastore": "^4.0.1",
"ipfs-repo": "^9.1.6",
"jest": "^26.6.3",
"jest-config": "^26.6.3",
"jest-puppeteer": "^4.4.0",
"jest-puppeteer": "^5.0.3",
"lint-staged": "^10.5.3",
"multihashing-async": "^2.1.2",
"prettier": "^1.19.1",
"prompt": "^1.1.0",
"puppeteer": "^5.5.0",
"puppeteer": "^9.1.1",
"replace-in-file": "^5.0.2",
"rimraf": "^3.0.2",
"rollup": "^2.37.1",
Expand All @@ -99,11 +102,12 @@
"yarn": "^1.22.4"
},
"dependencies": {
"base58-universal": "^1.0.0",
"borc": "^2.1.1",
"base58-universal": "https://github.com/digitalbazaar/base58-universal#de970560f005de0f7054723c35ef6e0ff4b328b7",
"borc": "^3.0.0",
"buffer": "^6.0.3",
"cids": "^1.1.5",
"fission-bloom-filters": "1.6.0",
"ipfs-core": "^0.6.1",
"ipfs-message-port-client": "https://ipfs.runfission.com/ipfs/bafybeigx6q4aezve7my76s5vvfuiinbxtepapqvmjf2jbgrozrut6cjape/p/ipfs-message-port-client.tar.gz",
"ipfs-message-port-protocol": "https://ipfs.runfission.com/ipfs/bafybeigx6q4aezve7my76s5vvfuiinbxtepapqvmjf2jbgrozrut6cjape/p/ipfs-message-port-protocol.tar.gz",
"ipld-dag-pb": "^0.20.0",
Expand Down
3 changes: 2 additions & 1 deletion shell.nix
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ in

pkgs.mkShell {
buildInputs = [
unstable.yarn
# https://github.com/NixOS/nixpkgs/issues/53820#issuecomment-617973476
(unstable.yarn.override { nodejs = null; })
unstable.nodejs-15_x
unstable.niv
];
Expand Down
7 changes: 5 additions & 2 deletions src/@types/index.d.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
declare module 'base58-universal/main.js'
declare module 'borc'
declare module 'base58-universal' {
export function encode(input: Uint8Array, maxline?: number): string
export function decode(input: string): Uint8Array
}
declare module 'ipld-dag-pb'
declare module 'borc'
3 changes: 1 addition & 2 deletions src/did/transformers.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import * as base58 from 'base58-universal/main.js'

import * as base58 from 'base58-universal'
import * as utils from 'keystore-idb/utils'

import { BASE58_DID_PREFIX, magicBytes, parseMagicBytes } from './util'
Expand Down
15 changes: 9 additions & 6 deletions src/fs/filesystem.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ type MutationOptions = {
export class FileSystem {

root: RootTree
localOnly: boolean
readonly localOnly: boolean

appPath: AppPath | undefined
proofs: { [_: string]: string }
Expand Down Expand Up @@ -115,11 +115,13 @@ export class FileSystem {
this.publishHooks.push(logCid)
this.publishHooks.push(updateDataRootWhenOnline)

// Publish when coming back online
globalThis.addEventListener('online', this._whenOnline)

// Show an alert when leaving the page while updating the data root
globalThis.addEventListener('beforeunload', this._beforeLeaving)
if (!this.localOnly) {
// Publish when coming back online
globalThis.addEventListener('online', this._whenOnline)

// Show an alert when leaving the page while updating the data root
globalThis.addEventListener('beforeunload', this._beforeLeaving)
}
}


Expand Down Expand Up @@ -170,6 +172,7 @@ export class FileSystem {
* The only function of this is to stop listing to online/offline events.
*/
deactivate(): void {
if (this.localOnly) return
const globe = (globalThis as any)
globe.filesystems = globe.filesystems.filter((a: FileSystem) => a !== this)
globe.removeEventListener('online', this._whenOnline)
Expand Down
33 changes: 18 additions & 15 deletions src/fs/protocol/private/mmpt.test.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import crypto from "crypto"
import Ipfs, { IPFS } from "ipfs"
import { IPFS } from "ipfs-core"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a size thing? (full IPFS includes more code than we use, etc?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, exactly. It's just much less dependencies.
I think the difference is that ipfs also contains the CLI and an http-server, which ipfs-core does not.


import MMPT from "./mmpt"
import * as ipfsConfig from "../../../ipfs/config"
import { createInMemoryIPFS } from "../../../../tests/helpers/in-memory-ipfs"

function sha256Str(str: string): string {
return crypto.createHash('sha256').update(str).digest('hex')
Expand All @@ -12,7 +13,20 @@ function encode(str: string): Uint8Array {
return (new TextEncoder()).encode(str)
}

let ipfs: IPFS;

let ipfs: IPFS | null = null

beforeAll(async done => {
ipfs = await createInMemoryIPFS()
ipfsConfig.set(ipfs)
done()
})

afterAll(async done => {
if (ipfs == null) return
await ipfs.stop()
done()
})

/*
Generates lots of entries for insertion into the MMPT.
Expand All @@ -22,8 +36,8 @@ The MMPT is a glorified key-value store.
This returns an array of key-values sorted by the key,
so that key collisions are more likely to be tested.
*/
async function generateExampleEntries(amount: number): Promise<{ name: string, cid: string }[]> {
let entries: { name: string, cid: string }[] = []
async function generateExampleEntries(amount: number): Promise<{ name: string; cid: string }[]> {
const entries: { name: string; cid: string }[] = []

for (const i of Array(amount).keys()) {
const hash = sha256Str(`${i}`)
Expand All @@ -39,17 +53,6 @@ async function generateExampleEntries(amount: number): Promise<{ name: string, c



beforeAll(async done => {
ipfs = await Ipfs.create({ offline: true, silent: true })
ipfsConfig.set(ipfs)
done()
})

afterAll(async done => {
await ipfs.stop()
done()
})

describe("the mmpt", () => {
it("can handle concurrent adds", async () => {
const mmpt = MMPT.create()
Expand Down
9 changes: 5 additions & 4 deletions src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ export type Continuation = {
permissions: Maybe<Permissions>

authenticated: true
newUser: false,
newUser: false
throughLobby: false
username: string

Expand Down Expand Up @@ -172,7 +172,7 @@ export async function initialise(
)

} else if (cancellation) {
const c = (_ => {
const c = (() => {
switch (cancellation) {
case "DENIED": return "User denied authorisation"
default: return "Unknown reason"
Expand Down Expand Up @@ -256,6 +256,7 @@ export * as ipfs from './ipfs'
export * as keystore from './keystore'
export * as machinery from './common'
export * as crypto from './crypto'
export * as cbor from 'borc'



Expand Down Expand Up @@ -345,7 +346,7 @@ async function importClassifiedInfo(
const secretsStr = await crypto.aes.decryptGCM(info.secrets, rawSessionKey, info.iv)
const secrets = JSON.parse(secretsStr)

const fsSecrets: Record<string, { key: string, bareNameFilter: string }> = secrets.fs
const fsSecrets: Record<string, { key: string; bareNameFilter: string }> = secrets.fs
const ucans = secrets.ucans

// Import read keys and bare name filters
Expand All @@ -365,7 +366,7 @@ async function importClassifiedInfo(
}

async function getClassifiedViaPostMessage(): Promise<string> {
const iframe: HTMLIFrameElement = await new Promise((resolve, reject) => {
const iframe: HTMLIFrameElement = await new Promise(resolve => {
const iframe = document.createElement("iframe")
iframe.id = "webnative-secret-exchange"
iframe.style.width = "0"
Expand Down
2 changes: 1 addition & 1 deletion src/ipfs/types.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
export type IPFS = {
add(data: FileContent, options?: unknown): UnixFSFile
add(data: FileContent, options?: unknown): Promise<UnixFSFile>
cat(cid: CID): AsyncIterable<FileContentRaw>
ls(cid: CID): AsyncIterable<UnixFSFile>
dns(domain: string): Promise<CID>
Expand Down
Loading