Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete revamp! Spawn js/go daemons locally or remote (from the browser) #176

Merged
merged 85 commits into from
Jan 16, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
fb0c80e
feat: add ability to start js-ipfs nodes
dryajov Nov 11, 2017
04476b0
fix: test & cleanup
dryajov Nov 13, 2017
8a306c8
feat: clean up and cosolidate tests
dryajov Nov 13, 2017
e5103ee
feat: add a factory to start/stop nodes from node and browser
dryajov Nov 22, 2017
c75389e
feat: reworking with new interface
dryajov Nov 29, 2017
aafcae6
more tests reworking
dryajov Nov 29, 2017
8e0b9a5
fix: timeouts
dryajov Nov 29, 2017
3848812
fix: timeouts
dryajov Nov 29, 2017
a291799
feat: making tests run in browser and node
dryajov Nov 30, 2017
0423702
feat: expose both, local and remote factories
dryajov Dec 2, 2017
0ef8413
feat: reworking exports
dryajov Dec 4, 2017
9c432d2
more cleanup
dryajov Dec 5, 2017
06c8410
fix: mapValues -> mapvalues
dryajov Dec 6, 2017
c8866e2
chore: updating ci files
dryajov Dec 6, 2017
7585650
test: timeouts
dryajov Dec 6, 2017
373d5b8
feat: enable cors in hapi
dryajov Dec 6, 2017
5d0adc3
updated comment
dryajov Dec 6, 2017
aa475d8
fix: run node and browser tests in appveyor
dryajov Dec 6, 2017
c408f57
fix: query string handling
dryajov Dec 7, 2017
5e4e69a
feat: use POST for spawn
dryajov Dec 7, 2017
3b82182
feat: move shared tests to its own file
dryajov Dec 7, 2017
da2ef07
test: skip tests untill js-ipfs pr 1134 is merged
dryajov Dec 7, 2017
a10b2c5
docs: updating readme
dryajov Dec 7, 2017
e766c22
docs: typo
dryajov Dec 7, 2017
259ad3d
docs: small change to readme
dryajov Dec 7, 2017
279f71a
docs: small change to readme
dryajov Dec 7, 2017
d60f288
chore: updating js-ipfs version
dryajov Dec 7, 2017
50e2b98
wip: examples
dryajov Dec 7, 2017
067dbc7
feat: add support for local node and fix examples
dryajov Dec 7, 2017
74eba98
feat: default params and allowing cmd args to spawn
dryajov Dec 8, 2017
78f37a1
several small changes
dryajov Dec 9, 2017
43ed821
feat: rework examples to have their own package.json
dryajov Dec 11, 2017
62832e1
feat: allow passing config values as object
dryajov Dec 12, 2017
3c0845b
docs: updating readme
dryajov Dec 13, 2017
14b4f3a
docs: small changes
dryajov Dec 13, 2017
5b8e044
docs: adding diagram
dryajov Dec 13, 2017
132ce96
feat: run non disposable nodes on default addresses/ports
dryajov Dec 13, 2017
e40e482
fix: NODE.JS to Node.js
dryajov Dec 13, 2017
088eb22
feat: split defaults into options and config
dryajov Dec 13, 2017
f10693f
docs: fixing naming conventions in readme
dryajov Dec 13, 2017
38026c6
wip: reworking factory instantiation
dryajov Dec 15, 2017
2f68fd0
feat: reworking with DaemonFactory
dryajov Dec 15, 2017
d970ee5
docs: small tweaks to the readme
dryajov Dec 15, 2017
28ac033
fix: naming conventions
dryajov Dec 16, 2017
5d1c5a4
wip: reworking with latest suggestions
dryajov Dec 18, 2017
79cfe25
feat: reworking endpoints
dryajov Dec 19, 2017
0ceea38
feat: use safe-json-parse
dryajov Dec 19, 2017
81e8d61
feat: rework spawn with new interface
dryajov Dec 19, 2017
1d81a0a
docs: small fixes to readme
dryajov Dec 19, 2017
689ec23
fix: init on different path
hacdias Dec 20, 2017
79a7194
chore: cleaning up unused deps
dryajov Dec 20, 2017
6687971
chore: update deps
dryajov Dec 21, 2017
2df79ec
feat: several changes
dryajov Dec 22, 2017
5f89431
chore: remove ipfs deps from optionalDependencies
dryajov Dec 22, 2017
591f9d9
chore: add daemons as peerdeps
dryajov Dec 23, 2017
9bf6bd5
chore: gire remove peer deps
dryajov Dec 23, 2017
2724038
docs: clarifying that ipfsd-ctl no longer bundles executables
dryajov Dec 23, 2017
67e0108
docs: small clarification
dryajov Dec 23, 2017
902b828
docs: removing replaceConfig from readme
dryajov Dec 23, 2017
9cf21f1
fix: hapi should be a `dependency` not a dev dep
dryajov Dec 24, 2017
f70ab83
feat: rework IpfsDaemonController as a class
dryajov Dec 26, 2017
d415a8a
feat: add ability to start in-proc nodes
dryajov Dec 27, 2017
8886cd8
test: add custom args test
dryajov Dec 28, 2017
8630ae9
docs: adding explanation for different daemon types
dryajov Dec 28, 2017
adc8d25
docs: small clarification
dryajov Dec 28, 2017
9e543ca
docs: small clarification
dryajov Dec 28, 2017
a90c19a
Update circle.yml
daviddias Dec 29, 2017
da9e17b
bump timeout
daviddias Dec 29, 2017
53bfd15
wip: tests and readme
dryajov Dec 30, 2017
cb7c957
wip: rework DaemonFactory to accept type
dryajov Jan 1, 2018
36cf27d
test: fixing routes test
dryajov Jan 1, 2018
a7728d8
docs: updating readme
dryajov Jan 2, 2018
5b1caae
fix: use const
dryajov Jan 2, 2018
177b1e5
timeouts
dryajov Jan 2, 2018
fcd1728
fix: dont run exec under coverage
dryajov Jan 4, 2018
19201d9
fix: exlectron example packaging
dryajov Jan 5, 2018
1a78f5f
wip: windows support
dryajov Jan 5, 2018
94c966f
test: fix start/stop test on mac/linux
dryajov Jan 5, 2018
ef28e17
fix: windows tests
dryajov Jan 5, 2018
b4522b7
fix: timeouts
dryajov Jan 5, 2018
85557c9
lint
dryajov Jan 5, 2018
c60dece
feat: allow to set the executable for DaemonController
dryajov Jan 8, 2018
dbe6b37
docs: review the readme (#181)
daviddias Jan 9, 2018
aa8d72b
fix: ipfs-repo is not a `devDependency`
dryajov Jan 8, 2018
c35f6ce
fix: various fixes from @vmx review
dryajov Jan 12, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .aegir.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
'use strict'

const createServer = require('./src').createServer

const server = createServer()
module.exports = {
karma: {
files: [{
pattern: 'test/fixtures/**/*',
watched: false,
served: true,
included: false
}],
singleRun: true
},
hooks: {
browser: {
pre: server.start.bind(server),
post: server.stop.bind(server)
}
}
}
23 changes: 14 additions & 9 deletions .appveyor.yml
Original file line number Diff line number Diff line change
@@ -1,23 +1,28 @@
version: "{build}"

environment:
matrix:
- nodejs_version: "6"
- nodejs_version: "8"

# cache:
# - node_modules

platform:
- x64
matrix:
fast_finish: true

install:
- ps: Install-Product node $env:nodejs_version $env:platform
# Install Node.js
- ps: Install-Product node $env:nodejs_version

# Upgrade npm
- npm install -g npm

# Output our current versions for debugging
- node --version
- npm --version

# Install our package dependencies
- npm install

test_script:
- npm test
- npm run test

build: off

version: "{build}"
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,5 @@ node_modules

dist
docs

.idea
228 changes: 212 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,33 @@
[![Appveyor CI](https://ci.appveyor.com/api/projects/status/4p9r12ch0jtthnha?svg=true)](https://ci.appveyor.com/project/wubalubadubdub/js-ipfsd-ctl-a9ywu)
[![Dependency Status](https://david-dm.org/ipfs/js-ipfsd-ctl.svg?style=flat-square)](https://david-dm.org/ipfs/js-ipfsd-ctl) [![js-standard-style](https://img.shields.io/badge/code%20style-standard-brightgreen.svg?style=flat-square)](https://github.com/feross/standard)

> Control an ipfs node daemon using Node.js
> Control an IPFS daemon using JavaScript in Node.js or in the Browser.

```
+-----+
| H |
| T |
+-----------------------------+ | T |
| Node.js | +-----------------------+ | P | +-----------------------------+
| | | | | | | BROWSER |
| +-----------------------+ | | IPFS Daemon | | S | | |
| | Local Daemon Ctrl | | | | | E | | +----------------------+ |
| | +------- -------- R -----|---- Remote Daemon Ctrl | |
| +-----------------------+ | +-----|-----------|-----+ | V | | | | |
| | | | | E | | +----------------------+ |
| +-----------------------+ | | | | R | | |
| | IPFS API | | | | +-----+ | +----------------------+ |
| | -------------+ | | | IPFS API | |
| +-----------------------+ | +-----------------------|---- | |
| | | +----------------------+ |
+-----------------------------+ +-----------------------------+
```

## Table of Contents

- [Install](#install)
- [Usage](#usage)
- [API](#api)
- [Contribute](#contribute)
- [License](#license)

Expand All @@ -30,35 +51,210 @@ npm install --save ipfsd-ctl

IPFS daemons are already easy to start and stop, but this module is here to do it from JavaScript itself.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a diagram explaining the difference between a local spawn and a remote spawn and why both exist

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.


### Spawn an IPFS daemon from Node.js

```js
// Start a disposable node, and get access to the api
// print the node id, and kill the temporary daemon

// IPFS_PATH will point to /tmp/ipfs_***** and will be
// cleaned up when the process exits.
// print the node id, and stop the temporary daemon

var ipfsd = require('ipfsd-ctl')
const DaemonFactory = require('ipfsd-ctl')
const df = DaemonFactory.create()

ipfsd.disposableApi(function (err, ipfs) {
ipfs.id(function (err, id) {
df.spawn(function (err, ipfsd) {
if (err) { throw err }

ipfsd.api.id(function (err, id) {
if (err) { throw err }

console.log(id)
process.exit()
ipfsd.stop()
})
})
```

If you need want to use an existing ipfs installation you can set `$IPFS_EXEC=/path/to/ipfs` to ensure it uses that.
### Spawn an IPFS daemon from the Browser using the provided remote endpoint

```js
// Start a remote disposable node, and get access to the api
// print the node id, and stop the temporary daemon

const DaemonFactory = require('ipfsd-ctl')

const port = 9999
const server = DaemonFactory.createServer(port)
const df = DaemonFactory.create({ remote: true, port: port })

server.start((err) => {
if (err) { throw err }

df.spawn((err, ipfsd) => {
if (err) { throw err }

ipfsd.api.id(function (err, id) {
if (err) { throw err }

console.log(id)
ipfsd.stop(server.stop)
})
})
})
```

## Disposable vs non Disposable nodes

`ipfsd-ctl` can create two types of node controllers, `disposable` and `non-disposable`. A disposable node will be created on a temporary repo which will be optionally initialized and started (the default), as well cleaned up on process exit. A non-disposable node on the other hand, requires the user to initialize and start the node, as well as stop and cleanup after wards. Additionally, a non-disposable will allow you to pass a custom repo using the `repoPath` option, if the `repoPath` is not defined, it will use the default repo for the node type (`$HOME/.ipfs` or `$HOME/.jsipfs`). The `repoPath` parameter is ignored for disposable nodes, as there is a risk of deleting a live repo.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🌟 for documenting all the things 👍

## IPFS executables

`ipfsd-ctl` no longer installs go-ipfs nor js-ipfs dependencies, instead it expects them to be provided by the parent project. In order to be able to use both go and js daemons, please make sure that your project includes these two npm packages as dependencies.

- `ipfs` - the js-ipfs implementation
- `go-ipfs-dep` - the packaged go-ipfs implementation

## API

### Daemon Factory Class

#### `DaemonFactory` - `const df = DaemonFactory.create([options])`

`DaemonFactory.create([options])` returns an object that will expose the `df.spawn` method

- `options` - an optional object with the following properties
- `remote` bool - indicates if the factory should spawn local or remote nodes. By default, local nodes are spawned in Node.js and remote nodes are spawned in Browser environments.
- `port` number - the port number to use for the remote factory. It should match the port on which `DaemonFactory.server` was started. Defaults to 9999.
- `type` - the daemon type to create with this factory. See the section bellow for the supported types
- `exec` - path to the desired IPFS executable to spawn, otherwise `ipfsd-ctl` will try to locate the correct one based on the `type`. In the case of `proc` type, exec is required and expects an IPFS coderef.

`ipfsd-ctl` allows spawning different IPFS implementations, such as:

- **`go`** - calling `DaemonFactory.create({type: 'go'})` will spawn a `go-ipfs` daemon.
- **`js`** - calling `DaemonFactory.create({type: 'js'})` will spawn a `js-ipfs` daemon.
- **`proc`** - calling `DaemonFactory.create({type: 'proc', exec: require('ipfs') })` will spawn an `in process js-ipfs node` using the provided code reference that implements the core IPFS API. Note that, `exec` option to `df.spawn()` is required if `type: 'proc'` is used.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to pass in exec when using type: 'proc'? Couldn't it default to require('ipfs')?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would force users to bring js-ipfs at all times and we learned that makes it hard for users that want to bundle go-ipfs in. Otherwise, it would be good.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call this embedded instead? Seems more consistent with e.g. ipfs-companion. (proc is also a pretty generic name.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine either way :) Unless anyone objects, I'll change it. Was trying to find a better name either way in-proc was another option.


#### DaemonFactory endpoint for remote spawning - `const server = `DaemonFactory.createServer([options]) `

`DaemonFactory.createServer` create an instance of the bundled REST API used by the remote controller.

- exposes `start` and `stop` methods to start and stop the http server endpoint.

#### Spawn a new daemon with `df.spawn`

Spawn either a js-ipfs or go-ipfs daemon

`df.spawn([options], callback)`

`options` is an optional object the following properties:
- `init` bool (default true) - should the node be initialized
- `start` bool (default true) - should the node be started
- `repoPath` string - the repository path to use for this node, ignored if node is disposable
- `disposable` bool (default false) - a new repo is created and initialized for each invocation, as well as cleaned up automatically once the process exits
- `args` - array of cmd line arguments to be passed to ipfs daemon
- `config` - ipfs configuration options

`callback` - is a function with the signature `function (err, ipfsd)` where:
- `err` - is the error set if spawning the node is unsuccessful
- `ipfsd` - is the daemon controller instance:
- `api` - a property of `ipfsd`, an instance of [ipfs-api](https://github.com/ipfs/js-ipfs-api) attached to the newly created ipfs node

### IPFS Daemon Controller (`ipfsd`)

The IPFS daemon controller (`ipfsd`) allows you to interact with the spawned IPFS daemon.

#### `ipfsd.apiAddr` (getter)

Get the address (multiaddr) of connected IPFS API. Returns a multiaddr

#### `ipfsd.gatewayAddr` (getter)

For more details see https://ipfs.github.io/js-ipfsd-ctl/.
Get the address (multiaddr) of connected IPFS HTTP Gateway. Returns a multiaddr.

#### `ipfsd.repoPath` (getter)

Get the current repo path. Returns string.

#### `ipfsd.started` (getter)

Is the node started. Returns a boolean.

#### `init([initOpts], callback)`

Initialize a repo.

`initOpts` (optional) is an object with the following properties:
- `keysize` (default 2048) - The bit size of the identity key.
- `directory` (default IPFS_PATH if defined, or ~/.ipfs for go-ipfs and ~/.jsipfs for js-ipfs) - The location of the repo.

`callback` is a function with the signature `function (Error, ipfsd)` where `err` is an Error in case something goes wrong and `ipfsd` is the daemon controller instance.

#### `ipfsd.cleanup(callback)`

Delete the repo that was being used. If the node was marked as `disposable` this will be called automatically when the process is exited.

`callback` is a function with the signature `function(err)`.

#### `ipfsd.start(flags, callback)`

Start the daemon.

`flags` - Flags array to be passed to the `ipfs daemon` command.

`callback` is a function with the signature `function(err, ipfsApi)` that receives an instance of `ipfs-api` on success or an instance of `Error` on failure


#### `ipfsd.stop([callback])`

Stop the daemon.

`callback` is a function with the signature `function(err)` callback - function that receives an instance of `Error` on failure

#### `ipfsd.killProcess([callback])`

Kill the `ipfs daemon` process.

First a `SIGTERM` is sent, after 10.5 seconds `SIGKILL` is sent if the process hasn't exited yet.

`callback` is a function with the signature `function()` called once the process is killed

#### `ipfsd.pid()`

Get the pid of the `ipfs daemon` process. Returns the pid number

#### `ipfsd.getConfig([key], callback)`

Returns the output of an `ipfs config` command. If no `key` is passed, the whole config is returned as an object.

`key` (optional) - A specific config to retrieve.

`callback` is a function with the signature `function(err, (Object|string))` that receives an object or string on success or an `Error` instance on failure

#### `ipfsd.setConfig(key, value, callback)`

Set a config value.

`key` - the key of the config entry to change/set

`value` - the config value to change/set

`callback` is a function with the signature `function(err)` callback - function that receives an `Error` instance on failure

#### `ipfsd.version(callback)`

Get the version of ipfs

`callback` is a function with the signature `function(err, version)`

### IPFS Client (`ipfsd.api`)

An instance of [ipfs-api](https://github.com/ipfs/js-ipfs-api#api) that is used to interact with the daemon.

This instance is returned for each successfully started IPFS daemon, when either `df.spawn({start: true})` (the default) is called, or `ipfsd.start()` is invoked in the case of nodes that were spawned with `df.spawn({start: false})`.

### Packaging

`ipfsd-ctl` can be packaged in Electron applications, but the ipfs binary
has to be excluded from asar (Electron Archives),
`ipfsd-ctl` can be packaged in Electron applications, but the ipfs binary has to be excluded from asar (Electron Archives).
[read more about unpack files from asar](https://electron.atom.io/docs/tutorial/application-packaging/#adding-unpacked-files-in-asar-archive).
`ipfsd-ctl` will try to detect if used from within an `app.asar` archive
and tries to resolve ipfs from `app.asar.unpacked`. The ipfs binary is part of
the `go-ipfs-dep` module.

`ipfsd-ctl` will try to detect if used from within an `app.asar` archive and tries to resolve ipfs from `app.asar.unpacked`. The ipfs binary is part of the `go-ipfs-dep` module.

```bash
electron-packager ./ --asar.unpackDir=node_modules/go-ipfs-dep
Expand Down
4 changes: 4 additions & 0 deletions circle.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
machine:
node:
version: stable

test:
post:
- npm run coverage -- --upload --providers coveralls

dependencies:
pre:
Expand Down
18 changes: 0 additions & 18 deletions examples/disposableApi.js

This file was deleted.

15 changes: 11 additions & 4 deletions examples/electron-asar/app.js
Original file line number Diff line number Diff line change
@@ -1,8 +1,13 @@
/* eslint no-console: 0 */
'use strict'

const { app, ipcMain, BrowserWindow } = require('electron')
const ipfsd = require('ipfsd-ctl')
const electron = require('electron')
const app = electron.app
const ipcMain = electron.ipcMain
const BrowserWindow = electron.BrowserWindow

const DaemonFactory = require('ipfsd-ctl')
const df = DaemonFactory.create()

app.on('ready', () => {
const win = new BrowserWindow({
Expand All @@ -15,20 +20,22 @@ ipcMain.on('start', ({ sender }) => {
console.log('starting disposable IPFS')
sender.send('message', 'starting disposable IPFS')

ipfsd.disposableApi((err, ipfs) => {
df.spawn((err, ipfsd) => {
if (err) {
sender.send('error', err)
throw err
}

console.log('get id')
sender.send('message', 'get id')
ipfs.id(function (err, id) {
ipfsd.api.id((err, id) => {
if (err) {
sender.send('error', err)
throw err
}
console.log('got id', id)
sender.send('id', JSON.stringify(id))
ipfsd.stop()
})
})
})
Loading