Verdaccio - A lightweight private npm proxy registry

[olizilla]

Verdaccio is a simple, zero-config-required local private npm registry. No need for an entire database just to get started! Verdaccio comes out of the box with its own tiny database, and the ability to proxy other registries (eg. npmjs.org), caching the downloaded modules along the way. For those looking to extend their storage capabilities, Verdaccio supports various community-made plugins to hook into services such as Amazon's s3, Google Cloud Storage or create your own plugin.

https://github.com/verdaccio/verdaccio

It'd be interesting to explore adding IPFS so that verdaccio servers could could share and co-host there caches via IPFS. If a bunch of verdaccio instances are in a swarm together then they could pool their individual caches (being careful not publish private modules). Combined with the npm-on-ipfs client, it could support installing and verifying packages via ipfs or ipns specifiers as per zkat/pacote#173

see also:

• Idea: Offline-capable Package Manager App #37
• npm-on-ipfs: the desktop app! A long lived client to encourage co-hosting ipfs-shipyard/npm-on-ipfs#94 (comment)

[olizilla]

by default, verdaccio caches the tarball and the (i assume) packument version of the package.json in a flat file in ~/.config/verdaccio/storage/ e.g.

$ ls -la ~/.config/verdaccio/storage/accepts/
total 176
drwxr-xr-x     4 oli  staff    128 Apr 10 11:39 .
drwxr-xr-x  1584 oli  staff  50688 Apr 10 11:39 ..
-rw-r--r--     1 oli  staff   5409 Apr 10 11:02 accepts-1.3.5.tgz
-rw-r--r--     1 oli  staff  80913 Apr 10 11:39 package.json

to try it out locally

$ npm i -g verdaccio@next
$ verdaccio
 warn --- config file  - /Users/oli/.config/verdaccio/config.yaml
 warn --- Plugin successfully loaded: verdaccio-htpasswd
 warn --- Plugin successfully loaded: verdaccio-audit
 warn --- http address - http://localhost:4873/ - verdaccio/4.0.0-alpha.7

then in another shell, configure npm to use your new local registry proxy, and

$ npm set registry http://localhost:4873/

# npm i wont hit the network unless you clear your cache...
# see: https://twitter.com/verdaccio_npm/status/1115926213296558080
$ npm cache clear --force

$ cd <to some project with a package.json>

$ npm i

now you have all the tarballs for that project in ~/.config/verdaccio/storage/

[olizilla]

docs on writing a storage plugin. https://verdaccio.org/docs/en/dev-plugins#storage-plugin

[olizilla]

UX hurdles

• managing the package-lock.json for an open source project becomes a chore. It's already tedious, but when some of you are using a local registry, and others are using a shared team registry, and many more using the central registry, the resolved field in the package-lock will continually thrash as each dev submits their subjective view of the true source of the package. I think npm-on-ipfs rewrites the resolved field back to the central registry, which is nice. This is a significant blocker to it's usefulness for an open source project.
• publishing modules, by default the go to your local registry, rather than the public one, and you have to remember to provide the central registry as an arg if you want to publish to the world. minor, but adds friction where there was none before.
• gotta clear your local cache to get npm to hit the network. minor, but undocumented.

[achingbrain]

you have to remember to provide the central registry as an arg if you want to publish to the world.

You can add a registry field to the publishConfig field in your package.json which should let you say where you want a module to be published. Any good? Or do you want to publish in multiple places?

[achingbrain]

gotta clear your local cache to get npm to hit the network

I think this is a feature. 😉

[achingbrain]

I think npm-on-ipfs rewrites the resolved field back to the central registry,

This is true, though it rewrites to the registry.js.ipfs.io mirror. This is because npm-on-ipfs spins it's proxy up on a random port so your package-lock.json would be full of http://localhost:53832 etc otherwise.

[andrew]

As mentioned on the package managers weekly call yesterday, there's a few different ways IPFS support could be added:

Without directly involving npm-on-ipfs:

• Verdaccio could ipfs add each tarball it downloads/caches from npm, providing extra hosting nodes to npm-on-ipfs without needing to be directly aware of it
• Verdaccio could use ipfs get fetch metadata from registry.js.ipfs.io as an upstream registry instead of http from registry.npmjs.org and to fetch packages from any other Verdaccio instances that already have them

For more direct integration with npm-on-ipfs:

• Verdaccio could act as a private registry.js.ipfs.io alternative server for npm-on-ipfs

All three are possible to do in combination as well.