pnpmfile declarative format

[vjpr]

Opinions wanted.

When supporting CRNA, I created a package called pnpmfile-file-read-package which supports a declarative format for the readPackage hook.

I think the most common case for readPackage can be done with a simple json object.

I was wondering if it would be a good idea to support this format in a readPackageJson hook.

Why

Otherwise, you have to rely on creating a project/pnpmfile directory with package.json and using recursive install which is a bit cumbersome. Or you have to copy in some code to parse the json format.

Supporting packages would just be a simple matter of copying a json file for each package that needs patching. Or you could distribute a pnpmfile-preset-crna.json for CRNA. Keeps things simple. Although the package approach allows for easier updates.

Format

Here is a format I use:

<package>@<version-range>: {
  dependencies: {}
  devDependencies: {}
  peerDependencies: {}
}

Example

export default {
  // `metro` pnpm support.
  'react-native@0.52.0': {
    dependencies: {
      metro: 'npm:metro-pnpm@0.24.7-vjpr.1',
      //'metro-resolver': '~/dev-live/metro/.dev/metro-resolver-0.28.0.tgz',
      uuid: '*', // Libraries/Blob/Blob.js
    },
  },

  'react-native-scripts': {
    dependencies: {
      expo: '*',
    },
  },

  'metro-core': {
    dependencies: {
      wordwrap: '*',
    },
  },

  // Cause: expo/src/Notifications.js
  expo: {
    dependencies: {
      fbjs: '^0.8.16',
      lodash: '*',
      'json-schema-traverse': '*',
    },
  },

  // react-native-gesture-handler@1.0.0-alpha.39
  'react-native-gesture-handler': {
    dependencies: {
      fbjs: '^0.8.16',
    },
  },

  // @expo/vector-icons@6.3.1
  '@expo/vector-icons': {
    dependencies: {
      'prop-types': '*',
    },
  },

  'metro-pnpm': {
    dependencies: {
      'babel-plugin-transform-flow-strip-types': '6',
      'babel-plugin-transform-object-rest-spread': '6',
      'babel-plugin-transform-class-properties': '6',
      'babel-plugin-transform-async-to-generator': '6',
      'babel-plugin-syntax-trailing-function-commas': '6',
      'babel-template': '6',
      resolve: '*',
      xtend: '*',
      errno: '*',
    },
  },

  /*
    pnpm ls babel-preset-expo

    ├─┬ expo@25.0.0
    │ └── babel-preset-expo@4.0.0
    └─┬ react-native-scripts@1.11.1
      └─┬ expo@25.0.0
        └── babel-preset-expo@4.0.0
    */
  '<root>': {
    dependencies: {
      'babel-preset-expo': '^4',
      'babel-plugin-transform-react-jsx-source': '6',
    },
  },
}

Questions

Should we use:

<package>@<version-range>: {
  dependencies: {}
  devDependencies: {}
  peerDependencies: {}
}

or

<package>: {
  <version-range>: {
    dependencies: {}
    devDependencies: {}
    peerDependencies: {}
  }
}

[zkochan]

I guess I am OK with it but then let's make it a non-executable file. The reason for that is because some toolings (like renovateapp) run pnpm with --ignore-scripts, --ignore-pnpmfile flags and so the hooks in pnpmfile.js are ignored.

[vjpr]

let's make it a non-executable file

You mean like .json file?

If so, I would prefer hjson to support comments. YAML is an option too I guess.

and so the hooks in pnpmfile.js are ignored

I don't understand. This is to get around renovate not running pnpmfile?

[zkochan]

Not just renovate. @etamponi also wanted to avoid an uncontrolled executable to be run by pnpm. That's why he implemented --ignore-pnpmfile.

I think YAML would be good because that's the format we mostly use. And I think it would be enough just to specify the overrides like this:

react-native@0.52.0:
  metro: npm:metro-pnpm@0.24.7-vjpr.1
  # 'metro-resolver': '~/dev-live/metro/.dev/metro-resolver-0.28.0.tgz'
  uuid: * # Libraries/Blob/Blob.js
react-native-scripts:
  expo: *

You can override optional dependencies but I don't think anyone would like to add new optional dependencies. Why would that be needed?

Possible name: pnpm-subdependencies.yaml

[vjpr]

I prefer having json option. Its easier to reuse if you want to add it to pnpmfile.js at some point, say if you want to publish as a package. Otherwise you need to pull in yaml parser lib.

[vjpr]

BTW:

The version should be a semver version range: react-native@0.52.0 or react-native@^0.52.0

And we need to support project root. E.g. <root>.

[vjpr]

Possible name: pnpm-subdependencies.yaml

This name doesn't fit because you can override root dependencies.

Yarn uses a resolutions field in package.json.

We should call it pnpm-resolutions.yaml|json|js.

---

Also, I want to be able to use packages in the config, so I think having a js option would be good, or some kind of mechanism to declaratively specify packages to be used.

[vjpr]

Yarn supports Selective dependency resolutions which allow:

  "resolutions": {
    "c/**/left-pad": "1.1.2"
  }

Do we need to handle this?

[aleclarson]

I like the idea of a pnpm namespace in package.json, since that's the standard AFAICT:

{
  "name": "foo",
  "version": "0.1.0",
  "pnpm": {
    "resolutions": {}
  }
}

It's a shame there's no way for pnpm to support Yarn resolutions. 😢

[vjpr]

Just want to add an issue with .js pnpmfile. When using Docker, if you want to cache the node_modules layer which is a common practice, you copy package.json and shrinkwrap.yaml and then run pnpm i. But when using pnpmfile.js, you need to make sure you copy that too, and any files it depends on. If pnpmfiles are complex, this approach becomes complex.

[Bessonov]

-1 for static files. Like for every tool, there is a set of files, which are needed to be commited or added to Dockerfile etc.

But from my perspective, pnpmfile.js is still not perfect. Is't not really reusable. I think it would be better to have pnpmfile.js for (dynamic) configuration, a some plugin system and install the plugins through pnpm (like @pnpm/licenses-plugin and make sure that they are available in pnpmfile.js).

But when using pnpmfile.js, you need to make sure you copy that too, and any files it depends on. If pnpmfiles are complex, this approach becomes complex.

Common, if you forgot to commit something, you get probably something wrong in production. Just don't do that :D

[zkochan]

Do we still want this or is it enough to support Yarn's resolutions field?

Related issue: #1221

[zkochan]

I will probably work on this soon. I plan to use Yarn's packageExtensions format.

[zkochan]

Ready for review: #3553

[zkochan]

We will probably use a shared compatibility database with Yarn. The question is how to ship this compatibility database.

Yarn bundles it with the CLI. Shipping it with the CLI probably means better dev experience, as Yarn fixes issues out-of-the-box. But it also means that new fixes are shipped rarely.

I'd suggest us to use a different approach and ask our users to download the packageExtensions json file and commit it to their repository. This way users can get fixes fast but it will also mean that sometimes new users will get a bad impression about pnpm not working.

Another solution could be to load the compatibility json before running installation.

[vjpr]

I like the idea of a separate file for the db outside of the cli. I’d also like to have a pnpmfile hook where I can return the declarative json. If there are bugs in the DB it would provide an opportunity to ignore/override them while still pulling down latest updates. There should probably be a way to check for updates to the db on every install (or once every 5 mins). Checking into source control is best though…you wouldn’t want something changing without your knowledge between installs - but I guess the lock file helps with that. A cli command to show active overrides would be useful too - show which rules have been applied. This could be cached in the lock file maybe.

…

On Wed 23. Jun 2021 at 01:57, Zoltan Kochan ***@***.***> wrote: We will probably use a shared compatibility database with Yarn. The question is how to ship this compatibility database. Yarn bundles it with the CLI. Shipping it with the CLI probably means better dev experience, as Yarn fixes issues out-of-the-box. But it also means that new fixes are shipped rarely. I'd suggest us to use a different approach and ask our users to download the packageExtensions json file and commit it to their repository. This way users can get fixes fast but it will also mean that sometimes new users will get a bad impression about pnpm not working. Another solution could be to load the compatibility json before running installation. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3172 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACEWRLIMYM3LHCF36URVSLTUEPPTANCNFSM47BQUBLQ> .

[zkochan]

We use now this DB: https://github.com/yarnpkg/berry/blob/master/packages/yarnpkg-extensions/sources/index.ts