"Cannot read directory" error after upgrading beyond 0.8.42

[evanleck]

The full error is:

 > error: Cannot read directory "node_modules/.pnpm/esbuild@0.8.46/node_modules/esbuild/bin/esbuild": not a directory

I use pnpm and its npx equivalent pnpx to execute esbuild like this:

pnpx esbuild  --metafile=tmp/manifest.json --public-path=/assets --bundle --format=esm --splitting --target=es2017 --charset=utf8 '--define:process.env.NODE_ENV="development"' --outdir=public/assets {big mess of scripts}

This error occurs starting with 0.8.43 and is still happening with 0.8.46.

[adamhp]

@evanleck
Assuming you're on some sort of Unix-based filesystem, what happens if you:

ls -la node_modules/.pnpm/esbuild@0.8.46/node_modules/esbuild/bin/

In other words, verify your assumption that node_modules/.pnpm/esbuild@0.8.46/node_modules/esbuild/bin actually exists?

[evanleck]

@adamhp good call, it exists and there's an executable in that directory. The error is complaining that bin/esbuild is not a directory, however, which is true but I'm unsure why it should be a directory.

# 0.8.42
ls -la node_modules/.pnpm/esbuild@0.8.42/node_modules/esbuild/bin
.rwxr-xr-x@ 7.3M evan 16 Feb  9:31 esbuild

# 0.8.46
ls -la node_modules/.pnpm/esbuild@0.8.46/node_modules/esbuild/bin 
.rwxr-xr-x@ 7.4M evan 16 Feb 10:12 esbuild

[adamhp]

I'd give the classic Microsoft fix a shot:

rm -rf node_modules
pnpm install

Or whatever the equivalent of a reinstall node_modules is—I'm not super familiar with pnpm.

I just did the following with no issues, so I doubt it is related to the latest version.

npm install -g pnpm
pnpx esbuild --version

Edit:
May have something to do with where the lockfile points too. Check out the contents of node_modules/.pnpm/lock.yaml and make sure that looks right.

dependencies:
  esbuild: 0.8.46
lockfileVersion: 5.2
packages:
  /esbuild/0.8.46:
    dev: false
    hasBin: true
    requiresBuild: true
    resolution:
      integrity: sha512-xck9sXNCNmjDHCCfxTCyhKTiFuEBweh+IDAhMLOJI990v1Fzii6MyIkT1LbkvjgoVgPX2SK1kpi5eZVGNrl8yg==
specifiers:
  esbuild: ^0.8.46

[evanleck]

@adamhp yup, I already tried running rm -rf node_modules ~/.pnpm-store and re-running pnpm install without luck. Not sure about the lock file, best I can tell it's identical to pnpm-lock.yaml. Could you elaborate on "where the lockfile points too" ?

[evanw]

The change in 0.8.43 that seems the most suspicious is the change to follow multiple levels of symlinks. Do you have instructions that would let me reproduce this? I tried using a similar command but it works for me. Also, what operating system are you using?

[evanleck]

@evanw I know pnpm uses hard links to its shared store so maybe something funky's happening there. I'm on Fedora 33 kernel 5.10.15-200.fc33.x86_64 with Node.js version 14.15.5, pnpm 5.17.2, and npm 7.5.4.

Let me see if I can't get you a standalone reproduction, right now I can only reproduce it in a private repository.

[evanleck]

@evanw mind if I shoot you an email with the code that's producing the error? It's not open source so I'd rather not share it publicly but it's not exactly top secret either haha

[evanw]

mind if I shoot you an email with the code that's producing the error?

Sure thing. Thanks for offering.

[evanw]

Just had time to look into this. Looks like this was actually due to the NODE_PATH change, not the symlinks change. pnpx inserts a path into NODE_PATH that is actually a file, not a directory. Now that esbuild respects NODE_PATH it hits an error when it tries to load a file as a directory. Here's the full contents of NODE_PATH when I build that code:

/Users/evan/Downloads/esbuild-816/node_modules/.pnpm/esbuild@0.8.48/node_modules/esbuild/bin/esbuild/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/.pnpm/esbuild@0.8.48/node_modules/esbuild/bin/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/.pnpm/esbuild@0.8.48/node_modules/esbuild/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/.pnpm/esbuild@0.8.48/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/.pnpm/node_modules
/Users/evan/Downloads/esbuild-816/node_modules
/Users/evan/Downloads/node_modules
/Users/evan/node_modules
/Users/node_modules
/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/esbuild/bin/esbuild/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/esbuild/bin/node_modules
/Users/evan/Downloads/esbuild-816/node_modules/esbuild/node_modules

The problematic path is /Users/evan/Downloads/esbuild-816/node_modules/esbuild/bin/esbuild/node_modules, which has a file as a parent directory. This is why this is failing. Kind of seems like a bug in pnpx to me.

A workaround for now could be to invoke esbuild directly instead of going through pnpx:

-pnpx esbuild $(ESBUILD_OPTIONS)
+./node_modules/esbuild/bin/esbuild $(ESBUILD_OPTIONS)

This bypasses the shim file in node_modules/.bin/esbuild that pnpm adds which has the problematic NODE_PATH value.

[evanleck]

@evanw thanks so much for figuring that out and working around it! I've filed an issue with pnpm as well and am referencing your comment above. pnpm/pnpm#3156

[richard-stafflink]

@evanw thanks so much for this - removing the NODE_PATH env var fixed it for me