Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate non-ASCII build file path on POSIX #14336

Closed
TimothyGu opened this issue Jul 18, 2017 · 12 comments
Closed

Investigate non-ASCII build file path on POSIX #14336

TimothyGu opened this issue Jul 18, 2017 · 12 comments
Labels
build Issues and PRs related to build files or the CI. inspector Issues and PRs related to the V8 inspector protocol python PRs and issues that require attention from people who are familiar with Python.

Comments

@TimothyGu
Copy link
Member

  • Version: master
  • Platform: macOS
  • Subsystem: build

At Code & Learn Shanghai we encountered some issues with non-ASCII (specifically, Chinese) file paths when building. Don't have the Python stack trace, but could someone look into that?

@TimothyGu TimothyGu added build Issues and PRs related to build files or the CI. macos Issues and PRs related to the macOS platform / OSX. labels Jul 18, 2017
@gibfahn
Copy link
Member

gibfahn commented Jul 18, 2017

Stack trace:

➜  弄得 git:(master) ❯ make                                                                                                                                                                  ~/wrk/com/弄得
/Library/Developer/CommandLineTools/usr/bin/make -C out BUILDTYPE=Release V=1
  LD_LIBRARY_PATH=/Users/gib/wrk/com/弄得/out/Release/lib.host:/Users/gib/wrk/com/弄得/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../.; mkdir -p /Users/gib/wrk/com/弄得/out/Release/obj/gen; python tools/js2c.py "/Users/gib/wrk/com/弄得/out/Release/obj/gen/node_javascript.cc" lib/internal/bootstrap_node.js lib/async_hooks.js lib/assert.js lib/buffer.js lib/child_process.js lib/console.js lib/constants.js lib/crypto.js lib/cluster.js lib/dgram.js lib/dns.js lib/domain.js lib/events.js lib/fs.js lib/http.js lib/_http_agent.js lib/_http_client.js lib/_http_common.js lib/_http_incoming.js lib/_http_outgoing.js lib/_http_server.js lib/https.js lib/inspector.js lib/module.js lib/net.js lib/os.js lib/path.js lib/process.js lib/punycode.js lib/querystring.js lib/readline.js lib/repl.js lib/stream.js lib/_stream_readable.js lib/_stream_writable.js lib/_stream_duplex.js lib/_stream_transform.js lib/_stream_passthrough.js lib/_stream_wrap.js lib/string_decoder.js lib/sys.js lib/timers.js lib/tls.js lib/_tls_common.js lib/_tls_legacy.js lib/_tls_wrap.js lib/tty.js lib/url.js lib/util.js lib/v8.js lib/vm.js lib/zlib.js lib/internal/buffer.js lib/internal/child_process.js lib/internal/cluster/child.js lib/internal/cluster/master.js lib/internal/cluster/round_robin_handle.js lib/internal/cluster/shared_handle.js lib/internal/cluster/utils.js lib/internal/cluster/worker.js lib/internal/errors.js lib/internal/freelist.js lib/internal/fs.js lib/internal/http.js lib/internal/linkedlist.js lib/internal/net.js lib/internal/module.js lib/internal/process/next_tick.js lib/internal/process/promises.js lib/internal/process/stdio.js lib/internal/process/warning.js lib/internal/process.js lib/internal/querystring.js lib/internal/process/write-coverage.js lib/internal/readline.js lib/internal/repl.js lib/internal/socket_list.js lib/internal/test/unicode.js lib/internal/url.js lib/internal/util.js lib/internal/v8_prof_polyfill.js lib/internal/v8_prof_processor.js lib/internal/streams/lazy_transform.js lib/internal/streams/BufferList.js lib/internal/streams/legacy.js lib/internal/streams/destroy.js deps/v8/tools/splaytree.js deps/v8/tools/codemap.js deps/v8/tools/consarray.js deps/v8/tools/csvparser.js deps/v8/tools/profile.js deps/v8/tools/profile_view.js deps/v8/tools/logreader.js deps/v8/tools/tickprocessor.js deps/v8/tools/SourceMap.js deps/v8/tools/tickprocessor-driver.js deps/node-inspect/lib/_inspect.js deps/node-inspect/lib/internal/inspect_client.js deps/node-inspect/lib/internal/inspect_repl.js ./config.gypi src/nolttng_macros.py src/perfctr_macros.py
  touch /Users/gib/wrk/com/弄得/out/Release/obj.host/node_js2c.stamp
  LD_LIBRARY_PATH=/Users/gib/wrk/com/弄得/out/Release/lib.host:/Users/gib/wrk/com/弄得/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/src/inspector; mkdir -p /Users/gib/wrk/com/弄得/out/Release/obj/gen/src; python ../../third_party/inspector_protocol/CheckProtocolCompatibility.py --stamp "/Users/gib/wrk/com/弄得/out/Release/obj/gen/src/js_protocol.stamp" js_protocol.json
  touch /Users/gib/wrk/com/弄得/out/Release/obj.target/deps/v8/src/inspector/protocol_compatibility.stamp
  touch cc205296965aac8499af8078773aa6a0978c1a60.intermediate
  LD_LIBRARY_PATH=/Users/gib/wrk/com/弄得/out/Release/lib.host:/Users/gib/wrk/com/弄得/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../deps/v8/src/inspector; mkdir -p /Users/gib/wrk/com/弄得/out/Release/obj/gen/src/inspector/protocol /Users/gib/wrk/com/弄得/out/Release/obj/gen/include/inspector; python ../../third_party/inspector_protocol/CodeGenerator.py --jinja_dir ../../third_party --output_base "/Users/gib/wrk/com/弄得/out/Release/obj/gen/src/inspector" --config inspector_protocol_config.json
Failed to parse config file: 'ascii' codec can't decode byte 0xe5 in position 19: ordinal not in range(128)

make[1]: *** [cc205296965aac8499af8078773aa6a0978c1a60.intermediate] Error 1
rm cc205296965aac8499af8078773aa6a0978c1a60.intermediate
make: *** [node] Error 2

@bnoordhuis bnoordhuis added the inspector Issues and PRs related to the V8 inspector protocol label Jul 18, 2017
@bnoordhuis
Copy link
Member

cc @nodejs/v8-inspector

@TimothyGu
Copy link
Member Author

I can reproduce this on Linux as well. This is because Python 2's sys.argv is an array of values of type str, which is basically an array of 8-bit values, instead of unicode. The file system functions see a str value and interpret it as ASCII, which fails when in Unix sys.argv is in fact encoded with UTF-8.

@refack
Copy link
Contributor

refack commented Jul 24, 2017

nodejs/node-gyp#1203 might solve this
(trying to upstream https://chromium-review.googlesource.com/c/509632/)

@TimothyGu
Copy link
Member Author

@refack It's unrelated.

@refack
Copy link
Contributor

refack commented Jul 24, 2017

@refack It's unrelated.

Hmmmm it's make not configure... so maybe #12786 (comment)

@Trott
Copy link
Member

Trott commented Jul 24, 2017

Editing title to say POSIX since this is on Linux as well as macOS.

@Trott Trott changed the title Investigate non-ASCII build file path under macOS Investigate non-ASCII build file path on POSIX Jul 24, 2017
@Trott Trott added python PRs and issues that require attention from people who are familiar with Python. and removed macos Issues and PRs related to the macOS platform / OSX. labels Jul 24, 2017
@SgtPooki
Copy link
Contributor

SgtPooki commented Oct 6, 2017

I was able to reproduce this and am looking into it.

sharkfisher added a commit to sharkfisher/node that referenced this issue Oct 6, 2017
sharkfisher added a commit to sharkfisher/node that referenced this issue Oct 6, 2017
sharkfisher added a commit to sharkfisher/node that referenced this issue Oct 6, 2017
mmarchini pushed a commit to mmarchini/node that referenced this issue Dec 12, 2017
The build breaks if there's a non-ASCII character on the path to the building
directory.

Ref: nodejs#16278
Ref: nodejs#14336
joyeecheung pushed a commit that referenced this issue Jan 18, 2018
The build breaks if there's a non-ASCII character on
the path to the building directory.

PR-URL: #16735
Refs: #16278
Refs: #14336
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
evanlucas pushed a commit that referenced this issue Jan 30, 2018
The build breaks if there's a non-ASCII character on
the path to the building directory.

PR-URL: #16735
Refs: #16278
Refs: #14336
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
evanlucas pushed a commit that referenced this issue Jan 30, 2018
The build breaks if there's a non-ASCII character on
the path to the building directory.

PR-URL: #16735
Refs: #16278
Refs: #14336
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
MylesBorins pushed a commit that referenced this issue Feb 27, 2018
The build breaks if there's a non-ASCII character on
the path to the building directory.

PR-URL: #16735
Refs: #16278
Refs: #14336
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
MylesBorins pushed a commit that referenced this issue Feb 27, 2018
The build breaks if there's a non-ASCII character on
the path to the building directory.

PR-URL: #16735
Refs: #16278
Refs: #14336
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
MayaLekova pushed a commit to MayaLekova/node that referenced this issue May 8, 2018
The build breaks if there's a non-ASCII character on
the path to the building directory.

PR-URL: nodejs#16735
Refs: nodejs#16278
Refs: nodejs#14336
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
@lundibundi
Copy link
Member

I will close this as it seems to be resolved (this seems to be the relevant comment #16735 (comment)). I've also built it locally with this path node/弄得/node just fine (Arch Linux).

@refack
Copy link
Contributor

refack commented Aug 25, 2018

@lundibundi were you able to run make test in that path?

@lundibundi
Copy link
Member

lundibundi commented Aug 26, 2018

@refack I've done a full ./configure && make -j4 test command successfully. Though I'm on Arch and it has python3 by default. I'll now try to do that in python2 env as I forgot about that.

@lundibundi
Copy link
Member

Upd: I've successfully built it via ./configure && make -j4 test in dev/node/弄得 folder using python2 env.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Issues and PRs related to build files or the CI. inspector Issues and PRs related to the V8 inspector protocol python PRs and issues that require attention from people who are familiar with Python.
Projects
None yet
Development

No branches or pull requests

7 participants