Handling encodings correctly #900

yan12125 · 2016-05-19T14:06:36Z

Fixes #457

On Travis CI pypy 2.5.0 and pypy3 2.4.0 failed. On my machine pypy 5.1 works fine:

$ pypy virtualenv.py 中文
New pypy executable in /home/yen/tmp/virtualenv/中文/bin/pypy
Installing setuptools, pip, wheel...done.

And pypy3 gives a strange error:

$ pypy3 virtualenv.py 中文
New pypy executable in /home/yen/tmp/virtualenv/中文/bin/pypy3
Also creating executable in /home/yen/tmp/virtualenv/中文/bin/pypy
debug: OperationError:
debug:  operror-type: LookupError
debug:  operror-value: no codec search functions registered: can't find encoding
debug: OperationError:
debug:  operror-type: AttributeError
debug:  operror-value: stdout
ERROR: The executable /home/yen/tmp/virtualenv/中文/bin/pypy3 is not functioning
ERROR: It thinks sys.prefix is '/home/yen/tmp/virtualenv' (should be '/home/yen/tmp/virtualenv/中文')
ERROR: virtualenv is not compatible with this system or executable

I guess it's related to a PyPy bug. Reported to https://bitbucket.org/pypy/pypy/issues/2300.

This was migrated from #875 to reparent it to the master branch. Please see original pull request for any previous discussion.

yan12125 · 2016-05-20T18:44:01Z

Now only PyPy 3 fails. Let's wait for responses from PyPy.

An question: is it OK to just skip the non-ASCII test on PyPy3 and list it as a known issue?

yan12125 · 2016-05-21T18:30:18Z

Skipped PyPy3. PyPy3 is incomplete and full of hacky codes. I don't think virtualenv should completely support it.

Ivoz · 2016-05-28T23:34:34Z

I'm worried this will only happen to work on linuxes with the default locale set as UTF-8. E.g, not on Windows.

yan12125 · 2016-05-29T05:23:26Z

You're right. Appveyor tests failed. I'll try to install Python on Windows and have a look.

yan12125 · 2016-05-29T14:15:09Z

Windows support fixed.

Ivoz · 2016-05-29T15:39:12Z

@yan12125 could you split out the part of this dealing specifically with pypy into a dependant / independent (whichever suits) PR?

yan12125 · 2016-05-29T16:06:53Z

Did you mean changes to .travis.yml?

Ivoz · 2016-05-29T16:09:01Z

and external/pypy-subprocess.patch and tests/test_cmdline.py

yan12125 · 2016-05-29T16:46:10Z

Done at #927

Ivoz · 2016-05-29T17:21:47Z

So I'm guessing you'd want to cherry-pick 27c8056 and 23f7268 onto a new branch and force push it onto encoding-fix, or something to that effect, to clean up this PR

yan12125 · 2016-05-29T17:44:12Z

Done :)

Ivoz · 2016-05-29T19:35:48Z

Uhh, maybe I wasn't clear enough. I mean remove all changes that are effectively in #927 from this PR / branch.

dholth · 2016-06-06T14:04:23Z

external/pypy-subprocess.patch

+-             ', out).write(sys.prefix.encode("utf-8"))']):  # 1.7.2 a9454bce
+             ', out).write(sys.prefix.encode("utf-8"))'],   # 1.7.2 a9454bce
+            ['-c', 'import sys;out=sys.stdout;prefix=sys.prefix;'
+             'prefix=prefix.encode("utf-8") if sys.version_info[0]==3 else prefix;'


Consider switching this to detect python 2 else do the python 3 behavior.

yan12125 · 2016-06-07T05:46:39Z

@Ivoz @dholth Done. Sorry for the late response.

yan12125 · 2016-07-26T05:29:34Z

Is there anything left that should be fixed?

pfmoore · 2016-07-26T07:51:45Z

virtualenv.py

@@ -1472,7 +1478,7 @@ def install_files(home_dir, bin_dir, prompt, files):
        content = content.replace('__VIRTUAL_ENV__', home_dir)
        content = content.replace('__VIRTUAL_NAME__', vname)
        content = content.replace('__BIN_NAME__', os.path.basename(bin_dir))
-        writefile(os.path.join(bin_dir, name), content)
+        writefile(os.path.join(bin_dir, name), content, use_fsenc=(name == 'activate.bat'))


Should also use fsenc for deactivate.bat surely?

Thanks for that. Changed.

pfmoore · 2016-07-26T08:10:05Z

Tests are failing on pypy. This should be fixed.

Also, I've added a number of comments. The most significant issue is that you're using the wrong encoding for bat files on Windows, but there also seem to be some unneeded changes which concern me.

yan12125 · 2016-07-26T10:25:45Z

Fixes for PyPy is in #927. Originally I put all changes in this PR, and split out supporting changes to #927, as requested by @Ivoz in #900 (comment)

pypy3 is no longer supported since 49721c1

yan12125 · 2017-01-18T13:42:24Z

A minor update after 49721c1. Can anyone have a look into #946, the first step to the whole solution?

yan12125 · 2017-02-07T13:29:17Z

@pfmoore in #1014 you said:

... which as noted in that PR, is stalled on being refactored out into separate PRs.

Is there something else I have to do other than splitting out pypy codes into #946?

pfmoore · 2017-02-07T14:36:31Z

Sorry, I can't see how to create a lint to an individual comment, but up above I said:

Agreed. There seem to be 3 things going on in this PR:
...
As @Ivoz suggests, we need multiple PRs here. Otherwise, this PR will simply sit here and never get applied, as no-one is in a position to understand and approve the whole thing. I realise this is more work for you - hopefully you can see why it would be helpful.

You split out the PyPy changes (which is waiting for someone to review and comment on, but I can't help there as I know nothing about PyPy). You still need (IMO) to split out the general encoding stuff and the Windows script file bits.

I'd happily review the Windows script file bits, I don't have an issue with that. For encodings, it's going to be more complex, as we lack a good test suite, but I'm willing to do what I can (with the proviso that I'd like to see tests demonstrating problem cases, not just reports that can't be reproduced elsewhere...)

gaborbernat · 2018-12-16T15:37:18Z

@yan12125 can you rebase this and fix the CI? thanks!

gaborbernat

@yan12125 can you rebase this and fix the CI? thanks!

yan12125 · 2018-12-17T02:17:29Z

It's my honor that someone is still looking at my PR. However, as this PR fixes a Python2-only issue, and Python 2 is going to be dead soon, I won't put more efforts into it. If another developer still needs it, feel free to take my branch.

dstufft added the migrated from develop label May 19, 2016

yan12125 mentioned this pull request May 21, 2016

fix error when invoking virtualenv with multibyte directory on windows #902

Closed

yan12125 mentioned this pull request May 28, 2016

Problem with non ASCII car in path #457

Closed

yan12125 mentioned this pull request May 29, 2016

Encoding fix dependency #927

Closed

Ivoz added the encoding label May 29, 2016

dholth reviewed Jun 6, 2016
View reviewed changes

yan12125 mentioned this pull request Jun 11, 2016

Support a different code page appveyor/ci#846

Closed

yan12125 mentioned this pull request Jul 26, 2016

Use the OEM encoding for bat files #943

Closed

pfmoore reviewed Jul 26, 2016
View reviewed changes

Yen Chi Hsuan added 14 commits January 18, 2017 21:25

Use zh-TW for appveyor tests

30010cf

Use FS encoding for CLI arguments

4d05e74

Drop unnecessary patches

e2147a1

Return back to "always-utf8" protocol

0506594

Restore changes in .travis.yml

1057d46

Correct PyPy patches

458f9f6

Try to test with multiple locales

b06c7f5

Copy libpypy-c.(so|dylib)

b885b43

Fix bash syntax error in .travis.yml

a2bd125

Fix for sequences with capital letters in system locale

ae62549

Fix ellipsis insertion

db42d11

Use Windows API GetOEMCP() to get the correct file encoding

fb49b95

Make tests run under CP437

07fb1ca

Drop pypy3 hack

b24dae6

pypy3 is no longer supported since 49721c1

yan12125 mentioned this pull request Feb 7, 2017

Not compatible with a directory having emojis in its path #1014

Closed

yan12125 mentioned this pull request Feb 25, 2017

Whitespace in root path of virtualenv breaks scripts #53

Closed

gaborbernat force-pushed the master branch 6 times, most recently from 2b3f876 to 9dfcb43 Compare October 27, 2018 11:31

gaborbernat requested changes Dec 16, 2018

View reviewed changes

yan12125 closed this Dec 17, 2018

yan12125 deleted the encoding-fix branch January 24, 2021 10:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handling encodings correctly #900

Handling encodings correctly #900

yan12125 commented May 19, 2016

yan12125 commented May 20, 2016

yan12125 commented May 21, 2016

Ivoz commented May 28, 2016 •

edited

Loading

yan12125 commented May 29, 2016

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016 •

edited

Loading

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016

dholth Jun 6, 2016

yan12125 commented Jun 7, 2016

yan12125 commented Jul 26, 2016

pfmoore Jul 26, 2016

yan12125 Jul 26, 2016

pfmoore commented Jul 26, 2016

yan12125 commented Jul 26, 2016

yan12125 commented Jan 18, 2017

yan12125 commented Feb 7, 2017

pfmoore commented Feb 7, 2017

gaborbernat commented Dec 16, 2018

gaborbernat left a comment

yan12125 commented Dec 17, 2018

Handling encodings correctly #900

Handling encodings correctly #900

Conversation

yan12125 commented May 19, 2016

yan12125 commented May 20, 2016

yan12125 commented May 21, 2016

Ivoz commented May 28, 2016 • edited Loading

yan12125 commented May 29, 2016

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016 • edited Loading

yan12125 commented May 29, 2016

Ivoz commented May 29, 2016

dholth Jun 6, 2016

Choose a reason for hiding this comment

yan12125 commented Jun 7, 2016

yan12125 commented Jul 26, 2016

pfmoore Jul 26, 2016

Choose a reason for hiding this comment

yan12125 Jul 26, 2016

Choose a reason for hiding this comment

pfmoore commented Jul 26, 2016

yan12125 commented Jul 26, 2016

yan12125 commented Jan 18, 2017

yan12125 commented Feb 7, 2017

pfmoore commented Feb 7, 2017

gaborbernat commented Dec 16, 2018

gaborbernat left a comment

Choose a reason for hiding this comment

yan12125 commented Dec 17, 2018

Ivoz commented May 28, 2016 •

edited

Loading

Ivoz commented May 29, 2016 •

edited

Loading