message_about_scripts_not_on_PATH does not expanduser()

[jflorian]

Environment

• pip version: 18.1
• Python version: 3.7
• OS: Fedora 29

from python3-pip-18.1-1.fc29.noarch

Description
I was trying to install PyGitUp with pip3 install --user git-up and ran into an issue where I was warned:

  The script git-up is installed in '/home/jflorian/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 

I callled BS because I knew this was on my PATH. But then I noticed my PATH actually had ~/.local/bin and not /home/jflorian/.local/bin as claimed. I looked at wheel.py where the error originates and sure enough, there was no usage of os.path.expanduser(). I changed my PATH to have this already expanded and the error message disappeared.

Expected behavior
Most tools seem to know how to expand the tilde in my PATH, I would expect Pip to do so as well.

How to Reproduce

1. export PATH with a local bin directory using a tilde, e.g., ~/.local/bin
2. Install any package with scripts using the --user option.
3. Observe the technically correct warning, but which is misleading w/o serious thought because it must be taken very literally and not as most users are likely to understand.
4. Repeat with PATH having the tilde already expanded, e.g., /home/jflorian/.local/bin and observe the behavior I'd expect in step 3.

[chrahunt]

The purpose of this message is to help users identify why a particular script scriptname may not be available for execution as scriptname in their shell after installation of the containing package.

I tested several shells and it appears that only bash expands a literal ~ in PATH by default (or at least with the default options on my machine).

$ cat test.sh
#!/bin/sh

echo "#!/bin/sh
echo found
" > ~/example
chmod +x ~/example

for shell in /bin/bash /bin/zsh /usr/bin/fish /bin/csh /bin/dash /bin/ksh; do
    echo "$shell plain ~"
    HOME=~ PATH='~' $shell -c example
    echo "$shell expanded ~"
    HOME=~ PATH=~ $shell -c example
done
$ ./test.sh
/bin/bash plain ~
found
/bin/bash expanded ~
found
/bin/zsh plain ~
zsh:1: command not found: example
/bin/zsh expanded ~
found
/usr/bin/fish plain ~
fish: Unknown command 'example'
fish:
example
^
/usr/bin/fish expanded ~
found
/bin/csh plain ~
example: Command not found.
/bin/csh expanded ~
found
/bin/dash plain ~
/bin/dash: 1: example: not found
/bin/dash expanded ~
found
/bin/ksh plain ~
/bin/ksh: example: not found
/bin/ksh expanded ~
found

Versions:

ii  bash                                      4.4.18-2ubuntu1.1         amd64                     GNU Bourne Again SHell
ii  csh                                       20110502-3                amd64                     Shell with C-like syntax
ii  dash                                      0.5.8-2.10                amd64                     POSIX-compliant shell
ii  fish                                      2.7.1-3                   amd64                     friendly interactive shell
ii  ksh                                       93u+20120801-3.1ubuntu1   amd64                     Real, AT&T version of the Korn shell
ii  zsh                                       5.4.2-3ubuntu3.1          amd64                     shell with lots of features

As a result, if expanduser was used here then bash users would not tell the difference, but users of any other shell would not get the benefit of the warning.

[chrahunt]

Do we want to implement anything related to this or should we leave it as-is with the justification given above?

[pfmoore]

I think the current behaviour is fine. We don't want to go down the rabbit hole of trying to second-guess subtleties in the user's shell.

Note that the OP was able to interpret the message and fix the issue (and actually, could probably have left their PATH unchanged and simply ignored the warning) so there's not actually a failure here.

[jflorian]

OP here. I guess I shouldn't be surprised this is tied to a bashism. If you look at the issue linked in the description you can see that I did not have the option of just ignoring this message -- there was a subtle failure in that I couldn't use git up, but git-up was fine. I think leaving things here "as is" is suboptimal simply because many users aren't going to be able to figure this out like I did. Even if they can, that can take time that feels like an utter waste once the problem is understood.

What would be better? I'm not sure exactly. Could the test for existence actually be delegated to the user's shell? I mean the trouble here is that some Python code is trying to replicate what the shell is doing and if we're to avoid implementing various shell subtleties then why not go straight to the shell for the answer wrapping around something like test -x FOO after FOO has been installed. At the very minimum, an enhanced message that hints to this type of problem would be better than no action at all.

[pfmoore]

there was a subtle failure in that I couldn't use git up, but git-up was fine.

How is that related to the pip install? That sounds to me like the problem that pip was warning you about - that something couldn't correctly find the git-up command, because it didn't expand ~ in PATH. The comments in the linked issue suggest that git behaves the same as pip (and the other shells that @chrahunt checked) in not expanding ~, so it's bash that's the odd one out here.

I'm confused as to what you're asking for here. The message about "not on PATH" is a warning, so if it's wrong or misleading, you can ignore it (we could of course improve the wording if there's something we can agree on is better, but that still wouldn't change the behaviour). The fact that git up didn't work isn't a pip issue, so it's not something we can fix.

[chrahunt]

What if we warn about this case if we see a path in PATH that starts with ~? For example:

The script git-up is installed in '/home/jflorian/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
NOTE: The current PATH contains path(s) starting with `~`, which may not be expanded by all applications.

[chrahunt]

Assuming no other comments - the changes here would be:

1. Update pip._internal.wheel.message_about_scripts_not_on_PATH here to check and trace a note as mentioned above if any path in PATH starts with a ~.
2. Add a test to tests.unit.test_wheel.TestMessageAboutScriptsNotOnPATH here to cover this case.

[jflorian]

@chrahunt I like what you've proposed and am quite convinced that my time-to-grok the issue would have been significantly shortened had this been in place.

[mdebi]

Hi. I would like to work on this PR.

[chrahunt]

Hi @mdebi! My comment here should still be accurate. Please let us know if you have any questions while working on this.

[mdebi]

Please review and let me know your comments.

I was not sure what to pick as the extension for the news fragment and chose feature since this issue was marked as enhancement. Please let me know if the name for news file should be different. Thank you.

[chrahunt]

Closed in #7345. Thanks @mdebi!