tox fails when running in a path containing a hash

[The-Compiler]

On Archlinux with 2.9.1, when trying to run it in a folder called test#foo, I get:

$ tox -e flake8
flake8 create: /home/florian/proj/qutebrowser/test#foo/.tox/flake8
flake8 installdeps: -r/home/florian/proj/qutebrowser/test#foo/requirements.txt, -r/home/florian/proj/qutebrowser/test#foo/misc/requirements/requirements-flake8.txt
flake8 installed: attrs==17.3.0,colorama==0.3.9,cssutils==1.0.2,flake8==3.5.0,flake8-bugbear==17.12.0,flake8-builtins==1.0.post0,flake8-comprehensions==1.4.1,flake8-copyright==0.2.0,flake8-debugger==3.0.0,flake8-deprecated==1.3,flake8-docstrings==1.1.0,flake8-future-import==0.4.3,flake8-mock==0.3,flake8-per-file-ignores==0.4,flake8-polyfill==1.0.1,flake8-string-format==0.2.3,flake8-tidy-imports==1.1.0,flake8-tuple==0.2.13,Jinja2==2.10,MarkupSafe==1.0,mccabe==0.6.1,pep8-naming==0.4.1,pycodestyle==2.3.1,pydocstyle==2.1.1,pyflakes==1.6.0,Pygments==2.2.0,pyPEG2==2.15.2,PyYAML==3.12,six==1.11.0,snowballstemmer==1.2.1
flake8 runtests: PYTHONHASHSEED='2112495524'
flake8 runtests: commands[0] | /home/florian/proj/qutebrowser/test
ERROR: invocation failed (errno 2), args: ['/home/florian/proj/qutebrowser/test'], cwd: /home/florian/proj/qutebrowser/test#foo
Traceback (most recent call last):
  File "/bin/tox", line 11, in <module>
    load_entry_point('tox==2.9.1', 'console_scripts', 'tox')()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 40, in main
    retcode = Session(config).runcommand()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 392, in runcommand
    return self.subcommand_test()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 583, in subcommand_test
    self.runtestenv(venv)
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 592, in runtestenv
    self.hook.tox_runtest(venv=venv, redirect=redirect)
  File "/usr/lib/python3.6/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/usr/lib/python3.6/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/usr/lib/python3.6/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/usr/lib/python3.6/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "/usr/lib/python3.6/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/usr/lib/python3.6/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/usr/lib/python3.6/site-packages/tox/venv.py", line 464, in tox_runtest
    venv.test(redirect=redirect)
  File "/usr/lib/python3.6/site-packages/tox/venv.py", line 384, in test
    ignore_ret=ignore_ret, testcommand=True)
  File "/usr/lib/python3.6/site-packages/tox/venv.py", line 414, in _pcall
    redirect=redirect, ignore_ret=ignore_ret)
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 140, in popen
    stdout=stdout, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 228, in _popen
    stdout=stdout, stderr=stderr, env=env)
  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/florian/proj/qutebrowser/test': '/home/florian/proj/qutebrowser/test'

Nothing too special in tox.ini:

[testenv:flake8]
basepython = {env:PYTHON:python3}
passenv =
deps =
    -r{toxinidir}/requirements.txt
    -r{toxinidir}/misc/requirements/requirements-flake8.txt
commands =
    {envpython} -m flake8 {posargs:qutebrowser tests scripts}

I'm guessing {envpython} gets replaced by the Python path (which contains a #) and only after that, comments are stripped out?

[jayvdb]

I'm guessing {envpython} gets replaced by the Python path (which contains a #) and only after that, comments are stripped out?

correct. Is it worth checking, are there any platforms that do not support # in paths? I scanned https://metacpan.org/source/XSAWYERX/PathTools-3.75/lib/File/Spec quickly and didnt see any.

Annoyingly this replacement does not occur within tox substitution system - instead it occurs in _ArgvlistReader.processcommand's use of shlex.shlex, which is the very last step of command processing.

Removing # from the shlex.shlex.commenters fixes this problem, but then many uses of # in commands fails.

diff --git a/src/tox/config/__init__.py b/src/tox/config/__init__.py
index ea37bdc..8cc8137 100644
--- a/src/tox/config/__init__.py
+++ b/src/tox/config/__init__.py
@@ -1936,6 +1938,7 @@ class _ArgvlistReader:
         # use all values as is in argv.
         shlexer = shlex.shlex(newcommand, posix=True)
         shlexer.whitespace_split = True
+        shlexer.commenters = ""
         shlexer.escape = ""
         return list(shlexer)
 

Only one test fails with the above, but there are way too many failure scenarios to seriously consider pre-processing the command so that '#' is correctly handled inside and outside of quotes.

Internally, tox stores paths as py.path objects (deprecated), so it should be possible to delay resolution of them until the moment of shlexing, and then wrap them in quotes. However that approach is complicated by the fact that these objects go through the regex engine (re.sub), which requires bytes or str objects only.

So, solving this seems to require adding quotes around path objects within the substitution engine, and then removing them before they are used in typical ways. They need to be removed after, otherwise the quotes appear everywhere especially in the cmdline UI and look ugly, and very likely breaking lots of plugins which are not expecting them. Hopefully this be slowly developed by activating this path quoting behaviour only when # is found in a path. There are lots of other chars that need to be quoted in paths, differing per platform, but I think we'd want to rely on a library which provides this host metadata. Or we can only add quoting around path chars when issues are raised, resulting in a core set where there was at least one vocal user presenting a use case.

Small aside, I didn't quickly find a way for py.path objects (or pathlib.Path objects) to handle quoted paths, as it interprets the quotes as literals.

Fun fact

[testenv:hashpath]
commands =
  echo {envdir}

results in an absolute path being shown on the reporter command, but the real command is not absolute. Anyone know if that bug has been raised, and/or if it is a regression and what the expected behaviour is?

hashpath run-test: commands[0] | echo /home/jayvdb/projects/tox/my-tox-tests/tests/.tox/hashpath
.tox/hashpath

[jayvdb]

Ah, #924 is another similar problem, with spaces in path.

[jayvdb]

commands = echo {envdir}
results in an absolute path being shown on the reporter command, but the real command is not absolute. Anyone know if that bug has been raised, and/or if it is a regression and what the expected behaviour is?

This looks like #1463, #1339, and perhaps others. Appears to have been an introduced change from absolute dirs prior to v3.8.0

[gaborbernat]

I think this has now been fixed in v3, though I'm sure they'll be some edge cases that will surprise us due to this.