Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

file parsing: check for CWD existence #5694

Merged
merged 2 commits into from
Aug 25, 2023
Merged

Conversation

hjoliver
Copy link
Member

@hjoliver hjoliver commented Aug 16, 2023

Close #5688

But note #5688 (comment)

Is it worth doing this?

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md entry included if this is a change that can affect users
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX. [not needed]
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@hjoliver hjoliver added this to the cylc-8.2.2 milestone Aug 16, 2023
@hjoliver hjoliver self-assigned this Aug 16, 2023
@oliver-sanders
Copy link
Member

Is it worth doing this?

It looks like applying this diff won't actually help because pkg_resources will fail e.g:

$ mkdir foo
$ cd foo
$ cylc version
8.3.0.dev
$ rm -r ../foo
$ cylc version
Traceback (most recent call last):
...
Traceback (most recent call last):
  File "~/mambaforge/envs/cylc-8.2.dev/bin/cylc", line 5, in <module>
    from cylc.flow.scripts.cylc import main
  File "~/cylc-flow/cylc/flow/scripts/cylc.py", line 26, in <module>
    import pkg_resources
...
    return os.path.normcase(os.path.realpath(os.path.normpath(_cygwin_patch(filename))))
  File "~/mambaforge/envs/cylc-8.2.dev/lib/python3.9/posixpath.py", line 393, in realpath
    return abspath(path)
  File "~/mambaforge/envs/cylc-8.2.dev/lib/python3.9/posixpath.py", line 380, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

@hjoliver
Copy link
Member Author

Yes that's what I found too (see the associated issue) - but @ColemanTom claims it works in his environment, which would be some kind of an improvement. Can you comment Tom?

@ColemanTom
Copy link
Contributor

ColemanTom commented Aug 22, 2023

$ mkdir tmpdir
$ cd tmpdir
$ cylc scan
rewind/run7 gadi-login-03.gadi.nci.org.au:43067 186324
rewind/run6 gadi-login-03.gadi.nci.org.au:43031 94705
rewind/run3 gadi-login-03.gadi.nci.org.au:43053 3954261
$ rm -fr ~/tmpdir
$ cylc scan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
ERROR: error getting working directory name: no such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<_AsyncPipe._generate() done, defined at /g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/async_util.py:148> exception=FileNotFoundError(2, 'No such file or directory')>
Traceback (most recent call last):
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/async_util.py", line 151, in _generate
    async for item in gen.func(*gen.args, **gen.kwargs):
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/network/scan.py", line 222, in scan
    max_depth = glbl_cfg().get(['install', 'max depth'])
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/cfgspec/glbl_cfg.py", line 22, in glbl_cfg
    return GlobalConfig.get_inst(cached=cached)
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/cfgspec/globalcfg.py", line 1903, in get_inst
    cls._DEFAULT.load()
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/cfgspec/globalcfg.py", line 1930, in load
    self._load(fname, conf_type)
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/cfgspec/globalcfg.py", line 1910, in _load
    self.loadcfg(fname, conf_type)
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/parsec/config.py", line 80, in loadcfg
    sparse = parse(
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/parsec/fileparse.py", line 565, in parse
    flines = read_and_proc(fpath, template_vars, opts=opts)
  File "/g/data/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/parsec/fileparse.py", line 404, in read_and_proc
    odir = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
$ vi ~/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/cylc/flow/parsec/fileparse.py
$ cylc scan
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
ERROR: error getting working directory name: no such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
rewind/run3 gadi-login-03.gadi.nci.org.au:43053 3954261
rewind/run7 gadi-login-03.gadi.nci.org.au:43067 186324
rewind/run6 gadi-login-03.gadi.nci.org.au:43031 94705

@ColemanTom
Copy link
Contributor

cylc version also works for me with this change.

$ cylc version
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
ERROR: error getting working directory name: no such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
8.2.1

@ColemanTom
Copy link
Contributor

What is your PYTHONPATH? There is a detailed discussion here about this issue. If you have a trailing or leading : in your PYTHONPATH, then the CWD will get added to your sys.path, or if you do an interactive call to Python.

$ ~/miniconda3/envs/cylc-8.2.0/bin/python ~/script.py
['/home/username', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python39.zip', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/lib-dynload', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages']
/home/username/script.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
$ ~/miniconda3/envs/cylc-8.2.0/bin/python -c 'import sys; print(sys.path); import pkg_resources'
['', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python39.zip', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/lib-dynload', '/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages']
<string>:1: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
Traceback (most recent call last):
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/pkgutil.py", line 417, in get_importer
    importer = sys.path_importer_cache[path_item]
KeyError: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3328, in <module>
    def _initialize_master_working_set():
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3302, in _call_aside
    f(*args, **kwargs)
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3340, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 622, in _build_master
    ws = cls()
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 615, in __init__
    self.add_entry(entry)
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 671, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 2075, in find_distributions
    importer = get_importer(path_item)
  File "/home/username/miniconda3/envs/cylc-8.2.0/lib/python3.9/pkgutil.py", line 421, in get_importer
    importer = path_hook(path_item)
  File "<frozen importlib._bootstrap_external>", line 1608, in path_hook_for_FileFinder
  File "<frozen importlib._bootstrap_external>", line 162, in _path_isdir
FileNotFoundError: [Errno 2] No such file or directory

@ColemanTom
Copy link
Contributor

Other comment, based on the above output, pkg_resources is deprecated. Can importlib.resources and/or importlib.metadata be used instead? They don't have the pkg_resources issue you identified above.

@oliver-sanders
Copy link
Member

oliver-sanders commented Aug 23, 2023

Strange.

What is your PYTHONPATH

Unset (I don't use PYTHONPATH)

I would expect any difference in behaviour is likely down to Python version

$ python --version
Python 3.9.16
$ pip list | grep setuptools
setuptools                     68.0.0
$ pip list | grep import
importlib-metadata             6.8.0
importlib-resources            6.0.0

Other comment, based on the above output, pkg_resources is deprecated

Note we still support Python 3.7 and will need to continue to support it for a while to come.

@ColemanTom
Copy link
Contributor

$ python --version
Python 3.9.17
$ pip list | grep 'setuptools\|import'
importlib-metadata        6.8.0
importlib-resources       6.0.1
setuptools                68.0.0

Installing matching versions to yours, and I don't get failures whether I run a script or do python -c approach I showed above.

I should perhaps note, it works for me on two completely different platforms, one an HPC login node, an another a VM, maintained by completely different organisations.

@oliver-sanders
Copy link
Member

oliver-sanders commented Aug 24, 2023

I've matched your versions but cannot get this to succeed on two different vms and three different filesystems, I'm really not sure what the difference is here, possible OS/FS.

@oliver-sanders
Copy link
Member

oliver-sanders commented Aug 24, 2023

There's no harm in us accepting this change, however, it doesn't look like this is behaviour you can rely on (especially as we cannot reproduce your results so cannot test this) as it may be OS/FS specific so sanitising working practices to avoid chopping off the branch you're standing on is going to be the safer option. I'm surprised that more commands don't fail under these circumstances as a wide range of filesystem operations are going to fail.

@hjoliver hjoliver changed the title file parsing: check for PWD existence file parsing: check for CWD existence Aug 24, 2023
@hjoliver
Copy link
Member Author

OK I've clarified and commented the code, and added a unit test. @oliver-sanders - can you do a quick review and merge it?

@oliver-sanders oliver-sanders merged commit 2888d95 into cylc:8.2.x Aug 25, 2023
25 checks passed
@hjoliver hjoliver deleted the chdir-err branch August 26, 2023 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants