symlink dirs: check for existing link target?

[hjoliver]

# global.cylc
[install]
   [[symlink dirs]]
      [[[localhost]]]
         run = /nesi/nobackup/$PROJECT/$USER

Now:

• install a new workflow from source (to blah/run1)
• mess with the installed files (by running the workflow, or otherwise)
• remove the run directory with rm -rf ~/cylc-run or rm -rf ~/cylc-run/blah
• install the workflow again from source (to blah/run1)

Result:

• the original messed-with files (those not overwritten by the new install) will exist in the new run directory (because rm does not follow symlinks).

Of course, this is one of the two reasons we should use cylc clean and not rm -rf (clean up job hosts, and clean up symlinked run dirs). But some users won't be aware of that, or will likely revert to using rm -rf sometimes without being aware of the consequences.

I think cylc install should abort if it finds the target of a new symlinked dir already exists, to ensure that run directories contain only the expected files (and those generated at run time).

(I can't think of any good reason why an existing target should be allowed - agree?)

[hjoliver]

This causes the cylc-rose plugin to crash.

(venv) oliverh@w-nwp01:~/cylc-src/foo$ ls
flow.cylc  rose-suite.conf

(venv) oliverh@w-nwp01:~/cylc-src/foo$ cylc install
INSTALLED foo/run1 from /scale_wlg_persistent/filesets/home/oliverh/cylc-src/foo

(venv) oliverh@w-nwp01:~/cylc-src/foo$ rm -rf ~/cylc-run/foo                    
                
(venv) oliverh@w-nwp01:~/cylc-src/foo$ cylc install                                             
INSTALLED foo/run1 from /scale_wlg_persistent/filesets/home/oliverh/cylc-src/foo
PluginError: Error in plugin cylc.post_install.rose_opts
AttributeError: 'Values' object has no attribute 'clear_rose_install_opts'

The AttributeError has something to do with rose-suite.conf already existing in the run directory. Not sure if the exception raised represents a bug in cylc-rose too.

[wxtim]

Reproduced, complete with the Cylc Rose Error message.

Explanation of the Cylc Rose message and unhandled situation

    if conf_filepath.is_file():
        if opts.clear_rose_install_opts:
            conf_filepath.unlink()
        else:
            ...

1. opts.clear_rose_install_opts is only set by cylc reinstall
2. If there is already a rose-suite-cylc-install.conf present and the script is not cylc reinstall this failure will occur.

How do we want this to behave? I think that if opts.clear_rose_install_opts is unset we should raise an exception because the workflow isn't cleaned properly.

It looks like the Cylc Rose problem arises from the absence of the option --clear-cylc-rose-options which gets used if we're trying to over-write an opt/rose-suite-cylc-install.conf file. This option is only provided to the plugin by cylc reinstall, unless

[dpmatthews]

I think cylc install should abort if it finds the target of a new symlinked dir already exists, to ensure that run directories contain only the expected files (and those generated at run time).

This sounds reasonable to me.
I assume we need to apply the same logic to remote init, not just cylc install.

[oliver-sanders]

I think the symlink dir issue is a consequence of #4790? These two could likely be addressed together.

[dpmatthews]

I think the symlink dir issue is a consequence of #4790? These two could likely be addressed together.

I think they are slightly different.
This issue refers to the case where a symlink needs to be created and the target already exists.
I think that should always be an error.

#4790 includes the case where the directory already exists (either as a directory or a symlink). We need to allow this in some cases.

[MetRonnie]

I've just had a thought - this should only apply for the run dir symlink, right? If e.g. the log dir symlink target exists, that shouldn't be a problem?

[dpmatthews]

I've just had a thought - this should only apply for the run dir symlink, right? If e.g. the log dir symlink target exists, that shouldn't be a problem?

In my view it should apply to any symlink target.
In all cases it implies the directories from a previous run were not cleaned up correctly and you could end up with files not generated by the current run.