Consolidate service files

[matthewrmshin]

1. Create service (variable) files under ~/cylc-run/SUITE/.service/.
   • source - symbolic link to suite source (can be ..)
     • Was path=SOURCE in ~/.cylc/REGDB/SUITE.
     • There is no need to store suite title. Read title from suite.rc is very fast.
   • ssl.pem - SSL private key file
     • Was in SOURCE/
   • ssl.cert - SSL certificate file
     • Was in SOURCE/
   • passphrase - passphrase file
     • Was in SOURCE/
   • contact - suite contact file
     • Was ~/cylc-run/SUITE/cylc-suite-env and ~/.cylc/ports/SUITE.
   • db - private suite run SQLite database file
     • Was ~/cylc-run/SUITE/cylc-suite-private.db
2. Suite contact file absorbs the port file.
   • Don't bother installing the contact file on job hosts that require polling.
   • Skip contact file installation for job host with shared file system with suite host.
   • On suite stop, remove suite contact files, local + remote ones.
   • Task message will no longer retry if suite contact file not found, as suite is unlikely to be running, and restart will poll any way.
3. Hierarchical suite name delimiter is now the path separator /.
   • Was .
   • It should now be natural to have suites in different levels of sub-directories under ~/cylc-run/
4. Remove subcommands: copy, unregister, rereregister, refresh.
   • (Happy to reinstate if this is a concern.)
5. cylc run|restart: automatic register, if necessary.
   • New --source=SOURCE option to specify alternate suite source.
6. cylc run|restart: (if previous run has left behind a contact file) remove old contact file and proceed with suite start up if it can prove that the old suite process is no longer running.
7. suite.rc.processed generation:
   • cylc validate: will no longer generate suite.rc.processed by default. New --output=FILENAME option to specify location to dump processed suite.rc file.
   • cylc run: will now dump suite.rc.processed under suite run directory, instead of the suite source directory.
8. Public suite run SQLite database file relocated to log/db, but will continue to have a symbolic link from top of suite run directory as cylc-suite.db.
9. Group suites of each test battery run under ~/cylc-run/cylctb-TIMESTAMP/ where TIMESTAMP is the start time of the test battery run. The hierarchy under which would also roughly reflect the hierarchy under $CYLC_DIR/tests/.

Close #2054. (Note: There are some minor changes from the original proposal.)

@cylc/core Change should be in a good state for review, but please discuss any concerns you may have. I expect #2063 to conflict with this change, so I think #2063 should probably go in before this.

[hjoliver]

@matthewrmshin - I'm hoping to have a good look at this before the end of the week.

[matthewrmshin]

Branch re-based and conflicts resolved.

[arjclark]

@matthewrmshin - this is on my to-do list - haven't forgotten just need space to examine and consider.

[hjoliver]

Very nice change.

A couple of minor questions on file naming:

• .cylc-var - what's the meaning of "variable" here? what about just .service?
• the "cylc" in the public db name cylc-suite.db is kind of redundant too - should it be just db like the new private db name? (could keep the top level symlink the same for back-compatibility)

[hjoliver]

Removing the old db category commands is fine with me.

[hjoliver]

Looks good; approved regardless of potential file name changes as per my question above.

[matthewrmshin]

Hi @hjoliver:

• On .cylc-var... At first, I proposed .cylc-service, and then I was trying to find a name with a meaning that roughly agrees with FHS. On hindsight, files in this directory are not going to change continuously during run time, so var is not really suitable either. Happy to change to .service.
• On log/cylc-suite.db... Make sense. I'll modify it to log/db.

[arjclark]

Can't spot anything obviously wrong or missing. Think I preferred the old cylc reg way of doing things but understand not necessarily needed. Test battery passing for me.