Prep 7.7.0 release #2661

hjoliver · 2018-05-12T03:13:19Z

Updates to change log etc. for 7.7.0 release.

[update] close #1113

matthewrmshin

Should also mention the directory restructure in CHANGES.md, e.g. new expected locations for site *.rc files and new locations for editor syntax highlight files.

matthewrmshin · 2018-05-14T10:36:57Z

doc/src/cylc-user-guide/job-host.tex

-\lstinline=rose suite-run= on \lstinline=vagrant@localhost=. The suite
-submits a single task called \lstinline=foo= to \lstinline=hobo@otherhost=.
-The operator polls then kills the task, then shuts the suite down.
+Security-conscious sites may want to restrict access to job hosts with SSH


Can we tone this down slightly? It is not that other sites are not security conscious. It is that sites may have different security audit requirements.

matthewrmshin · 2018-05-14T10:38:24Z

doc/src/cylc-user-guide/job-host.tex

+\lstset{language=sh}
+{\scriptsize
+\dirtree{%
+.1 /home/hjoliver/cylc-run/suitex.


Use ~/ to make it more general?

matthewrmshin · 2018-05-14T10:38:47Z

doc/src/cylc-user-guide/job-host.tex

+Result:
+{\scriptsize
+\dirtree{%
+.1 /home/hjoliver/cylc-run/suitex.


matthewrmshin · 2018-05-14T10:39:21Z

doc/src/cylc-user-guide/job-host.tex

+Result:
+{\scriptsize
+\dirtree{%
+.1 /home/hjoliver/cylc-run/suitex.


matthewrmshin · 2018-05-14T10:55:09Z

tests/tutorial/test_header_tutorial

-    perl -pi -e "s/(initial cycle point\s*=)\s*.*/\1 20130808T00/g" suite.rc
-    perl -pi -e "s/(final cycle point\s*=)\s*.*/\1 20130808T12/g" suite.rc
-
+    if grep "cycling mode = integer" suite.rc > /dev/null; then


grep -q ...?

hjoliver · 2018-05-14T12:39:16Z

@matthewrmshin - feedback all addressed. Over 'n out for today.

matthewrmshin · 2018-05-14T14:52:20Z

See also #1113 (comment) for issue with global.rc.

matthewrmshin · 2018-05-14T14:55:51Z

(I think it is worth moving site global to etc/global.rc, with back compat logic to also search for the file under conf/global.rc.)

matthewrmshin

Looks good enough.

matthewrmshin · 2018-05-14T15:24:11Z

(My bad.) You may want to update the docs for cylc message with something like this:

diff --git a/bin/cylc-message b/bin/cylc-message
index 8f172c086..08578d8dc 100755
--- a/bin/cylc-message
+++ b/bin/cylc-message
@@ -15,7 +15,7 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-"""cylc [task] message [OPTIONS] -- ARGS
+r"""cylc [task] message [OPTIONS] -- ARGS
 
 Record task job messages.
 
@@ -30,7 +30,22 @@ and to report registered task outputs.
 
 Messages can be specified as arguments. A '-' indicates that the command should
 read messages from STDIN. When reading from STDIN, multiple messages are
-separated by empty lines.
+separated by empty lines. E.g.:
+
+Single message as an argument:
+  % cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" 'Hello world!'
+Multiple messages as arguments:
+  % cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" \
+  %     'Hello world!' 'Hi' 'WARNING:Hey!'
+Multiple messages on STDIN:
+  % cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" - <<'__STDIN__'
+  % Hello
+  % world!
+  %
+  % Hi
+  %
+  % WARNING:Hey!
+  % __STDIN__
 
 Each message can be prefixed with a severity level using the syntax 'SEVERITY:
 MESSAGE'.
diff --git a/doc/src/cylc-user-guide/cug.tex b/doc/src/cylc-user-guide/cug.tex
index f2df8cf41..955da6312 100644
--- a/doc/src/cylc-user-guide/cug.tex
+++ b/doc/src/cylc-user-guide/cug.tex
@@ -5503,14 +5503,14 @@ Normal severity messages are printed to \lstinline=job.out= and logged by the
 suite server program:
 \lstset{language=bash}
 \begin{lstlisting}
-cylc message "Hello from ${CYLC_TASK_ID}"
+cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" "Hello from ${CYLC_TASK_ID}"
 \end{lstlisting}
 
 CUSTOM severity messages are printed to \lstinline=job.out=, logged by the
 suite server program, and can be used to trigger {\em custom} event handlers:
 \lstset{language=bash}
 \begin{lstlisting}
-cylc message -p CUSTOM "data available for ${CYLC_TASK_CYCLE_POINT}"
+cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" "CUSTOM:data available for ${CYLC_TASK_CYCLE_POINT}"
 \end{lstlisting}
 Task output messages, used for triggering other tasks, can also be sent with
 custom severity if need be
@@ -5518,13 +5518,13 @@ custom severity if need be
 WARNING severity messages are printed to \lstinline=job.err=, logged by the
 suite server program, and can be passed to {\em warning} event handlers:
 \begin{lstlisting}
-cylc message -p WARNING "Uh-oh, something's not right here."
+cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" "WARNING:Uh-oh, something's not right here."
 \end{lstlisting}
 
 CRITICAL severity messages are printed to \lstinline=job.err=, logged by the
 suite server program, and can be passed to {\em critical} event handlers:
 \begin{lstlisting}
-cylc message -p CRITICAL "ERROR occurred in process X!"
+cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" "CRITICAL:ERROR occurred in process X!"
 \end{lstlisting}
 
 \subsection{Aborting Job Scripts on Error}
@@ -5563,7 +5563,7 @@ For failures detected by the scripting you could send a critical message back
 before aborting, potentially triggering a {\em critical} task event handler:
 \begin{lstlisting}
 if ! /bin/false; then
-  cylc message -p CRITICAL "ERROR: /bin/false failed!"
+  cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" "CRITICAL:/bin/false failed!"
   exit 1
 fi
 \end{lstlisting}
diff --git a/etc/examples/message-triggers/suite.rc b/etc/examples/message-triggers/suite.rc
index 32d423c6f..8ec50fe64 100644
--- a/etc/examples/message-triggers/suite.rc
+++ b/etc/examples/message-triggers/suite.rc
@@ -12,9 +12,9 @@
     [[foo]]
         script = """
 sleep 5
-cylc message "file 1 done"
+cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" 'file 1 done'
 sleep 10
-cylc message "file 2 done"
+cylc message -- "${CYLC_SUITE_NAME}" "${CYLC_TASK_JOB}" 'file 2 done'
 sleep 10"""
         [[[outputs]]]
             out1 = "file 1 done"

The new interface of the cylc message command is backward compatible, so no big deal.

The shell variable would not normally be defined in user environments.

hjoliver · 2018-05-15T04:32:35Z

cylc message help and docs updated.

hjoliver · 2018-05-15T04:34:31Z

(I think it is worth moving site global to etc/global.rc, with back compat logic to also search for the file under conf/global.rc.)

OK, I've done that now. Note I've dropped the deprecated site.rc and user.rc filename variants - they are years old now, no one should be using them.

hjoliver · 2018-05-15T04:41:58Z

See also #1113 (comment) for issue with global.rc.

Well, you asked for it. I've had a crack at this now. My new solution mimics the site update method (keep it version specific):

~/.cylc/x.y.z/global.rc

where x.y.z is cylc version (and is truncated beyond z, for use in git clones); falling back to ~/.cylc/global.rc.

New file etc/global.rc.eg explains the rationale. I think this is better than what we have now, because it allows users to work with multiple versions at once, which may not be possible at the moment depending on your global settings. On the downside, they have to know to copy over the global config file before using a new version. But mitigating that, if they only have simple global.rc needs (e.g. set editor) they can stick with the non version-specific location. The original proposal of #1113 would probably be better, but that is at least arguable because it can't distinguish between errors and version incompatibilities.

Thoughts?

matthewrmshin · 2018-05-15T08:26:05Z

3 tests broken by directory restructure. Need to make this change:

diff --git a/lib/cylc/version.py b/lib/cylc/version.py
index 8c339bd0b..26a91190d 100644
--- a/lib/cylc/version.py
+++ b/lib/cylc/version.py
@@ -34,8 +34,8 @@ def _get_cylc_version():
         # We're running in a cylc git repository, so dynamically determine
         # the cylc version string.  Enclose the path in quotes to handle
         # avoid failure when cylc_dir contains spaces.
-        is_ok, outlines = run_get_stdout(
-            '"%s"' % os.path.join(cylc_dir, "etc", "bin", "get-repo-version"))
+        is_ok, outlines = run_get_stdout('"%s"' % os.path.join(
+            cylc_dir, "etc", "dev-bin", "get-repo-version"))
         if is_ok and outlines:
             return outlines[0]
         else:

matthewrmshin · 2018-05-15T08:48:02Z

I think we should close #1113 after this - the intended requirement is met by this solution.

hjoliver · 2018-05-15T09:28:44Z

(Marked #1113 to close when this is merged)

(These were regenerated a few commits ago without updated the generating script with --no-detach)

hjoliver · 2018-05-15T10:15:12Z

Final commit makes the regenerated tutorial reference tests non-detaching (as reference tests are supposed to be ... but unfortunately suite_run_ok cylc run foo --reference-test just passes if --no-detach is omitted).

That should be it ... hopefully I can make the release tomorrow.

matthewrmshin · 2018-05-15T11:11:13Z

@oliver-sanders please do a final sanity check.

oliver-sanders

All seems good to me.

oliver-sanders · 2018-05-15T15:19:43Z

doc/src/cylc-user-guide/cug.tex

@@ -3779,7 +3794,7 @@ \subsubsection{Trigger Types}
 message. Other tasks can trigger off the externally triggered task as required,
 of course.

-\lstinline=$CYLC_DIR/examples/satellite/ext-triggers/suite.rc= is a working
+\lstinline=<cylc-dir>/etc/examples/satellite/ext-triggers/suite.rc= is a working


I like the <substitution> syntax, in the rose documentation we now have a pygments lexer for highlighting these in code blocks e.g: http://metomi.github.io/rose/doc/html/cheat-sheet.html

Nice, looking foward to going cylc docs going this way...

hjoliver added 3 commits May 11, 2018 18:38

Updated change log for 7.6.1.

b3501c9

Use new sbin directory for stuff.

464f01c

Update docs a bit.

f3902ab

hjoliver force-pushed the prep-7.7.0 branch 13 times, most recently from 34749d5 to f8c3945 Compare May 12, 2018 11:35

Updated changes.md for 7.7.0 [skip ci]

7961bd5

hjoliver force-pushed the prep-7.7.0 branch from f8c3945 to 7961bd5 Compare May 12, 2018 11:37

Update remote job management documentation.

108433a

hjoliver self-assigned this May 12, 2018

hjoliver added this to the next release milestone May 12, 2018

Fixed cut-n-paste error. [skip ci]

f183527

hjoliver force-pushed the prep-7.7.0 branch from 0a387f9 to f183527 Compare May 12, 2018 22:07

hjoliver added 5 commits May 13, 2018 13:44

Moved some things into etc/.

1a85ed8

Moved examples into etc dir.

a310949

Moved syntax files to etc dir.

86cb49e

Moved 5to6 dir to etc.

f04f6d9

Moved bash completion to etc dir.

7fcf433

hjoliver requested review from matthewrmshin and sadielbartholomew May 13, 2018 02:28

oliver-sanders self-requested a review May 14, 2018 10:33

matthewrmshin suggested changes May 14, 2018

View reviewed changes

hjoliver added 2 commits May 15, 2018 00:02

Address review feedback.

e57b549

Final update to change log [skip ci].

3e39279

hjoliver force-pushed the prep-7.7.0 branch from 79887f5 to 3e39279 Compare May 14, 2018 12:37

matthewrmshin approved these changes May 14, 2018

View reviewed changes

hjoliver added 4 commits May 15, 2018 09:59

Corrected "cylc message" help and docs.

abdfaaf

Moved global config to etc.

c039207

Changed $CYLC_DIR to <cylc-dir> in docs.

d0c77b8

The shell variable would not normally be defined in user environments.

pep8

5d759a4

change log update

063b5a6

hjoliver added 2 commits May 15, 2018 21:10

Fix post dir restructure.

7fee31d

Tidy text (etc/global.rc.eg only). [skip ci]

97a4642

hjoliver mentioned this pull request May 15, 2018

parsec should ignore unknown items, not abort #1113

Closed

Fix tutorial ref tests.

31cb150

(These were regenerated a few commits ago without updated the generating script with --no-detach)

oliver-sanders approved these changes May 15, 2018

View reviewed changes

oliver-sanders merged commit e154833 into cylc:master May 15, 2018

hjoliver deleted the prep-7.7.0 branch May 15, 2018 20:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prep 7.7.0 release #2661

Prep 7.7.0 release #2661

hjoliver commented May 12, 2018 •

edited

Loading

matthewrmshin left a comment

matthewrmshin May 14, 2018

matthewrmshin May 14, 2018

matthewrmshin May 14, 2018

matthewrmshin May 14, 2018

matthewrmshin May 14, 2018

hjoliver commented May 14, 2018

matthewrmshin commented May 14, 2018

matthewrmshin commented May 14, 2018

matthewrmshin left a comment

matthewrmshin commented May 14, 2018

hjoliver commented May 15, 2018

hjoliver commented May 15, 2018

hjoliver commented May 15, 2018 •

edited

Loading

matthewrmshin commented May 15, 2018

matthewrmshin commented May 15, 2018

hjoliver commented May 15, 2018

hjoliver commented May 15, 2018

matthewrmshin commented May 15, 2018

oliver-sanders left a comment

oliver-sanders May 15, 2018

hjoliver May 15, 2018

Prep 7.7.0 release #2661

Prep 7.7.0 release #2661

Conversation

hjoliver commented May 12, 2018 • edited Loading

matthewrmshin left a comment

Choose a reason for hiding this comment

matthewrmshin May 14, 2018

Choose a reason for hiding this comment

matthewrmshin May 14, 2018

Choose a reason for hiding this comment

matthewrmshin May 14, 2018

Choose a reason for hiding this comment

matthewrmshin May 14, 2018

Choose a reason for hiding this comment

matthewrmshin May 14, 2018

Choose a reason for hiding this comment

hjoliver commented May 14, 2018

matthewrmshin commented May 14, 2018

matthewrmshin commented May 14, 2018

matthewrmshin left a comment

Choose a reason for hiding this comment

matthewrmshin commented May 14, 2018

hjoliver commented May 15, 2018

hjoliver commented May 15, 2018

hjoliver commented May 15, 2018 • edited Loading

matthewrmshin commented May 15, 2018

matthewrmshin commented May 15, 2018

hjoliver commented May 15, 2018

hjoliver commented May 15, 2018

matthewrmshin commented May 15, 2018

oliver-sanders left a comment

Choose a reason for hiding this comment

oliver-sanders May 15, 2018

Choose a reason for hiding this comment

hjoliver May 15, 2018

Choose a reason for hiding this comment

hjoliver commented May 12, 2018 •

edited

Loading

hjoliver commented May 15, 2018 •

edited

Loading