show and scan: add JSON option and metadata

[sadielbartholomew]

Supersede #2507. Close #2459.

[matthewrmshin]

As discussed off line, please make sure that:

• STDOUT contains a single JSON data structure.
• The returned JSON data structure should just replicate the data structure as returned by the API. There is no need to do any fancy processing. (This is noting that the --raw option actually returns something that is not that raw, but we'll leave that alone for now.)

[sadielbartholomew]

@matthewrmshin I have just started playing around and I can in fact return the simplest non-pretty ('raw') JSON output with just 2 basic lines (plus a one-line change to prevent further printout)!

    if options.json_format:
        print json.dumps(scan_many(args, options.comms_timeout), indent=4)

Simplicity! The moral of the story: don't mess with (the) JSON.

[sadielbartholomew]

Whilst I am responding to feedback I am improving my 'cmp_json_ok()' method in the 'test_header' as currently it gives a horrible stderr output with unicode and single-quotes (invalid for JSON) such as [[u'localhost', 43135, {u'group': u'', u'description': u'describing ... not equal to {u'cylctb-20180124T120254Z/authentication/00-identity': {u'owner': u'sbarth', u'host': u'localhost', u'port': 43094}}. Adding a processing function and new line characters to allow for simple lifting and pasting (into and straight out of a validator) of expected outputs, as required when creating and amending the tests.

[hjoliver]

(is this one ready for review again?)

[matthewrmshin]

(I think it is hold pending fixes to my comments.)

[sadielbartholomew]

@hjoliver as Matt said, not yet. I should hopefully complete & commit this today - I will comment to let you know when I am ready for review. Thanks.

[sadielbartholomew]

(in progress, cylc scan i.e. authentication tests will now fail until next commit to deal with arbitrary floating point value for 'update-time' in JSON structure)

[sadielbartholomew]

@hjoliver @matthewrmshin - ready for re-review! To summarise my new amendments:

1. scan JSON is no longer an adaptation of the 'not that raw' --raw but a 'dump' of the Pythonic list/dict structures outputted straight from the API as requested by Matt - though not entirely trivial as I had to ensure that undesired tests were filtered out when--name and/or --suite-owner patterns were specified. Tests updated accordingly;
2. Test header now contains a function to replace arbitrary time floats with a standard marker so they do not cause the comparison tests to fail by default. I have also improved it so it gives more helpful stderr e.g. allowing for simple copying and pasting to tests as likely required in future;
3. show now (I believe) prints an exclusive JSON structure for any valid argument group, notably even with an input of a combination of name and ID formats for TASKID. Tests updated accordingly.

[sadielbartholomew]

Further thought: I imagine should update the documentation to include the new --json options once this code is deemed ready to merge?

[oliver-sanders]

I shouldn't think so, the command line help should suffice, the command reference section of the documentation (http://cylc.github.io/cylc/doc/cylc-user-guide.pdf#appendix.F) is auto-generated.

[sadielbartholomew]

Ah I see. That's useful!

[hjoliver]

Looks good, no problems found. Tests as working.