Merge branch 'workflow-more-good-stuff'

guide/blueprints/workflow/common.md

@@ -50,6 +50,8 @@ steps:
     not: { equals: true }
 ```
 
+Care should be taken when using `:` in a step with shorthand.  YAML will parse it as a map if it is not quoted in YAML. However at runtime, if the step looks like it came from an accidental colon causing a map, it will be reverted to a string with the colon re-introduced, so you can write steps with shorthand `- log Your name is: ${name}`. 
+
 All steps support a number of common properties, described below.
 
 ### Explicit IDs and Name

guide/blueprints/workflow/examples/aws-cfn-stacks/aws-cfn-type.bom

@@ -0,0 +1,15 @@
+brooklyn.catalog:
+  bundle: aws-cfn-discovery-sample
+  version: 1.0.0-SNAPSHOT
+  items:
+    - id: aws-cfn-discovered-stack-sample
+      item:
+        type: org.apache.brooklyn.entity.stock.BasicEntity
+        brooklyn.initializers:
+          - type: workflow-effector
+            name: on_update
+            steps:
+              - set-entity-name ${item.StackName}
+              - set-sensor data = ${item}
+              - set-sensor stack_status = ${item.StackStatus}
+              # above is just a start, you can check drift, explore resources, etc

guide/blueprints/workflow/examples/aws-cfn-stacks/aws-discoverer.yaml

@@ -0,0 +1,16 @@
+name: AWS CloudFormation Discoverer
+
+services:
+  - type: workflow-software-process
+    location: localhost
+    name: Stacks
+
+    brooklyn.policies:
+      - type: workflow-policy
+        brooklyn.config:
+          name: periodically update children
+          period: 1m
+          steps:
+            - ssh aws cloudformation describe-stacks
+            - transform stdout | json | set describe_stacks_output_json
+            - update-children type aws-cfn-discovered-stack-sample id ${item.StackId} from ${describe_stacks_output_json.Stacks}

guide/blueprints/workflow/examples/aws-cfn-stacks/index.md

@@ -0,0 +1,35 @@
+---
+title: AWS CloudFormation Stack Discovery
+title_in_menu: AWS CFN Stacks
+layout: website-normal
+---
+
+The `update-children` step makes it straightforward to keep an Apache Brooklyn model
+in synch with external resources, from a cloud, GitHub or Jira tickets, or any data source you choose.
+The Brooklyn blueprint can then be used to attach management logic, including for example
+automatically deploying branched resources into ephemeral test environments.
+
+This example shows how CloudFormation stacks in AWS can be synchronized.
+
+Firstly, we define our type to represent discovered stack and be able to refresh `on_update`:
+
+{% highlight yaml %}
+{% readj aws-cfn-type.yaml %}
+{% endhighlight %}
+
+This should be added to the catalog.
+
+We can then deploy our Brooklyn application to discover and monitor stacks: 
+
+{% highlight yaml %}
+{% readj aws-discoverer.yaml %}
+{% endhighlight %}
+
+Create and delete stacks, and see them update in Brooklyn.
+Then consider:
+
+* Modify the `ssh aws` step in the "discoverer" to filter based on your preferred tags.
+* Use the `transform ... | merge` operator to combine lists from different regions.
+* Add other policies to check for drift on stacks and show failures in AMP if there is drift.
+* Create a similar workflow to monitor pull requests using the `gh` CLI;
+  then create, update, delete, and track ephemeral test deployments based on those

guide/blueprints/workflow/examples/index.md

@@ -6,6 +6,7 @@ children:
 - ansible-bash/
 - git-latest/
 - oauth/
+- aws-cfn-stacks/
 ---
 
 The following examples are available:

guide/blueprints/workflow/nested-workflow.md

@@ -56,14 +56,53 @@ as follows:
   inclusive (so the string `1..4` is equivalent to the list `[1,2,3,4]`)
 
 The scratch variables `target` and `target_index` are available referring to to the specific target
-and its 0-indexed position.
+and its 0-indexed position. These names can be overridden with the `target_var_name` and `target_index_var_name` keys. 
 
 Where a list is supplied, the result of the step is the list collecting the output of each sub-workflow.
 
 If a `condition` is supplied when a list is being used, the `workflow` step will always run,
 and the `condition` will be applied to entries in the list.
 An example of this is included below.
 
+The `foreach` type is a simplified variant of `workflow` when recursing over a list,
+taking the same.
+
+#### Example
+
+```
+- step: foreach x in 1..3
+  steps:
+  - return ${x}
+```
+
+The above loop will return `[1,2,3]`.
+
+
+### Reducing
+
+Each nested workflow runs in its own scope and does not share workflow variables with the parent,
+apart from values specified as `input`, or with other iterations of a loop.
+Where it is desired to share variables across iterations, the key `reducing` can be supplied,
+giving a map of variable names to be shared and their initial values.
+
+When `reducing`, the output of the workflow is this set of variables with their final values.
+
+
+#### Example
+
+```
+- step: foreach x in 1..3
+  reducing:
+    sum: 0
+  steps:
+  - let sum = ${sum} + ${x}
+```
+
+The above loop will return `6`.
+
+
+### Concurrency
+
 By default nested workflows with list targets run sequentially over the entries,
 but this can be varied by setting `concurrency`.
 The following values are allowed:
@@ -80,6 +119,8 @@ and always allowing 1.
 This might be used for example to upgrade a cluster in situ, leaving the larger of 10 instances or half the cluster alone, if possible.  
 If the concurrency expression evaluates to 0, or to a negative number whose absolute value is larger than the number of values, the step will fail before executing, to ensure that if e.g. "-10" is specified when there are fewer than 10 items in the target list, the workflow does not run.  (Use "max(1, -10)" to allow it to run 1 at a time if there are 10 or fewer.)
 
+Note: Concurrency cannot be specified when `reducing`.
+
 #### Example
 
 This example invokes an effector on all children which are `service.isUp`,
@@ -115,6 +156,11 @@ It also accepts the standard step keys such as `input`, `timeout` on `on-error`.
 A user of the defined step type can also supply `output` which, as per other steps,
 is evaluated in the context of the outer workflow, with visibility of the output from the current step.
 
+When supplying a workflow in contexts where a `workflow` is already expected,
+such as in a config key that takes a `workflow` (a Java `CustomWorkflowStep`),
+it is not necessary to specify the `type: workflow`, and additionally, if the only things being set is `steps`, those steps can be provided as a list without the `steps` keyword.
+Internally a _list_ will coerce to a `workflow` by interpreting the list as the steps.
+
 
 #### Shorthand Template Syntax
 
@@ -182,7 +228,7 @@ boolean isDefaultIdempotent();
 ```
 
 The first of these does the work of the step, resolving inputs and accessing context as needed via `context`.
-The second handles providing a cusotm shorthand, as described above;
+The second handles providing a custom shorthand, as described above;
 it can call to a superclass method `populateFromShorthandTemplate(TEMPLATE, value)`
 with the `TEMPLATE` for the class, if shorthand is to be supported.
 Finally, the third returns whether the step is idempotent, that is if the custom step is interrupted,

guide/blueprints/workflow/settings.md

@@ -10,7 +10,7 @@ Some of the common properties permitted on [steps](common.md) also apply to work
 including `condition`, `timeout`, and `on-error`.
 
 This rest of this section describes the remaining properties for more advanced use cases 
-including mutex locking and resilient workflows with replay points.
+including mutex locking and resilient workflows with replay points, and some tips on optimizing.
 
 
 ## Locks and Mutual Exclusion Behavior
@@ -477,3 +477,22 @@ on-error:
 - workflow retention parent
 ```
 
+## Optimizing for Workflows
+
+Workflows can generate a huge amount of data which can impact memory usage, persistence, and the UI.
+The REST API and UI do some filtering (e.g. in the body of the `internal` sensors used by workflow),
+but when working with large `ssh` `output` and `http` `content` payloads, and with `update-children`,
+performance can be dramatically improved by following these tips:
+
+* Optimize external calls to return the minimal amount of information needed
+  * Use `jq` to filter when using `ssh` or `container` steps
+  * Pass filter argumetns to `http` endpoints that accept them
+  * Use small page sizes with `retry from` steps
+
+* Optimize the data which is stored
+  * Override the `output` on `ssh` and `http` steps to remove unnecessary objects;
+    for example `http` returns several `content*` fields, and often just one is needed.
+    Simply settings `output: { content: ${content} }` will achieve this.
+  * Set `retention: 1` or `retention: 0` on workflows that use a large amount of information
+    and can simply be replayed from the start
+