Merge apache-brooklyn/master as at commit 5237ea6

apache · duncangrant · May 21, 2021 · Sep 4, 2018 · Sep 4, 2018 · Sep 4, 2018
commit a6c7aafcf39c79851f8f3470f1dc92844cf672d0
diff --git a/.gitignore b/.gitignore
@@ -40,4 +40,5 @@ style/js/catalog/items.js
 javadoc
 _book
 node_modules
-
+javadoc/stylesheet.css
+javadoc/javadoc.log
diff --git a/guide/blueprints/entity-configuration.md b/guide/blueprints/entity-configuration.md
@@ -88,7 +88,7 @@ management hierarchy. This applies to entities and locations. In a future releas
 extended to also apply to policies and enrichers.
 
 When a blueprint author defines a config key, they can explicitly specify the rules for inheritance 
-(both for super/sub-types, and for the runtime management hiearchy). This gives great flexibilty,
+(both for super/sub-types, and for the runtime management hierarchy). This gives great flexibilty,
 but should be used with care so as not to surprise users of the blueprint.
 
 The default behaviour is outlined below, along with examples and details of how to explilcitly 
@@ -123,7 +123,7 @@ is inherited unchanged.
 It will write out: `Sub-type launch command: Goodbye`.
 
 
-#### Inheriting Configuration from a Parent in the Runtime Management Hieararchy
+#### Inheriting Configuration from a Parent in the Runtime Management Hierarchy
 
 Configuration passed to an entity is inherited by all child entities, unless explicitly overridden.
 
@@ -244,7 +244,7 @@ itself.
 When deploying to a jclouds location, one can specify `templateOptions` (of type map). Rather than
 overriding, these will be merged with any templateOptions defined on the location.
 
-In the example below, the VM will be provisioned with minimum 2G ram and minimum 2 cores. It will 
+In the example below, the VM will be provisioned with minimum 2GB RAM and minimum 2 cores. It will 
 also use the merged template options value of 
 `{placementGroup: myPlacementGroup, securityGroupIds: sg-000c3a6a}`:
 
@@ -342,7 +342,7 @@ services:
 #### Never Inherited
 
 For some configuration values, the most logical behaviour is for the value to never be inherited
-in the runtime management hiearchy.
+in the runtime management hierarchy.
 
 Some common config keys that will never inherited include:
 

diff --git a/guide/blueprints/java/java_app/ExampleWebApp.java b/guide/blueprints/java/java_app/ExampleWebApp.java
@@ -33,7 +33,7 @@ public void init() {
                 .displayName("Cluster")
                 .configure(DynamicCluster.MEMBER_SPEC, EntitySpec.create(TomcatServer.class)
                         .configure(TomcatServer.ROOT_WAR, 
-                                "http://search.maven.org/remotecontent?filepath=org/apache/brooklyn/example/brooklyn-example-hello-world-sql-webapp/0.8.0-incubating/brooklyn-example-hello-world-sql-webapp-0.8.0-incubating.war")
+                                "https://search.maven.org/remotecontent?filepath=org/apache/brooklyn/example/brooklyn-example-hello-world-sql-webapp/0.12.0/brooklyn-example-hello-world-sql-webapp-0.12.0.war" /* BROOKLYN_VERSION */ )
                         .configure(TomcatServer.JAVA_SYSPROPS.subKey("brooklyn.example.db.url"),
                                 DependentConfiguration.formatString("jdbc:%s%s?user=%s&password=%s",
                                         DependentConfiguration.attributeWhenReady(db, MySqlNode.DATASTORE_URL),

diff --git a/guide/blueprints/multiple-services.md b/guide/blueprints/multiple-services.md
@@ -73,7 +73,7 @@ to include the credentials for the database (which are defined in the database c
 
 ### An Aside: Substitutability
 
-Don't like JBoss?  Is there something about Maria?
+Don't like JBoss?  Is there something about MariaDB?
 One of the modular principles we follow in Brooklyn is substitutability:
 in many cases, the config keys, sensors, and effectors are defined
 in superclasses and are portable across multiple implementations.

diff --git a/guide/blueprints/winrm/client.md b/guide/blueprints/winrm/client.md
@@ -34,24 +34,25 @@ and it will be used to instantiate a `org.apache.brooklyn.util.core.internal.win
 
 ## WinRM Connectivity Diagnostics
 
-If you are experiencing problems with a Windows blueprint against a jclouds location 
-where Apache Brooklyn complains about failing to connect to the IP you should check those things.
+If you are experiencing problems with a Windows blueprint,
+with an error about failing to connect (or about an authorization conduit),
+try the following quick list:
 
 1. Apache Brooklyn is using correct username and password
 1. Apache Brooklyn can reach the IP of the provisioned machine. WinRM port 5985 or 5986 is also reachable from Apache Brooklyn.
 1. Check whether `WinRmMachineLocation#getDefaultUserMetadataString(ConfigurationSupportInternal)` is applied on the VM.
    This script should be passed to the cloud and executed in order to configure WinRM according to Apache Brooklyn requirements for authentication.
    So far Windows startup script are known to be supported on AWS EC2 and VCloud Director.
    If your cloud doesn't use this script then tune WinRM parameters accordingly.
-1. Check whether you use WinRM over HTTP or over HTTPS.
-  1. If you are using WinRM over HTTP then make sure WinRM service on target VM has `AllowUnencrypted = true`
+1. Check whether you use WinRM over HTTP or over HTTPS. If you are using WinRM over HTTP then make sure WinRM service on target VM has `AllowUnencrypted = true` (see below).
 
-If the quick list above doesn't help then follow the steps bellow.
+If the quick list above doesn't help then follow the steps below.
 
-To speed up diagnosing the problem we advice to trigger a deployment with the JcloudsLocation flag `destroyOnFailure: false` so you can check status of the provisioned machine
+To speed up diagnosing the problem if you don't already have a static machine to target,
+we advise to trigger a deployment with the JcloudsLocation flag `destroyOnFailure: false` so you can check status of the provisioned machine
 or try later different WinRM parameters with a Apache Brooklyn [BYON Location](../../locations/index.html#byon).
 
-After you determined what is the username and the password you can proceed with next steps.
+After you determine what is the username and the password you can proceed with next steps.
 *(Notice that for cloud providers which use Auto Generated password will not be logged.
 For these cases use Java Debug to retrieve ot or provision a VM manually with the same parameters when using Apache Brooklyn to provision a jclouds location.)*
 
@@ -103,8 +104,9 @@ Use an Apache Brooklyn BYON blueprint to try easily other connection options.
 
 1. Check IP is reachable from Apache Brooklyn instance
    Check whether `telnet 10.0.0.1 5985` makes successfully a socket.
-1. If AllowUnencrypted is false and you are using WinRM over HTTP then apply `winrm set winrm/config/service @{AllowUnencrypted="true"}`
-   *If jclouds or the cloud provider doesn't support passing `sysprep-specialize-script-cmd` then consider modifying Windows VM Image.* 
+1. Check that WinRM works, before delving deep in to the client: `Test-WSMan TARGET` and/or `winrs -r:10.0.2.15 -unencrypted -u:Administrator -p:pa55w0rd ipconfig`;
+   many of the tips below will fix underlying WinRM problems, not just Winrm4j.
+   *If the cloud provider doesn't support passing `sysprep-specialize-script-cmd` it may be necessary to modify the source Windows VM image to enable WinRM.* 
 1. Check your username and password. Notice in Windows passwords are case sensitive.
    Here is how it looks log from a wrong password:
 
@@ -114,12 +116,12 @@ Use an Apache Brooklyn BYON blueprint to try easily other connection options.
         org.apache.cxf.interceptor.Fault: Could not send Message.
         at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
 
-1. When having wrong password you may want to try logging on a different domain
-   This is possible from `brooklyn.winrm.config.winrm.computerName` location config.
+1. Try `./User` instead of `User`.
+1. Check whether you need to specify a different domain: this is possible from `brooklyn.winrm.config.winrm.computerName` location config.
+1. Ensure all Windows machines consider the other side a "trusted host". On a private subnet, it may be appropriate to run: `Set-Item wsman:\localhost\client\trustedhosts *`
+   whereas in other environments you will need to specify the list of machines.
+1. Restart WinRM on both machines (some changes need a restart to take effect): `Restart-Service WinRM`
 1. If you want to configure Windows target host with HTTPS then check the article [Configuring WINRM for HTTPS](https://support.microsoft.com/en-us/kb/2019527)
-1. If you are still seeing authorization errors then try connecting via WinRM with the embedded winrs client.
-   First make sure you have the server in trusted hosts.
 
-Then execute a simple command like
+In some cases the problems may be outwith the client, and it might be useful to look at [Troubleshooting](troubleshooting.md).
 
-    winrs -r:10.0.2.15 -unencrypted -u:Administrator -p:pa55w0rd ipconfig
diff --git a/guide/blueprints/winrm/index.md b/guide/blueprints/winrm/index.md
@@ -42,7 +42,6 @@ In particular, you will most likely want to set these properties on your locatio
 
 In your YAML blueprint:
 
-    ...
     location:
       jclouds:aws-ec2:
         region: us-west-2
@@ -53,17 +52,16 @@ In your YAML blueprint:
         hardwareId: m3.medium
         useJcloudsSshInit: false
         templateOptions: {mapNewVolumeToDeviceName: ["/dev/sda1", 100, true]}
-    ...
 
-Alternatively, you can define a new named location in `brooklyn.properties`:
+Or for an existing Windows machine:
 
-    brooklyn.location.named.AWS\ Oregon\ Win = jclouds:aws-ec2:us-west-2
-    brooklyn.location.named.AWS\ Oregon\ Win.displayName = AWS Oregon (Windows)
-    brooklyn.location.named.AWS\ Oregon\ Win.imageNameRegex = Windows_Server-2012-R2_RTM-English-64Bit-Base-.*
-    brooklyn.location.named.AWS\ Oregon\ Win.imageOwner = 801119661308
-    brooklyn.location.named.AWS\ Oregon\ Win.hardwareId = m3.medium
-    brooklyn.location.named.AWS\ Oregon\ Win.useJcloudsSshInit = false
-    brooklyn.location.named.AWS\ Oregon\ Win.templateOptions = {mapNewVolumeToDeviceName: ["/dev/sda1", 100, true]}
+    location:
+      byon:
+        hosts:
+        - winrm: 10.0.0.1
+          user: Administrator
+          password: pa55w0rd
+          osFamily: windows
 
 
 
@@ -79,34 +77,25 @@ Entity authors are strongly encouraged to write Windows PowerShell or Batch scri
 files, to configure these to be uploaded, and then to configure the appropriate command as a 
 single line that executes given script.
 
-For example - here is a simplified blueprint (but see [Tips and Tricks](#tips-and-tricks) below!):
+For example here is a simplified blueprint:
 
     name: Server with 7-Zip
 
-    location:
-      jclouds:aws-ec2:
-        region: us-west-2
-        identity: AKA_YOUR_ACCESS_KEY_ID
-        credential: <access-key-hex-digits>
-        imageNameRegex: Windows_Server-2012-R2_RTM-English-64Bit-Base-.*
-        imageOwner: 801119661308
-        hardwareId: m3.medium
-        useJcloudsSshInit: false
-        templateOptions: {mapNewVolumeToDeviceName: ["/dev/sda1", 100, true]}
+    location: windows-machine       # register this, or inject the above instead
 
     services:
     - type: org.apache.brooklyn.entity.software.base.VanillaWindowsProcess
       brooklyn.config:
         templates.preinstall:
-          file:///Users/richard/install7zip.ps1: "C:\\install7zip.ps1"
+          /path/to/install7zip.ps1: "C:\\install7zip.ps1"
         install.command: powershell -command "C:\\install7zip.ps1"
         customize.command: echo true
         launch.command: echo true
         stop.command: echo true
         checkRunning.command: echo true
         installer.download.url: http://www.7-zip.org/a/7z938-x64.msi
 
-The installation script - referred to as `/Users/richard/install7zip.ps1` in the example above - is:
+The installation script - referred to as `/path/to/install7zip.ps1` in the example above (but put this on your Brooklyn server or in the bundle classpath) - is:
 
     $Path = "C:\InstallTemp"
     New-Item -ItemType Directory -Force -Path $Path
@@ -118,10 +107,6 @@ The installation script - referred to as `/Users/richard/install7zip.ps1` in the
 
     Start-Process "msiexec" -ArgumentList '/qn','/i',$Dl -RedirectStandardOutput ( [System.IO.Path]::Combine($Path, "stdout.txt") ) -RedirectStandardError ( [System.IO.Path]::Combine($Path, "stderr.txt") ) -Wait
 
-Where security-related operation are to be executed, it may require the use of `CredSSP` to obtain
-the correct Administrator privileges: you may otherwise get an access denied error. See the sub-section 
-[How and Why to re-authenticate within a PowerShell script](#how-and-why-to-re-authenticate-within-a-powershell-script) for more details.
-
 This is only a very simple example. A more complex example can be found in the [Microsoft SQL Server blueprint in the
 Brooklyn source code]({{ site.brooklyn.url.git }}/software/database/src/main/resources/org/apache/brooklyn/entity/database/mssql).
 
@@ -602,11 +587,14 @@ entities are installed.
 Blueprint authors are strongly encourages to explicitly specific directories for file
 uploads and in their PowerShell scripts.
 
-### Windows template settings for an Unattended Installation
 
-Windows template needs certain configuration to be applied to prevent Windows setup UI from being displayed.
-The default behavior is to display it if there are incorrect or empty settings. Showing Setup UI will prevent the proper
-deployment, because it will expect interaction by the user such as agreeing on the license agreement or some of the setup dialogs.
+Learn More
+----------
+
+A few other WinRM resources are available:
 
-Detailed instruction how to prepare an Unattended installation are provided at [https://technet.microsoft.com/en-us/library/cc722411%28v=ws.10%29.aspx](https://technet.microsoft.com/en-us/library/cc722411%28v=ws.10%29.aspx).
+* [Tips and Tricks](tips.md)
+* [About the Winrm4j Client](client.md)
+* [Troubleshooting](troubleshoot.md)
+* [Limitations](limitations.md)
 
diff --git a/guide/blueprints/winrm/limitations.md b/guide/blueprints/winrm/limitations.md
@@ -0,0 +1,115 @@
+Known Limitations
+-----------------
+
+WinRM 2.0 supports encryption mechanisms on top of HTTP. However those are not supported in Apache Brooklyn.
+For production adoptions please make sure you follow Microsoft Guidelines https://msdn.microsoft.com/en-us/library/ee309366(v=vs.85).aspx
+
+### Apache Brooklyn limitations on using WinRM over HTTP and HTTPS
+
+By default Apache Brooklyn is currently using unencrypted HTTP for WinRM communication. It does not support encrypted HTTP for WinRM.
+
+HTTPS is supported but there is no mechanism of specifying which certificates to trust.
+Currently Apache Brooklyn will accept any certificate used in a HTTPS WinRM connection.
+
+### Incorrect Exit Codes
+
+Some limitations with WinRM (or at least the chosen WinRM Client!) are listed below:
+
+##### Single-line PowerShell files
+
+When a PowerShell file contains just a single command, the execution of that file over WinRM returns exit code 0
+even if the command fails! This is the case for even simple examples like `exit 1` or `thisFileDoesNotExist.exe`.
+
+A workaround is to add an initial command, for example:
+
+    Write-Host dummy line for workaround 
+    exit 1
+
+##### Direct Configuration of PowerShell commands
+
+If a command is directly configured with PowerShell that includes `exit`, the return code over WinRM
+is not respected. For example, the command below will receive an exit code of 0.
+
+    launch.powershell.command: |
+      echo first
+      exit 1
+
+##### Direct Configuration of Batch commands
+
+If a command is directly configured with a batch exit, the return code over WinRM
+is not respected. For example, the command below will receive an exit code of 0.
+
+    launch.command: exit /B 1
+
+##### Non-zero Exit Code Returned as One
+
+In some configurations, scripts can report any non-zero exit code as `1`.
+It may be possible to workaround this where the exit code is needeed by using
+e.g. `install.command: powershell -command "C:\\installmssql.ps1"`
+instead of `install.powershell.command: "C:\\installmssql.ps1"`
+If this is problematic, please consider submitting a patch to `VanillaWindowsProcess`!
+
+### PowerShell "Preparing modules for first use"
+
+The first command executed over WinRM has been observed to include stderr saying "Preparing 
+modules for first use", such as that below:
+
+    < CLIXML
+    <Objs Version="1.1.0.1" xmlns="http://schemas.microsoft.com/powershell/2004/04"><Obj S="progress" RefId="0"><TN RefId="0"><T>System.Management.Automation.PSCustomObject</T><T>System.Object</T></TN><MS><I64 N="SourceId">1</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj><Obj S="progress" RefId="1"><TNRef RefId="0" /><MS><I64 N="SourceId">2</I64><PR N="Record"><AV>Preparing modules for first use.</AV><AI>0</AI><Nil /><PI>-1</PI><PC>-1</PC><T>Completed</T><SR>-1</SR><SD> </SD></PR></MS></Obj></Objs>
+
+The command still succeeded. This has only been observed on private clouds (e.g. not on
+AWS). It could be related to the specific Windows images in use. It is recommended that 
+VM images are prepared carefully, e.g. so that security patches are up-to-date and the
+VM is suitably initialised.
+
+### WinRM executeScript failed: httplib.BadStatusLine: ''
+
+As described in https://issues.apache.org/jira/browse/BROOKLYN-173, a failure has been
+observed where the 10 attempts to execute the command over WinRM failed with:
+
+    httplib.BadStatusLine: ''
+
+Subsequently retrying the command worked. It is unclear what caused the failure, but could 
+have been that the Windows VM was not yet in the right state.
+
+One possible workaround is to ensure the Windows VM is in a good state for immediate use (e.g. 
+security updates are up-to-date). Another option is to increase the number of retries, 
+which defaults to 10. This is a configuration option on the machine location, so can be set on
+the location's brooklyn.properties or in the YAML: 
+
+    execTries: 20
+
+### Direct Configuration of Multi-line Batch Commands Not Executed
+
+If a command is directly configured with multi-line batch commands, then only the first line 
+will be executed. For example the command below will only output "first":
+
+    launch.command: |
+      echo first
+      echo second
+
+The workaround is to write a file with the batch commands, have that file uploaded, and execute it.
+
+Note this is not done automatically because that could affect the capture and returning
+of the exit code for the commands executed.
+
+### Install location
+
+Work is required to better configure a default install location on the VM (e.g. so that 
+environment variables are set). The installation pattern for Linux-based blueprints,
+of using brooklyn-managed-processes/installs, is not used or recommended on Windows.
+Files will be uploaded to C:\ if no explicit directory is supplied, which is untidy, 
+unnecessarily exposes the scripts to the user, and could cause conflicts if multiple 
+entities are installed.
+
+Blueprint authors are strongly encourages to explicitly specific directories for file
+uploads and in their PowerShell scripts.
+
+### Windows template settings for an Unattended Installation
+
+Windows template needs certain configuration to be applied to prevent Windows setup UI from being displayed.
+The default behavior is to display it if there are incorrect or empty settings. Showing Setup UI will prevent the proper
+deployment, because it will expect interaction by the user such as agreeing on the license agreement or some of the setup dialogs.
+
+Detailed instruction how to prepare an Unattended installation are provided at [https://technet.microsoft.com/en-us/library/cc722411%28v=ws.10%29.aspx](https://technet.microsoft.com/en-us/library/cc722411%28v=ws.10%29.aspx).
+