Re-land "Update docs and command line flags for MB."

This patch re-lands {#338951}

  https://crrev.com/747cb269970294b7928ca30a3cde192c90072d13

with the necessary fix (a missing command-line arg).

TBR=maruel@chromium.org

Review URL: https://codereview.chromium.org/1234343005

Cr-Commit-Position: refs/heads/master@{#339164}

tools/mb/README.md

@@ -18,5 +18,5 @@ one place, in `//tools/mb/mb_config.pyl`.
 
 For more information, see:
 
-* [the user guide](docs/user_guide.md)
-* [the design spec](docs/design_spec.md)
+* [The User Guide](docs/user_guide.md)
+* [The Design Spec](docs/design_spec.md)

tools/mb/docs/README.md

@@ -0,0 +1,4 @@
+# The MB (Meta-Build wrapper) documentation
+
+* The [User Guide](user_guide.md)
+* The [Design Spec](design_spec.md)

tools/mb/docs/design_spec.md

@@ -20,6 +20,9 @@ Ninja files, and the `analyze` step that takes a list of modified files
 and a list of targets to build and returns which targets are affected by
 the files.
 
+For more information on how to actually use MB, see
+[the user guide](user_guide.md).
+
 ## Design
 
 MB is intended to be as simple as possible, and to defer as much work as
@@ -122,7 +125,7 @@ This design allows us to determine easily all of the different sets
 of flags that we need to support, but *not* which flags are used on which
 host/target combinations.
 
-It may be that we should really track the latter. Doing so is just a 
+It may be that we should really track the latter. Doing so is just a
 config file change, however.
 
 ### Non-goals
@@ -137,7 +140,7 @@ config file change, however.
   the GYP->GN migration is done, and so we should not add things for
   developers that can't easily be added to GN itself.
 
-* MB is not intended to replace the 
+* MB is not intended to replace the
   [CR tool](https://code.google.com/p/chromium/wiki/CRUserManual). Not
   only is it only intended to replace the gyp\_chromium part of `'gclient
   runhooks'`, it is not really meant as a developer-facing tool.
@@ -147,7 +150,7 @@ config file change, however.
 * Some common flags (goma\_dir being the obvious one) may need to be
   specified via the user, and it's unclear how to integrate this with
   the concept of build\_configs.
-  
+
   Right now, MB has hard-coded support for a few flags (i.e., you can
   pass the --goma-dir flag, and it will know to expand "${goma\_dir}" in
   the string before calling out to the tool. We may want to generalize

tools/mb/docs/user_guide.md

@@ -14,26 +14,11 @@ that Chromium is built in.
 
 Ideally this tool will no longer be needed after the migration is complete.
 
-## `mb gen`
+For more discussion of MB, see also [the design spec](design_spec.md).
 
-`mb gen` is responsible for generating the Ninja files by invoking either GYP
-or GN as appropriate. It takes arguments to specify a build config and
-a directory, then runs GYP or GN as appropriate:
-
-```
-% mb gen -m tryserver.chromium.linux -b linux_rel //out/Release
-% mb gen -c linux_rel_trybot //out/Release
-```
-
-Either the `-c/--config` flag or the `-m/--master` and `-b/--builder` flags
-must be specified so that `mb` can figure out which config to use.
-
-The path must be a GN-style "source-absolute" path (as above).
-
-If gen ends up using GYP, the path must have a valid GYP configuration as the
-last component of the path (i.e., specify `//out/Release_x64`, not `//out`).
+## MB subcommands
 
-## `mb analyze`
+### `mb analyze`
 
 `mb analyze` is reponsible for determining what targets are affected by
 a list of files (e.g., the list of files in a patch on a trybot):
@@ -45,7 +30,6 @@ mb analyze -c chromium_linux_rel //out/Release input.json output.json
 Either the `-c/--config` flag or the `-m/--master` and `-b/--builder` flags
 must be specified so that `mb` can figure out which config to use.
 
-
 The first positional argument must be a GN-style "source-absolute" path
 to the build directory.
 
@@ -71,27 +55,93 @@ fields:
     * `"Found dependency (all)"` (build everything, in which case
       `targets` and `build_targets` are not returned).
 
-## `mb help`
+The `-b/--builder`, `-c/--config`, `-f/--config-file`, `-m/--master`,
+`-q/--quiet`, and `-v/--verbose` flags work as documented for `mb gen`.
+
+
+### `mb gen`
+
+`mb gen` is responsible for generating the Ninja files by invoking either GYP
+or GN as appropriate. It takes arguments to specify a build config and
+a directory, then runs GYP or GN as appropriate:
+
+```
+% mb gen -m tryserver.chromium.linux -b linux_rel //out/Release
+% mb gen -c linux_rel_trybot //out/Release
+```
+
+Either the `-c/--config` flag or the `-m/--master` and `-b/--builder` flags
+must be specified so that `mb` can figure out which config to use.
+
+By default, MB will look in `//tools/mb/mb_config.pyl` to look up the config
+information, but you can specify a custom config file using the
+`-f/--config-file` flag.
+
+The path must be a GN-style "source-absolute" path (as above).
+
+You can pass the `-n/--dryrun` flag to mb gen to see what will happen without
+actually writing anything.
+
+You can pass the `-q/--quiet` flag to get mb to be silent unless there is an
+error, and pass the `-v/--verbose` flag to get mb to log all of the files
+that are read and written, and all the commands that are run.
+
+If the build config will use the Goma distributed-build system, you can pass
+the path to your Goma client in the `-g/--goma-dir` flag, and it will be
+incorporated into the appropriate flags for GYP or GN as needed.
+
+If gen ends up using GYP, the path must have a valid GYP configuration as the
+last component of the path (i.e., specify `//out/Release_x64`, not `//out`).
+
+### `mb help`
 
 Produces help output on the other subcommands
 
-## `mb lookup`
+### `mb lookup`
 
 Prints what command will be run by `mb gen` (like `mb gen -n` but does
 not require you to specify a path).
 
-## `mb validate`
+The `-b/--builder`, `-c/--config`, `-f/--config-file`, `-m/--master`,
+`-q/--quiet`, and `-v/--verbose` flags work as documented for `mb gen`.
+
+### `mb validate`
 
 Does internal checking to make sure the config file is syntactically
 valid and that all of the entries are used properly. It does not validate
 that the flags make sense, or that the builder names are legal or
 comprehensive, but it does complain about configs and mixins that aren't
 used.
 
+The `-f/--config-file` and `-q/--quiet` flags work as documented for
+`mb gen`.
+
 This is mostly useful as a presubmit check and for verifying changes to
 the config file.
 
-## mb_config.pyl
+## Isolates and Swarming
+
+`mb gen` is also responsible for generating the `.isolate` and
+`.isolated.gen.json` files needed to run test executables through swarming
+in a GN build (in a GYP build, this is done as part of the compile step).
+
+If you wish to generate the isolate files, pass `mb gen` the
+`--swarming-targets-file` command line argument; that arg should be a path
+to a file containing a list of ninja build targets to compute the runtime
+dependencies for (on Windows, use the ninja target name, not the file, so
+`base_unittests`, not `base_unittests.exe`).
+
+MB will take this file, translate each build target to the matching GN
+label (e.g., `base_unittests` -> `//base:base_unittests`, write that list
+to a file called `runtime_deps` in the build directory, and pass that to
+`gn gen $BUILD ... --runtime-deps-list-file=$BUILD/runtime_deps`.
+
+Once GN has computed the lists of runtime dependencies, MB will then
+look up the command line for each target (currently this is hard-coded
+in [mb.py](https://code.google.com/p/chromium/codesearch?q=mb.py#chromium/src/tools/mb/mb.py&q=mb.py%20GetIsolateCommand&sq=package:chromium&type=cs)), and write out the
+matching `.isolate` and `.isolated.gen.json` files.
+
+## The `mb_config.pyl` config file
 
 The `mb_config.pyl` config file is intended to enumerate all of the
 supported build configurations for Chromium. Generally speaking, you
@@ -178,7 +228,7 @@ translate into a call to `gyp_chromium -G Release` with `GYP_DEFINES` set to
 (From that you can see that mb is intentionally dumb and does not
 attempt to de-dup the flags, it lets gyp do that).
 
-## Debugging (-v/--verbose and -n/--dry-run)
+## Debugging MB
 
 By design, MB should be simple enough that very little can go wrong.
 

tools/mb/mb.py

@@ -65,8 +65,8 @@ def AddCommonOptions(subp):
                         help='Do a dry run (i.e., do nothing, just print '
                              'the commands that will run)')
       subp.add_argument('-q', '--quiet', action='store_true',
-                        help='Do not print anything, just return an exit '
-                             'code.')
+                        help='Do not print anything on success, '
+                             'just return an exit code.')
       subp.add_argument('-v', '--verbose', action='count',
                         help='verbose logging (may specify multiple times).')
 
@@ -108,7 +108,13 @@ def AddCommonOptions(subp):
 
     subp = subps.add_parser('validate',
                             help='validate the config file')
-    AddCommonOptions(subp)
+    subp.add_argument('-f', '--config-file', metavar='PATH',
+                      default=self.default_config,
+                      help='path to config file '
+                          '(default is //tools/mb/mb_config.pyl)')
+    subp.add_argument('-q', '--quiet', action='store_true',
+                      help='Do not print anything on success, '
+                           'just return an exit code.')
     subp.set_defaults(func=self.CmdValidate)
 
     subp = subps.add_parser('help',