Add documentation for parsing test results.

Bug: 892307
Change-Id: I8153fb41b1c53fbf972a7fe3c2982d4145f8c95b
Reviewed-on: https://chromium-review.googlesource.com/c/1269079
Commit-Queue: Erik Chen <erikchen@chromium.org>
Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#597687}

docs/README.md

@@ -107,6 +107,8 @@ used when committed.
     builder migration for Chromium
 *   [Tour of Continuous Integration UI](tour_of_luci_ui.md) - A tour of our
     the user interface for LUCI, our continuous integration system
+*   [Parsing Test Results](parsing_test_results.md) - An introduction for how to
+    understand the results emitted by polygerrit and CI builds.
 *   [Closure Compilation](closure_compilation.md) - The _Closure_ JavaScript
     compiler
 *   [Threading and Tasks in Chrome](threading_and_tasks.md) - How to run tasks
@@ -290,7 +292,7 @@ used when committed.
     Enabling spoken feedback (ChromeVox) on desktop Linux.
 *   [Offscreen, Invisible and Size](accessibility/offscreen.md) - How Chrome
     defines offscreen, invisible and size in the accessibility tree.
-*   [Text to Speech](accessibility/tts.md) - Overview of text to speech in 
+*   [Text to Speech](accessibility/tts.md) - Overview of text to speech in
     Chrome and Chrome OS.
 *   [BRLTTY in Chrome OS](accessibility/brltty.md) - Chrome OS integration with
     BRLTTY to support refreshable braille displays

docs/parsing_test_results.md

@@ -0,0 +1,89 @@
+# Parsing Test Results
+
+Chromium runs over 500,000 tests for each CL. There are many layers of UI for
+parsing and interpreting these test results. This doc provides a brief guide
+for navigating these UI layers.
+
+## Polygerrit UI
+
+Tests are segmented by build and test configurations. The segments are usually
+referred to as *builds*. In the example below, each green and red rectangle
+refers to a *build*.
+
+![Example polygerrit build status](images/parsing_test_results_polygerrit.png)
+
+The name of each build usually contains enough information to get a rough idea
+of the configuration. Some examples:
+
+* *android_compile_dbg* is a compile-only [no tests] build of Chromium for
+  Android, using the *debug* configuration.
+* *android-kitkat-arm-rel* builds and runs tests for Chromium for Android,
+  using the *release* configuration on a kitkat device.
+* *win7_chromium_rel_ng* builds and runs tests for Chromium on Windows, using
+  the release configuration on a Windows 7 device. *ng* stands for next
+  generation, but this has no meaning as the previous generation was already
+  phased out.
+
+Green boxes refer to builds that passed. Red boxes refer to builds that failed.
+Some failed builds get automatically retried by the CQ. In this example,
+*linux_chromium_rel_ng* and *mac_chromium_rel_ng* were automatically retried
+[hence the two **X**s], but *win7_chromium_rel_ng* was not. The **X** on the
+left is the first build, and the **X** on the right is the second build.
+
+Each of these boxes is a link that provides more information about the build
+failure.
+
+## Build Results UI
+
+Selecting any of the build results from the previous section will navigate to
+the build results UI. Each build is implemented by a [recipe] --
+effectively a Python script. Each recipe is divided into *steps*. Each *step*
+represents a well-defined action, such as updating the repository to point to
+tip of tree, or compiling the necessary build artifacts.
+
+[recipe]: https://chromium.googlesource.com/external/github.com/luci/recipes-py/+/master/doc/user_guide.md
+
+![Example 1 Build Results UI](images/parsing_test_results_build_results_1.png)
+
+Under the **Steps and Logfiles** heading is a list of numbered *steps*. Each
+*step* has a color (red, green or purple) which indicates whether the *step*
+failed, succeeded, or encountered an unexpected condition. Failing steps are
+also grouped into the **Results** section at the very top for convenience.
+
+## Build Results UI -- Overview
+
+Most builds follow a similar pattern. The key *steps* are listed here.
+
+* **bot_update** Update the repository to point to tip of tree. Apply the CL
+  as a patch.
+* **analyze** Analyze dependencies of test suites to determine which test
+  suites are affected by the patch.
+* **compile (with patch)** Builds test suites and associated artifacts.
+* **isolate tests** Archives test suite binaries and artifacts.
+* **test_pre_run.[trigger] webkit_layout_tests (with patch)** Triggers a test
+  suite on swarming [remote execution framework] -- in this case,
+  webkit_layout_tests.
+* **webkit_layout_tests (with patch)** Collects the results from swarming for a
+  test suite.
+
+If all test suites pass, then the *build* is marked as a success and no further
+steps are run. If at least one test suite has failures, then the failing tests
+are rerun with the patch deapplied. This allows the recipe to determine if the
+test failure is due to the CL or due to a problem with tip of tree.
+
+* **bot_update [without patch]** Deapplies the CL patch.
+* **compile [without patch]** Compiles test suites.
+* **isolate tests (2)** Archives test suite binaries and artifacts.
+* **test_pre_run.[trigger] webkit_layout_tests (without patch)** Triggers test
+  suite on swarming. Only failing tests are rerun.
+* **webkit_layout_tests (without patch)** Collects results from swarming.
+
+**Important safety notice**. When test suites are run with the patch applied,
+each test is run up to N times -- any success will mark the test as a success.
+When test suites are run without the patch, each failing test is run exactly N
+times. Any failure will mark the test as a failure.
+
+If there are tests that failed with the patch applied, but not with the patch
+deapplied, then that implies that it's likely that the CL broke a test. Just to
+confirm, the first suite of steps is run again, this time with the suffix
+**(retry with patch)**.