[Fuchsia] Added more documentation for comparative_tester scripts

This CL adds a README.md file to the tools/fuchsia/comparative_tester/
directory, explaining how all the scripts interact with each other, and
how they should be used.

Change-Id: Ib9f668fb09f5fa1ba50d6da396251b2c8113d4a5
Reviewed-on: https://chromium-review.googlesource.com/1176366
Commit-Queue: Stephan Stross <stephanstross@google.com>
Reviewed-by: Sergey Ulanov <sergeyu@chromium.org>
Cr-Commit-Position: refs/heads/master@{#583814}

tools/fuchsia/comparative_tester/README.md

@@ -0,0 +1,77 @@
+# Comparative Testing Scripts for Fuchsia
+
+The collection of python scripts inside of `//tools/fuchsia/comparative_tester`
+exist to facilitate the automated building, deployment, and execution of tests
+on Linux and Fuchsia devices on the same LAN as the hosting PC which will run
+the scripts, as well as generating comparisons and valuable statistical data and
+displaying it in an easily viewed form.
+
+## Test Support
+The automated test building and launching should currently work for any test
+target within the base `chromium/src` directory. Work is also underway to
+support executing Catapult tests automatically, with the same stats collection
+capabilities.
+
+## Usage
+The general usage pattern for these scripts is as follows:
+1. Check `target_spec.py`. Make sure that the output directories are where you
+      want them, and that you are running the chromium test targets that you're
+      interested in collecting results from. Also check to make sure that the
+      specifications for the Linux and Fuchsia devices are appropriate for your
+      specific network and OS configuration.
+2. Execute `comparative_tester.py`, with any flags necessary to collect the data
+      of interest.
+3. Run `generate_perf_report.py`. This should require no extra configuration on
+      your part.
+4. View your results by loading `display_perf_results.html` in any web browser
+      and giving it the result JSONs in the output directory you specified.
+
+### test_results.py
+(_Non-Invokable_)
+
+This file is used internally to parse test output and return objects that can be
+manipulated easily by the rest of the code.
+
+### target_spec.py
+(_Non-Invokable_)
+
+This file contains constant definitions used by other files in this folder to
+represent what tests to run, where the output will live, where the test binaries
+live, and so on. To add more tests for automatic building and deploying, they
+should be added here.
+
+### comparative_tester.py
+_Invocation_: `comparative_tester.py --do-config --do-build --is-debug --num-repetitions=1`
+
+This is where tests are actually executed. It has four flags of note:
+  * `--do-config`: makes the test script to generate an args.gn file for
+        the output directory, and over-writes any existing `args.gn` file. This
+        option is off by default, and no files will be generated or changed.
+      * `--is-debug`: requires the do_config flag to be set as well. Makes
+            the args.gn files that the script will generate have the
+            `is_debug = true` line. This option is off by default, and will
+            cause the line `is_debug = false` to appear in the `args.gn` file
+  * `--do-build`: makes the test script build the targets specified inside of
+        `target_spec.py`. By default, this is off, and the targets will not be
+        rebuilt for the test cases.
+  * `--num-repetitions`: tells the script how many times to run each test in the
+        battery of tests specified in `target_spec.py`. By default, this is one,
+        so tests will only be executed one time each.
+More complex configuration options are present in `target_spec.py.`
+
+### generate_perf_report.py
+_Invocation_: `generate_perf_report.py`
+This script takes no command line flags, but works off of many of the same
+fields inside of `target_spec.py` that `comparative_tester.py` does, namely
+fields instructing it where the raw data lives, and where to place the generated
+statistics when it's finished generating them. It generates the mean, standard
+deviation, and coefficient of variance for each target, test, and individual
+line scraped from the test output, and writes them to appropriately named JSONs.
+
+### display_perf_results.html and .js
+The HTML file is just a thin shim around `display_perf_results.js` that can be
+opened in any web browser to view the data. Due to the isolation between
+javascript interpreters and the host's filesystem, the web page must be manually
+given the final JSON files that you want to display, at which point it will draw
+a table full of the data, and in a more human readable format. It accepts
+multiple files at a time.

tools/fuchsia/comparative_tester/comparative_tester.py

@@ -23,7 +23,20 @@
 
 
 def RunCommand(command: List[str], msg: str) -> str:
-  "One-shot start and complete command with useful default kwargs"
+  """Runs a command and returns the standard output.
+
+  Args:
+    command (List[str]): The list of command chunks to use in subprocess.run.
+        ex: ['git', 'grep', 'cat'] to find all instances of cat in a repo.
+    msg (str): An error message in case the subprocess fails for some reason.
+
+  Raises:
+    subprocess.SubprocessError: Raises this with the command that failed in the
+        event that the return code of the process is non-zero.
+
+  Returns:
+    str: the standard output of the subprocess.
+  """
   command = [piece for piece in command if piece != ""]
   proc = subprocess.run(
       command,
@@ -42,6 +55,16 @@ def RunCommand(command: List[str], msg: str) -> str:
 
 # TODO(crbug.com/848465): replace with --test-launcher-filter-file directly
 def ParseFilterFile(filepath: str) -> str:
+  """Takes a path to a filter file, parses it, and constructs a gtest_filter
+  string for test execution.
+
+  Args:
+    filepath (str): The path to the filter file to be parsed into a
+        --gtest_filter flag.
+
+  Returns:
+    str: The properly-joined together gtest_filter flag.
+  """
   positive_filters = []
   negative_filters = []
   with open(filepath, "r") as file:
@@ -77,14 +100,36 @@ def __init__(self, target: str) -> None:
       self._filter_flag = ParseFilterFile(self._filter_file)
 
   def ExecFuchsia(self, out_dir: str, run_locally: bool) -> str:
+    """Execute this test target's test on Fuchsia, either with QEMU or on actual
+    hardware.
+
+    Args:
+      out_dir (str): The Fuchsia output directory.
+      run_locally (bool): Whether to use QEMU(true) or a physical device(false)
+
+    Returns:
+      str: The standard output of the test process.
+    """
+
     runner_name = "{}/bin/run_{}".format(out_dir, self._name)
     command = [runner_name, self._filter_flag, "--exclude-system-logs"]
     if not run_locally:
       command.append("-d")
     return RunCommand(command,
-                      "Test {} failed on fuchsia!".format(self._target))
+                      "Test {} failed on Fuchsia!".format(self._target))
 
   def ExecLinux(self, out_dir: str, run_locally: bool) -> str:
+    """Execute this test target's test on Linux, either with QEMU or on actual
+    hardware.
+
+    Args:
+      out_dir (str): The Linux output directory.
+      run_locally (bool): Whether to use the host machine(true) or a physical
+          device(false)
+
+    Returns:
+      str: The standard output of the test process.
+    """
     command = []  # type: List[str]
     user = target_spec.linux_device_user
     ip = target_spec.linux_device_ip
@@ -102,6 +147,14 @@ def ExecLinux(self, out_dir: str, run_locally: bool) -> str:
     return RunCommand(command, "Test {} failed on linux!".format(self._target))
 
   def TransferDependencies(self, out_dir: str, host: str):
+    """Transfer the dependencies of this target to the machine to execute the
+    test.
+
+    Args:
+      out_dir (str): The output directory to find the dependencies in.
+      host (str): The IP address of the host to receive the dependencies.
+    """
+
     gn_desc = ["gn", "desc", out_dir, self._target, "runtime_deps"]
     out = RunCommand(
         gn_desc, "Failed to get dependencies of target {}".format(self._target))
@@ -152,6 +205,17 @@ def TransferDependencies(self, out_dir: str, host: str):
 
 
 def RunTest(target: TestTarget, run_locally: bool = False) -> None:
+  """Run the given TestTarget on both Linux and Fuchsia
+
+  Args:
+    target (TestTarget): The TestTarget to run.
+    run_locally (bool, optional): Defaults to False. Whether the test should be
+        run on the host machine, or sent to remote devices for execution.
+
+  Returns:
+    None: Technically an IO (), as it writes to the results files
+  """
+
   linux_out = target.ExecLinux(target_spec.linux_out_dir, run_locally)
   linux_result = test_results.TargetResultFromStdout(linux_out.splitlines(),
                                                      target._name)
@@ -168,6 +232,19 @@ def RunTest(target: TestTarget, run_locally: bool = False) -> None:
 
 
 def RunGnForDirectory(dir_name: str, target_os: str, is_debug: bool) -> None:
+  """Create the output directory for test builds for an operating system.
+
+  Args:
+    dir_name (str): The name to use for the output directory. This will be
+        created if it does not exist.
+    target_os (str): The operating system to initialize this directory for.
+    is_debug (bool): Whether or not this is a debug build of the tests in
+        question.
+
+  Returns:
+    None: It has a side effect of replacing args.gn
+  """
+
   if not os.path.exists(dir_name):
     os.makedirs(dir_name)
 
@@ -185,6 +262,16 @@ def RunGnForDirectory(dir_name: str, target_os: str, is_debug: bool) -> None:
 
 def GenerateTestData(do_config: bool, do_build: bool, num_reps: int,
                      is_debug: bool):
+  """Initializes directories, builds test targets, and repeatedly executes them
+  on both operating systems
+
+  Args:
+    do_config (bool): Whether or not to run GN for the output directories
+    do_build (bool): Whether or not to run ninja for the test targets.
+    num_reps (int): How many times to run each test on a given device.
+    is_debug (bool): Whether or not this should be a debug build of the tests.
+  """
+
   DIR_SOURCE_ROOT = os.path.abspath(
       os.path.join(os.path.dirname(__file__), *([os.pardir] * 3)))
   os.chdir(DIR_SOURCE_ROOT)

tools/fuchsia/comparative_tester/display_perf_results.js

@@ -7,6 +7,7 @@ const headers = ["Test", "Line", "Units", "Fuchsia Avg", "Fuchsia Dev",
                        "F-L Avgs"];
 
 function generateHeader() {
+  // Generates the header for a table of statistics.
   let header_row = document.createElement("tr");
   for (let i = 0; i < headers.length; i++){
     let header = document.createElement("th");
@@ -17,6 +18,8 @@ function generateHeader() {
 }
 
 function generateTableHtmlFromOutputRows(output_rows) {
+  // Takes a list of rows of data, and collects them together into a single
+  // table.
   let table = document.createElement("table");
   table.appendChild(generateHeader());
   output_rows.forEach(function(row){
@@ -32,6 +35,7 @@ function generateTableHtmlFromOutputRows(output_rows) {
 }
 
 function extractStats(line) {
+  // Deconstructs the JSON file given into a list of the relevant values.
   return [line.fuchsia_avg,
           line.fuchsia_dev,
           line.fuchsia_cv,
@@ -42,6 +46,8 @@ function extractStats(line) {
 }
 
 function renderTable(obj_dict) {
+  // Returns an HTML table object by taking the TargetResults JSON and using it
+  // to populate a table of statistics.
   let rows = []
 
   const name = obj_dict['name']
@@ -64,6 +70,8 @@ function renderTable(obj_dict) {
 }
 
 function loadTable() {
+  // Reads the result files, and adds the tables generated to the table_div HTML
+  // object as they finish loading.
   let files = document.getElementById('files').files;
   let table_div = document.getElementById('stats_div');
   // Clear the table div of all prior stats tables