The collection of python scripts inside of //tools/fuchsia/comparative_tester
exist to facilitate the automated building, deployment, and execution of tests
on Linux and Fuchsia devices on the same LAN as the hosting PC which will run
the scripts, as well as generating comparisons and valuable statistical data and
displaying it in an easily viewed form.
The automated test building and launching should currently work for any test
target within the base chromium/src
directory. Work is also underway to
support executing Catapult tests automatically, with the same stats collection
capabilities.
The general usage pattern for these scripts is as follows:
- Check
target_spec.py
. Make sure that the output directories are where you want them, and that you are running the chromium test targets that you're interested in collecting results from. Also check to make sure that the specifications for the Linux and Fuchsia devices are appropriate for your specific network and OS configuration. - Execute
comparative_tester.py
, with any flags necessary to collect the data of interest. - Run
generate_perf_report.py
. This should require no extra configuration on your part. - View your results by loading
display_perf_results.html
in any web browser and giving it the result JSONs in the output directory you specified.
(Non-Invokable)
This file is used internally to parse test output and return objects that can be manipulated easily by the rest of the code.
(Non-Invokable)
This file contains constant definitions used by other files in this folder to represent what tests to run, where the output will live, where the test binaries live, and so on. To add more tests for automatic building and deploying, they should be added here.
Invocation: comparative_tester.py --do-config --do-build --is-debug --num-repetitions=1
This is where tests are actually executed. It has four flags of note:
--do-config
: makes the test script to generate an args.gn file for the output directory, and over-writes any existingargs.gn
file. This option is off by default, and no files will be generated or changed.--is-debug
: requires the do_config flag to be set as well. Makes the args.gn files that the script will generate have theis_debug = true
line. This option is off by default, and will cause the lineis_debug = false
to appear in theargs.gn
file
--do-build
: makes the test script build the targets specified inside oftarget_spec.py
. By default, this is off, and the targets will not be rebuilt for the test cases.--num-repetitions
: tells the script how many times to run each test in the battery of tests specified intarget_spec.py
. By default, this is one, so tests will only be executed one time each.--gtest_filter
: Works just like it does on the tests. Positive filters go before the '-' and negative ones after. All individual filters are separated from each other via a ':'. Support globbing.- Ex:
--gtest_filter=TestFoo.Case1:TestBar.*-TestBaz.*:TestQuux.Case2
filters out all cases in TestBaz, Case2 of TestQuux, and only runs TestFoo.Case1, and all cases in TestBar.
- Ex:
More complex configuration options are present in target_spec.py.
Invocation: generate_perf_report.py
This script takes no command line flags, but works off of many of the same
fields inside of target_spec.py
that comparative_tester.py
does, namely
fields instructing it where the raw data lives, and where to place the generated
statistics when it's finished generating them. It generates the mean, standard
deviation, and coefficient of variance for each target, test, and individual
line scraped from the test output, and writes them to appropriately named JSONs.
The HTML file is just a thin shim around display_perf_results.js
that can be
opened in any web browser to view the data. Due to the isolation between
javascript interpreters and the host's filesystem, the web page must be manually
given the final JSON files that you want to display, at which point it will draw
a table full of the data, and in a more human readable format. It accepts
multiple files at a time.