Skip to content

Commit

Permalink
Clarify artifact documentation for test results
Browse files Browse the repository at this point in the history
This clarifies the meaning of forward slashes in artifact file paths,
and also clarifies the directory these paths are relative to, when no
artifact permanent location is set.

Bug: 775619
Change-Id: I1f41d86273455123823350a39b239cc04b6a0216
Reviewed-on: https://chromium-review.googlesource.com/802092
Commit-Queue: Stephen Martinis <martiniss@chromium.org>
Reviewed-by: Ned Nguyen <nednguyen@google.com>
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Cr-Commit-Position: refs/heads/master@{#521525}
  • Loading branch information
martiniss authored and Commit Bot committed Dec 4, 2017
1 parent 8e7c5e4 commit e1e5bf9
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/testing/json_test_results_format.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ results for each invocation in the `actual` field.
|-------------|-----------|-------------|
| `actual` | string | **Required.** An ordered space-separated list of the results the test actually produced. `FAIL PASS` means that a test was run twice, failed the first time, and then passed when it was retried. If a test produces multiple different results, then it was actually flaky during the run. |
| `expected` | string | **Required.** An unordered space-separated list of the result types expected for the test, e.g. `FAIL PASS` means that a test is expected to either pass or fail. A test that contains multiple values is expected to be flaky. |
| `artifacts` | dict | **Optional.** A dictionary describing test artifacts generated by the execution of the test. The dictionary maps the name of the artifact (`screenshot`, `crash_log`) to a list of relative locations of the artifact (`screenshot/page.png`, `logs/crash.txt`). There is one entry in the list per test execution. If `artifact_permanent_location` is specified, then this location is relative to that path. Otherwise, it is assumed that this test file is in a known location by whatever is processing this test file, and so the location is relative to a known directory. |
| `artifacts` | dict | **Optional.** A dictionary describing test artifacts generated by the execution of the test. The dictionary maps the name of the artifact (`screenshot`, `crash_log`) to a list of relative locations of the artifact (`screenshot/page.png`, `logs/crash.txt`). Any '/' characters in the file paths are meant to be platform agnostic; tools will replace them with the appropriate per platform path separators. There is one entry in the list per test execution. If `artifact_permanent_location` is specified, then this location is relative to that path. Otherwise, the path is assumed to be relative to the location of the json file which contains this. |
| `bugs` | string | **Optional.** A comma-separated list of URLs to bug database entries associated with each test. |
| `is_unexpected` | bool | **Optional.** If present and true, the failure was unexpected (a regression). If false (or if the key is not present at all), the failure was expected and will be ignored. |
| `time` | float | **Optional.** If present, the time it took in seconds to execute the first invocation of the test. |
Expand Down

0 comments on commit e1e5bf9

Please sign in to comment.