diff --git a/docs/testing/web_tests.md b/docs/testing/web_tests.md index faa428ca938798..33963609749a40 100644 --- a/docs/testing/web_tests.md +++ b/docs/testing/web_tests.md @@ -217,7 +217,7 @@ on this. There are two ways to run web tests with additional command-line arguments: -* Using `--additional-driver-flag`: +* Using `--additional-driver-flag` or `--flag-specific`: ```bash third_party/blink/tools/run_web_tests.py --additional-driver-flag=--blocking-repaint @@ -300,8 +300,15 @@ There are two ways to run web tests with additional command-line arguments: with the flags without creating any virtual tests. For flags whose implementation is still in progress, virtual test suites and -flag-specific expectations represent two alternative strategies for testing. -Consider the following when choosing between them: +flag-specific expectations represent two alternative strategies for testing both +the enabled code path and not-enabled code path. They are preferred to only +setting a [runtime enabled feature](../../third_party/blink/renderer/platform/RuntimeEnabledFeatures.md) +to `status: "test"` if the feature has substantially different code path from +production because the latter would cause loss of test coverage of the production +code path. + +Consider the following when choosing between virtual test suites and +flag-specific expectations: * The [waterfall builders](https://dev.chromium.org/developers/testing/chromium-build-infrastructure/tour-of-the-chromium-buildbot) @@ -309,7 +316,8 @@ Consider the following when choosing between them: will run all virtual test suites in addition to the non-virtual tests. Conversely, a flag-specific expectations file won't automatically cause the bots to test your flag - if you want bot coverage without virtual test suites, - you will need to set up a dedicated bot for your flag. + you will need to set up a dedicated bot ([example](https://chromium-review.googlesource.com/c/chromium/src/+/1850255)) + for your flag. * Due to the above, virtual test suites incur a performance penalty for the commit queue and the continuous build infrastructure. This is exacerbated by diff --git a/third_party/blink/renderer/platform/RuntimeEnabledFeatures.md b/third_party/blink/renderer/platform/RuntimeEnabledFeatures.md index d49533e69b46f3..fa98c8494b13b1 100644 --- a/third_party/blink/renderer/platform/RuntimeEnabledFeatures.md +++ b/third_party/blink/renderer/platform/RuntimeEnabledFeatures.md @@ -14,7 +14,7 @@ Example: ``` The status of the feature controls when it will be enabled in the Blink engine. -| Status Value | Web feature enabled during web tests with content_shell [1] | Web feature enabled as part of web experimental features [2] | Web feature enabled in stable release | Non-web exposed feature enabled through a command line flag [3] +| Status Value | Web feature enabled during [web tests] with content_shell [1] | Web feature enabled as part of web experimental features [2] | Web feature enabled in stable release | Non-web exposed feature enabled through a command line flag [3] |:---:|:---:|:---:|:---:|:---:| | | No | No | No | Yes | | `test` | Yes | No | No | No | @@ -134,16 +134,17 @@ interface ExistingObject { **Warning:** You will not be able to change the enabled state of these at runtime as the V8 object templates definitions are created during start up and will not be updated during runtime. ## Web Tests (JavaScript) -Test whether a feature is enabled using: + +In [web tests], you can test whether a feature is enabled using: ```javascript internals.runtimeFlags.amazingNewFeatureEnabled ``` -This attribute is read only and cannot be changed. +This attribute is read only and cannot be changed, unless `settable_from_internals: true` is specified for the feature. -**Note:** The `internals` JavaScript API is only available in ContentShell for use by web tests and does not appear in released versions of Chromium. +**Note:** The `internals` JavaScript API is only available in content_shell for use by web tests and does not appear in Chromium. In content_shell's browser mode, `--expose-internals-for-testing` is needed to have the `internals` JavaScript API. ### Running Web Tests -When content_shell is run with `--stable-release-mode` flag, test-only features (ones listed in [runtime_enabled_features.json5] with `test`) are turned off. +When content_shell is run for web tests with `--stable-release-mode` flag, test-only and experimental features (ones listed in [runtime_enabled_features.json5] with `status: "test"` or `status: "experimental"`) are turned off. The [virtual/stable] suite runs with the flag, which is one of the ways to ensure test coverage of production code path for these features. ## Generated Files [renderer/build/scripts/make_runtime_features.py][make_runtime_features.py] uses [runtime_enabled_features.json5] to generate: @@ -173,7 +174,7 @@ After applying most other feature settings, the features requested feature setti https://groups.google.com/a/chromium.org/d/msg/blink-dev/JBakhu5J6Qs/re2LkfEslTAJ - +[web tests]: [supportedPlatforms]: [cssProperties]: [virtual test suite]: @@ -187,3 +188,4 @@ https://groups.google.com/a/chromium.org/d/msg/blink-dev/JBakhu5J6Qs/re2LkfEslTA [runtime_enabled_features.json5]: [make_internal_runtime_flags.py]: [code_generator_v8.py]: +[virtual/stable]: