Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More Test System documentation #106899

Merged
merged 1 commit into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Add more docs on the test system
  • Loading branch information
jkoritzinsky committed Aug 23, 2024
commit 9419a287ca838cb0f012a6a31a89dad864ff03c8
2 changes: 2 additions & 0 deletions docs/workflow/ci/disabling-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ in the [issues.targets](../../../src/tests/issues.targets) file. Additionally, t
link to a GitHub issue in the `<Issue>` element. Disabling a test here can be conditioned on processor
architecture, runtime, and operating system.

### Disabling runtime tests (src/tests) with test configuration properties

However, some test configurations must be disabled by editing the `.csproj` or `.ilproj` file for the test,
and inserting a property in a `<PropertyGroup>`, as follows:

Expand Down
31 changes: 30 additions & 1 deletion docs/workflow/testing/coreclr/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ On macOS and Linux:

We have multiple different mechanisms of executing tests.

Our test entrypoints are generally what we call "merged test runners", as they provide an executable runner project for multiple different test assemblies. These projects can be identified by the `<Import Project="$(TestSourceDir)MergedTestRunner.targets" />` line in their .csproj file. These projects provide a simple experience for running tests. When executing a merged runner project, it will run each test sequentially and record if it passes or fails in an xunit results file. The merged test runner support runtime test filtering. If specified, the first argument to the test runner is treated as a `dotnet test --filter` argument following the xUnit rules in their documentation. Today, the runner only supports the simple form, a substring of a test's fully-qualified name, in the format `Namespace.ContainingTypeName.TypeName.Method`. If support for further filtering options is desired, please open an issue requesting it.
Our test entrypoints are generally what we call "merged test runners", as they provide an executable runner project for multiple different test assemblies. These projects can be identified by the `<Import Project="$(TestSourceDir)MergedTestRunner.targets" />` line in their .csproj file. These projects provide a simple experience for running tests. When executing a merged runner project, it will run each test sequentially and record if it passes or fails in an xunit results file. The merged test runner support runtime test filtering. If specified, the first argument to the test runner is treated as a `dotnet test --filter` argument following the xUnit rules in their documentation. Today, the runner only supports the simple form, a substring of a test's fully-qualified name, in the format `Namespace.ContainingTypeName.TypeName.Method`. Additionally, tests can be filtered using the `FullyQualifiedName=Namespace.ContainingTypeName.TypeName.Method` syntax or the `DisplayName=TestDisplayName` syntax. The display name of a test is the name printed out on the console when the test runs. Additionally, a `~` can be used instead of an `=` to specify a substring search. If support for further filtering options is desired, please open an issue requesting it.

Some tests need to be run in their own process as they interact with global process state, they have a custom test entrypoint, or they interact poorly with other tests in the same process. These tests are generally marked with `<RequiresProcessIsolation>true</RequiresProcessIsolation>` in their project files. These tests can be run directly, but they can also be invoked through their corresponding merged test runner. The merged test runner will invoke them as a subprocess in the same manner as if they were run individually.

Expand All @@ -150,6 +150,35 @@ To filter tests on a merged test runner built as standalone, you can set the `Te

If you wish to use the Standalone runner described in the [previous section](#the-standalone-test-runner-and-build-time-test-filtering), you can set the `BuildAllTestsAsStandalone` environment variable to `true` when invoking the `./src/tests/build.sh` or `./src/tests/build.cmd` scripts (for example, `export BuildAllTestsAsStandalone=true` or `set BuildAllTestsAsStandalone=true`). This will build all tests that are not directly in a merged test runner's project as separate executable tests and build only the tests that are compiled into the runner directly. If a runner has no tests that are built directly into the runner, then it will be excluded.

### I added a test, which project do I run to run it?

Now that we run multiple tests in a single process, determining which project corresponds to the test to run can be a bit tricky. Here's some basic steps to determine which project to run to execute a test:

1. Look at the project file.

If the project file has `<RequiresProcessIsolation>true</RequiresProcessIsolation>` or `<Import Project="$(TestSourceDir)MergedTestRunner.targets" />`, then to run the test, you should build this project and run the `.cmd` or `.sh` script that corresponds to this project file.

2. Look at .csproj files in parent directories.

In a parent directory, you will find a `.csproj` file marked with `<Import Project="$(TestSourceDir)MergedTestRunner.targets" />`. In that project file, you'll see one or more `MergedTestProjectReference` items. If one of the glob patterns in the `Include` attribute matches and the `Exclude` attribute on the same item, then the test is included in this merged test runner. To run the test, you should build this merged runner project and run the `.cmd` or `.sh` script that corresponds to this project file. You can filter the tests in this runner using the instructions in the [Test Executors section](#test-executors).

### When to make a test RequiresProcessIsolation

The following are common reasons to mark a test as requiring process isolation:

- The test manipulates process-wide state, such as setting environment variables or changing the current directory.
- The test requires the ability to parse command line arguments.
- The test needs a custom main method.
- The test requires special information, such as an app manifest, in its executable.
- The test launches through a native executable.
- The test sets one of the configuration properties that are checked in the test run scripts, such as those in [test-configuration.md](test-configuration.md#adding-test-guidelines).

When a test is marked as `<RequiresProcessIsolation>true</RequiresProcessIsolation>`, it will be run in its own process and have its own `.cmd` and `.sh` scripts generated as test entrypoints. In CI, it will be executed as out of process by whichever merged test runner it is referenced by.

#### Main methods in RequiresProcessIsolation tests

If a custom main is not provided, the test can use `[Fact]` and `[Theory]` attributes internally. The test will use the "standalone" generator to create the test entrypoint. If you want to provide your own `Main` method for your test, set the `<ReferenceXUnitWrapperGenerator>false</ReferenceXUnitWrapperGenerator>` property in the test project file.

### Building C++/CLI Native Test Components Against the Live Ref Assemblies

By default, the _C++/CLI_ native test components build against the _ref pack_ from the SDK specified in the `global.json` file in the root of the repository. To build these components against the _ref assemblies_ produced in the build, pass the `-cmakeargs -DCPP_CLI_LIVE_REF_ASSEMBLIES=1` parameters to the test build. For example:
Expand Down