diff --git a/docs/workflow/testing/libraries/testing-android.md b/docs/workflow/testing/libraries/testing-android.md index 23b6e07d92414..e3c1c4e79c38d 100644 --- a/docs/workflow/testing/libraries/testing-android.md +++ b/docs/workflow/testing/libraries/testing-android.md @@ -1,18 +1,23 @@ # Testing Libraries on Android +## Prerequisites + The following dependencies should be installed in order to be able to run tests: +- OpenJDK - Android NDK - Android SDK -- OpenJDK -- OpenSSL + +To manage the dependencies, you can install them via terminal or using Android Studio. + +### Using a terminal OpenJDK can be installed on Linux (Ubuntu) using `apt-get`: ```bash sudo apt-get install openjdk-8 zip unzip ``` -Android SDK, NDK and OpenSSL can be automatically installed via the following script: +Android SDK and NDK can be automatically installed via the following script: ```bash #!/usr/bin/env bash set -e @@ -21,7 +26,6 @@ NDK_VER=r21b SDK_VER=6200805_latest SDK_API_LEVEL=29 SDK_BUILD_TOOLS=29.0.3 -OPENSSL_VER=1.1.1g-alpha-1 if [[ "$OSTYPE" == "darwin"* ]]; then HOST_OS=darwin @@ -47,8 +51,21 @@ yes | ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROI ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROID_SDK_ROOT} "platform-tools" "platforms;android-${SDK_API_LEVEL}" "build-tools;${SDK_BUILD_TOOLS}" ``` +### Using Android Studio + +Android Studio offers a convenient UI: +- to install all the dependencies; +- to manage android virtual devices; +- to make easy use of adb logs. + ## Building Libs and Tests for Android +Before running a build you might want to set the Android SDK and NDK environment variables: +``` +export ANDROID_SDK_ROOT= +export ANDROID_NDK_ROOT= +``` + Now we're ready to build everything for Android: ``` ./build.sh mono+libs -os Android -arch x64 @@ -66,8 +83,35 @@ The following shows how to run tests for a specific library ./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64 ``` +### Running the functional tests + +There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform. + +A functional test can be run the same way as any library test suite, e.g.: +``` +./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj +``` + +Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. + +### Testing various configurations + +It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoForceInterpreter`, and some more. + +1. AOT + +To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoForceInterpreter=false` to a build command. + +2. AOT-LLVM + +To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoForceInterpreter=false /p:MonoEnableLLVM=true` to a build command. + +3. Interpreter + +To build for Interpreter mode, add `/p:RunAOTCompilation=false /p:MonoForceInterpreter=true` to a build command. + ### Test App Design -Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test +Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed. ### Obtaining the logs @@ -75,7 +119,7 @@ XHarness for Android doesn't talk much and only saves test results to a file. Ho ``` adb logcat -s "DOTNET" ``` -Or simply open `logcat` window in Android Studio or Visual Stuido. +Or simply open `logcat` window in Android Studio or Visual Studio. ### AVD Manager If Android Studio is installed, [AVD Manager](https://developer.android.com/studio/run/managing-avds) can be used from the IDE to create and start Android virtual devices. Otherwise, the Android SDK provides the [`avdmanager` command line tool](https://developer.android.com/studio/command-line/avdmanager). diff --git a/docs/workflow/testing/libraries/testing-apple.md b/docs/workflow/testing/libraries/testing-apple.md index 99adf6a87176b..c303fc737362d 100644 --- a/docs/workflow/testing/libraries/testing-apple.md +++ b/docs/workflow/testing/libraries/testing-apple.md @@ -1,34 +1,117 @@ -# Testing Libraries on iOS and tvOS +# Testing Libraries on iOS, tvOS, and MacCatalyst -In order to build libraries and tests for iOS or tvOS you need recent version of XCode installed (e.g. 11.3 or higher). +## Prerequisites -Build Libraries for iOS Simulator: +- XCode 11.3 or higher +- a certificate and provisioning profile if using a device +- a simulator with a proper device type and OS version. +Go `XCode > Window > Devices and Simulators` to revise the list of the available simulators and then `"+" button on bottom left > OS Version dropdown selection > Download more simulator runtimes` in case you need to download more simulators. + +## Building Libs and Tests + +You can build and run the library tests: +- on a simulator; +- on a device. + +Run the following command in a terminal: +``` +./build.sh mono+libs -os -arch +``` +where `` is one of the following: +- iOSSimulator +- tvOSSimulator +- MacCatalyst +- iOS +- tvOS + +and `` is one of the following: +- x64 +- arm64 (for device) + +e.g., to build for an iOS simulator, run: ``` ./build.sh mono+libs -os iOSSimulator -arch x64 ``` + Run tests one by one for each test suite on a simulator: ``` ./build.sh libs.tests -os iOSSimulator -arch x64 -test ``` + +### Building for a device + In order to run the tests on a device: -- Set the os to `iOS` instead of `iOSSimulator` -- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`): +- Set the `-os` parameter to a device-related value (see above) +- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`). + +For example: ``` ./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5 ``` -[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices. +[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices. ### Running individual test suites + - The following shows how to run tests for a specific library: ``` ./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64 ``` +Also you can run the built test app through Xcode by opening the corresponding `.xcodeproj` and setting up the right scheme, app, and even signing if using a local device. + +There's also an option to run a `.app` through `xcrun`, which is simulator specific. Consider the following shell script: + +``` + xcrun simctl shutdown + xcrun simctl boot + open -a Simulator + xcrun simctl install + xcrun simctl launch --console booted +``` + +where +`` is a name of the simulator to start, e.g. `"iPhone 11"`, +`` is a path to the iOS test app bundle, +`` is a name of the iOS test app + +### Running the functional tests + +There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform. + +A functional test can be run the same way as any library test suite, e.g.: +``` +./dotnet.sh build /t:Test -c Release /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj +``` + +Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one. + +### Viewing logs +- see the logs generated by the XHarness tool +- use the `Console` app on macOS: +`Command + Space`, type in `Console`, search the appropriate process (System.Buffers.Tests for example). + +### Testing various configurations + +It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoEnableInterpreter`, and some more. + +1. Interpreter Only + +This configuration is necessary for hot reload scenarios. + +To enable the interpreter, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=true` to a build command. + +2. AOT only + +To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command. + +3. AOT-LLVM + +To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command. + ### Test App Design -iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test +iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets. ### Existing Limitations -- Most of the test suites crash on devices due to #35674 - Simulator uses JIT mode only at the moment (to be extended with FullAOT and Interpreter) - Interpreter is not enabled yet. diff --git a/docs/workflow/testing/mono/testing.md b/docs/workflow/testing/mono/testing.md index 4c37f634de45c..3e2eec366153d 100644 --- a/docs/workflow/testing/mono/testing.md +++ b/docs/workflow/testing/mono/testing.md @@ -79,7 +79,20 @@ For example, the following command is for running System.Runtime tests: make run-tests-corefx-System.Runtime ``` ### Mobile targets and WebAssembly -Build and run library tests against Webassembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/) +Build and run library tests against WebAssembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/) + +## Running the functional tests + +There are the [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on Android, iOS-like platforms (iOS/tvOS + simulators, MacCatalyst), and WebAssembly. + +A functional test can be run the same way as any library test suite, e.g.: +``` +./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj +``` + +Currently the functional tests are expected to return `42` as a success code so please be careful when adding a new one. + +For more details, see instructions located in [Library testing document folder](../libraries/). # Running the Mono samples There are a few convenient samples located in `$(REPO_ROOT)/src/mono/sample`, which could help you test your program easily with different flavors of Mono or do a sanity check on the build. The samples are set up to work with a specific configuration; please refer to the relevant Makefile for specifics. If you would like to work with a different configuration, you can edit the Makefile.