Skip to content

Commit

Permalink
[VFX] Add Performance Test (#203)
Browse files Browse the repository at this point in the history
* *Prepare performance test for editor (adding another asmdef)

* Fix two issue with testing.graphics-performance package

- EditorShaderStaticAnalysisAsset is only available if shaderAnalysis is present
- Runtime meta conflicting with someting else (?)

* *Add dummy performance runtime test

* Add some editor performance test :

- Compilation
- Backup (Undo/Redo)
- Load Library

* - Add VFXViewWindow_Open_And_Render test & Reorder test (Order isn't suported, it would be nice...)

* Fix performance test build setup, testScenesAsset can be null

* First iteration of GraphicTest run as performance test

* Clean some code

* Workaround to avoid double run from test result (class doesn't appear in Name result)

* Add custom sampling implementation (require C++ change to enable GPU timing)

* Add basic capture of memory : global, VisualEffect & VisualEffectAsset (TODO : find a way to capture allocated memory from VFXManager)

* Add Measure marker for performance editorTest

* Add missing marker (unused in runtime anyway)

* Rename Unit.Testing.VisualEffectGraph => Unity.Testing.VisualEffectGraph (it was a typo)

* *Fix package reference (adding shaderanalysis)

* Force ShaderUtil.allowAsyncCompilation to be false while running performance test

* Fix malformed json

* Add --testfilter="VFX.Test" to exclude performanceTest from execution in runtime verification

* Add performanceTest variant

VFX_HDRP => Edit/PlayMode/Standalone
VFX_URP => Edit/PlayMode/Standalone

* *Regenerate yml with performance test addition

* Fix issue with multiple vfx_in_assetbundle loaded (scene can now be loaded twice)

* *Use indepedant VFXPerformanceUseGraphicsTestCasesAttribute to avoid issue with multiple AssetBundle loading

* Fix missing filter in vfx_hdrp playmode_cache

* Add --performance-project-version=./Graphics --report-performance-data --performance-project-id=VFX in performanceTest

* *regenerate metafile

* Fix --performance-project-version path

* *Regenerate metafiles

* Add old playmode tests back

* Add perf tests to URP and HDRP nightlies

* Add --testFilter argument to Standalone_Build jobs

* Use preserve directive for "SetupGraphicsTestCases"

* No real change : move VFXPerformanceUseGraphicsTestCasesAttribute to performanceTest runtime

It's more consitent

* Remove preserve in favor of explicit link.xml

Fix il2cpp platform with old performance test package

* Fix unexpected stripping in Unity.PerformanceTesting

* Use WaitForEndOfFrame instad of yield return null to have always a consistent behavior

* Reduce complexity/time for editor test

* *Improve memoryTest tracking same memory Type than HDRP (+VFX)

* *Regenerate yml after merge

artifactory change its path

* Increase performance test timeout

Some shader could be really long to import

* Remove dynamic test filtering from all vfx configurations

* Minor : gather allowAsyncCompilation override in OneTimeSetUp/OneTimeTearDown & remove disable log

* Fix previously hidden error : "EditMode test can only yield null'. Use UnityEngine.TestTools.LogAssert.Expect"

* Fix playerbuild

* Fix player build (missing staged change -_-')

Co-authored-by: Theo Penavaire <theo.penavaire@unity3d.com>
  • Loading branch information
PaulDemeulenaere and theopnv committed Jan 20, 2021
1 parent aa0195c commit eaf4683
Show file tree
Hide file tree
Showing 46 changed files with 3,763 additions and 118 deletions.
12 changes: 12 additions & 0 deletions .yamato/all-vfx_hdrp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@ Nightly_VFX_HDRP_trunk:
rerun: on-new-revision
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_playmode_cache_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_editmode_PerformanceTests_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_playmode_PerformanceTests_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_Standalone_PerformanceTests_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_hdrp-win-dx12.yml#VFX_HDRP_Win_DX12_playmode_cache_mono_Linear_trunk
rerun: on-new-revision
variables:
Expand Down Expand Up @@ -57,6 +63,12 @@ Nightly_VFX_HDRP_CUSTOM-REVISION:
rerun: always
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_playmode_cache_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_editmode_PerformanceTests_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_playmode_PerformanceTests_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_hdrp-win-dx11.yml#VFX_HDRP_Win_DX11_Standalone_PerformanceTests_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_hdrp-win-dx12.yml#VFX_HDRP_Win_DX12_playmode_cache_mono_Linear_CUSTOM-REVISION
rerun: always
variables:
Expand Down
12 changes: 12 additions & 0 deletions .yamato/all-vfx_urp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,12 @@ Nightly_VFX_URP_trunk:
rerun: on-new-revision
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_playmode_cache_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_editmode_PerformanceTests_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_playmode_PerformanceTests_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_Standalone_PerformanceTests_mono_Linear_trunk
rerun: on-new-revision
- path: .yamato/vfx_urp-osx-metal.yml#VFX_URP_OSX_Metal_playmode_cache_mono_Linear_trunk
rerun: on-new-revision
variables:
Expand All @@ -45,6 +51,12 @@ Nightly_VFX_URP_CUSTOM-REVISION:
rerun: always
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_playmode_cache_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_editmode_PerformanceTests_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_playmode_PerformanceTests_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_urp-win-dx11.yml#VFX_URP_Win_DX11_Standalone_PerformanceTests_mono_Linear_CUSTOM-REVISION
rerun: always
- path: .yamato/vfx_urp-osx-metal.yml#VFX_URP_OSX_Metal_playmode_cache_mono_Linear_CUSTOM-REVISION
rerun: always
variables:
Expand Down
38 changes: 33 additions & 5 deletions .yamato/config/vfx_hdrp.metafile
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,21 @@ test_platforms:
- type: Standalone
utr_flags:
- [OSX_Metal]: --timeout=2400
utr_flags_build:
- [all]: --testfilter="VFX.Test"
- type: Standalone
name: Standalone_PerformanceTests
utr_flags:
- [all]: --performance-project-version=../.. --report-performance-data --performance-project-id=VFX
utr_flags_build:
- [all]: --testfilter="VFX.PerformanceTest"
- type: playmode
utr_flags:
- [all]: --testfilter="VFX.Test"
- type: playmode
name: playmode_cache
utr_flags:
- [all]: --testfilter="VFX.Test"
- [all]: --extra-editor-arg="-adb2"
- [all]: --extra-editor-arg="-enableCacheServer"
- [all]: --extra-editor-arg="-cacheServerEndpoint" --extra-editor-arg="cacheserver-slo.hq.unity3d.com"
Expand All @@ -18,8 +29,21 @@ test_platforms:
- type: playmode
name: playmode_XR
utr_flags:
- [all]: --extra-editor-arg="-xr-reuse-tests"
- [all]: --testfilter="VFX.Test"
- [all]: --extra-editor-arg="-xr-reuse-tests"
- type: playmode
name: playmode_PerformanceTests
utr_flags:
- [all]: --testfilter="VFX.PerformanceTest"
- [all]: --performance-project-version=../.. --report-performance-data --performance-project-id=VFX
- type: editmode
utr_flags:
- [all]: --testfilter="VFX.Test"
- type: editmode
name: editmode_PerformanceTests
utr_flags:
- [all]: --testfilter="VFX.PerformanceTest"
- [all]: --performance-project-version=../.. --report-performance-data --performance-project-id=VFX
- type: Standalone
name: Standalone_cache
utr_flags:
Expand All @@ -37,7 +61,7 @@ test_platforms:
- [all]: --extra-editor-arg="-cacheServerNamespacePrefix" --extra-editor-arg="SRP"
- [all]: --extra-editor-arg="-cacheServerEnableDownload" --extra-editor-arg="true"
- [all]: --extra-editor-arg="-cacheServerEnableUpload" --extra-editor-arg="true"
expression_trigger:
expression_trigger:
expression: 'pull_request.target eq "master" AND NOT pull_request.draft AND pull_request.changes.any match ["com.unity.visualeffectgraph/**/*", "TestProjects/VisualEffectGraph_HDRP/**/*"]'
platforms:
- name: Win
Expand Down Expand Up @@ -69,7 +93,7 @@ platforms:
type: Unity::VM
image: sdet/gamecode_win10:stable
flavor: b1.xlarge
- name: OSX
- name: OSX
apis:
- name: Metal
exclude_test_platforms:
Expand All @@ -79,7 +103,7 @@ platforms:
color_spaces:
- Linear
- name: Linux
apis:
apis:
- name: Vulkan
exclude_test_platforms:
- name: playmode_XR
Expand Down Expand Up @@ -113,9 +137,13 @@ nightly:
test_platforms:
- editmode
- playmode_cache
- editmode_PerformanceTests
- playmode_PerformanceTests
- Standalone_PerformanceTests
- platform: Win
api: DX12
build_config: mono
color_space: Linear
test_platforms:
- playmode_cache
- playmode_cache

38 changes: 33 additions & 5 deletions .yamato/config/vfx_urp.metafile
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,40 @@ test_platforms:
- type: Standalone
utr_flags:
- [OSX_Metal]: --timeout=2400
utr_flags_build:
- [all]: --testfilter="VFX.Test"
- type: Standalone
name: Standalone_PerformanceTests
utr_flags:
- [all]: --performance-project-version=../.. --report-performance-data --performance-project-id=VFX
utr_flags_build:
- [all]: --testfilter="VFX.PerformanceTest"
- type: playmode
utr_flags:
- [all]: --testfilter="VFX.Test"
- type: playmode
name: playmode_cache
utr_flags:
- [all]: --testfilter="VFX.Test"
- [all]: --extra-editor-arg="-adb2"
- [all]: --extra-editor-arg="-enableCacheServer"
- [all]: --extra-editor-arg="-cacheServerEndpoint" --extra-editor-arg="cacheserver-slo.hq.unity3d.com"
- [all]: --extra-editor-arg="-cacheServerNamespacePrefix" --extra-editor-arg="SRP"
- [all]: --extra-editor-arg="-cacheServerEnableDownload" --extra-editor-arg="true"
- [all]: --extra-editor-arg="-cacheServerEnableUpload" --extra-editor-arg="true"
- type: playmode
name: playmode_PerformanceTests
utr_flags:
- [all]: --testfilter="VFX.PerformanceTest"
- [all]: --performance-project-version=../.. --report-performance-data --performance-project-id=VFX
- type: editmode
utr_flags:
- [all]: --testfilter="VFX.Test"
- type: editmode
name: editmode_PerformanceTests
utr_flags:
- [all]: --testfilter="VFX.PerformanceTest"
- [all]: --performance-project-version=../.. --report-performance-data --performance-project-id=VFX
- type: Standalone
name: Standalone_cache
utr_flags:
Expand All @@ -33,7 +57,7 @@ test_platforms:
- [all]: --extra-editor-arg="-cacheServerEnableDownload" --extra-editor-arg="true"
- [all]: --extra-editor-arg="-cacheServerEnableUpload" --extra-editor-arg="true"
# - type: editmode
expression_trigger:
expression_trigger:
expression: 'pull_request.target eq "master" AND NOT pull_request.draft AND pull_request.changes.any match ["com.unity.visualeffectgraph/**/*", "TestProjects/VisualEffectGraph_URP/**/*"]'
platforms:
- name: Win
Expand All @@ -45,7 +69,7 @@ platforms:
- name: mono
color_spaces:
- Linear
- name: OSX
- name: OSX
apis:
- name: Metal
exclude_test_platforms:
Expand All @@ -60,7 +84,7 @@ platforms:
color_spaces:
- Linear
- name: Linux
apis:
apis:
- name: OpenGLCore
exclude_test_platforms:
- name: Standalone
Expand All @@ -86,7 +110,7 @@ platforms:
color_spaces:
- Linear
# - name: Android
# apis:
# apis:
# - name: Vulkan
# - name: OpenGLES3
# build_configs:
Expand All @@ -111,10 +135,14 @@ nightly:
color_space: Linear
test_platforms:
- playmode_cache
- editmode_PerformanceTests
- playmode_PerformanceTests
- Standalone_PerformanceTests
- platform: OSX
api: Metal
build_config: mono
color_space: Linear
test_platforms:
# - Standalone
- playmode_cache
# - Standalone_cache
- playmode_cache
Loading

0 comments on commit eaf4683

Please sign in to comment.