Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 9 Regressions on 12/1/2022 11:48:13 PM #79337

Closed
performanceautofiler bot opened this issue Dec 6, 2022 · 8 comments
Closed

[Perf] Linux/x64: 9 Regressions on 12/1/2022 11:48:13 PM #79337

performanceautofiler bot opened this issue Dec 6, 2022 · 8 comments
Labels
arch-x64 area-Codegen-Interpreter-mono os-linux Linux OS (any supported distro)

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreatePerspectiveBenchmark - Duration of single invocation 46.88 ns 106.40 ns 2.27 0.23 False
CreatePerspectiveOffCenterBenchmark - Duration of single invocation 56.08 ns 100.36 ns 1.79 0.19 False
CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation 33.20 ns 61.18 ns 1.84 0.33 False
CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation 50.28 ns 76.30 ns 1.52 0.23 False
CreatePerspectiveFieldOfViewBenchmark - Duration of single invocation 63.69 ns 152.52 ns 2.39 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.40167473232303 > 50.089019744049914.
IsChangePoint: Marked as a change because one of 10/19/2022 7:27:15 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -29.54742807413155 (T) = (0 -96.53403777573979) / Math.Sqrt((2.0842638368238435 / (31)) + (50.35869810254004 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.0185334186441561 = (47.82384917886644 - 96.53403777573979) / 47.82384917886644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 100.36403988735348 > 58.79696350489112.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -62.96910756137589 (T) = (0 -101.91213463508836) / Math.Sqrt((3.1128228061074505 / (31)) + (8.126852446138692 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.8149874229088963 = (56.15032553324964 - 101.91213463508836) / 56.15032553324964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.175483380092 > 33.07993844635303.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/19/2022 7:27:15 PM, 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -240.36780006315195 (T) = (0 -61.18321579634298) / Math.Sqrt((0.7338720353181785 / (48)) + (0.00011958052215698276 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.9482623727783919 = (31.403991911567015 - 61.18321579634298) / 31.403991911567015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 76.29638516883118 > 52.98744532824259.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -20.23418898038416 (T) = (0 -75.14045784284106) / Math.Sqrt((5.187993144752031 / (48)) + (2.67233596594134 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.47845555759071756 = (50.82361621020892 - 75.14045784284106) / 50.82361621020892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveFieldOfViewBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.5239787610467 > 66.84254185778312.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -50.33783026021656 (T) = (0 -143.0980608155679) / Math.Sqrt((1.8023112844920022 / (31)) + (46.10575932368977 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.2446768917145479 = (63.749959445729196 - 143.0980608155679) / 63.749959445729196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor4 - Duration of single invocation 54.92 ns 95.97 ns 1.75 0.22 False
Ctor3 - Duration of single invocation 51.64 ns 73.53 ns 1.42 0.18 False
Ctor2 - Duration of single invocation 48.49 ns 63.28 ns 1.31 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Ctor4

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 95.9653964201535 > 57.81517049815903.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -68.53791289679593 (T) = (0 -96.72656882046915) / Math.Sqrt((2.166112228046841 / (31)) + (5.923382867603497 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.7785079664400922 = (54.38635679213716 - 96.72656882046915) / 54.38635679213716 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.53258813463972 > 54.44255171823497.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -38.789721575245764 (T) = (0 -75.65424811264478) / Math.Sqrt((2.1752788781519157 / (31)) + (5.739031977705032 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.45520962769791584 = (51.98855661251143 - 75.65424811264478) / 51.98855661251143 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.27801788496673 > 52.23769541648208.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -27.21061402026845 (T) = (0 -65.16909851640705) / Math.Sqrt((1.518428932024682 / (31)) + (5.053331740985409 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.3060337808413552 = (49.898478486846415 - 65.16909851640705) / 49.898478486846415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in PerfLabTests.BlockCopyPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CallBlockCopy - Duration of single invocation 86.43 ns 99.44 ns 1.15 0.15 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.BlockCopyPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.43981666225224 > 87.0023639795907.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -23.47984704244632 (T) = (0 -101.44930096610167) / Math.Sqrt((5.209829064913729 / (31)) + (8.017543922096761 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.21622082875210036 = (83.41355333487704 - 101.44930096610167) / 83.41355333487704 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added refs/heads/main untriaged New issue has not been triaged by the area owner labels Dec 6, 2022
@kotlarmilos
Copy link
Member

@SingleAccretion looks like regression by c782569. Do you have an idea what could have caused the regression?

@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Dec 7, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@kotlarmilos kotlarmilos added area-Codegen-Interpreter-mono and removed untriaged New issue has not been triaged by the area owner labels Dec 7, 2022
@ghost
Copy link

ghost commented Dec 7, 2022

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreatePerspectiveBenchmark - Duration of single invocation 46.88 ns 106.40 ns 2.27 0.23 False
CreatePerspectiveOffCenterBenchmark - Duration of single invocation 56.08 ns 100.36 ns 1.79 0.19 False
CreateScaleFromScalarWithCenterBenchmark - Duration of single invocation 33.20 ns 61.18 ns 1.84 0.33 False
CreateScaleFromVectorWithCenterBenchmark - Duration of single invocation 50.28 ns 76.30 ns 1.52 0.23 False
CreatePerspectiveFieldOfViewBenchmark - Duration of single invocation 63.69 ns 152.52 ns 2.39 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 106.40167473232303 > 50.089019744049914.
IsChangePoint: Marked as a change because one of 10/19/2022 7:27:15 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -29.54742807413155 (T) = (0 -96.53403777573979) / Math.Sqrt((2.0842638368238435 / (31)) + (50.35869810254004 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.0185334186441561 = (47.82384917886644 - 96.53403777573979) / 47.82384917886644 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveOffCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 100.36403988735348 > 58.79696350489112.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -62.96910756137589 (T) = (0 -101.91213463508836) / Math.Sqrt((3.1128228061074505 / (31)) + (8.126852446138692 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.8149874229088963 = (56.15032553324964 - 101.91213463508836) / 56.15032553324964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromScalarWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.175483380092 > 33.07993844635303.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/19/2022 7:27:15 PM, 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -240.36780006315195 (T) = (0 -61.18321579634298) / Math.Sqrt((0.7338720353181785 / (48)) + (0.00011958052215698276 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.9482623727783919 = (31.403991911567015 - 61.18321579634298) / 31.403991911567015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreateScaleFromVectorWithCenterBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 76.29638516883118 > 52.98744532824259.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/5/2022 3:14:07 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -20.23418898038416 (T) = (0 -75.14045784284106) / Math.Sqrt((5.187993144752031 / (48)) + (2.67233596594134 / (2))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (48) + (2) - 2, .025) and -0.47845555759071756 = (50.82361621020892 - 75.14045784284106) / 50.82361621020892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Matrix4x4.CreatePerspectiveFieldOfViewBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.5239787610467 > 66.84254185778312.
IsChangePoint: Marked as a change because one of 10/17/2022 12:47:34 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -50.33783026021656 (T) = (0 -143.0980608155679) / Math.Sqrt((1.8023112844920022 / (31)) + (46.10575932368977 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -1.2446768917145479 = (63.749959445729196 - 143.0980608155679) / 63.749959445729196 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor4 - Duration of single invocation 54.92 ns 95.97 ns 1.75 0.22 False
Ctor3 - Duration of single invocation 51.64 ns 73.53 ns 1.42 0.18 False
Ctor2 - Duration of single invocation 48.49 ns 63.28 ns 1.31 0.23 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.Ctor4

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 95.9653964201535 > 57.81517049815903.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -68.53791289679593 (T) = (0 -96.72656882046915) / Math.Sqrt((2.166112228046841 / (31)) + (5.923382867603497 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.7785079664400922 = (54.38635679213716 - 96.72656882046915) / 54.38635679213716 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.53258813463972 > 54.44255171823497.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -38.789721575245764 (T) = (0 -75.65424811264478) / Math.Sqrt((2.1752788781519157 / (31)) + (5.739031977705032 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.45520962769791584 = (51.98855661251143 - 75.65424811264478) / 51.98855661251143 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.Ctor2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.27801788496673 > 52.23769541648208.
IsChangePoint: Marked as a change because one of 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -27.21061402026845 (T) = (0 -65.16909851640705) / Math.Sqrt((1.518428932024682 / (31)) + (5.053331740985409 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.3060337808413552 = (49.898478486846415 - 65.16909851640705) / 49.898478486846415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 776a87d9f0c9105c2729d1ab781d74bffa2a182b
Compare 9d7ffb5b0f9c56376460a0a45b90cb3ca90302ea
Diff Diff

Regressions in PerfLabTests.BlockCopyPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CallBlockCopy - Duration of single invocation 86.43 ns 99.44 ns 1.15 0.15 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.BlockCopyPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.BlockCopyPerf.CallBlockCopy(numElements: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 99.43981666225224 > 87.0023639795907.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/1/2022 7:29:59 PM, 12/6/2022 5:44:36 AM falls between 11/27/2022 4:37:47 PM and 12/6/2022 5:44:36 AM.
IsRegressionStdDev: Marked as regression because -23.47984704244632 (T) = (0 -101.44930096610167) / Math.Sqrt((5.209829064913729 / (31)) + (8.017543922096761 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.21622082875210036 = (83.41355333487704 - 101.44930096610167) / 83.41355333487704 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

untriaged, area-Codegen-Interpreter-mono, refs/heads/main, ubuntu 18.04, Regression, x64
Milestone: -

@kotlarmilos kotlarmilos mentioned this issue Dec 7, 2022
Closed
@kotlarmilos
Copy link
Member

Same regression in MonoAOT - dotnet/perf-autofiling-issues#10359

@SingleAccretion
Copy link
Contributor

SingleAccretion commented Dec 7, 2022 •
edited
Loading

The Matrix4x4 regressions are clearly from #79172. Probably Mono's codegen for Vector3 operations is not as good as for simple field assigns. It is exactly the opposite with RyuJit, but, oh well, I think we can find a(nother) workaround.

The other regressions I do not immediately see being related to #79172.

Edit: I checked the commit diffs for non-Matrix4x4 regressions and indeed #79172 is not in them (the diff is the same as in dotnet/perf-autofiling-issues#10359 (comment)), so it would probably be a good idea to split them into a different issue.

@kotlarmilos
Copy link
Member

@SingleAccretion Thanks for the update. Let me check the instruction stream for Matrix4x4 regressions on mono interpreter. I will close this issue and open relevant optimization issues.

@kotlarmilos
Copy link
Member

The other regressions are same as in #79336

@kotlarmilos
Copy link
Member

Same cause as in #78805. Closing it, I will open tracking issue for span optimizations on mono interpreter.

@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jan 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-Codegen-Interpreter-mono os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jeffhandley @kotlarmilos @SingleAccretion