Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf][Mono] Regression on Serailizer Benchmarks Changes at 5/20/2022 2:58:35 PM #70219

Closed
performanceautofiler bot opened this issue May 24, 2022 · 17 comments
Closed

[Perf][Mono] Regression on Serailizer Benchmarks Changes at 5/20/2022 2:58:35 PM #70219

performanceautofiler bot opened this issue May 24, 2022 · 17 comments
Assignees
@pavelsavara
Labels
arch-x64 area-Codegen-meta-mono os-linux Linux OS (any supported distro)
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented May 24, 2022 •
edited by naricc
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 339.59 μs 421.04 μs 1.24 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<Location>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 421.0374860830527 > 360.3277714008774.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -107.88968173006991 (T) = (0 -416462.0605698214) / Math.Sqrt((7115911.3609417435 / (52)) + (5958088.944844954 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.2153584322035255 = (342666.0395277367 - 416462.0605698214) / 342666.0395277367 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 91.36 μs 110.78 μs 1.21 0.05 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<LoginViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.7758675595238 > 95.6215819958911.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -73.77410290263704 (T) = (0 -110421.01200474823) / Math.Sqrt((447954.9437939434 / (52)) + (1046265.9580128257 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.20860866019041133 = (91362.08902172839 - 110421.01200474823) / 91362.08902172839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 140.42 ms 162.04 ms 1.15 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 162.0418461153846 > 147.2387700525.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -50.65564926711963 (T) = (0 -161808805.247385) / Math.Sqrt((1525019236549.4724 / (52)) + (2701224736616.268 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.15286568999852104 = (140353561.26140988 - 161808805.247385) / 140353561.26140988 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 138.23 μs 173.17 μs 1.25 0.05 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 173.17193392114126 > 143.30756264913916.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -123.68629024049936 (T) = (0 -172349.59492403342) / Math.Sqrt((798426.8210777837 / (51)) + (1162097.3777697405 / (18))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (18) - 2, .025) and -0.2551079131388944 = (137318.5469709971 - 172349.59492403342) / 137318.5469709971 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 51.35 ms 63.41 ms 1.23 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.412730711538465 > 54.04382307692309.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -92.50614155633478 (T) = (0 -63774875.54960827) / Math.Sqrt((169876198041.4083 / (52)) + (260174152808.59546 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.23929893333817115 = (51460445.768176764 - 63774875.54960827) / 51460445.768176764 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 66.06 μs 81.61 μs 1.24 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.60717073011733 > 68.58656458326523.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -94.00964525821348 (T) = (0 -80908.31417726286) / Math.Sqrt((141504.47169674604 / (52)) + (453180.8178477467 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.2408073785567304 = (65206.18395368745 - 80908.31417726286) / 65206.18395368745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 47.90 μs 55.09 μs 1.15 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;XmlElement&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.08972981220658 > 50.05582197315684.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -59.45488956905215 (T) = (0 -55204.83342682804) / Math.Sqrt((108191.57482139874 / (52)) + (258770.1440727731 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.16030812297205438 = (47577.73589089803 - 55204.83342682804) / 47577.73589089803 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 114.30 ms 129.32 ms 1.13 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.31821425 > 119.478555035.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -60.79137700487244 (T) = (0 -128549734.39655069) / Math.Sqrt((531701764641.9763 / (50)) + (888723260188.9719 / (18))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (18) - 2, .025) and -0.13102219526990483 = (113658012.13642305 - 128549734.39655069) / 113658012.13642305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 10.72 ms 12.16 ms 1.13 0.05 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.158967846428569 > 11.065523925217391.
IsChangePoint: Marked as a change because one of 3/28/2022 7:37:41 AM, 3/28/2022 9:07:19 PM, 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -110.78835983024476 (T) = (0 -12171679.268167479) / Math.Sqrt((4287678531.936606 / (52)) + (2107489848.1284816 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.1475457039261136 = (10606705.446697546 - 12171679.268167479) / 10606705.446697546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<ClassImplementingIXmlSerialiable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 51.78 μs 59.42 μs 1.15 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;ClassImplementingIXmlSerialiable&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<ClassImplementingIXmlSerialiable>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.42197409958046 > 54.294100581261134.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -88.65312670653857 (T) = (0 -59374.90491959243) / Math.Sqrt((56724.90276240548 / (52)) + (112787.69486814098 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.14687701134139686 = (51770.943468600046 - 59374.90491959243) / 51770.943468600046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 4.70 ms 5.36 ms 1.14 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.355766666666667 > 4.910791719603663.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -57.92374158271294 (T) = (0 -5352328.436258438) / Math.Sqrt((540630615.6445099 / (50)) + (2229093302.1526623 / (18))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (18) - 2, .025) and -0.1436145810868225 = (4680185.549201294 - 5352328.436258438) / 4680185.549201294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 56.93 ms 63.67 ms 1.12 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.67307148214286 > 59.581869385625005.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -46.901649881421 (T) = (0 -63506493.06560009) / Math.Sqrt((87468678174.13133 / (51)) + (336620379325.5662 / (17))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (17) - 2, .025) and -0.12149262425712384 = (56626759.456101425 - 63506493.06560009) / 56626759.456101425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 160.67 μs 198.08 μs 1.23 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<Location>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 198.07930025600388 > 167.48253939087434.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -91.306682889007 (T) = (0 -196045.6815454997) / Math.Sqrt((996200.490846422 / (51)) + (2506293.6081947945 / (18))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (18) - 2, .025) and -0.22786826739665736 = (159663.44823061384 - 196045.6815454997) / 159663.44823061384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.
Lookjs like one of these two changes based on dates: https://github.com/dotnet/runtime/compare/8d4a724883715cc5ae71e020b22762136f7ce50b...810a7f9cd78d6611c0473940cb733231f3eabc06

Looks like one of these two changes: https://github.com/dotnet/runtime/compare/8d4a724883715cc5ae71e020b22762136f7ce50b...810a7f9cd78d6611c0473940cb733231f3eabc06

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added CompliationMode=wasm untriaged New issue has not been triaged by the area owner labels May 24, 2022
@naricc
Copy link
Member

naricc commented Jun 3, 2022

Related to one of these two: 8d4a724...810a7f9

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 3, 2022
@naricc naricc transferred this issue from dotnet/perf-autofiling-issues Jun 3, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 3, 2022
@ghost
Copy link

ghost commented Jun 3, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 339.59 μs 421.04 μs 1.24 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<Location>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 421.0374860830527 > 360.3277714008774.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -107.88968173006991 (T) = (0 -416462.0605698214) / Math.Sqrt((7115911.3609417435 / (52)) + (5958088.944844954 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.2153584322035255 = (342666.0395277367 - 416462.0605698214) / 342666.0395277367 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 91.36 μs 110.78 μs 1.21 0.05 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<LoginViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.7758675595238 > 95.6215819958911.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -73.77410290263704 (T) = (0 -110421.01200474823) / Math.Sqrt((447954.9437939434 / (52)) + (1046265.9580128257 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.20860866019041133 = (91362.08902172839 - 110421.01200474823) / 91362.08902172839 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 140.42 ms 162.04 ms 1.15 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 162.0418461153846 > 147.2387700525.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -50.65564926711963 (T) = (0 -161808805.247385) / Math.Sqrt((1525019236549.4724 / (52)) + (2701224736616.268 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.15286568999852104 = (140353561.26140988 - 161808805.247385) / 140353561.26140988 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 138.23 μs 173.17 μs 1.25 0.05 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 173.17193392114126 > 143.30756264913916.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -123.68629024049936 (T) = (0 -172349.59492403342) / Math.Sqrt((798426.8210777837 / (51)) + (1162097.3777697405 / (18))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (18) - 2, .025) and -0.2551079131388944 = (137318.5469709971 - 172349.59492403342) / 137318.5469709971 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 51.35 ms 63.41 ms 1.23 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<CollectionsOfPrimitives>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.412730711538465 > 54.04382307692309.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -92.50614155633478 (T) = (0 -63774875.54960827) / Math.Sqrt((169876198041.4083 / (52)) + (260174152808.59546 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.23929893333817115 = (51460445.768176764 - 63774875.54960827) / 51460445.768176764 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 66.06 μs 81.61 μs 1.24 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<SimpleStructWithProperties>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.60717073011733 > 68.58656458326523.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -94.00964525821348 (T) = (0 -80908.31417726286) / Math.Sqrt((141504.47169674604 / (52)) + (453180.8178477467 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.2408073785567304 = (65206.18395368745 - 80908.31417726286) / 65206.18395368745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 47.90 μs 55.09 μs 1.15 0.02 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;XmlElement&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<XmlElement>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 55.08972981220658 > 50.05582197315684.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -59.45488956905215 (T) = (0 -55204.83342682804) / Math.Sqrt((108191.57482139874 / (52)) + (258770.1440727731 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.16030812297205438 = (47577.73589089803 - 55204.83342682804) / 47577.73589089803 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 114.30 ms 129.32 ms 1.13 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 129.31821425 > 119.478555035.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -60.79137700487244 (T) = (0 -128549734.39655069) / Math.Sqrt((531701764641.9763 / (50)) + (888723260188.9719 / (18))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (18) - 2, .025) and -0.13102219526990483 = (113658012.13642305 - 128549734.39655069) / 113658012.13642305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractJsonSerializer_ - Duration of single invocation 10.72 ms 12.16 ms 1.13 0.05 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.DataContractJsonSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.158967846428569 > 11.065523925217391.
IsChangePoint: Marked as a change because one of 3/28/2022 7:37:41 AM, 3/28/2022 9:07:19 PM, 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -110.78835983024476 (T) = (0 -12171679.268167479) / Math.Sqrt((4287678531.936606 / (52)) + (2107489848.1284816 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.1475457039261136 = (10606705.446697546 - 12171679.268167479) / 10606705.446697546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<ClassImplementingIXmlSerialiable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 51.78 μs 59.42 μs 1.15 0.03 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;ClassImplementingIXmlSerialiable&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<ClassImplementingIXmlSerialiable>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 59.42197409958046 > 54.294100581261134.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -88.65312670653857 (T) = (0 -59374.90491959243) / Math.Sqrt((56724.90276240548 / (52)) + (112787.69486814098 / (18))) is less than -1.9954689314290597 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (52) + (18) - 2, .025) and -0.14687701134139686 = (51770.943468600046 - 59374.90491959243) / 51770.943468600046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 4.70 ms 5.36 ms 1.14 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<IndexViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.355766666666667 > 4.910791719603663.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -57.92374158271294 (T) = (0 -5352328.436258438) / Math.Sqrt((540630615.6445099 / (50)) + (2229093302.1526623 / (18))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (50) + (18) - 2, .025) and -0.1436145810868225 = (4680185.549201294 - 5352328.436258438) / 4680185.549201294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 56.93 ms 63.67 ms 1.12 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<MyEventsListerViewModel>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.67307148214286 > 59.581869385625005.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -46.901649881421 (T) = (0 -63506493.06560009) / Math.Sqrt((87468678174.13133 / (51)) + (336620379325.5662 / (17))) is less than -1.9965644189515832 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (17) - 2, .025) and -0.12149262425712384 = (56626759.456101425 - 63506493.06560009) / 56626759.456101425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 3914bcc230134ec3e1c9be903c6bd8ff60d92501
Compare 6069680d08199b862eeb372082ca12f47a203124
Diff Diff

Regressions in MicroBenchmarks.Serializers.Xml_FromStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DataContractSerializer_ - Duration of single invocation 160.67 μs 198.08 μs 1.23 0.04 True

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'MicroBenchmarks.Serializers.Xml_FromStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Xml_FromStream<Location>.DataContractSerializer_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 198.07930025600388 > 167.48253939087434.
IsChangePoint: Marked as a change because one of 5/20/2022 10:46:48 AM, 5/24/2022 7:58:36 AM falls between 5/15/2022 3:27:46 PM and 5/24/2022 7:58:36 AM.
IsRegressionStdDev: Marked as regression because -91.306682889007 (T) = (0 -196045.6815454997) / Math.Sqrt((996200.490846422 / (51)) + (2506293.6081947945 / (18))) is less than -1.9960083540247138 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (51) + (18) - 2, .025) and -0.22786826739665736 = (159663.44823061384 - 196045.6815454997) / 159663.44823061384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: kunalspathak
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, ubuntu 18.04, RunKind=micro, Regression, x64, WASM, CompliationMode=wasm
Milestone: -

@naricc naricc assigned naricc and unassigned kunalspathak Jun 3, 2022
@naricc naricc removed the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 3, 2022
@naricc
Copy link
Member

naricc commented Jun 3, 2022

Looks like one of these changes: 8d4a724...810a7f9

@naricc naricc removed their assignment Jun 3, 2022
@naricc naricc changed the title [Perf] Changes at 5/20/2022 2:58:35 PM [Perf][Mono] Regressi on Serailizer Benchmarks Changes at 5/20/2022 2:58:35 PM Jun 3, 2022
@naricc naricc changed the title [Perf][Mono] Regressi on Serailizer Benchmarks Changes at 5/20/2022 2:58:35 PM [Perf][Mono] Regression on Serailizer Benchmarks Changes at 5/20/2022 2:58:35 PM Jun 3, 2022
@naricc
Copy link
Member

naricc commented Jun 3, 2022

@naricc

@SamMonoRT
Copy link
Member

@pavelsavara, I went through the changes in the diff and I think these regressions maybe caused by your changes in #64845 -- please investigate.

@SamMonoRT SamMonoRT removed the untriaged New issue has not been triaged by the area owner label Jun 7, 2022
@SamMonoRT SamMonoRT added this to the 7.0.0 milestone Jun 7, 2022
@SamMonoRT
Copy link
Member

cc @radical can help provide instructions to run locally.

@lewing
Copy link
Member

lewing commented Jun 8, 2022 •
edited
Loading

I see the regression here in this range 8d4a724...810a7f9 so I suspect it is #69213 causing the problem. A very quick glance at that pr makes me think it might be because of the use of Reverse there which is a known regression #68667 but that is just a guess.

@SamMonoRT
Copy link
Member

cc @AaronRobinsonMSFT is it possible to investigate.

@AaronRobinsonMSFT
Copy link
Member

@SamMonoRT That change introduced a new marshalling type, that is all. No one in the .NET runtime repo, that I know of, is using the new BstrStringMarshaller as it was added because of ASP.Net Core, not anything in NET core app. None of the other changes impact anything at run-time.

@lewing What Reverse call in that PR are you referring to?

@AaronRobinsonMSFT
Copy link
Member

@lewing What Reverse call in that PR are you referring to?

Oh, in the NativeExport/Strings.cs? If so, that is a test and not a product change.

@lewing
Copy link
Member

lewing commented Jun 8, 2022

Yes, I see that now. I did try to be clear I had only glanced at the pr 😬, sorry. The range I see from https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu%2018.04_CompliationMode=wasm_RunKind=micro/MicroBenchmarks.Serializers.Xml_FromStream(MyEventsListerViewModel).DataContractSerializer_.html that regressed on the 20th is just those two changes though.

@lewing
Copy link
Member

lewing commented Jun 8, 2022 •
edited
Loading

#69879
so the other change was partially responsible but the fix doesn't seem to have brought it back to baseline

cc @eerhardt

@AaronRobinsonMSFT
Copy link
Member

Yes, I see that now. I did try to be clear I had only glanced at the pr 😬, sorry.

No worries. Always happy to help investigate performance regressions.

@eerhardt
Copy link
Member

eerhardt commented Jun 8, 2022

so the other change was partially responsible but the fix doesn't seem to have brought it back to baseline

Is this just on WASM? Do we have special "knowledge" of Stream.Read in WASM that makes it faster? I know I've seen things in the past where certain library APIs were optimized specifically for WASM. If that is the case, maybe the new ReadAtLeast and ReadAll need to be optimized as well?

For CoreCLR, my understanding is that the fix brought it back to baseline on all supported architectures.

@lewing
Copy link
Member

lewing commented Jun 10, 2022

It does appear to be wasm specific but I don't think there is anything special going on for wasm in this case. Which streams types are being used in this test?

@lewing
Copy link
Member

lewing commented Jun 10, 2022

There was a stretch of failed runs over the period where the fix landed so it is entirely possible your change handled the regression and we picked up a different regression during that time. Without better granularity we'd need to bisect

6761c44...748dfb7

@lewing
Copy link
Member

lewing commented Jun 10, 2022

There is at least one wasm regression in that range and main is fairly close to the previous baseline once that was fixed so I'm ok with considering this closed

@lewing lewing closed this as completed Jun 10, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jul 11, 2022
@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@pavelsavara pavelsavara

Labels
arch-x64 area-Codegen-meta-mono os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
8 participants
@lewing @pavelsavara @jeffhandley @eerhardt @kunalspathak @naricc @AaronRobinsonMSFT @SamMonoRT