Skip to content

[Perf] Linux/x64: 59 Regressions on 11/2/2022 4:25:40 PM #9599

Description

@performanceautofiler

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConditionalSelectBenchmark - Duration of single invocation 3.11 ns 9.71 ns 3.13 0.47 False
AbsBenchmark - Duration of single invocation 2.90 ns 3.91 ns 1.35 0.51 False
AndNotBenchmark - Duration of single invocation 3.16 ns 9.42 ns 2.99 0.46 False
OnesComplementOperatorBenchmark - Duration of single invocation 4.12 ns 9.17 ns 2.23 0.27 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Byte>.ConditionalSelectBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.708599310662363 > 3.2421146326666.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -211.36363440761716 (T) = (0 -9.726078993075308) / Math.Sqrt((0.029687571625891588 / (31)) + (0.00014679510447676626 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.078985206910099 = (3.1588586301899997 - 9.726078993075308) / 3.1588586301899997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.AbsBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 3.9129686826572416 > 3.725767779511884.
IsChangePoint: Marked as a change because one of 10/30/2022 5:36:44 PM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -4.054607871924009 (T) = (0 -3.627839770912456) / Math.Sqrt((0.1349751557855404 / (16)) + (0.08619022070421639 / (34))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (34) - 2, .025) and -0.13258478075541288 = (3.203150733221715 - 3.627839770912456) / 3.203150733221715 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.419579978245908 > 3.2796371459389317.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -133.03736436305078 (T) = (0 -9.3427405922146) / Math.Sqrt((0.06433403671264686 / (31)) + (0.0024052918712732956 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.0137277633725263 = (3.1000612284102136 - 9.3427405922146) / 3.1000612284102136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Byte&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.165525510754637 > 3.0523499734213257.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -128.90273591456324 (T) = (0 -9.181027720534507) / Math.Sqrt((0.06117518887247477 / (31)) + (0.008129593530573503 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.2051702306606145 = (2.864443090326037 - 9.181027720534507) / 2.864443090326037 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementBenchmark - Duration of single invocation 2.75 ns 9.20 ns 3.35 0.20 False
OnesComplementOperatorBenchmark - Duration of single invocation 2.74 ns 9.30 ns 3.40 0.21 False
AndNotBenchmark - Duration of single invocation 3.17 ns 9.28 ns 2.93 0.47 False
ConditionalSelectBenchmark - Duration of single invocation 3.13 ns 9.73 ns 3.11 0.48 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int32>.OnesComplementBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.202677352135359 > 2.9502145492997687.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -248.00953875505516 (T) = (0 -9.202923728668273) / Math.Sqrt((0.00862566119909591 / (31)) + (0.007274336171924177 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.256401236266472 = (2.8261025165374294 - 9.202923728668273) / 2.8261025165374294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.300238498048524 > 3.037126283011585.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -201.20657594153576 (T) = (0 -9.215734058318192) / Math.Sqrt((0.010539935948370065 / (31)) + (0.012538993605785309 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.2299659209618574 = (2.8531985425944746 - 9.215734058318192) / 2.8531985425944746 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.283824817783087 > 3.0972113041108225.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -227.13130703954934 (T) = (0 -9.330820770573203) / Math.Sqrt((0.020916367833117098 / (31)) + (0.0016967368263415652 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.056589622806297 = (3.0526900637732477 - 9.330820770573203) / 3.0526900637732477 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.734824400368963 > 3.486777361921763.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -190.8991652303114 (T) = (0 -9.731284459193885) / Math.Sqrt((0.03612852924777575 / (31)) + (0.00012551151062415055 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.0449926656270754 = (3.195831822205673 - 9.731284459193885) / 3.195831822205673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
OnesComplementBenchmark - Duration of single invocation 3.00 ns 9.17 ns 3.06 0.13 False
ConditionalSelectBenchmark - Duration of single invocation 3.11 ns 9.72 ns 3.12 0.41 False
OnesComplementOperatorBenchmark - Duration of single invocation 2.89 ns 9.34 ns 3.23 0.29 False
AndNotBenchmark - Duration of single invocation 3.16 ns 9.39 ns 2.97 0.47 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int16>.OnesComplementBenchmark


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.172717175891426 > 3.049656375257257.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -246.86746275678226 (T) = (0 -9.173159684128953) / Math.Sqrt((0.0022958017063680054 / (31)) + (0.011034594572024737 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.211973100888315 = (2.8559266830696655 - 9.173159684128953) / 2.8559266830696655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.716598034074371 > 3.2686649787112256.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -261.6728171781317 (T) = (0 -9.722972205760604) / Math.Sqrt((0.018564885566641934 / (31)) + (0.00029948102803120386 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.004949266336037 = (3.235652699593133 - 9.722972205760604) / 3.235652699593133 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.OnesComplementOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.343872076223343 > 3.113629355130062.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -184.9989879169057 (T) = (0 -9.2238995106291) / Math.Sqrt((0.02140720577441967 / (31)) + (0.008815269318567473 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -2.1395474982816105 = (2.9379710024064543 - 9.2238995106291) / 2.9379710024064543 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 9.392419924154426 > 3.306382396365186.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -218.13552842981684 (T) = (0 -9.383920588226768) / Math.Sqrt((0.020067899561086965 / (30)) + (0.0033510075510733124 / (19))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (19) - 2, .025) and -2.084945477632499 = (3.041843253394655 - 9.383920588226768) / 3.041843253394655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Collections.IterateForEach<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 295.58 ns 459.48 ns 1.55 0.04 False
ReadOnlySpan - Duration of single invocation 458.10 ns 682.08 ns 1.49 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Collections.IterateForEach<String>.Array(Size: 512)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 459.48332969450746 > 310.46955687713313.
IsChangePoint: Marked as a change because one of 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/18/2022 9:26:37 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -1373.7272007225604 (T) = (0 -459.30916066966785) / Math.Sqrt((0.052255858195927866 / (31)) + (0.23743961627529306 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.5532334700790591 = (295.7116039009185 - 459.30916066966785) / 295.7116039009185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;String&gt;.ReadOnlySpan(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 682.0759055516708 > 481.2748606447134.
IsChangePoint: Marked as a change because one of 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/29/2022 8:01:59 PM, 10/18/2022 9:26:37 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -1508.6741425375667 (T) = (0 -681.990433380907) / Math.Sqrt((0.28338834589491785 / (31)) + (0.2436811312352892 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.4878069078130593 = (458.38638723849647 - 681.990433380907) / 458.38638723849647 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Tests.Perf_Char

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Char_ToLower - Duration of single invocation 11.52 ns 13.10 ns 1.14 0.16 False
Char_ToUpperInvariant - Duration of single invocation 100.26 ns 116.89 ns 1.17 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Char*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_Char.Char_ToLower(c: 'a', cultureName: en-US)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 13.10145478250563 > 12.095987005954012.
IsChangePoint: Marked as a change because one of 10/5/2022 3:48:29 AM, 10/11/2022 8:42:28 PM, 10/18/2022 9:26:37 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -9.141166508971919 (T) = (0 -13.152515264124235) / Math.Sqrt((0.0634678049070142 / (30)) + (0.525738948116271 / (19))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (19) - 2, .025) and -0.13629875847655515 = (11.574874271408971 - 13.152515264124235) / 11.574874271408971 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Char.Char_ToUpperInvariant(input: "Hello World!")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 116.88501989544638 > 105.18496469591452.
IsChangePoint: Marked as a change because one of 9/2/2022 11:09:40 AM, 9/8/2022 12:20:47 PM, 9/29/2022 8:01:59 PM, 10/27/2022 9:35:54 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -13.43493501181134 (T) = (0 -117.30063966661007) / Math.Sqrt((22.255410235265717 / (31)) + (5.58412929643243 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.13021040562850203 = (103.78655078952313 - 117.30063966661007) / 103.78655078952313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Log2_ulong - Duration of single invocation 895.22 ns 1.32 μs 1.47 0.03 False
LeadingZeroCount_ulong - Duration of single invocation 746.19 ns 1.10 μs 1.48 0.02 False
LeadingZeroCount_uint - Duration of single invocation 667.49 ns 1.10 μs 1.65 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'

Related Issues

Regressions

Improvements

Details

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_BitOperations.Log2_ulong


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.3199362918110857 > 939.9019055566016.
IsChangePoint: Marked as a change because one of 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -1659.2904419196575 (T) = (0 -1320.5936904490882) / Math.Sqrt((0.1081329163430295 / (31)) + (1.1835436526965102 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.4754815659054461 = (895.0255435002272 - 1320.5936904490882) / 895.0255435002272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.1025752622227607 > 783.081472002204.
IsChangePoint: Marked as a change because one of 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -1611.1063054104666 (T) = (0 -1103.1841508442424) / Math.Sqrt((0.1746362845219137 / (31)) + (0.8276779267500464 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.4791161651317811 = (745.8401015757642 - 1103.1841508442424) / 745.8401015757642 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 1.099078617094986 > 700.7997576002152.
IsChangePoint: Marked as a change because one of 8/30/2022 12:00:50 PM, 9/10/2022 11:12:52 PM, 9/21/2022 9:54:19 PM, 10/3/2022 8:45:37 PM, 10/18/2022 9:26:37 AM, 11/2/2022 9:35:40 AM, 11/8/2022 2:29:58 AM falls between 10/30/2022 12:44:54 AM and 11/8/2022 2:29:58 AM.
IsRegressionStdDev: Marked as regression because -885.4453215244073 (T) = (0 -1100.2172150692463) / Math.Sqrt((0.20491164629965838 / (31)) + (4.414028089355247 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.6484881582571643 = (667.4098382559399 - 1100.2172150692463) / 667.4098382559399 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions