Reduce CI noise sensitivity in daily CLI performance benchmarks - #53777
Merged
Conversation
…nment-noise guard Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Investigate performance regression pattern and environmental factors
Reduce CI noise sensitivity in daily CLI performance benchmarks
Aug 18, 2026
Contributor
PR Triage
Draft PR reducing CI benchmark noise sensitivity. Low risk, small diff. Needs undraft before scheduling review.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Improves daily CLI benchmark stability and reduces false regression alerts.
Changes:
- Uses time-based benchmark sampling.
- Switches historical baseline to median.
- Adds environment-noise detection and suppresses related issues.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Increases benchmark sampling duration. |
.github/workflows/daily-cli-performance.md |
Adds median baselines and noise classification. |
.github/workflows/daily-cli-performance.lock.yml |
Updates generated workflow metadata. |
Review details
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 3/3 changed files
- Comments generated: 4
- Review effort level: Balanced
Comment on lines
364
to
+365
| recent_history = historical_values[-7:] if len(historical_values) >= 7 else historical_values | ||
| avg_historical = sum(recent_history) / len(recent_history) | ||
| avg_historical = statistics.median(recent_history) |
Comment on lines
+433
to
+436
| likely_noise = ( | ||
| regressions >= NOISE_MIN_REGRESSIONS | ||
| and total > 0 | ||
| and degraded / total >= NOISE_REGRESSION_RATIO |
Comment on lines
+560
to
+565
| if analysis['summary'].get('likely_environment_noise'): | ||
| print(f"⏭️ {len(regressions)} regression(s) detected across unrelated benchmarks — " | ||
| "classified as likely environment noise, no regression issues will be opened.") | ||
| with open('/tmp/gh-aw/agent/benchmarks/regressions.json', 'w') as f: | ||
| json.dump([], f, indent=2) | ||
| return |
Comment on lines
+362
to
+365
| # Use the median of recent history (last 7 data points) as the baseline — | ||
| # the median is far less sensitive to one-off noisy runs than the mean | ||
| recent_history = historical_values[-7:] if len(historical_values) >= 7 else historical_values | ||
| avg_historical = sum(recent_history) / len(recent_history) | ||
| avg_historical = statistics.median(recent_history) |
Contributor
|
🎉 This pull request is included in a new release. Release: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The daily performance workflow reported 33%–328% "regressions" across unrelated micro-benchmarks (e.g.
ExtractWorkflowNameFromFile, a filename parse) with no plausible shared code path. Root cause is measurement methodology, not code:bench-performanceran each benchmark 3 times and compared against a mean baseline, so a single busy shared runner produces large phantom deltas.Makefile:
bench-performance-benchtime=3x→-benchtime=2sfor the workflow benchmarks, and1s→2sfor the CLI helpers.daily-cli-performance.md: noise-resistant trend analysislikely_environment_noisesignal: set when ≥3 benchmarks regress and regressions+warnings cover ≥50% of tracked benchmarks — broad simultaneous slowdown across unrelated code is a runner symptom, not a code symptom.When flagged,
regressions.jsonis emitted empty and the agent is instructed to report the noise and recommend a re-run instead of filing per-benchmark issues. Isolated regressions (1–2 benchmarks) still file issues as before.Not changed
The 2026-07-06 → 2026-08-18 history gap is expected:
pre_activationskips the run unlessgo.mod,go.sum,Makefile,pkg, orcmdchanged in the prior 24h. The skip logic is working as designed.daily-cli-performance.lock.ymlrecompiled.