Repository navigation
v1.90.2.0 perf: halve local free-suite time and preserve coverage - #2972
Merged
Merged
Conversation
|
😎 This pull request was merged. |
Contributor
E2E Evals: ⚠ MANUAL ACCEPTED (unscored)106 automated passed / 107 final results | 0 failed, 1 manual accepted (unscored; no score-cache credit) | 107 executed, 0 reused | $35.14 total cost | reconcile exit: 0 | ⚠ 1 cases with multiple attempts Profile: pr / full-fallback; selected behaviors: 84, judges: 26. Deferred to scheduled/release coverage: 123 behaviors and 0 changed prompt files. Deferred checks did not run and receive no PR-pass credit.
Fail-closed reconciliationSliced lane: declared PR profile or broad fallback via scripts/test-paid-shards.ts (planner → 6 executors → fail-closed report). Reused scores retain their original provenance and expiry. |
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cut local test waiting without removing scenarios, reducing budgets or moving coverage to a later tier.
/land-and-deployapproval, merge/queue readback, staging, deployment reporting and rollback paths. Head or destination changes invalidate approval; confirmed merges are not replayed; unknown deployment status remains unknown.Measured performance
The free comparison used the same four-CPU Linux machine, Bun 1.4.0 and Node 24.18.0, with identical build/display/Git prerequisites and no worker override. Its pass count increased from 25,504 to 25,547, with zero failures and the same 60 skipped identities. These matched measurements precede the additional repair regressions; they are not whole-paid-suite or CI speedup claims. Exact component selectors and coverage ownership are in
docs/TEST_PORTFOLIO.md.Validation
Integrated revision:
c32df847430d65bf736ff87a63ee29672439303d, merging maina84b0b5b6d73fc1d7f97d95dcce484f2ffd334dcinto this PR without rewriting history. Fresh merged-head GitHub CI is green: 55 checks passed, zero failed or pending. Four manual-dispatch/release-only jobs were intentionally skipped. All 20 Linux groups, all six AI E2E groups, Windows and their aggregate gates passed. The paid report records 106 automated passes plus one manual acceptance (unscored, no score-cache credit) across 107 executed results; none were reused. The manual case is the existing upstream-approvedsetup-browser-cookiesjudge exception for an exact provider refusal, not a new waiver.shared-libs-review-path-eligibilitytimed out on its first attempt and passed its configured retry; both attempts remain recorded.Retained pre-integration evidence
The preceding head
4082ef73had 55 successful CI checks, zero failures or pending checks, plus two intentionally skipped release-only jobs. Its paid run reconciled 106/106 final results: 92 executed and 14 reused, and its native Windows run passed both shards across 664 files.Local selected validation before this catch-up covered 90 gate callbacks, 25 quality judges and two explicit periodic callbacks. Of those 117 results, 98 retained prior/upstream evidence after consumed-input comparison, 16 retained compatible integrated evidence and three ran on
95333c80. Original source, time, attempts and cache provenance remain recorded. The three AUQ consistency samples and both A/B arms retained 7/7 format and substance 5. The fresh merged-head CI results are reported separately above; these historical results are not relabeled as executions on the merge commit.Original failures and retries remain preserved, including the pre-repair symlink detector failure, Windows alias timeout and upstream Design UI timeout. The alias repair retains both install passes and the original hook deadline; its negative control still detects write-through without changing checkout skill files. Outside-model review timeouts remain unavailable coverage, not clean-review credit.
Coverage limits
Current local acceptance is complete Linux free validation. Selected paid, quality and periodic evidence above is attributed to its original pre-integration source; fresh merged-head CI is green with the explicit manual/unscored exception and retry noted above. The 123 scheduled/release deferrals remain explicit and receive no pass credit. The two all-skipped paid shards (
skill-routing-e2eandllm-judge-recommendation, 11 skipped tests total) likewise receive no coverage credit. Full native macOS/Windows/iOS, the complete periodic census and whole-paid/CI performance gains are not claimed. The retained Windows evidence covers the curated Windows suite, not the complete test portfolio. Deployment fixtures simulate workflows; no real merge, deployment or rollback was performed.No merge or auto-merge is requested.