Repository navigation
fix: make the required check measure what we actually depend on (#1472, #1491, #1568) - #1575
Merged
Merged
Conversation
…d on Three ways the one required status check reported more confidence than it had. #1472 — a shard that never ran left the check green. On 2026-08-14 a codeload outage made setup-dotnet unfetchable and shard 0 never started; the five surviving shards produced trx with no failures, the evidence gate saw a non-zero trx count, and `Consolidate test results` reported success on five sixths of the suite. "No results because it never started" and "no failures" are different inputs and counting failures cannot tell them apart. The consolidation now counts REPORTERS against the statically-known matrix size (CI_SHARD_TOTAL) and names the missing shards, plus a direct check on the matrix job's own result. No `if:` on artifact presence, no continue-on-error — the only condition is the reuse path, which is a run mode, not evidence. #1491 — three honest greens, none covering the tree that went red. A run that reuses a green marker concludes success without executing anything, so "last green main run" is not "last main run that executed the tests", and the natural `git log <last-green>..<red>` window pointed at the wrong PR. Marker refs now carry provenance (`refs/ci-green/<tree>/<epoch>-<run_id>`; readers split on the first `-`, legacy bare-epoch markers still parse), a reusing run names the run that actually gathered the evidence, and a main run whose suite genuinely ran and passed records `refs/ci-executed/main/<epoch>-<run_id>` at its commit. A red run then STATES its bisect window from that ref instead of leaving it to be reconstructed from a run list that cannot distinguish the two kinds of green. #1568 — PR #1555 merged with the entire Clients workflow red, including Portal-next, which is the same `npm run build` main-cd's portal-next image leg runs. promote is all-or-nothing, so CD published nothing for ~1h and a production 42P18 fix was stranded. Clients cannot simply be marked required: it is path-filtered, and a legitimately-skipped required check blocks merges forever. So a new always-runs `clients-gate` job reports that workflow's verdict into the required check. Zero runs means the path filter did not match — a fact about the diff, not missing evidence — and is only believed after a 180s creation-grace window so it cannot race run creation. main-cd's loud-failure half of #1568 already exists (alert-on-failure files a ci-failure issue, verify-images asserts the complete set); the missing half was the merge gate, which this adds. Verified: actionlint clean against the origin/main baseline (the single remaining SC2016 is pre-existing), `bash -n` over every run: block, YAML parses. Fixes #1472 Fixes #1491 Fixes #1568 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmBFaHTqhy7h742wkAaJKj
Contributor
There was a problem hiding this comment.
Pull request overview
This PR tightens the repo’s merge gate (Consolidate test results, the only required check) so it reflects the real evidence we depend on, and makes CI diagnostics more trustworthy when runs reuse prior greens or when filtered workflows don’t run.
Changes:
- Adds explicit shard coverage assertions so “no failures” cannot be mistaken for “every shard ran and reported” (#1472).
- Records and surfaces provenance for green-tree reuse, and records “last main commit whose tests actually executed” to produce a correct bisect window on red runs (#1491).
- Introduces an always-runs
clients-gatejob that folds the path-filteredClientsworkflow result into the required check (#1568).
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
Contributor
Contributor
Test Results (shard 4)1 924 tests - 88 1 914 ✅ - 90 6m 57s ⏱️ +21s Results for commit 9336744. ± Comparison against base commit 179d66b. This pull request removes 436 and adds 348 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Contributor
Test Results (shard 3) 11 files ± 0 11 suites ±0 5m 58s ⏱️ -16s Results for commit 9336744. ± Comparison against base commit 179d66b. This pull request removes 348 and adds 438 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Contributor
Contributor
Contributor
Copilot review on #1575: the clients gate collapsed a gh api failure (auth / rate limit / network) into total_count=0, which after the grace window reported 'not applicable' — a verdict it had never obtained. That is the exact pass-on-no-evidence trapdoor this job exists to close, one level up. Fetch failure and a genuine zero are now tracked separately: a failed query is retried inside the poll, and if the query never succeeds once, the job fails RED naming the last error. Only a SUCCESSFUL query returning zero runs can reach the not-applicable verdict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KmBFaHTqhy7h742wkAaJKj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three filings, one theme: the required status check reported more confidence than it had.
#1472 — a shard that never ran left the check green
A codeload outage made
setup-dotnetunfetchable and shard 0 never started. The five surviving shards produced trx with no failures, the existing evidence gate saw a non-zero trx count, andConsolidate test results— the repo's only required check — reported success on five sixths of the suite.mergeStateStatuswas merelyUNSTABLE, which does not block."No results because the shard never started" and "no failures" are different inputs, and counting failures cannot tell them apart.
Every shard must have reportedstep compares reporters against the statically-known matrix size (CI_SHARD_TOTAL: 6) and names the missing shards.Fail if any shard job did not succeedstep covers the same defect from the other side (a shard that failed to start, was cancelled, or died before uploading).if:on artifact presence and neither iscontinue-on-error. The only condition is the reuse path — a run mode, not a property of the evidence.#1491 — three honest greens, none covering the tree that went red
A run that reuses a green tree marker concludes
successwithout executing anything. So "last green main run" ≠ "last main run that executed the tests", andgit log <last-green>..<red>produced a window containing exactly one PR — the wrong one.refs/ci-green/<tree>/<epoch>-<run_id>. Readers split on the first-; legacy bare-epoch markers still parse, and the prune parser was updated to match.refs/ci-executed/main/<epoch>-<run_id>at its commit.git log <sha>..<red>), instead of leaving it to be reconstructed from a run list that cannot distinguish the two kinds of green.Neither weakens the reuse — it stays a real ~22 min saving and it is correct. They make the diagnostic honest about what a given green covers.
#1568 — a red Clients workflow stopped all image publication
PR #1555 merged with the entire Clients workflow red, including
Portal-next (typecheck + test + build)— the samenpm run buildthatmain-cd's portal-next image leg runs.promoteis all-or-nothing, so CD published nothing for ~1 h and a production42P18fix was stranded.Clients cannot simply be marked required: it is path-filtered, so on a server-only PR it never runs, and a legitimately-skipped required check blocks merges forever (exactly why
Build solution (once)and the shards are not required). So this adds the shape the issue itself prescribes — a small always-runs job that reports the filtered workflow's result:clients-gateresolves the Clients workflow's conclusion for the PR head sha and fails on anything but success/skipped.if:asking whether a secret is set). The job always runs, always reports, and never tests its own inputs.The "narrower alternative" in the issue — main-cd failing loudly — already exists (
alert-on-failurefiles aci-failureissue,verify-imagesasserts the complete four-image set). The missing half was the merge gate; that is what this adds.Verification
actionlintclean against theorigin/mainbaseline — the one remainingSC2016info is pre-existing (same step, unchanged).bash -nover everyrun:block in the workflow (GH expressions stripped) — clean.precheck, preflight, build, test, plugin-gate, clients-gate, doc-gate, licence-gate → collect-results.dotnet-test.yml, soclients-gateexercises its own not-applicable branch on this very run.No What's New entry: CI-internal, no user-visible effect.
Fixes #1472, #1491, #1568
🤖 Generated with Claude Code
https://claude.ai/code/session_01KmBFaHTqhy7h742wkAaJKj