Skip to content

A red Clients workflow merged and silently stopped ALL image publication for ~1h — it is not a required check #1568

Closed

Description

@rbuergi

On 2026-08-14, PR #1555 merged with the entire Clients workflow RED — all 7 client jobs failing, including Portal-next (typecheck + test + build). Nothing blocked it, because Consolidate test results is the only required status check and it is green independently of Clients.

The consequence was not confined to the clients.

What it cost

main-cd.yml builds four images and its promote job applies real tags only after every leg succeeds (all-or-nothing, by design — a partial image set is worse than none). The portal-next leg builds by running npm run build in the image. #1555 bumped next ^15.5.22 → ^16.3.0, where next build defaults to Turbopack and hard-errors on a webpack config with no turbopack config. That leg has died in ~1.1 s on every CD run since.

So CD published nothing at all from 12:05Z onward:

CD run commit promote
#3681 66822b3f (11:35Z) ✅ succeeded — last published tag 3.0.0-rc2.ci.3681
— fcbfe8b2a (12:05Z, PR #1555 merged) —
— 3cf807f3 (12:25Z) ❌ portal-next leg failed, promote skipped
— de1d108c (13:00Z) ❌ portal-next leg failed, promote skipped

.github/scripts/check-image-set.sh de1d108c → all four images missing.

Every self-updating install stayed on the previous image, and a production fix was stranded: de1d108c6 fixes 42P18 (which wedges a portal's NodeType bake — see the memex-cloud incident the same day), and it could not ship. That namespace ran on a live config override instead.

Why this is a required-checks problem, not a dependabot problem

The bump itself is fine and probably wanted. The defect is that a workflow whose failure can stop image publication is not a merge gate. Clients had exactly the signal needed — Portal-next (typecheck + test + build) was red, naming the build that CD would later run — and it was advisory.

This is the same family as #1472 (a shard that never ran left the required check green) and #1491 (three honest greens, none covering the tree that went red): the required check is not measuring what we actually depend on.

What to decide

Make Clients (or at least its Portal-next job) a required status check on main.

🚨 Before doing that, check it against the reason AGENTS.md keeps exactly one required check: a required check that gets legitimately skipped blocks merges forever. Build solution (once) and the test shards are skipped when a run reuses an already-green tree, which is why they are not required. So the question to answer per job is: can it skip? A job with no path filter and no input-shaped if: cannot, and is safe to require. If Clients is path-filtered to clients/**, it WILL skip on server-only PRs and must not be required as-is — the fix is then a small always-runs job that reports the clients result, not requiring the filtered one.

A narrower alternative worth weighing: have main-cd.yml fail loudly and visibly when a leg fails (it currently skips promote, which looks like "nothing to do"), plus an alert on "main tip has no complete image set for N minutes". That catches any cause of a stranded image set, not just this one. The reconciler already checks the image set — it just does not shout when the answer is "no".

Related: #1472, #1491. Surfaced by the portal-next Turbopack break (fix in #1567).

Activity

  1. added a commit that references this issue on Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions