ci-health: detect checks that degrade mergeStateStatus without completing (stuck runner-label / orphaned status / never-settling)

[kyle-sexton]

Problem

Some checks degrade mergeStateStatus to UNSTABLE without ever completing, blocking a
merge-readiness read even when every REQUIRED check is green. Three observed classes:

1. A job stuck QUEUED forever on an unmatched self-hosted runner label (post-migration sentinel).
2. An orphaned StatusContext: posted pending with null creator / empty target_url / no backing
   run — never resolves, no branch-side run to cancel.
3. A non-required check that never settles but still holds UNSTABLE.

Dominant operational wall of the autopilot run — the only thing that blocked reaching literal zero.
Cancelling makes it worse (CANCELLED is a failure state). Evidence: medley#1567 (sentinel),
medley#1597 (orphaned codex-review); retrospective
.work/handoffs/babysit-autopilot-retro.md §1.2 / §1.3 / §1.4 / §2.1.

Recommended shape (re-derived blind to the original "skill" framing)

NOT a skill. Engine signal (detect) + a routing reference doc. Detection is the engine's job;
routing/escalation is a judgment call prose owns.

Detection — in the engine, zero new fetches

mergeStateStatus == UNSTABLE is itself the trigger: it already means "mergeable, required gates
satisfied, a non-required commit status not passing" (per the pull-request skill's
reference/readiness.md). The snapshot engine already captures mergeStateStatus and normalizes
every check with type (StatusContext vs CheckRun), effective_state, target_url/details_url,
and a timestamp — so all three classes are detectable with no new fetch:

• (b) orphaned StatusContext: type==StatusContext + pending + empty target_url.
• (a) stuck QUEUED: type==CheckRun + QUEUED + age past a threshold (age = snapshot
  generated_at − check createdAt/startedAt). The root cause (unmatched runner label) needs an
  Actions-jobs API call — that is remediation diagnosis, fetched on demand by the playbook, not
  baked into every cycle.
• (c) non-required never-settler: under UNSTABLE, anything still pending/failing that isn't (a)/(b)
  past the age threshold.

Emit as a new snapshot per-PR JSON field (e.g. checks.stuck[] = {name, type, class, target_url, details_url, age_seconds}). Formalize the internal sort timestamp into a public age_seconds;
consumers cite the snapshot CLI + JSON field name, never the internal classifier module.

Correctness guard (critical)

The classified stuck signal must surface as a material_findings entry (report/escalate), never a
blockers string — a sticky blocker would pin classification==active and re-dispatch a worker
every cycle (the fan-out trap the suppressible/unsuppressible delta model exists to prevent).
Branch-config-fixable cases (wrong runs-on: in the branch's own workflow YAML) ride the normal
head_sha_changed delta; org/settings-class (runner pool, orphaned status, branch protection) stay
material + escalate. Fire only under UNSTABLE + age threshold (configurable, mirroring
max_quiet_recheck_seconds) so normal in-flight CI and pending required checks are never flagged.

Routing — in prose

New reference/stuck-checks.md (sibling of freshness.md), wired into SKILL.md References +
operational runbook. Points at — never restates — safety.md's Stop-and-Ask / Never-Do-Automatically
rules (runners/settings/branch-protection), the merge wrapper's requiredChecks JSON (confirm
required-green), and the new checks.stuck field. Route CI-config fixes upstream (branch CI /
ci-workflows select-runner); orphaned-status/settings to github-iac / app config.

Files

• scripts/babysit_checks.py — per-check stuck-class from type/state/target_url/timestamp (the
  shared classifier, so the merge gate benefits too)
• scripts/babysit_delta.py — compute the signal in the per-PR classifier that already derives
  blockers/material_findings and holds mergeStateStatus
• reference/stuck-checks.md (new) + SKILL.md (References + runbook step)

Rejected shapes

Standalone script (re-implements check normalization = a 2nd source of truth), pure prose skill
(can't detect → reintroduces the survey-without-classifying failure), subagent (no benefit for a pure
function over data in hand).

SSOT note

The fact "UNSTABLE from a stuck non-required check ≠ a required-check failure" currently lives in the
pull-request skill's readiness reference. Optionally lift it to the plugin-scope shared reference/
(where review-discipline.md already serves both skills) and point both at one copy.

Related

Merge-gate stuck-check policy (retro §1.4): the gate correctly refuses UNSTABLE and forbids an
--admin/gh pr merge bypass — this auditor is the clean path, not a gate-bypass mode.

[kyle-sexton]

This was generated by AI during triage.

T2 triage planning comment. Implementation plan verified against the real code (not the issue's approximated paths). Ready for an autonomous or human implementer with no other context.

---

Verification summary (what grounds this plan)

Real file locations (the issue's scripts/babysit_checks.py / scripts/babysit_delta.py are under the skill dir):

• plugins/source-control/skills/babysit-prs/scripts/babysit_checks.py — pure shared check-rollup classifier (normalize_check, classify_checks). The guarded merge gate (babysit_merge.py:172) also calls classify_checks, so anything added here must stay pure (rollup-only, no PR/merge/state context).
• plugins/source-control/skills/babysit-prs/scripts/babysit_delta.py — classify_pr() holds mergeStateStatus (as merge_state), generated_at/observed_at, previous-cycle state, and builds both blockers and material_findings.
• plugins/source-control/skills/babysit-prs/scripts/babysit_state.py — persisted_pr_state() is where per-PR cross-cycle state (incl. check identities) is written.
• plugins/source-control/skills/babysit-prs/scripts/pr_queue_snapshot.py — orchestrator + CLI flags; build_config() maps flags → ClassifyConfig.
• Reference docs: plugins/source-control/skills/babysit-prs/reference/ (siblings freshness.md, safety.md). Plugin-scope shared refs: plugins/source-control/reference/ (holds review-discipline.md).

Zero-new-fetch claim — verified live against real rollups (gh pr view --json statusCheckRollup). Authoritative per-type field sets:

• CheckRun: __typename, name, status, conclusion, startedAt, completedAt, detailsUrl, workflowName. No createdAt, no targetUrl.
• StatusContext: __typename, context, state, targetUrl, startedAt. No creator, no detailsUrl.

normalize_check() already exposes everything needed per check: type, effective_state, state/status/conclusion, target_url, details_url, workflow_name, and a private _sort_key (= completedAt || startedAt || createdAt). So classification needs no new API call.

Two corrections to the issue's sketch (both from the live field sets):

1. The issue's class-(b) "null creator" signal is not available zero-fetch — gh's StatusContext rollup carries no creator. Use empty target_url alone as the orphan discriminator (that field is present).
2. The issue's age formula "generated_at − createdAt/startedAt" is insufficient for its own flagship class (a): a job stuck QUEUED on an unmatched runner label never starts, so its CheckRun has null startedAt/completedAt (and gh gives CheckRun no createdAt) → _sort_key == "" → no self-timestamp to age from. StatusContext (class b) and started-then-hung checks (class c) do carry startedAt. Resolution below: dual-source age (own timestamp when present, else a persisted first-observed-stuck anchor). This stays zero-fetch. (The precise null-startedAt-for-QUEUED behavior is plausible but I could not capture a live stuck-QUEUED rollup to confirm; the dual-source design is correct either way, so it is not a blocker.)

---

What actually changes (and, deliberately, what does not)

This is a purely additive diagnostic overlay. It renames nothing and re-gates nothing:

• A stuck pending/failing check already appends a blockers entry (babysit_delta.py:462-465) and already forces classification == "active" (:537). The new signal does not remove those, does not change merge gating (the merge gate's UNSTABLE-refusal at babysit_merge.py:58 READY_MERGE_STATES = {CLEAN, HAS_HOOKS} is correct and stays), and does not add anything to babysit_merge.py (the "so the merge gate benefits" line in the issue is an over-claim — the gate needs no change).
• The real anti-fan-out guard is "not a delta arm," not "material vs blockers." In this engine a sticky blocker does not re-dispatch every cycle — the suppressible/unsuppressible delta model already prevents that (a still-pending check produces no checks_changed, hence no delta). So the issue's stated rationale ("a sticky blocker re-fires a worker every cycle") is mechanically inaccurate; the correct enforcement is: the stuck signal is never added to suppressible_delta, unsuppressible_delta, or needs_worker_reasons, and never drives needs_worker. material_findings is purely its reporting bucket (and, unlike blockers, appending to it does not set classification=active). Honest consequence: a stuck PR still receives a worker roughly every quiet_recheck window (default 4h, via quiet_recheck_due) — periodic re-escalation, not zero dispatch. The queue reaches literal zero only when the orphan/runner is fixed out-of-band; this signal is the escalation surface that makes that fix discoverable.
• Class (c) needs no required/non-required bit. normalize_check carries no such flag and fetching one would break zero-fetch. It is unnecessary: under UNSTABLE, every still-pending/failing check is non-required by definition — a non-green required check yields BLOCKED, not UNSTABLE (see the pull-request skill's reference/readiness.md:70). So "under UNSTABLE" is the non-required filter.

---

Change 1 — babysit_checks.py: pure per-check stuck classifier

Add a pure, context-free helper (no age, no merge_state — the caller owns those):

def stuck_class(check: dict[str, Any]) -> str | None:
    """Classify one normalized check as a stuck-candidate class, or None.

    Pure: sees only a normalized-check dict. Age gating and the UNSTABLE
    trigger are the caller's job (babysit_delta), because this same helper
    feeds the pure rollup classifier that the merge gate also uses.
    """
    typ = check.get("type")
    category = check.get("category")            # "failing" | "pending" | "success"
    if typ == "StatusContext" and category == "pending" and not check.get("target_url"):
        return "orphaned_status"
    if typ == "CheckRun" and check.get("effective_state") == "QUEUED":
        return "stuck_queued"
    if category in {"pending", "failing"}:
        return "never_settling"
    return None

• classify_checks() is left unchanged (it must stay the pure shared classifier the merge gate calls). The stuck fields already flow through checks["checks"] (the normalized list) that classify_checks returns.
• Also promote the private _sort_key to a documented public field on each normalized check, e.g. add "observed_ts": sort_key in normalize_check's return dict (keep _sort_key for existing sort callers, or rename its consumers — one call site: dedupe_latest_checks). Consumers must cite observed_ts, never _sort_key.

Change 2 — babysit_delta.py: assemble the signal in classify_pr

Add near the other constants (mirror DEFAULT_MAX_QUIET_RECHECK_SECONDS):

DEFAULT_STUCK_CHECK_AGE_SECONDS = 30 * 60   # 30 min; tunable via --stuck-check-age-seconds

Add stuck_check_age_seconds: float = DEFAULT_STUCK_CHECK_AGE_SECONDS to ClassifyConfig.

In classify_pr, only when merge_state == "UNSTABLE", iterate checks["checks"]:

1. cls = stuck_class(check); skip if None.
2. Age (dual-source, zero-fetch):
   • If check["observed_ts"] is non-empty → age = observed_at − observed_ts.
   • Else (null-timestamp stuck-QUEUED) → look up a persisted first-observed-stuck anchor keyed by check_identity_key(check) ((type, name, workflow_name)) from prev.get("stuck_first_seen"); if absent, record observed_at as the anchor for this cycle and treat age = 0; if present, age = observed_at − anchor.
   • Anchor map is reset whenever head_sha_changed (a new head = fresh CI = fresh clock), so a stale anchor from a superseded head never fires.
   • age_seconds semantic: true wall-age when self-timestamped; else "seconds since first observed stuck" — document this in reference/stuck-checks.md.
3. If age >= config.stuck_check_age_seconds, append to a local stuck list an entry (shape below). Apply the age gate to all three classes uniformly (decision: yes — including class (b) orphan; this avoids flagging a legitimately-just-posted pending StatusContext in its first seconds before it settles).
4. After the loop, attach checks["stuck"] = stuck (so it rides the existing pr["checks"] object into JSON) and, if stuck is non-empty, append exactly one material_findings entry — e.g. material.append(f"{len(stuck)} stuck check(s) degrading UNSTABLE without settling: {', '.join(...classes...)}"). Never touch blockers.

checks["stuck"][] entry shape (public snapshot contract):

{ "name": "...", "type": "CheckRun|StatusContext", "class": "orphaned_status|stuck_queued|never_settling",
  "target_url": "", "details_url": "...", "age_seconds": 1830 }

Do not add the stuck signal to suppressible_delta, unsuppressible_delta, needs_worker, or needs_worker_reasons — leave all fan-out arms untouched.

Change 3 — babysit_state.py: persist the anchor

In persisted_pr_state(), write "stuck_first_seen": <map of check_identity_key→timestamp for the current head>, built from this cycle's stuck candidates (reset when head changes, per Change 2). This is the only new persisted field.

Change 4 — pr_queue_snapshot.py: CLI plumbing

• Add --stuck-check-age-seconds (type=float, default=delta.DEFAULT_STUCK_CHECK_AGE_SECONDS), mirroring --max-quiet-recheck-seconds exactly.
• Wire it into build_config() → ClassifyConfig(stuck_check_age_seconds=...).
• Optional: surface stuck entries in print_text() (text mode), e.g. under each PR: stuck checks: <name> (<class>, <age>s).

Change 5 — new reference/stuck-checks.md + SKILL.md wiring

New plugins/source-control/skills/babysit-prs/reference/stuck-checks.md (sibling of freshness.md). Outline:

1. What the signal is — the checks.stuck[] field on the snapshot (cite the snapshot CLI + JSON field name; never the internal classifier module), the three classes, the UNSTABLE-only trigger, and the age_seconds semantic (incl. the anchored "since first observed" case).
2. Why it is diagnostic, not gating — it does not change merge readiness; the merge gate's UNSTABLE refusal is the correct, un-bypassable path (no --admin); this signal is how the operator finds the out-of-band fix, not a bypass.
3. Routing — point at, never restate, safety.md's Stop And Ask / Never Do Automatically rules for runners / settings / branch-protection; and the merge wrapper's requiredChecks JSON to confirm required-green. Per class: stuck_queued → CI-config fix upstream (branch's own runs-on: rides the normal head_sha_changed delta; runner-pool issues escalate to ci-workflows select-runner / org settings); orphaned_status → github-iac/app config; never_settling → report + escalate.
4. Cancelling makes it worse — CANCELLED is a failure state; never auto-cancel.

SKILL.md edits:

• References section (:466): add a bullet for reference/stuck-checks.md.
• Operational runbook (:385, the step that reacts to blockers and material_findings): add "load reference/stuck-checks.md only when the snapshot reports checks.stuck[]" alongside the existing conditional-load list (freshness/feedback/etc.).

SSOT note (optional, from the issue)

The fact "UNSTABLE from a stuck non-required check ≠ a required-check failure" currently lives only in the pull-request skill's reference/readiness.md:70. Optionally lift it to the plugin-scope shared reference/ (alongside review-discipline.md) and have both stuck-checks.md and readiness.md point at one copy. Defer if it widens scope.

---

Test / verification plan (extend existing files)

• scripts/tests/test_babysit_checks.py — unit-test the pure stuck_class(): orphaned StatusContext (pending + empty target_url) → orphaned_status; StatusContext pending with target_url → None; CheckRun QUEUED → stuck_queued; CheckRun pending/failing non-queued → never_settling; success/neutral → None. Assert normalize_check exposes the public observed_ts.
• scripts/tests/test_babysit_delta.py — (a) stuck signal lands in material_findings, never blockers; (b) fires only under merge_state == "UNSTABLE" (no emission under CLEAN/BLOCKED/BEHIND); (c) age threshold gates emission (below threshold → nothing); (d) cross-cycle anchor: a null-timestamp QUEUED CheckRun records an anchor cycle 1, fires once observed_at − anchor >= threshold, and the anchor resets on head_sha_changed; (e) needs_worker stays False and needs_worker_reasons is unchanged when the stuck signal is the only thing present (no new delta).
• scripts/tests/test_pr_queue_snapshot.py — assert the JSON snapshot carries prs[].checks.stuck[] with the documented shape and that --stuck-check-age-seconds plumbs into ClassifyConfig.

Run: plugins/source-control/skills/babysit-prs/scripts/engine.test.sh (the existing test entrypoint) must stay green.

Confidence

HIGH. All load-bearing claims verified against real code and live rollups; the one genuine wrinkle (null startedAt on stuck-QUEUED CheckRuns) is resolved with a concrete zero-fetch dual-source age design that fits the existing state-persistence pattern. No open design question blocks implementation.

[kyle-sexton]