stuck-automerge alert: prove the issue-filing path fires and fail loudly on mint errors

[kyle-sexton]

Summary

The scheduled standards-sync-stuck-automerge-alert workflow has ZERO non-failure runs in its entire visible history (last 80 runs, back to 2026-07-31) — the stuck-automerge monitor has been dark the whole time and would not have alerted on anything.

Root cause (from run logs): the App-token mint step 404s on the app-installation endpoint while requesting repositories .github,ci-runner,ci-workflows,claude-code-plugins,dotfiles,github-iac,medley,provisioning — the App appears uninstalled on at least one requested repo (medley / provisioning are the newest suspects, since a mint scoped to a repo outside the installation's selection fails the whole request).

Note: these failures attach check-runs to main's head SHA, which makes unrelated merged commits (e.g. f96b2e57) show red — cosmetic fallout worth knowing when reading CI history.

Test plan

• Compare the workflow's requested repository list against the App installation's actual selection; reconcile (drop the uninstalled repo from the request, or extend the installation — the latter is an org-owner action).
• After the fix: one green scheduled run, and an induced stuck-automerge condition (or a dry-run assertion) proving the alert path actually fires.

Related

• Surfaced by a read-only session-ledger audit (ci-workflows .work/agent-returns/session-ledger-audit.md).

[kyle-sexton]

Root cause corrected: the 404 was on the caller repo, not on any requested target

The hypothesis in the description — that one of the eight requested repositories sits outside the App installation's selection — is refuted.

The failing mint names standards, not a target. Last failing run 30870238869 (2026-08-04T01:55Z):

Failed to create token for "melodic-software/standards" (attempt 1): Not Found
  - https://docs.github.com/rest/apps/apps#get-a-repository-installation-for-the-authenticated-app
status: 404

That is the reusable's second mint — the one for the tracking issue — which carried no repositories input and so defaulted to the calling repository. standards is the sync source, deliberately not a target, and therefore outside the App's selected access.

The eight-repo mint succeeds, with the list unchanged. Latest scheduled run 30916292521 (2026-08-04T13:55Z) requests the identical list .github,ci-runner,ci-workflows,claude-code-plugins,dotfiles,github-iac,medley,provisioning, and both mint steps report success:

Step	Conclusion
Mint read-only App token scoped to every target	success
Mint App token for the tracking-issue repository (repositories: medley)	success

Already fixed

PR #319 (merged 2026-08-04T03:03:30Z) added tracking-issue-repository: medley and re-pinned the reusable to ed6d410c. The first run after that merge was 03:03:44Z, and every run since has been green — 11 consecutive successes through 13:55Z. #273 tracked the same defect and is closed by that PR.

Do not widen the App installation

Adding standards to the installation's selected repositories would clear the 404 and break sync.yml for all eight targets. standards-sync.yml in ci-workflows at 8202e03f compares the installation's repository list against the derived target set by strict count equality:

if (totalCount !== expectedSet.size) {
  fail(`installation reports ${totalCount} repositories; expected ${expectedSet.size}`);
}

A ninth repository fails attest, which gates the sync job.

What is still open

The first item of this issue's test plan is settled; the second is not.

• One green scheduled run — satisfied, with the mint step itself proven rather than inferred from a green run.
• An induced stuck-automerge condition proving the alert path actually fires — not satisfied. All eight manifest targets carry automerge: false, so the never-armed half is inert by design and the armed-but-blocked half has nothing to find. In the latest green run, Find existing tracking issue and Open or update tracking issue are both skipped. The issue-filing path has still never executed.

Separately, #274's item 3 — make the alert fail loudly on a token-mint error, so the next occurrence is visible without someone reading run history — remains unaddressed. That residual is why the defect went unnoticed across the whole failure window.

Recommend keeping this issue open, rescoped to the unexercised alert path, rather than closing it as a duplicate of #273.

[kyle-sexton]

Rescoped: the 404 is fixed; this issue now tracks the two residuals

The mint 404 this issue was filed for is fixed by #319 (merged 2026-08-04T03:03:30Z: tracking-issue-repository: medley, reusable re-pinned to ed6d410c). Independent adversarial verification confirmed the root-cause comment above and added two corrections:

• The monitor was not dark "since 2026-07-31" — it was dark its entire life. The workflow has 270 runs; the oldest on record (29963472498, 2026-07-22T22:37Z) failed at the same mint step. 2026-07-31 was only this issue's 80-run visible-history horizon. Zero successes before the fix; the alert has never filed an issue.
• Green runs since fix(ci): re-pin the watchdog and file its tracking issue in medley #319 already exercise more than assumed: "Close recovered tracking issue" runs (both stuck counts zero), so the medley-scoped mint AND an issues search against medley are proven live. The genuinely unproven surface is narrower: the issues write path (open/update, or closing a real issue) and the loud-failure step.

This issue now carries the two genuinely open items, consolidated here from #274 (closed as duplicate — its suggested fix, widening the App installation to include standards, is foreclosed: sync.yml's attest job fails when the installation reports more than the 8 expected repos, standards-sync.yml@ed6d410c readRepositorySnapshot):

1. Prove the issue-filing path fires. All 8 manifest targets carry automerge: false, so "Find existing tracking issue" / "Open or update tracking issue" / "Fail so the scheduled run notifies" have executed zero times in 270 runs. Trigger: once any target restores automerge: true, induce (or wait for) a stuck-automerge condition and confirm a tracking issue is actually opened in medley.
2. Fail loudly on token-mint error (was fix(ci): standards-sync-stuck alert fails hourly — App token minting 404s on /standards/installation #274 item 3). The whole outage went unnoticed because a mint failure kills the run without any alert-liveness signal. ed6d410c has no failure() handler; the liveness pattern exists (queue-monitor-liveness.yml, from ci-workflows feat(distribution): retarget transferred repos to melodic-software #105) and was never applied to this alert.

[kyle-sexton]

Orchestration handoff (class C — host/operator)

This issue is classified as not fully completable from the cloud-agent environment for the drain pass running now. Work stays parked here; no new tracking issue will be filed.

Operator checklist will be consolidated at the end of the drain session (Wave 4 packet). Do not treat silence as abandonment.

🤖

[kyle-sexton]

Wave 4 operator host packet

Why parked: Mint path fixed (#319). Proving the alert issue-filing path still needs either an automerge: true sync target or a dry-run harness in ci-workflows, plus App behavior validation. Cannot fully verify from this cloud environment alone.

Operator actions:

1. Compare App installation repo selection vs workflow requested repos (already partially reconciled).
2. Induce a stuck-automerge condition (or land a ci-workflows dry-run assertion) and confirm an issue is filed.
3. Ensure mint errors fail loudly in the reusable (ci-workflows) and re-pin the caller.

No new issue filed. 🤖

[kyle-sexton]

Lease claim — implementing/closing in-session (no new issues). 🤖