Skip to content

fix(standards-sync-stuck-automerge-alert)!: author the tracking issue with the ambient token - #347

Closed
kyle-sexton wants to merge 2 commits into
mainfrom
fix/watchdog-ambient-issue-authorship
Closed

kyle-sexton wants to merge 2 commits into
mainfrom
fix/watchdog-ambient-issue-authorship

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

No linked issue in this repository. The two issues that track this defect live
in melodic-software/standards (#273, #274) and are deliberately not closed by
a native keyword here: this PR is only the ci-workflows half, and cross-repo
closing keywords fire on merge.

What was broken

standards-sync-stuck-automerge-alert.yml has failed every run since it was
first scheduled
— 254 of 254 at the time of investigation, 100 of the last
100 confirmed by gh run list. Zero successes, ever.

Detection was never the problem. The Mint App token for this repository's tracking issue step minted a second App token with owner and
repositories omitted, which actions/create-github-app-token resolves to the
calling repository. The only caller is melodic-software/standards — the
sync source, which distribution/sync-manifest.yml states is "manifest
source, not a target" and is therefore outside the sync App's 8-repo selected
installation. Live evidence from run
30862951664:

Failed to create token for "melodic-software/standards" (attempt 1): Not Found
  url: 'https://api.github.com/repos/melodic-software/standards/installation'
  status: 404

The first token mint — owner: melodic-software, repositories: <manifest targets> — completed fine two seconds earlier. Only the caller-scoped mint
404s.

The fix

Converge this workflow onto the ambient-GITHUB_TOKEN authorship its sibling
rolling-report workflows already use, and retire the App identity for
tracking-issue authorship.

Ambient GITHUB_TOKEN is scoped to the calling repository by construction,
which is precisely what a 404 on the caller's installation says the App
identity cannot be. This workflow was the sole outlier among the repo's
tracking-issue workflows, and the outlier is the broken one:

Workflow Tracking-issue identity
link-check.yml ambient GITHUB_TOKEN (link-check.yml:129)
queue-monitor-liveness.yml ambient GITHUB_TOKEN (queue-monitor-liveness.yml:157)
tool-version-drift-check.yml ambient GITHUB_TOKEN (tool-version-drift-check.yml:446)
pulumi-version-drift-check.yml ambient GITHUB_TOKEN, no token step at all
this workflow, before second App token — 404

README.md:337-342 already describes the ambient shape ("a scheduled caller
that grants issues: write; the tracking issue lands in the caller's own
repository"). It needed no edit — the App-token detour is what had drifted from
the documented contract.

Changes:

  • Delete the issue-token step. The three consumers run on the ambient token
    (github-script defaults to github.token;
    peter-evans/create-issue-from-file defaults to ${{ github.token }}).
  • ISSUE_AUTHOR_LOGIN → github-actions[bot]. It is declared once at workflow
    level, so the single change covers both the find and the close filters.
  • Job permissions gain issues: write.
  • Fold in the sibling guard shape: Find existing tracking issue now runs
    unconditionally, and Close recovered tracking issue consumes its
    issue-number behind an != '' guard — matching link-check.yml:246-247 —
    replacing a duplicated lookup that carried its own author filter.

The App secrets stay required: true and keep their names. They still mint the
cross-repository read token that scans the 8 targets; only the second,
caller-scoped mint is gone.

BREAKING: workflow_call consumer contract

The caller's job must now grant issues: write. Previously it needed no
write permission at all, because the App token supplied the write capability. A
called workflow can only downgrade the caller's grant, never elevate it, so a
caller that does not grant it gets a hard failure on the issue write.

Affected callers — full org sweep (gh search code --owner melodic-software
for standards-sync-stuck-automerge-alert.yml@):

Repository Path Kind
melodic-software/standards .github/workflows/standards-sync-stuck-automerge-alert.yml the one and only workflow_call caller
melodic-software/standards components/runner-policy/policy.json contract entry (not a caller)
melodic-software/{claude-code-plugins,github-iac,dotfiles,provisioning,medley} .github/standards/runner-policy/policy.json sync-materialized copies of that contract

melodic-software/ci-runner references this workflow only as the subject its
queue monitor watches, not as a caller.

Merging this PR is inert for that caller: it pins @8202e03, so nothing
changes in production until the companion standards PR re-pins. There is no
interim breakage window.

Required companion change in melodic-software/standards

One PR, and all three parts must land together — a re-pin alone yields a
403 on the issue write:

  1. Re-pin .github/workflows/standards-sync-stuck-automerge-alert.yml's uses:
    to this PR's merge SHA, with the matching short-SHA trailing comment.
  2. Add job-level permissions: {contents: read, issues: write} to the alert
    job (it currently inherits workflow-level contents: read). Same shape as
    that repo's own link-check.yml caller.
  3. Add a reviewed contract entry for the new SHA in
    components/runner-policy/policy.json, keeping the current shape:
"melodic-software/ci-workflows/.github/workflows/standards-sync-stuck-automerge-alert.yml@<merge-sha>": {
  "routing": "runner-input",
  "runnerInput": "runner",
  "allowedInputs": ["runner", "manifest", "standards-ref", "threshold-hours"],
  "allowedSecrets": {
    "app-client-id": "${{ secrets.STANDARDS_SYNC_APP_CLIENT_ID }}",
    "app-private-key": "${{ secrets.STANDARDS_SYNC_APP_PRIVATE_KEY }}"
  }
}

Auto-approval cannot admit this entry — a secret-capable runner-input contract
declines it by design — so it is a human contract review either way.

On the secret rename this PR deliberately does not make

The brief for this work carried a premise that granting the caller issues: write forces an allowedCallerPermissions entry in the contract, which in
turn forces the kebab-case secret inputs (app-client-id, app-private-key)
to be renamed to identity-passthrough form. That premise does not hold, and
the rename is therefore not made.

Verified by executing the real validator
(components/runner-policy/runner-policy.mjs) against the real repo config,
four variants plus a negative control:

Variant Caller issues: write Contract Result
A yes kebab secrets, no allowedCallerPermissions passes
B yes kebab secrets + allowedCallerPermissions ConfigurationError — passthrough required
C yes renamed passthrough secrets + allowedCallerPermissions passes
D no (status quo) kebab secrets, no allowedCallerPermissions passes
E yes contract missing an input the caller passes fails (runner-target-contract) — the probe genuinely exercises the caller

The passthrough requirement (runner-policy.mjs:204-213) fires if and only
if
the contract declares allowedCallerPermissions. Nothing forces that
field: standards is visibility: public, selfHostedCi: false, so the
privileged-hosted machinery that would demand the waiver never engages, and
reusableWorkflowStatus checks caller permissions only when the contract
declares them.

The comment this PR removes (old lines 78-88) was not false — it correctly
stated that a contract with allowedCallerPermissions needs passthrough
secrets. It was a non-sequitur: nothing required adding that field. The real
tradeoff is now recorded in the workflow header instead of being deleted
silently — link-check.yml and pulumi-version-drift-check.yml pin their
callers' grants via allowedCallerPermissions because they take no secrets;
this workflow's contract can pin the secrets or the grant, not both, until the
inputs are renamed. Renaming them is available later at the cost of a genuinely
breaking secret-interface change; it is not needed to fix this.

Semantic changes a diff reader would not notice

Stated rather than slipped in:

  • Decoy resistance widens. The author filter survives — every adopt/update/
    close path still acts only on an issue matching ISSUE_AUTHOR_LOGIN with
    user.type === 'Bot'. What widens is who can author a qualifying decoy:
    from "only the sync App" to "any workflow in the caller repo". Both require
    write access to that repo. This is exactly the posture
    queue-monitor-liveness.yml already ships (ambient identity, marker + author
    filter, no label filter), so no label filter is added here.
    link-check.yml's label filter exists because it is a multi-invocation
    reusable whose marker is keyed by a title hash — a different problem.
  • Healthy runs can now fail closed on marker ambiguity. The lookup's
    matches.length > 1 fail-closed guard previously ran only on alert runs; the
    close path used .find() and silently took the first match. Now a
    duplicate-marker state turns a healthy run red. Stricter, and identical to
    link-check.yml.
  • The close path can now close a title-fallback match. It previously
    required the marker in the body; it now closes whatever the lookup resolved,
    which includes the pre-marker title fallback. Still constrained to
    github-actions[bot]-authored issues with the exact title. No such issue
    exists today — the workflow has never had a successful run, so it has never
    authored an issue at all.

Verification

  • actionlint — clean.
  • zizmor v1.28.0 — no findings.
  • typos — clean.
  • node --test .github/scripts/*.test.cjs — 496 pass, 0 fail.
  • biome ci @2.5.4 with the repo's config, exactly as ci.yml invokes it —
    clean.
  • comment-hygiene scan — 18 findings, byte-identical to the origin/main
    baseline; this change adds none.

Two close-path tests asserted the old semantics and are rewritten. Because the
close step's decoy protection now lives in an if: expression — which no mock
can reach — it is asserted structurally, together with the lookup running
unconditionally, since the two halves are only correct together. The new
assertion was mutation-tested: re-adding an if: to the lookup step fails it.

A fresh-context verifier (rationale withheld) audited the change independently
and found the test-suite breakage above, which is fixed in the second commit.
Its remaining verdicts: the 404 is genuinely eliminated rather than relocated
(no surviving path mints or consumes a caller-scoped App token); both outcome
branches receive a usable token and issue reference; every issue mutation stays
behind the author filter; and no silently-green failure mode exists — every
partial-landing state it traced fails red, and an alert run always ends red via
the Fail so the scheduled run notifies step.

Residue — stated plainly

The 404 is fully eliminated from this reusable. The watchdog is not
functional until the standards companion change lands.
That is not residue
in this PR; it is the cross-repo shape of the fix. Nothing here defers the
failure or converts it into a quieter one.

One item deferred with a trigger: a private, selector-routed consumer is
the only case that would need allowedCallerPermissions on this contract, and
no current caller is selector-routed. Should one appear, the resolution is not
necessarily the secret rename — splitting the cross-repository scan off the App
secrets is the likelier fix — so nothing is pre-committed.

Not done, deliberately, per the constraint that the attest step requires the
App installation's selected set to equal the derived target set: standards
is not added to the sync App's installation or as a manifest target. Any
addition would fail the sync for every target.

Related

kyle-sexton and others added 2 commits August 3, 2026 20:53
…with the ambient token

Every run of this workflow has failed since it was first scheduled. The
`issue-token` step minted a second App token with `owner`/`repositories`
omitted, which `actions/create-github-app-token` resolves to the calling
repository. The only caller is melodic-software/standards, the sync SOURCE,
which is deliberately not a sync target and so is outside the App's selected
installation — the mint 404s on
`GET /repos/melodic-software/standards/installation`.

Converge on the ambient-`GITHUB_TOKEN` authorship its sibling rolling-report
workflows already use (link-check.yml, queue-monitor-liveness.yml,
pulumi-version-drift-check.yml). Ambient `GITHUB_TOKEN` is scoped to the
calling repository by construction, which is exactly what the 404 says the App
identity cannot be. The App secrets stay required and unrenamed: they still
mint the cross-repository read token that scans the targets.

Fold in the sibling's guard shape while here: `Find existing tracking issue`
now runs unconditionally and `Close recovered tracking issue` consumes its
`issue-number` behind an `!= ''` guard, replacing a duplicate lookup that
carried its own author filter.

BREAKING CHANGE: the caller's job must now grant `issues: write`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFFtZLysg75gm7hviQM5wU
…contract

The close step no longer resolves an issue of its own, so the two tests that
drove it with a list of open issues asserted semantics it no longer has. Its
decoy protection now lives entirely in its `if:` — the author-filtered lookup
result plus the `issue-number != ''` conjunct — and no mock can reach an `if:`
expression, so assert it structurally, together with the lookup running
unconditionally. The two halves are only correct together: guarding the lookup
on the alert condition would starve the close step on exactly the healthy runs
it serves.

Also record in the workflow header what the reusable's kebab-case secret input
names cost a governed consumer: a runner-policy contract can pin the caller's
`issues: write` grant only when every mapped secret's input name equals the
repository secret it reads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TFFtZLysg75gm7hviQM5wU
@cursor

cursor Bot commented Aug 4, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@claude

claude Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 3m 43s —— View job


Review: fix(standards-sync-stuck-automerge-alert)!: author the tracking issue with the ambient token

Scope reviewed: .github/workflows/standards-sync-stuck-automerge-alert.yml and .github/scripts/standards-sync-stuck-automerge-alert.test.cjs (the two files this PR touches). Security-model implications (App-token vs. ambient-token authorship, decoy resistance) are already covered by this repo's dedicated security lane and are also stated explicitly by the PR author, so I did not re-litigate them here — I focused on correctness, control flow, and test coverage.

No blocking issues found. This is a careful, well-reasoned fix. Specifics:

  • Root-cause fix is correct. Deleting the second, caller-scoped create-github-app-token mint removes the only path that 404s (a token for melodic-software/standards, which is outside the App's 8-target installation). The remaining App mint (Mint read-only App token scoped to every target, line 158) is untouched and still does the cross-repo scan reads.
  • Control-flow trace checks out for all four reachable states (healthy/first-run, alert/first-run, alert/still-active, alert/recovering) — I traced steps.scan outputs through the now-unconditional Find existing tracking issue step (L471) into both the Open or update (L521) and Close (L532) branches. Every path resolves a usable token (ambient) and either skips cleanly or acts on the correct issue number; no state leaves the close step racing an empty issue-number since it's gated on steps.tracking.outputs.issue-number != ''.
  • Sibling-shape claim verified by direct comparison. I diffed this against link-check.yml's own Find existing tracking issue/Close recovered tracking issue steps: the unconditional lookup, the issue-number != '' close guard, and the github-actions[bot] author filter are structurally identical patterns, not just similar in spirit.
  • No dangling references to the removed issue-token step or steps.issue-token.outputs.token remain anywhere in the file (grepped).
  • SHA-pinning is intact — no uses: lines were added or had their pin/comment touched; all three actions used here (create-github-app-token, github-script, create-issue-from-file) keep their existing 40-char SHA + trailing # vX.Y.Z, per CLAUDE.md's pin-everything rule.
  • Test coverage matches the semantic changes. The rewritten close-path tests correctly shift decoy-resistance assertions onto the lookup step (author-filter tests at lines 846/864 of the test file already covered that), and the new structural test (L918) asserting the lookup runs unconditionally and the close step's if: carries the issue-number != '' guard is a reasonable way to cover an if: expression no mock can reach — I confirmed by reading the extraction helpers (extractStepScript, the workflow const loaded straight from the on-disk YAML) that this test reads the real file content, not a fixture.
  • README.md:337-342 claim confirmed — I read that range directly; it already says "grants issues: write" and needed no edit.

What I could not independently verify: I was unable to execute node --test, actionlint, or zizmor in this sandbox (the node --test … invocation required approval I don't have in this lane, and actionlint/zizmor aren't installed here). So the PR body's "496 pass, 0 fail", actionlint/zizmor/typos/biome ci/comment-hygiene results, and the mutation-testing claim are author-claimed and unverified by me — everything above reflects my own static reading of the diff and cross-file comparison, not a re-run of those tools. If this lane should have node/actionlint/zizmor available, an --allowedTools update would let a future review run them directly.

Minor, non-blocking observations (not filed as inline findings):

  • ISSUE_AUTHOR_LOGIN: 'github-actions[bot]' (L102) is now single-quoted where the old value was bare; purely cosmetic, no functional effect.
  • The header comment block (L35-48) is dense but earns its length given the exact tradeoff it's documenting (kebab-case secrets vs. allowedCallerPermissions) — consistent with this file's existing comment density elsewhere.

Nothing here should block merge. The breaking-change framing (caller must grant issues: write) is accurate and well-documented in both the PR body and the workflow header.

@claude

claude Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Claude finished @kyle-sexton's task in 2m 8s —— View job


Security review

  • Gather context (PR diff, CLAUDE.md security rules)
  • Review standards-sync-stuck-automerge-alert.yml diff for auth/logic/injection issues
  • Review test file diff
  • Post findings as review

This PR only touches .github/workflows/standards-sync-stuck-automerge-alert.yml and its test file. Findings below cover logic/authorization/trust-boundary concerns only — permission grants, action pins, and trigger patterns are zizmor's lane and are not re-reported here.

SUGGESTION — decoy-authorship surface widens from one App identity to every workflow with issues: write in the caller repo

Confidence: confirmed as designed, not a defect

ISSUE_AUTHOR_LOGIN changes from the narrow App identity melodic-standards-sync[bot] to the shared ambient github-actions[bot], which every workflow running in the caller repo authors as. The lookup step's author filter — and therefore what the close step acts on — now trusts that shared identity rather than a single App.

Concretely: any other workflow in the caller repo that can open an issue (already requires issues: write in that repo, so no new privilege is needed to reach this) can pre-create an issue with the exact title [Alert] standards-sync auto-merge PR(s) needing attention or embedding the `` marker. If it's the sole match, this workflow adopts it as the tracking issue — future real incident reports get written into that issue instead of a fresh one, and a subsequent healthy scan will comment on and close it. If two matching issues exist, the lookup fails closed (matches.length > 1), so the failure mode is a stuck/noisy watchdog rather than a silent takeover — bounded impact, no token/secret exposure, no privilege escalation beyond what `issues: write` already grants.

This is explicitly called out and reasoned through in the PR description and workflow comments (lines 86–101), and it converges this workflow onto the same posture queue-monitor-liveness.yml already ships (ambient identity + marker/title filter, no label filter) rather than introducing a new pattern. Flagging only because the task asks for authorization/trust-boundary observations — no action needed unless the calling repos' set of issues: write-capable workflows is broader/less trusted than assumed.

No other issues found

  • The App-token removal is a straightforward reduction in blast radius: the surviving App token is scoped permission-pull-requests: read on the 8 sync targets only (L158-L166); it no longer carries issues: write on the caller at all.
  • The issue number flowing into the close step (L536-L539) is passed via an env var and only ever a GitHub-assigned integer from the prior step's own output — not attacker-controlled text interpolated into the script body, so no script-injection surface here.
  • The unconditional-lookup + issue-number != ''-guarded close (L471, L533) correctly inherits the implicit success() gate GitHub Actions adds to any if: without an explicit status function, so a core.setFailed in the lookup (ambiguous-marker case) or the scan step still fails the job closed — no path mutates an issue on an ambiguous or failed scan.
  • Test changes are a faithful mechanical follow of the workflow changes; no behavioral drift between the two.
    · branch fix/watchdog-ambient-issue-authorship

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Claude has reviewed this PR 1 time. The lane skips further automatic reviews after 5; deleting this comment resets the count.

@kyle-sexton

Copy link
Copy Markdown
Contributor Author

Closing in favor of #346 (shape 2). Shape 1' authors the tracking issue with the ambient GITHUB_TOKEN, which is scoped to the repo where the workflow runs — melodic-software/standards, which is public. The alert body embeds a per-PR table of repo names and PR URLs, and 4 of the 8 sync targets are private (dotfiles, github-iac, medley, provisioning). Shape 1' therefore structurally republishes private repository names and PR URLs into a public issue, and that is not fixable within the shape. Shape 2's caller-named destination is.

Branch fix/watchdog-ambient-issue-authorship is left on origin for reference only; it should not be merged.

@kyle-sexton kyle-sexton closed this Aug 4, 2026
@kyle-sexton
kyle-sexton deleted the fix/watchdog-ambient-issue-authorship branch August 13, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant