Skip to content

Canary blocker: apply-repo-settings next->ring0 (cum_fail=0, UNRESOLVED) #1042

Closed
#1043
Closed
Canary blocker: apply-repo-settings next->ring0 (cum_fail=0, UNRESOLVED)#1042
#1043
Labels
canary-blockercanary-rollout automationdev-leadFor dev-lead agent pickup

Description

@combsmith-release-manager
combsmith-release-manager
bot
opened on Aug 31, 2026

Automated canary-rollout blocker. The release gate is holding apply-repo-settings and will not promote it until this clears. Filed + maintained by the Canary Rollout workflow (gate standard: .github#548); this issue is regenerated each run and auto-closes when the gate passes — do not edit the table below by hand.

field value
agent apply-repo-settings
transition next->ring0
candidate 4bb3e7aa986c
host repo petry-projects/.github
cumulative failures 0 (startup_failures: 0)
triage UNRESOLVED

⚠️ UNRESOLVED (gate blind on a ring member, #1224) — at least one ring member is an ADR-0007 collapsed repo (its per-role workflow is gone, Agent Ingress is present) and this agent's runs there could not be attributed to its role job. That member is not counted as passing evidence, so the gate holds instead of promoting on incomplete evidence. This is not a detected run failure (cumulative failures: 0). Fix: register the agent's ingress_job (its job key in the ingress) in standards/canary-rings.json. If the evidence below says job reads were capped, the ingress has more runs in the window than CANARY_INGRESS_JOBS_MAX allows: raise that knob (default 300) instead. This issue auto-closes once every member resolves.

Unresolved ring members

  • petry-projects/.github-private — no 'Apply repo settings' workflow but 'Agent Ingress' is present and the agent registers no ingress_job
  • petry-projects/markets — no 'Apply repo settings' workflow but 'Agent Ingress' is present and the agent registers no ingress_job

Whole-fleet status is in the Canary Rollout workflow run's job summary (Actions → Canary Rollout → latest run → Summary).

Activity

  1. combsmith-release-manager
    added
    canary-blockercanary-rollout automation
    dev-leadFor dev-lead agent pickup
    on Aug 31, 2026

  2. don-petry commented on Sep 1, 2026

    @don-petry
    don-petry
    on Sep 1, 2026
    Contributor

    Dev-Lead Implementation Plan

    Issue: #1042 — Canary blocker: apply-repo-settings ring0->ring1 (cum_fail=1, PRE_EXISTING)

    Root cause (from the failing run)

    The failing step Apply settings on petry-projects/TalkTerm (run 33446111321) is TalkTerm's own, stale forked apply-repo-settings.yml + scripts/apply-repo-settings.sh, invoked with the GH_PAT_WORKFLOWS token. That token lacks the classic-PAT capability the legacy check-suites/preferences endpoint requires, so the call returns HTTP 403 ("You must authenticate with a personal access token … to change check suite permissions"), and TalkTerm's old script does not tolerate it → exit 1.

    This matches the gate's PRE_EXISTING / environmental triage: the reusable is byte-identical to the prior channel; the candidate did not introduce it. The canonical scripts/apply-repo-settings.sh in this repo already degrades gracefully on exactly this 403 (apply_check_suite_prefs, tracking petry-projects/.github-private#1209) — but that graceful-degradation path has zero bats coverage, which is why this class keeps recurring unguarded.

    Scope

    Lock in the canonical apply_check_suite_prefs 403 graceful-degradation (the exact TalkTerm failure mode) with bats tests, so the reusable the canary is promoting cannot regress this environmental-failure class. In-repo, fix-forward, additive.

    Implementation Checklist

    • Extend the gh stub in test/scripts/apply-repo-settings/apply-repo-settings.bats to model the check-suites/preferences GET/PATCH (env-toggled, defaults preserve existing behavior)
    • No production code change expected — canonical apply_check_suite_prefs already handles the 403 (verified against the exact run error string)

    Tests to Write

    • 403 on the check-suites PATCH → graceful skip (rc 0), warns + names the tracking issue (the exact TalkTerm failure)
    • non-403 PATCH error → hard fail (rc 1)
    • auto-trigger already disabled → no-op (rc 0), no PATCH issued
    • --dry-run → no PATCH issued (rc 0)

    Test Command

    bats --print-output-on-failure test/scripts/apply-repo-settings/

    Note on TDD phase

    Per the Phase-2 exception, this is behavior-preserving regression lock-in of an already-correct code path; the new tests pass against current HEAD. If a genuine detection gap surfaces while writing them, the code is fixed first (red→green).

    Out of scope (reported, not actioned here)

    The run only clears once TalkTerm adopts the reusable caller stub (standards/workflows/apply-repo-settings.yml) instead of its stale local fork — a change in the TalkTerm repo, outside this repo's dev-lead scope.

  3. don-petry commented on Sep 1, 2026

    @don-petry
    don-petry
    on Sep 1, 2026
    Contributor

    Dev-Lead: Implementation Complete

    Root cause (confirmed from the failing run)

    Failing step Apply settings on petry-projects/TalkTerm (run 33446111321) is TalkTerm's own stale forked workflow + scripts/apply-repo-settings.sh, run with GH_PAT_WORKFLOWS. The log ends with:

    gh: You must authenticate with a personal access token, or basic auth, or via a GitHub App in order to change check suite permissions. (HTTP 403)
{"message":"...","status":"403"}
##[error]Process completed with exit code 1.

    The legacy check-suites/preferences endpoint accepts only a classic PAT / GitHub App; the token in use gets HTTP 403, and TalkTerm's old script fork does not tolerate it → exit 1. This is exactly the gate's PRE_EXISTING / environmental triage — the reusable is byte-identical to the prior channel; the candidate did not introduce it.

    The canonical scripts/apply-repo-settings.sh in this repo already absorbs this 403 (apply_check_suite_prefs, tracking petry-projects/.github-private#1209), but that graceful-degradation path had zero bats coverage — the recurring-regression gap. Fix-forward: lock it in with tests so the reusable the canary promotes cannot regress this class.

    Plan Execution

    • Reproduced + root-caused the 403 against the exact run error string
    • Confirmed the canonical apply_check_suite_prefs already degrades gracefully (no production change needed)
    • Modelled the check-suites/preferences GET/PATCH in the bats gh stub (env-toggled; defaults preserve prior behavior)
    • Added 5 tests: 403→skip (rc 0), non-403→hard-fail (rc 1), already-disabled no-op, never-run/"missing" no-op, --dry-run no PATCH
    • Mutation-verified: disabling the 403 branch turns the 403 test red (test is not trivial)
    • Full bats suite (39) + dev-lead-lint.sh green

    Test Results

    Command: bats --print-output-on-failure test/scripts/apply-repo-settings/
...
ok 19 a 403 on the check-suites PATCH is absorbed — skips loudly, rc 0 (issue #1042)
ok 20 a non-403 error on the check-suites PATCH still fails hard, rc 1
ok 21 apply_check_suite_prefs is a no-op when auto-trigger is already disabled
ok 22 apply_check_suite_prefs treats a never-run app (missing) as no orphaned suite
ok 23 apply_check_suite_prefs --dry-run issues no PATCH
...
39 tests, 0 failures

dev-lead-lint.sh: [lint] all checks passed

    Files Changed

    • test/scripts/apply-repo-settings/apply-repo-settings.bats: extend the gh stub with check-suites/preferences GET/PATCH modelling + add 5 tests for apply_check_suite_prefs's 403 graceful-degradation, hard-fail, no-op, and dry-run paths. No production script changed (byte-identical).

    Notes

    • Phase-2 exception applied: this is behavior-preserving regression lock-in of an already-correct code path, so the new tests pass at HEAD rather than starting red. Meaningfulness was instead proven by mutation testing (removing the 403 branch turns test 19 red).
    • What actually clears this canary is out of scope for this repo: the TalkTerm run only goes green once TalkTerm adopts the reusable caller stub (standards/workflows/apply-repo-settings.yml, which runs .github's canonical script under GH_PAT_DON_PETRY) instead of its stale local fork. That is a change in the TalkTerm repo. Per the gate ("Report only … armed timer auto-promotes once clean"), no rollback/advance is needed here.
  4. don-petry
    added a commit that references this issue on Sep 1, 2026

    feat: implement issue #1042 — Canary blocker: apply-repo-settings rin…

    9d14c7d
  5. don-petry
    closed this as completedin #1043on Sep 1, 2026
  6. combsmith-release-manager
    reopened this on Oct 3, 2026
  7. combsmith-release-manager
    changed the title [-]Canary blocker: apply-repo-settings ring0->ring1 (cum_fail=1, PRE_EXISTING)[/-] [+]Canary blocker: apply-repo-settings next->ring0 (cum_fail=0, UNRESOLVED)[/+] on Oct 3, 2026

  8. combsmith-release-manager commented on Oct 11, 2026

    @combsmith-release-manager
    combsmith-release-managerbot
    on Oct 11, 2026
    Author

    ✅ Gate cleared — apply-repo-settings is no longer BLOCKED. Closed automatically by canary-rollout.

  9. combsmith-release-manager
    closed this as completedon Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    canary-blockercanary-rollout automationdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions