Skip to content

canary-rollout: deployment-record step 422s for cross-repo promotions (creates on GITHUB_REPOSITORY, not the SHA's host) + only records the last of a promote-all #1059

Closed

Description

@don-petry

Summary

The first live promote-all run (28681440491) promoted dependabot-automerge/stable + dependabot-rebase/stable successfully (tags moved on petry-projects/.github via cross-repo gh api), but the job went red on the Record deployment step:

gh: No ref found for: b7e6b67063f4b5673ca6955b2630b315adc1ccd3 (HTTP 422)
{"message":"No ref found for: b7e6b670...","status":"422"}

Root cause

The step creates the Deployment on ${GITHUB_REPOSITORY} (= .github-private) with ref = <promoted_sha>. For a cross-repo agent (the 6 #482 reusables), the candidate SHA lives on the host (petry-projects/.github), not .github-private → No ref found (422). Two defects:

  1. Wrong repo — the deployment must be created on the repo that owns the moved commit (the agent's host).
  2. Last-wins — promote-all does N promotions but only the last one's GITHUB_OUTPUT survives, so only one deployment is even attempted.

Also: a traceability failure should never mark a successful tag move as a failed run.

Fix (this PR)

  • cmd_promote emits promoted_host (the owning repo) and appends one TSV line per move (agent ring sha host) to CANARY_PROMOTIONS_LOG.
  • The workflow step reads that log and records one deployment per promotion on the correct host, guarded (|| ::warning) and if: always() so a hiccup can't fail a good run.
  • +test coverage (promoted_host output + promotions-log line). 56/56 bats passing.

No behaviour change to the tag moves themselves — those already succeeded. This only fixes the traceability step so scheduled promote-all runs go green.

Refs: #1045 (promote path) · #1054 (cross-repo move) · #495 / #502 (canary rollout + observability).

Activity

  1. github-actions commented on Jul 3, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up and for already shipping a fix — this is a great root-cause analysis! Before we merge, could you confirm whether the CANARY_PROMOTIONS_LOG approach handles edge cases where a single agent is promoted to multiple rings in one promote-all run (i.e., does each ring get its own TSV line and its own deployment record)?

  2. don-petry commented on Jul 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #1059 — canary-rollout: deployment-record step 422s for cross-repo promotions + only records the last of a promote-all

    Scope

    Record one GitHub Deployment per promotion on the repo that OWNS the moved commit (the agent's host), driven by a per-move promotions log, so cross-repo promote-all runs go green.

    Implementation Checklist

    • scripts/canary-rollout.sh (cmd_promote): emit promoted_host (owning repo = host, else THIS_REPO) to GITHUB_OUTPUT, and append one TSV line per move (agent<TAB>ring<TAB>sha<TAB>host) to CANARY_PROMOTIONS_LOG when set.
    • .github/workflows/canary-rollout.yml: set CANARY_PROMOTIONS_LOG in the run step; rewrite the "Record deployment" step to read the log, record one deployment per line on the correct host, guarded with || ::warning and if: always() so a traceability hiccup cannot fail a run whose tag moves already succeeded.

    Tests to Write

    • tests/canary_rollout.bats: cross-repo promote emits promoted_host=petry-projects/.github to GITHUB_OUTPUT.
    • tests/canary_rollout.bats: cross-repo promote appends the expected agent<TAB>ring<TAB>sha<TAB>host line to CANARY_PROMOTIONS_LOG.

    Test Command

    npx bats tests/canary_rollout.bats

  3. don-petry commented on Jul 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • cmd_promote emits promoted_host (owning repo = host, else THIS_REPO) to GITHUB_OUTPUT
    • cmd_promote appends one TSV line per move (agent<TAB>ring<TAB>sha<TAB>host) to CANARY_PROMOTIONS_LOG when set
    • Workflow run step exports CANARY_PROMOTIONS_LOG (runner.temp)
    • "Record deployment" step rewritten to read the log and record one Deployment per promotion on the owning host, guarded (|| ::warning) and if: always()
    • Added 2 bats tests (promoted_host output + promotions-log line)

    Test Results

    ok 55 orchestrator: cross-repo promote (real) moves the host channel tag via gh api + records the output (#1054)
    ok 56 orchestrator: cross-repo promote --dry-run shows the host move but touches neither git nor the API (#1054)
    ok 57 orchestrator: cross-repo promote emits promoted_host = the owning host (#1059)
    ok 58 orchestrator: cross-repo promote appends one TSV promotions-log line per move (#1059)
    
    58/58 canary_rollout.bats passing; full tests/*.bats suite green (0 failures).
    dev-lead-lint.sh: all checks passed (shellcheck 87 files + agent frontmatter).
    

    Files Changed

    • scripts/canary-rollout.sh (cmd_promote): emit promoted_host output; append per-move TSV line to CANARY_PROMOTIONS_LOG.
    • .github/workflows/canary-rollout.yml: export CANARY_PROMOTIONS_LOG; record one Deployment per promotion on the correct host, if: always() + || ::warning.
    • tests/canary_rollout.bats: +2 tests (promoted_host output, promotions-log TSV line).

    Notes

    • Root cause test issue from agent #1 (wrong repo) fixed by recording on the owning host instead of GITHUB_REPOSITORY; Go-live improvements for PR review agent #2 (last-wins) fixed by the per-move promotions log rather than single GITHUB_OUTPUT.
    • A traceability failure now emits a ::warning and, via if: always(), can never mark a successful tag move as a failed run.
    • runner.temp is per-job, so no stale-log carryover between runs. Blank/host-less log lines are defensively skipped / fall back to GITHUB_REPOSITORY.
    • No behaviour change to the tag moves themselves.
  4. added 18 commits that reference this issue on Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickupneeds-human-reviewFlagged by automated PR review agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions