Skip to content

Reconcile workflow-stub pin drift: templates vs compliance-audit vs fleet are three-way inconsistent #482

Closed

Description

@don-petry

Summary

A controlled verification of the PR-based deploy (#478/#480) surfaced a three-way inconsistency between (a) the deploy templates in standards/workflows/, (b) the compliance audit's expected pins in check_centralized_workflow_stubs (scripts/compliance-audit.sh), and (c) the actual pins deployed in consumer repos. As a result, deploy-standard-workflows.sh reports 28 stubs as "drifted" — but a blind re-sync would push downgrades, pin-loosenings, and audit-breaking changes. The drift cannot be safely swept until the canonical pin per reusable is reconciled across these three sources.

Evidence

Audit (check_centralized_workflow_stubs) expected pins vs. template uses: vs. fleet (the 3 repos sampled — google-app-scripts, markets, ContentTwin):

Workflow Audit expects Template uses: Fleet (sampled) Problem
pr-review-mention @v2 @v2 (tag) SHA # v2 / @v2 re-sync would loosen SHA→tag
agent-shield @v1 @208ec2… # v1 (SHA) SHA #v1 / @v1 template SHA-pinned ≠ audit @v1 → deploying it fails the centralized-stub audit
auto-rebase @v1 @v1 (tag) SHA #v1 / SHA #v2 re-sync loosens SHA→tag; markets is on a v2-SHA
dependabot-automerge @v1 @v1 (tag) @v2 (all 3) fleet ahead (v2) of template+audit (v1) → re-sync = downgrade
dependabot-rebase @v1 @9a694e5… # main (SHA→main) SHA #v1 template pins a main-branch SHA ≠ audit @v1
dependency-audit @v1 @v1 (tag) @v2 (all 3) fleet ahead (v2) → re-sync = downgrade

Root causes

  1. No single source of truth for the per-reusable pin. Three places encode it independently (template, audit array, deployed stubs) and they have diverged.
  2. SHA-pin vs tag-pin tension is unresolved. Security/SonarCloud pushed SHA pins (fix(standards): pin auto-rebase stub to SHA to satisfy SonarCloud security gate #166), the centralized-stub check was relaxed to exempt internal reusable refs (Compliance: unpinned-actions-dependabot-automerge.yml #239), but templates and the audit still mostly encode @vX tags — so the fleet, templates, and audit each landed in different states.
  3. The deploy script's is_already_compliant is a crude substring match on the template's uses: line — it disagrees with check_centralized_workflow_stubs, so "drift" per the deploy script ≠ "non-compliant" per the audit. This is why 28 stubs look drifted while the fleet is largely functional.

Why it matters

  • A blind deploy-standard-workflows.sh run (now that it opens real PRs) would open ~28 PRs, several of which are downgrades (dependency-audit/dependabot-automerge v2→v1) or pin-loosenings, and the agent-shield/dependabot-rebase templates would deploy stubs the audit then flags non-compliant. So the deploy tooling is safe to run only per-workflow after the canonical pin is settled, not as a fleet sweep.

Recommendation

  1. Pick one source of truth for each reusable's canonical pin (proposal: the <name>/stable channel where it exists; otherwise a single @vX tag), and make the template, the audit array, and the fleet agree.
  2. Decide SHA vs channel/tag deliberately and apply uniformly (reconcile fix(standards): pin auto-rebase stub to SHA to satisfy SonarCloud security gate #166 ↔ Compliance: unpinned-actions-dependabot-automerge.yml #239), updating the action-pinning exemption + check_centralized_workflow_stubs accordingly.
  3. Align is_already_compliant in deploy-standard-workflows.sh with check_centralized_workflow_stubs (ideally share the expected-pin logic) so "drift" and "non-compliant" mean the same thing.
  4. Only then run the deploy per-workflow to reconcile the fleet (it will now open auditable PRs).

Acceptance

  • One documented canonical pin per reusable; template + audit + fleet agree.
  • deploy-standard-workflows.sh drift detection matches the compliance audit.
  • A dry-run reports compliant for the fleet (no spurious "would update"); intentional updates are real upgrades, not downgrades.

References

Activity

  1. don-petry commented on Jun 25, 2026

    @don-petry
    ContributorAuthor

    Resolved ✅ — three-way pin inconsistency closed

    All three acceptance criteria met; the full-fleet deploy-standard-workflows.sh --dry-run now reports 0 non-compliant stubs.

    Root cause & fix

    The per-reusable canonical pin was encoded independently in three places (template, compliance audit, deploy sweep) and diverged. The audit became ring-aware in #529/#870, but the deploy sweep's is_already_compliant was still a literal substring match on the template's @<name>/stable pin — so it falsely flagged every ring1/next repo and would have reverted their intentional ring pins.

    #531 — extracted the canary-ring model into scripts/lib/ring-pins.sh (single source of truth: ring_tier_for_repo, ring_canonical_ref, ring_accepted_refs, ring_legacy_csv, RING_REUSABLES) and sourced it from both compliance-audit.sh and deploy-standard-workflows.sh. "Drift" (deploy) and "non-compliant" (audit) are now computed from the same code and can't diverge again. +7 bats.

    Fleet reconciliation (the genuine off-channel pins the dry-run surfaced)

    Both needed the org-standard SonarCloud S7637 rule exemption (sonar.issue.ignore.multicriteria, per #498/#502) added to their sonar-project.properties — a channel tag is a mutable action ref, and sonar.exclusions alone quiets the quality gate but not the GitHub-Advanced-Security code-scanning SARIF.

    Acceptance

    • One documented canonical pin per reusable; template + audit + fleet agree.
    • Deploy drift detection matches the compliance audit (shared ring-pins.sh).
    • Fleet dry-run reports compliant — 0 non-compliant stubs.

    Closing.

  2. added a commit that references this issue on Jun 26, 2026
  3. added 3 commits that reference this issue on Jul 4, 2026
  4. added 7 commits that reference this issue on Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsciCI/CD pipeline issues

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions