[Phase 5] DRY_RUN soak, threshold tuning without overfitting, and go/no-go to enable label/close #1206

Story

As a maintainer,
I want progress spam-pr-guard through the fleet canary rings — spam-pr-guard/next → ring0 → ring1 → stable — using the standard #548 graduated dwell/sample gate, with require_confirmation: true on ring1→stable as the human go/no-go to enable label/close, tuning thresholds via standards/spam-pr-guard.json without overfitting the frozen regression fixtures,
so that label and close are enabled only after the scorer is validated on real traffic with a bounded false-positive rate, keeping legitimate first-timers safe, and the enable gate ships via the fleet pattern rather than a one-off DRY_RUN flip.

Reframed from "DRY_RUN soak + go/no-go code flip" to "canary ring progression" — the fleet pattern already encodes dwell-and-sample soak at every ring boundary (scripts/canary-rollout.sh, graduated gate per #548). The dry_run input is a stub-level default (ON initially), overridable per-stub, and flipped at a specific cut-standards-release.sh cut once require_confirmation clears on ring1→stable.

Acceptance Criteria

1. Ring progression. spam-pr-guard passes the standard #548 graduated gate at each ring boundary (next → ring0: dwell_hours 4 + sample_fraction_permille 250 clamped 3..15; ring0 → ring1: dwell_hours 8 + waive_sample; ring1 → stable: dwell_hours 12 + sample_min 1 + require_confirmation: true). Promotion is driven by scripts/canary-rollout.sh promote-all on its scheduled cadence; a human runs scripts/canary-rollout.sh promote spam-pr-guard --confirm once to clear the ring1→stable confirmation gate (dispatch audit-logged; no state store).
2. DRY_RUN defaults, per ring. While on next and ring0, the stub's effective with.dry_run stays true — the staged action logs the intended action to the step summary and makes zero write API calls. The flip is a stub-default change (with.dry_run: false) that ships via the next scripts/cut-standards-release.sh cut AFTER the --confirm on ring1→stable. Individual consumers may override via their own stub's with: block.
3. Soak evidence. The soak log is reviewed against at least 10 real outside-PR events on next + ring0; the review is captured as an evidence comment on the Phase 5 canary-confirm issue that scripts/canary-rollout.sh sync-issues files for spam-pr-guard (labeled needs-human, auto-closes on promote or candidate change — see canary-rings.json § _confirm_note).
4. Fixture discipline. Thresholds are tuned only via standards/spam-pr-guard.json; the frozen regression fixtures (#1976-shaped → high; legitimate-first-time → low with the raw-score assertion from [Phase 2] Deterministic spam-PR scoring script and config #1203; security-sensitive → escalate) are never edited to force a score — they are the immutable baseline and must still pass at whatever thresholds are chosen.
5. Held-out validation. Threshold changes are validated on held-out real DRY_RUN events (from the next/ring0 soak) distinct from the bats fixtures, so tuning cannot overfit or reward-hack the fixture set.
6. Metric gate before --confirm. The legitimate-first-time counterfactual scores below the medium threshold at the enabled values, and the observed false-positive rate on legitimate PRs is under 2% on next + ring0 BEFORE the ring1→stable --confirm. These metrics are recorded on the canary-confirm issue.
7. Decisions recorded on the epic. The decision on whether high-score PRs auto-close vs label-only, and any close delay (discussion open Q1), is recorded on the epic Org standard: automated spam / low-effort outside-PR handling (spam-pr-guard) #1200 at --confirm time rather than silently baked into earlier code.
8. Anti-reward-hacking guard. A CI guard (or ack-gate label, mirroring .github/workflows/test-deletion-guard.yml) fails any PR that modifies a frozen baseline fixture under tests/fixtures/spam-pr-guard/ without an explicit maintainer ack, so thresholds cannot be tuned by quietly editing the fixtures to pass.

Tasks / Subtasks

• Verify scripts/canary-rollout.sh promote-all cycles spam-pr-guard through next → ring0 → ring1 under the standard #548 gate (AC: Addressing PR comments #1)
• Capture the ~2-week next + ring0 soak log against >= 10 real events on the canary-confirm issue (AC: Add coding standards & principles section to AGENTS.md #3)
• Tune thresholds via standards/spam-pr-guard.json while keeping the regression fixtures frozen; validate on held-out real events (AC: Add workflow, environment, and orchestration guidance #4, Add stacked PR strategy and Epic-level workflow guidance #5)
• Confirm the counterfactual stays below medium and FP rate < 2% before --confirm (AC: feat: add Structured Logging and CQRS standards #6)
• Record the auto-close-vs-label decision on the epic (AC: docs: document branch protection rules, rulesets, and auto-merge policy #7)
• Run scripts/canary-rollout.sh promote spam-pr-guard --confirm (dispatch; audit-logged) to clear ring1→stable (AC: Addressing PR comments #1)
• Cut standards/vX.Y.Z with the with.dry_run: false default once --confirm clears, so the stable default ships to all stub consumers (AC: Add multi-agent isolation strategy using git worktrees #2)
• Add the anti-reward-hacking CI guard against silent edits to tests/fixtures/spam-pr-guard/ (AC: Evolve Agentic Development: Integrate Claude Agent Teams with BMAD Workflow #8)

Dev Notes

• The soak IS the canary ring progression; the fleet pattern does not have a separate "DRY_RUN phase" distinct from next/ring0. See standards/canary-rings.json § _confirm_note for how require_confirmation holds a ring1→stable transition and how sync-issues surfaces evidence.
• --confirm is NOT --override: it advances ONLY a reliability-clean AWAITING_CONFIRMATION state and can never bypass a BLOCKED gate. Reliability gates (next → ring0 → ring1) still clear unattended on the standard #548 graduated gate.
• The anti-reward-hacking guard mirrors .github/workflows/test-deletion-guard.yml (which gates tests/ deletions behind an ack-test-deletion label): extend the same shape to cover EDITS of the frozen tests/fixtures/spam-pr-guard/ fixtures, since tuning against the fixtures rather than held-out events is the reward-hacking failure mode.
• Held-out validation = the real next/ring0 events, which are distinct from the training fixtures — the guard against overfitting the fixtures. Capture a summary on the canary-confirm issue so the --confirm decision has a durable record.
• Discussion open Q1 (auto-close vs label-only) is decided at --confirm time, not baked into the scorer or workflow earlier; enhancement recommends auto-close with a clear reopen comment plus a delay, but this remains the recorded human decision on Org standard: automated spam / low-effort outside-PR handling (spam-pr-guard) #1200.
• The dry_run flip is a stub-default change (edit standards/workflows/spam-pr-guard.yml to set with: { dry_run: false }) that ships via scripts/cut-standards-release.sh cut vX.Y.Z → standards/v1-stable. A consumer that opts out of the flip can override in their own stub's with: block.

Project Structure Notes

Primarily an operational/decision story (dev-lead:hands-off): config threshold edits in standards/spam-pr-guard.json, a recorded --confirm decision, a one-time stub-default change, and the CI guard. No new scripts; touches only the config file, the stub default, and canary-rings.json if any gate knobs need tweaking.

References

• scripts/canary-rollout.sh (promote / promote-all / promote --confirm / sync-issues; AC Addressing PR comments #1, Add coding standards & principles section to AGENTS.md #3)
• standards/canary-rings.json § _confirm_note + .agents["pr-auto-review"] as a correctness prior art
• standards/standards-versioning.md (how the with.dry_run: false default ships; AC Add multi-agent isolation strategy using git worktrees #2)
• .github/workflows/test-deletion-guard.yml (anti-reward-hacking guard pattern; AC Evolve Agentic Development: Integrate Claude Agent Teams with BMAD Workflow #8)
• AGENTS.md § Fleet stub-drift remediation loop
• Org standard for automatically handling spam / low-effort outside PRs #1198

Likely target surface

• standards/spam-pr-guard.json (threshold tuning edits)
• standards/workflows/spam-pr-guard.yml (one-shot with.dry_run: false default change at --confirm time)
• .github/workflows/spam-pr-guard-tests.yml (or a sibling fixture-ack-gate workflow for AC Evolve Agentic Development: Integrate Claude Agent Teams with BMAD Workflow #8)

Story prepared by the BMAD Scrum Master (Bob) for epic #1200. Realigned from one-off DRY_RUN flip to fleet canary ring progression + require_confirmation go/no-go. Status: inert pending Phase 3 landing + first promote-all cycle.