Signals scouts: a deprecation lifecycle for canonical scouts #104389

Summary

Give canonical scouts a deprecation lifecycle, so retiring one is a state the fleet moves through instead of a flag entry and a directory delete.

Today there are two ways to stop a canonical signals-scout-* scout, and neither was built for retirement:

• The signals-scout flag's withheld_skills denylist. Built to hide unreleased scouts from teams not on their allowlist. Used for a retirement, it makes the scout vanish: the roster row disappears with no reason shown, the scout page 404s, and nobody can turn it back on. The inactivity sweep keeps warning the hidden config. Report checks bound to the scout's lane get a non-retryable skill_withheld refusal and land on the report as an errored check result. Teams that edited their copy lose it too, because the holdback is by name.
• Deleting the skill directory. sync_canonical_skills(prune=True) tombstones the seeded, unedited LLMSkill rows on the next tick, which is right. It never touches SignalScoutConfig, so every config is left enabled=true with no skill behind it: a ghost row that lingers in the roster with an empty description and can never run. The only thing keeping those ghosts out of sight is a flag entry that then has to stay forever.

The first retirement of a canonical scout that teams actually ran (signals-scout-health-checks, 2026-09-22, via the holdback) hit all of this. Context: https://posthog.slack.com/archives/C09SK2PAGKF/p1789994089452429

What a person sees

Phase 1, announced. The scout keeps running for everyone who has it on. Its roster row gains a "Retiring on {date}" chip. The scout page shows a banner with the reason and what replaces it. It stops being offered: no suggestion card, no seeding into new projects, and the toggle to turn it on is disabled with the reason.

Phase 2, retired. On the sunset date the scout stops running. The row stays in the roster for a grace window, now reading "Retired", with the same banner. The reports it wrote stay in the inbox with their attribution. Open checks on those reports run on the fallback scout instead of erroring. After the grace window the row leaves the default roster view.

Phase 3, removed. The skill is gone from the skills tab as well. Nothing else changes for the person, because the config was already retired in phase 2.

A team that edited its copy of the scout is not affected by any phase. Their copy keeps running as a custom scout, and the row shows a note that the PostHog version was retired.

Proposed copy (sentence case, no dashes):

• Chip: "Retiring on {date}" and "Retired"
• Banner: "PostHog is retiring this scout on {date}. {reason}" followed by the replacement sentence when one is set, for example "PostHog's health checks now report to your inbox directly, so this scout is no longer needed."
• Disabled toggle: "This scout is being retired and can't be turned on."
• Custom fork note: "PostHog retired its version of this scout on {date}. Your copy keeps running."

Design decisions

1. The marker lives on disk with the skill. SKILL.md frontmatter gains scout-status: deprecated and a scout-deprecation block with reason, optional superseded_by, and sunset_at. discover_canonical_skills parses it onto CanonicalSkill; the sync writes it into LLMSkill.metadata so per-team surfaces read it without a disk read; the config serializer exposes it as a read-only deprecation object with the computed phase. The PR that deprecates a scout is then a reviewable frontmatter change.
2. A flag layer for the cases a deploy is too slow for. default_team_config.deprecated_skills on the signals-scout flag, a map of skill name to {reason, sunset_at}, resolved the same most-specific-first way as withheld_skills. The flag entry wins over the disk marker when both exist. withheld_skills goes back to meaning "unreleased" only, and its docstring says so.
3. Phase 1 stops intake, not runs. register_missing_configs seeds nothing for a deprecated scout; the suggestions producer skips it; SignalScoutConfigViewSet create and any update that sets enabled=true return 400 with the reason. sync_canonical_skills keeps updating the row so the marker reaches every team. Dispatch is unchanged.
4. Phase 2 is a system transition, not a hidden row. On the tick after sunset_at, the coordinator moves each enabled config to PAUSED_BY_SYSTEM through transition_status_by_system with a new PauseReason.RETIRED. That keeps the status enum small (see feat(signals): auto-pause scouts after 14 days of inactivity #74866), lands in the activity log, and gives the roster a real state to render. The dispatch gate, scout-run-now, the inactivity sweep and the failure breaker all skip RETIRED. resolve_check_skill_name falls through to FALLBACK_CHECK_SKILL_NAME when the lane is retired, withheld, or has no live skill.
5. The prune pass finishes the job, and defends against skipped phases. When sync_canonical_skills(prune=True) tombstones a skill row, it also retires that team's config if it is still active. So a scout deleted from disk without going through phases 1 and 2 still leaves no ghost, and the flag entry can always be removed.
6. Forks are untouched. The marker applies to rows the sync owns (seeded_by tag plus matching canonical_hash), exactly as the update path decides today. A fork keeps running; the roster shows the soft note.
7. Removal is a normal PR. Delete the directory, drop the scout from BILLING_EXEMPT_SCOUT_SKILLS and from the sibling scout bodies that hand off to it, update products/signals/skills/AGENTS.md. A test asserts every key in BILLING_EXEMPT_SCOUT_SKILLS names a skill on disk, so a removal cannot leave the exemption behind.

Sequence

1. PR 1, backend lifecycle. Frontmatter parse, deprecation on the config serializer, PauseReason.RETIRED, flag key, phase 1 gates (seed, suggestions, create and enable), check-lane fallback. Dispatch behavior unchanged.
2. PR 2, roster and scout page. Chip, banner, disabled toggle, fork note, "Retired" filter and the grace-window hide.
3. PR 3, sunset and prune. Coordinator sunset transition, sweep and breaker skip, prune-time config retirement, telemetry (signals_scout_retired per transition, a retired_skills count on the fleet sync event).
4. Apply it to the health-checks scout. It is already withheld. Once PR 3 lands: add the disk marker with sunset_at in the past, let a day of ticks retire the configs, remove the withheld_skills entries, then the removal PR.

Non-goals

• An in-app announcement beyond the roster chip and scout page banner. Use the change-notice pattern separately if a retirement warrants it.
• Cleaning up the scout's scratchpad keys, notes, or run history. They age out and nothing reads them once the scout is gone.
• A retirement path for custom scouts. Their owner deletes them.

Open questions

• Grace window before a retired row leaves the default roster view. 30 days is the proposal.
• Whether phase 1 should also switch the scout to dry-run for teams that have it on, or leave emit posture alone. The proposal leaves it alone.
• Whether a deprecation with no sunset_at should mean "announced indefinitely" or "retire on the next tick". The proposal is the latter, so a marker is never silently half-applied.