feat(signals): let research and scouts author report checks #102283

Problem

#99783 specs scheduled follow-up checks on a report. Phases 1 and 2 have merged: #99797 added the SignalReportCheck model, the check_result artefact, the REST endpoint and deterministic metric_threshold execution on the coordinator tick; #102033 added agent checks, the triggered_by="check" dispatch path, the scout-check-record-result tool and the signals-scout-inbox-validation fallback lane.

Nothing creates a check in production. The only create surface is POST /projects/:id/signals/reports/:report_id/checks/, which no pipeline stage and no scout prompt calls. The signals-report-checks-* entries in products/signals/mcp/tools.yaml are enabled: false, and there is no scout-scoped create tool. So the follow-up machinery is fully built and idle, while follow-up still runs as prose in three places:

• The "Follow up on your own past work" section every scout prompt carries (_SELF_VALIDATION_FOLLOWUPS_TEMPLATE in products/signals/backend/scout_harness/prompt.py) still asks scouts to keep followup:<skill>:<entity> scratchpad entries with a validate-after date, a probe and a baseline, and says outright there is no schedule and no harness trigger.
• signals-scout-inbox-validation (products/signals/skills/signals-scout-inbox-validation/SKILL.md) still enqueues newly resolved reports as pending:inbox_validation:* scratchpad entries in Stage 1 and re-derives the probe from the live report in Stage 2, at a sandbox turn per validation.
• The research agent writes a prose ## Verification plan note (FixVerificationOutput in products/signals/backend/report_generation/research.py, from feat(signals): add verification plans to research #91726) that names a metric, a baseline and a comparison criterion, and then nothing executes it. The implementation agent is told to re-measure by hand.

derived_metadata.has_self_validation still reads scratchpad followup: writes, so once scouts stop writing those entries the fleet loses its validation signal.

This is phase 3 of #99783, filed as its own issue because the phases are independently shippable and the earlier two have landed.

Proposal

Make checks the way follow-up is expressed, and retire the scratchpad queues.

1. Scout create tool

Add scout-report-check-create (and scout-report-check-list / scout-report-check-cancel) to the scout harness, in products/signals/backend/scout_harness/tools/checks.py next to record_check_result, exposed through tools.yaml under the signal_scout_internal:write scope. Gate it by the report-channel opt-in the way scout-emit-report is gated, since a check is a write on a report. The tool accepts the same body as the REST endpoint (title, rationale, kind, config, next_run_at, optional interval and runs_remaining) and is attributed from_task like every other scout write. Validation and limits stay in report_checks.py; the tool adds nothing the endpoint does not enforce.

2. Research agent authors checks

Add checks: list[CheckSpec] to ReportResearchOutput and ReportPresentationOutput, persisted in mark_report_ready_activity in the same transaction as title, summary, charts, metrics and the verification note (products/signals/backend/temporal/agentic/report.py, where verification_note becomes an artefact draft). Attribute them to research_task_id.

Source them from the verification turn. FixVerificationOutput already asks for the evidence to collect and the result that supports a conclusion; extend it so that when the "confirm the outcome" section names a metric, a baseline and a threshold, the turn also emits a metric_threshold CheckSpec referencing one of the report's own metrics by metric_id, so the check rides a query the report already validated. When the plan's method is a replay, a test, a code review or a manual step, it stays prose only. Offer the agent kind only when the team is enrolled in signals-scout (check_fleet_gates), so a team without scouts degrades to deterministic checks rather than storing checks that can never run.

The verification plan note and the check should read as one thing in the timeline: the plan is the prose, the check is its executable part. Name it that way in the note heading and in the check_result renderer.

3. Arm research-authored checks on resolve, not at creation

A research-authored check is written before any fix exists, so next_run_at cannot be "merge plus soak" at write time. And many fixes land with no pull request at all: a skills-store edit, a config change, a manual resolve with a note. There is no merge timestamp to soak from.

Proposal: the research turn writes the check with a soak duration instead of an absolute next_run_at, and the check is created active with next_run_at set when the report transitions to resolved, whatever caused the transition (PR-merge webhook, manual resolve, MCP inbox-reports-set-state). That makes the resolve transition the clock for every kind of fix. Default soak 24 hours; the research turn may set longer when the fix is client-side or mobile. Until the report resolves the spec sits on the report (an artefact, or a pending status on the row if adding one to SignalReportCheck.Status is cleaner) and the timeline shows it as "will check after resolve".

This is the open design question from the original issue. If a different clock is preferred, decide it in review rather than defaulting inside the implementation run.

4. A failed deterministic check on a resolved report re-surfaces

The on_fail actions in the original proposal did not ship in phases 1 or 2. An agent check has a scout in the loop that can author a fresh report. A metric_threshold check has nobody, so a failed verdict on a resolved report is an artefact on a report the inbox no longer lists.

Add the minimal action: on failed for a report whose status is resolved, emit a source_product="signals_check" signal carrying the verdict and the origin report id, so the pipeline authors a fresh report linked by related_to. This matches the existing "a recurrence starts a fresh report" rule and adds no resolved -> ready transition. notify and run_scout stay out of scope.

5. Migrate the prompts and the follow-up scout

• _SELF_VALIDATION_FOLLOWUPS_TEMPLATE becomes "create a check": when a run surfaces something whose fix would show in data, create a metric_threshold check on the report when a bounded query and a baseline exist, otherwise an agent check with the probe as instructions and the entities as probe_hints. Drop the queue-reading step, the validate-after bookkeeping and the janitor rule. Keep the re-surface clauses (_FOLLOWUP_RESURFACE_*) for the run that answers a check, since they say what to do when a fix did not hold. Remove the FOLLOWUP_KEY_PREFIX special case in the scratchpad rules once nothing writes it.
• signals-scout-inbox-validation Stage 1 becomes: for each newly resolved report with no active check, capture the baseline and create one check (the same probe plan it builds today, as instructions and probe_hints, with next_run_at from the real merge time or resolve time). Stage 2 disappears; the coordinator dispatches the check to this scout when due and the run closes it with scout-check-record-result, which feat(signals): add agent checks for scheduled report follow-ups #102033 already wired. The dispatched-check section of the skill stays.
• derived_metadata.has_self_validation reads check writes: a run that created a SignalReportCheck or recorded a check_result with its run_id counts, a scratchpad followup: write no longer does. Keep the same field name so dashboards do not move.

Tests

• Research output with a checks list persists rows in the same transaction as metrics and rejects a spec whose metric_id names no report metric.
• A check written with a soak duration is not due before the report resolves and becomes due at resolve time plus soak for each transition path (merge webhook, manual resolve, MCP set-state).
• A failed metric_threshold verdict on a resolved report emits one signals_check signal linked to the origin report; a failed verdict on a ready report emits none.
• scout-report-check-create refuses a run without the report-channel opt-in and attributes the row to the run's task.
• has_self_validation is true for a run that created a check or recorded a verdict, and false for one that only wrote a scratchpad entry.
• Prompt snapshot: the follow-ups section no longer mentions followup: keys or "no schedule and no harness trigger".

Out of scope

• A detail-rail Checks section or checks_summary on the report serializer.
• notify and run_scout on_fail actions.
• Enabling the public signals-report-checks-* MCP tools.
• Deleting existing followup: and pending:inbox_validation: scratchpad rows. They expire on their own, and a run that finds one can still act on it.

Pointers

• Spec and phasing: feat(signals): scheduled follow-up checks on a report #99783. Phase 1: feat(signals): schedule follow-up checks on a report #99797. Phase 2: feat(signals): add agent checks for scheduled report follow-ups #102033. Verification plans: feat(signals): add verification plans to research #91726.
• products/signals/ARCHITECTURE.md, "Report checks".
• products/signals/backend/report_checks.py (config schemas, limits), report_check_execution.py (the verdict funnel), report_check_agent.py (dispatch lane).
• products/signals/backend/scout_harness/tools/checks.py, scout_harness/prompt.py, scout_harness/derived_metadata.py.
• products/signals/backend/report_generation/research.py, products/signals/backend/temporal/agentic/report.py, products/signals/backend/temporal/summary.py (mark_report_ready_activity).
• products/signals/skills/signals-scout-inbox-validation/SKILL.md.

This can land as one PR or two. If two, the first is the backend (sections 1 to 4 and the has_self_validation change) and the second is the prompt and skill migration (section 5). The second depends on the first.