feat(signals): skip scout suggestion scans for projects with too little recent activity #104517

Context

#85963 added the scheduled producer that pre-computes scout suggestions per project, and #101698 stopped it planning projects that have never ingested an event (Team.ingested_event). That gate is a lifetime check. A project that ingested once and went quiet, or a project that was created last week and has sent a handful of test events, still passes it, still gets a full sandbox and model session, and still ends the scan with an empty batch. The agent is right to refuse: there is nothing to suggest a scout for.

Reading the producer at tier 2 across both regions, one week after the never-ingested gate deployed:

• About one in five projects whose latest scan completed holds an empty batch. The gate removed the never-ingested half of the empties; what is left is projects with data, just very little of it.
• The empty batches are almost all on quiet projects. The median empty-batch project had a few dozen events in the last 14 days. The median productive-batch project had over ten thousand.
• About a third of the empty-batch projects were created in the last 14 days, so a good share of this is projects that are still being set up rather than projects that went dark.
• The strip is not being used on quiet projects. Over 14 days, projects with fewer than 100 events in the window account for about 1% of strip impressions and one scout created, across both regions.
• Failed scans are a different population (median in the thousands of events) and are not addressed here.

Skipping quiet projects frees roughly a fifth of scheduled scans at steady state, which matters because the US coordinator has run at its per-tick cap since tier 2 opened.

Sizing the rule

Each candidate rule was scored against the latest scheduled scan of every project in the tier 1+2 pool, joined to project-level event counts over the same 14 days (now a standing internal warehouse view, so the table below is repeatable). Shares are US / EU.

"Productive batches lost" counts batches that would not have been generated. Almost none of them were ever seen: the adoption column is the real cost, and it is near zero up to the 100-event line. Past 500 events the rule starts removing batches people act on.

The recommended default is the bold row. It removes about two thirds of the empties for a cost of one created scout in two weeks. Both thresholds are payload knobs so the line can move after the first read.

Proposal

A dispatch-time activity check on the candidates the planner picks, in the shape #101681 described for its second gate. The check runs on the picked set only, never over the pool, so it costs one bounded ClickHouse read per candidate per tick.

Payload knobs on signals-scout-suggestions

• activity_window_days (default 14)
• min_events_in_window (default 100; 0 disables the volume check)
• min_active_days_in_window (default 3; 0 disables the spread check)

A project is skipped when either threshold is not met. Parsed in parse_suggestion_settings with the same defensive clamping as the other knobs.

The read

After plan_suggestion_runs orders the candidates and before the coordinator dispatches a child, one query per picked project, over the project and its child environments together (the batch is per canonical project):

SELECT count(), uniqExact(toDate(timestamp))
FROM events
WHERE team_id IN (<project and child environment ids>)
  AND timestamp >= now() - INTERVAL <activity_window_days> DAY
SETTINGS max_rows_to_read = 100000, read_overflow_mode = 'break'

The row cap keeps the read bounded on large projects. A read that hits the cap means the project has at least that many events in the window and is treated as active without looking at the day spread; only a read that finishes under the cap has exact numbers, and those are the small projects the rule is about. Overselect the picked set (twice the cap) so a tick still fills when several candidates are skipped.

The stamp

A skipped project gets a new SignalScoutSuggestionSet.Status, low_activity, with last_requested_at stamped, so the planner leaves it alone until the refresh window passes and then re-checks it. The check is cheap, so a project that picks up traffic is scanned one refresh window after it does. Prior items on the row, if any, are kept, the same as failed. The read API serves low_activity the way it serves empty: nothing on the strip.

Emit $scout_suggestions_generated with status = skipped, skip_reason = low_activity and the two measured numbers, so the daily read (signals_scout_suggestion_runs) sees the saving without a new column.

Unchanged on purpose

• team_allowlist projects are scanned regardless (dogfood and support).
• The manual refresh endpoint is untouched. A person who presses "Suggest a scout" on a quiet project asked for the scan and gets the honest "nothing to suggest" outcome.
• The agent's behavior on a sparse project. Refusing to suggest is correct.

Verification

• Unit tests in products/signals/backend/test/test_scout_suggestions.py: a project under either threshold is stamped low_activity and dispatches no child; a project over both is dispatched; a capped read counts as active; an allowlisted project under the threshold is still scanned; a low_activity row is re-checked after the refresh window; min_events_in_window: 0 and min_active_days_in_window: 0 each disable their check.
• python manage.py run_scout_suggestions --plan on a stack with one set-up project holding a handful of events: the tick records low_activity on its row and the next candidate takes the slot.
• After deploy, on the internal warehouse layer: skip_reason = low_activity should land at roughly a fifth of scheduled runs per region, and the zero-item share of completed runs should fall from about one in five to under one in ten within one refresh window. The skipped projects' actual activity and the strip's creations per day are readable from the same layer, so the threshold can be moved on evidence rather than re-derived.
• Update the suggestions.py bullet in products/signals/backend/scout_harness/AGENTS.md so the planner's eligibility description names the check and the knobs.

Out of scope

• A recency pre-filter over the whole pool. The check runs on the picked candidates only.
• Persisting the batch notes field (the model's own reason for an empty batch). It is asked for in the prompt and dropped on write; storing it would say what the remaining empties are about. Small, separate change.
• Raising max_children_per_tick or widening to tier 3. Both get cheaper once this lands, and each is its own decision.

Files

• products/signals/backend/scout_harness/suggestions.py: SuggestionSettings and parse_suggestion_settings for the three knobs; the low_activity status and its persistence helper next to mark_generation_failed.
• products/signals/backend/models.py: SignalScoutSuggestionSet.Status.LOW_ACTIVITY (a TextChoices addition; check whether the column has a DB-level constraint before assuming no migration).
• products/signals/temporal/agentic/scout_suggestions.py (the coordinator): the per-candidate check before dispatch, with the ClickHouse read in its own activity, and the overselect.
• products/signals/backend/scout_suggestions_api.py: serve the new status.
• products/signals/backend/scout_harness/AGENTS.md: the suggestions.py bullet.
• products/signals/backend/test/test_scout_suggestions.py: the cases above.