feat(tasks): nudge Ask AI discussion runs and scouts to keep a useful task summary #104516

Summary

Add surface-specific nudges for the agent-maintained task summary (task_summary_update, from #94290) in the two places where the generic nudge is not doing the job: Ask AI discussion runs on an inbox report, and scout runs. Prompt text only, no new tools or endpoints.

Why

The harness ships one shared paragraph, "Keeping the task summary", appended to every cloud system prompt. It asks the agent to call the tool when the goal changes, when an approach is dropped, about every ten turns, and before it ends a turn. That works where a turn has a natural "I'm done" moment and nothing else competes for it. It does not work the same way everywhere, because each product that builds on tasks shapes its runs differently:

• Ask AI discussion runs (ai_stage = inbox, interactive) are chats. The agent answers the question and the turn ends. In a live check on 2026-09-22 a fresh discussion run answered in nine tool calls and never called the tool; asked explicitly in the next message, it called it, the call succeeded, and the stored summary was a good two-sentence recap. So the tool is available and allowed, and the nudge just does not register after a chat answer. These are the runs where a summary is most useful to a person: a recap of what was asked and found, readable from the report's Runs row without opening the transcript (feat(signals): show the task summary as a tooltip on inbox report run rows #104500).
• Scout runs already call the tool on most runs, but the text is formulaic: skill version, eligibility, what was checked, verdict, which scratchpad keys were refreshed, "no friction". The verdict line carries all the value. Scouts also already write a second summary: the JSON close-out's summary field, which the harness parses into the scout run row with "lead with the verdict" guidance. The two summaries should say the same thing, and the task one should read like the scout run one.

Proposed changes

1. Ask AI discussion runs

Two touch points, both in the prompt the discussion run receives:

• products/desktop/packages/harness/src/extensions/task-system-prompt/cloud.ts, the repo-less branch that answers questions about an inbox report (the one that carries the business-knowledge search instruction). Add a short chat-shaped nudge alongside it, for example:

  After you answer, update the task summary with a recap of the conversation so far: what was asked, what you found, and what is still open. A teammate reads it on the report's Runs row without opening this chat. Replace the previous summary; keep it to a few sentences.

• products/signals/frontend/inbox/inboxTaskKickoffLogic.ts, the kickoff prompt ("Answer this question about the PostHog Inbox report at …"). Consider one trailing sentence pointing at the summary, since the first turn's user message is what the model reads most closely. Optional if the system prompt change alone moves the needle.

Also soften the shared block in task-system-prompt/task-summary.ts so "before you end a turn" clearly includes answering a question in a conversation, not only finishing a work turn. Keep it one sentence; this block is always loaded.

2. Scout runs

In products/signals/backend/scout_harness/prompt.py, the close-out step (_CLOSE_OUT_STEP_TEMPLATE) already tells the scout to end with the JSON object and points at Writing the summary for the summary field. Extend it so the task summary is the same text:

Before the JSON close-out, call task_summary_update with the same verdict-first summary you are about to put in the summary field. It shows on the task and in the fleet's task list, so lead with the verdict and what you wrote or edited (report ids), and leave out the run ritual: skill version, eligibility, scratchpad keys, tool friction, unless one of them changed the outcome.

Keep it a nudge, not a rule, in line with how the rest of that prompt is written.

Alternative worth deciding here: the harness already parses the close-out JSON and persists summary to the scout run row. It could write the same string to the task run through set_summary and drop the extra tool call on scout runs entirely. That is deterministic and saves a model step on every scout run. The trade is that the task summary then only exists at run end, which is fine for a one-turn scout. If that route is preferred, this issue becomes prompt-only for Ask AI and a harness change for scouts.

Out of scope

• Enforcing the call at turn end (a Claude Code Stop hook, or an agent-server check on turn completion). That is the deterministic fix for chat surfaces and is tracked separately; this issue is the prompt-level improvement that should land regardless.
• Research-stage pipeline runs, which end every turn on a JSON finding envelope and are pipeline-internal.
• The always-allow lists for the tool in non-bypass permission modes.
• Any UI. The tooltip that reads these summaries is feat(signals): show the task summary as a tooltip on inbox report run rows #104500.

Acceptance

• A fresh Ask AI discussion run on an inbox report writes a summary after its first answer without being asked, and the summary reads as a recap of the exchange (question, findings, open points).
• Scout task summaries lead with the verdict and name what was written or edited; the run-ritual lines are gone unless they changed the outcome.
• The shared block stays one paragraph; surface-specific text lives in the surface's own prompt.
• Existing prompt tests updated (prompt.test.ts in the harness, the scout harness prompt tests), including the HARNESS_PROMPT_VERSION bump the scout prompt derives from its content.

Context

• Primitive and its rollout: feat(tasks): agent-maintained task summary so you can drop into a task without re-reading it #94290, feat(tasks): let the agent keep a running task summary #94362, feat(tasks): let cloud agents keep a running task summary #95970, chore(tasks): bump sandbox @posthog/agent to 2.4.208 #104347.
• Reader that depends on this for discussion rows: feat(signals): show the task summary as a tooltip on inbox report run rows #104500.
• Origin thread: https://posthog.slack.com/archives/C0C1T3V1RJA/p1789638920497979