session-flow/running-retro: detached observer's distilled observations never preserve enough structure to compute sequencing/batching/delegation/dependency claims

[kyle-sexton]

Context

Filed from PR #1482 (fix for #1473 recommendation 1, the running-retro compute-don't-assert prompt
rule). Both claude[bot]'s PR review and Codex's automated review independently flagged the same
consistency gap: the detached observer's headless analysis prompt (observer.py's
_analysis_prompt) instructs the analyzer to derive tool-call sequencing/batching/delegation claims
by "grouping tool-use events by API message id" — but summarize_record() (the function that
produces the distilled observations the headless run actually receives) never preserves a
message-id field. It extracts only type, timestamp, tools, a truncated text snippet, and
stop_reason.

PR #1482 closed the immediate correctness gap defensively: the prompt now states explicitly that an
absent message-id field makes the claim uncomputable, so the analyzer drops it rather than asserting
it from impression (the compute-don't-assert rule's fallback already covered this). That is safe,
but it means the headless observer path can never actually compute a sequencing/batching/delegation
finding — it can only ever drop that finding class. The in-session checkpoint path (which reads the
full transcript directly) is unaffected.

Codex's suggested remediation is a genuine capability restoration, not just a prompt clarification:
preserve enough structure in the distilled observations (either the real API message id, or an
explicit per-assistant-turn grouping boundary) so the headless path can compute these findings
instead of always dropping them. That is a change to summarize_record()'s output schema and the
observations' on-disk shape — a larger, separately-scoped piece of work, not a text-only fix, so it
was deferred out of #1482 rather than expanding that PR's scope.

Addendum (from claude[bot]'s review of PR #1482 commit 8f29a28): the same gap extends to the
dependency check PR #1482 later added (a computed sequencing fact isn't proof of a missed batching
opportunity when the calls are dependent). That check tells the headless agent to inspect the calls'
tool inputs and result content — but summarize_record() strips both: it preserves only tool
names (out["tools"] = [c.get("name") ...]) and result counts (out["tool_results"] = results),
never the actual input arguments or result values. So the dependency check is equally uncomputable
on the headless path today — same interim fallback applies (drop the Efficiency finding rather than
infer dependency from tool names alone), same underlying schema gap this item tracks.

Proposed work

• Add a stable grouping key to each distilled observation summarize_record() emits — either the
  transcript record's own message id (if present in the raw JSONL) or an explicit per-assistant-turn
  boundary marker the headless prompt can group on.
• Preserve enough of each tool call's input arguments and result content (or a bounded/truncated
  summary of them) for the headless path to check for a data/control/resource/side-effect dependency
  between calls, not just their names and result counts.
• Update observer.py's _analysis_prompt to reference the new fields instead of (or in addition
  to) "API message id" / "tool inputs/results," and drop the corresponding "may not carry..." caveats
  once those fields are reliably present.
• Verify token-cost impact is acceptable — summarize_record()'s whole design goal is a token-cheap
  distillation; any added field should stay minimal (an integer/short id, a truncated argument/result
  summary, not full message objects).

Acceptance criteria

• summarize_record()'s output carries a grouping key sufficient to reconstruct tool-call
  batching/sequencing without the raw transcript.
• summarize_record()'s output carries enough of each tool call's inputs/results (or a bounded
  summary) to support a dependency check between calls without the raw transcript.
• The headless analysis prompt in observer.py can compute a sequencing/batching/delegation
  finding, and a data/control/resource/side-effect dependency check backing an Efficiency
  finding, from the distilled observations alone — verified by a test that builds observations
  from real transcript-shaped records and checks both round-trip correctly.
• No meaningful token-cost regression to the distilled observations file.

References

• session-flow/running-retro analyzer: stenographer bias — accurate on self-declared findings, 0/2 on independent inference; structural claims asserted not computed #1473 (recommendation 1 — the compute-don't-assert rule this gap was found underneath)
• PR fix(session-flow): add compute-don't-assert rule to running-retro analysis prompts #1482 (the fix that added the drop-if-uncomputable caveats as interim mitigations, for both the
  message-id gap and the tool-inputs/results gap)
• plugins/session-flow/skills/running-retro/scripts/observer.py: summarize_record() (~line 51),
  _analysis_prompt() (~line 621)

Metadata

Field	Value
Category	session-flow / running-retro
Area	session-flow
Ecosystem	python

---

This was generated by AI during work-loop execution.

[kyle-sexton]

This was generated by AI during triage.

Triage

Verified. Read summarize_record() and _analysis_prompt() on current main: confirmed each distilled
observation carries only event type, timestamp, tool names, a truncated text snippet, a tool-result
count, and stop reason — no message-id-equivalent grouping key, and no tool-call input arguments or
result content. The headless prompt's "group by API message id" / dependency-via-tool-inputs guidance
is genuinely uncomputable from that shape today, matching PR #1482's and this issue's description.

Redundancy check: no existing code path preserves this structure elsewhere; not already implemented.
No cluster with other open intake (this is a standalone schema-shape gap distinct from #1483, #1479,
#1478).

Routing: the message-id-vs-synthetic-boundary-marker choice named in "Proposed work" is an ordinary
reversible implementation choice — the acceptance criteria (round-trip test, token-cost check) are
satisfied regardless of which the implementer picks, so this doesn't need a human decision gate.
Briefed as delegable.

Priority: set to priority: medium (assessed-default tier — real value, no stated deadline or
blocking relationship to other work).

Note: .work-item-tracker.json has no config.role_labels entry — resolving the autonomous-eligible
role to its default agent-ready label.

Agent Brief

Type: Task
Summary: Add a stable grouping key and enough tool-call detail to the running-retro detached
observer's distilled observations so the headless analysis prompt can compute (not just drop)
sequencing/batching/delegation and dependency findings.

Current behavior: The function producing each distilled observation the headless analysis prompt
consumes extracts only event type, timestamp, tool names, a truncated text snippet, a tool-result
count, and stop reason. It preserves no message-id-equivalent grouping key and no tool-call input
arguments or result content. The headless prompt instructs the analyzer to group tool-use events by
API message id and to check computed sequencing claims against a dependency signal in tool
inputs/results — neither is possible from the current distilled shape, so the analyzer can only ever
drop those finding classes, never compute them. The in-session checkpoint path (reads the full
transcript directly) is unaffected.

Desired behavior: Each distilled observation carries (a) a stable per-turn grouping key — the
transcript record's own message id when present in the raw JSONL, or an explicit per-assistant-turn
boundary marker otherwise — sufficient to reconstruct tool-call batching/sequencing without the raw
transcript; and (b) enough of each tool call's input arguments and result content (bounded/truncated,
not full objects) to support a data/control/resource/side-effect dependency check between two calls.
The headless analysis prompt's guidance is updated to reference the new fields, and the "may not be
computable" caveats are dropped once the fields are reliably present.

Key interfaces:

• The distillation function's output schema (one dict per transcript record) — gains the grouping key
  and bounded tool-call input/result fields
• The headless analysis prompt's text — references the new fields instead of "API message id" / "tool
  inputs/results", drops the now-obsolete "may not carry..." caveats

Acceptance criteria:

• The distillation output carries a grouping key sufficient to reconstruct tool-call
  batching/sequencing without the raw transcript.
• The distillation output carries enough of each tool call's inputs/results (or a bounded summary)
  to support a dependency check between calls without the raw transcript.
• A test builds observations from real transcript-shaped records and verifies both the grouping
  key and the dependency-relevant fields round-trip correctly.
• The headless analysis prompt text is updated to use the new fields; the drop-if-uncomputable
  caveats for message-id grouping and tool-inputs/results are removed.
• No meaningful token-cost regression to the distilled observations file (spot-check size delta on
  a representative session).

Out of scope:

• The in-session checkpoint path (unaffected; reads the full transcript directly)
• Model-tier defaults (tracked separately under session-flow/running-retro: raise default analysis model tier above cheap/fast class (cost/quality decision) #1480)
• Ledger coverage-truncation disclosure (tracked separately under session-flow/running-retro observer: mtime-idle end detection fires mid-session (default 900s), analyzes a partial transcript, never resumes #1471)

Implementation note: the choice between a real message id (if present in the raw JSONL) and a
synthesized per-turn boundary marker is left to the implementer — pick whichever is reliably present
and cheapest to preserve; either satisfies the acceptance criteria as long as round-trip grouping is
correct.

[kyle-sexton]