session-flow/running-retro: raise default analysis model tier above cheap/fast class (cost/quality decision)

[kyle-sexton]

This was generated by AI during triage.

Context

Filed as a follow-up split out of #1473 during triage — that issue validated the running-retro observer's headless analysis (session-flow running-retro, observer's --model default claude-haiku-4-5) as accurate on findings the session had already self-declared but 0/2 on independent structural inference, and recommended (as recommendation 2) defaulting the analysis model one capability tier above the cheap/fast class, on the grounds that the observed failure mode (cross-referencing distant transcript passages, deriving structural claims) is capability-shaped rather than purely a prompting gap.

This is kept separate from #1473's delegable "compute-don't-assert prompt rule" fix because it is a genuinely open cost/quality tradeoff — raising the default model tier for every autonomous analysis run has a direct cost implication across every consumer of this plugin, not a scoped/reversible code change a brief can default unilaterally. observer_analysis_model already lets a consumer override the model; this is about the default.

Proposed work

• Decide whether to raise observer.py's --model default (currently claude-haiku-4-5, the cheap/fast tier) one capability class, expressed as a capability class resolved at run time rather than a pinned model ID.
• If raised, update the default in both arm_observer.py and observer.py (--model argparse defaults) and the SessionStart auto-arm hook path, and document the cost implication in the plugin's config reference.
• If declined, record the rationale (e.g. cost outweighs the accuracy gain, or the compute-don't-assert prompt fix from session-flow/running-retro analyzer: stenographer bias — accurate on self-declared findings, 0/2 on independent inference; structural claims asserted not computed #1473 is judged sufficient on its own) so the question isn't re-litigated.

Acceptance criteria

• A maintainer decision is recorded on whether the default analysis model tier changes.
• If changed, the new default is consistent across every entry point that sets --model for the observer's analysis run, and the change is documented.
• If declined, the rationale is captured on this item and it closes as not planned rather than sitting open indefinitely.

References

• session-flow/running-retro analyzer: stenographer bias — accurate on self-declared findings, 0/2 on independent inference; structural claims asserted not computed #1473 (stenographer bias finding this decision was split from), recommendation 2 specifically.
• plugins/session-flow/skills/running-retro/scripts/observer.py (--model argparse default, _run_analysis).
• plugins/session-flow/skills/running-retro/scripts/arm_observer.py (--model forwarded verbatim).

Metadata

Field	Value
Category	general
Area	unspecified
Ecosystem	python

[kyle-sexton]

This was generated by AI during triage.

Triage

State: raw → verified → briefed → human-gated

Context: split out of #1473 (recommendation 2) by that item's own triage pass, on the grounds that raising the default analysis-model tier is a cost/quality tradeoff affecting every consumer's spend, not a scoped/reversible text change a brief could default unilaterally. This pass verifies and routes #1480 on its own merits rather than inheriting the sibling's framing uncritically.

Verified (source inspection): both entry points that set the observer's analysis model default to the cheap/fast tier —

• plugins/session-flow/skills/running-retro/scripts/observer.py: p.add_argument("--model", default="claude-haiku-4-5")
• plugins/session-flow/skills/running-retro/scripts/arm_observer.py: p.add_argument("--model", default="claude-haiku-4-5") (forwarded verbatim to the detached observer process)

observer_analysis_model already exists as a per-consumer override, so the open question is specifically the shared default every non-overriding consumer inherits. No docs/out-of-scope/ ledger exists in this repo, so there's no prior rejection of this concept to check against. No other open item shares this exact decision (checked: nothing else references the observer's model-tier default), so no cluster carrier is needed — this item stands alone.

Classification: Task (already carries native Issue Type Task, matching a decision-recording exercise — not a bug against stated intent, not a new feature). priority: medium — no directive, blocking relationship, or imminent deadline pushes it off the assessed-default tier; consistent with sibling #1473.

Routing test — human-gated, not decision-defaulted: the decision-defaulted exit requires a well-grounded recommendation whose alternative is reversible/maintainer-vetoable at low stakes. This doesn't clear that bar:

• The change is a recurring, silent cost multiplier applied to every autonomous checkpoint run for every consumer who doesn't set observer_analysis_model — not a one-time, easily-inspected code change a maintainer can veto after the fact in a PR review.
• There's no objectively correct default: the tradeoff is between per-run cost and catching a specific class of structural-inference errors (per session-flow/running-retro analyzer: stenographer bias — accurate on self-declared findings, 0/2 on independent inference; structural claims asserted not computed #1473's validation: 0/2 on independent structural inference at the current tier). Reasonable maintainers could land on either side depending on cost tolerance — that's product intent / policy, not a fact a brief can settle.
• The issue's own title and session-flow/running-retro analyzer: stenographer bias — accurate on self-declared findings, 0/2 on independent inference; structural claims asserted not computed #1473's triage both already frame this as "an open cost/quality decision" — that framing holds up under independent review, it isn't just inherited.

This matches the human-gated criteria (open design space / product intent) rather than decision-defaulted.

Agent Brief

Type: Task
Summary: decide whether the running-retro observer's default analysis-model tier should be raised one capability class above the current cheap/fast default, and if so, apply and document the change consistently everywhere the default is set.

Current behavior:
The observer's headless analysis run defaults to a cheap/fast-tier model (capability class, not a pinned model ID) at every entry point that launches it — the standalone observer invocation and the auto-arm hook path. A consumer can override this per-installation via observer_analysis_model, but nothing currently raises the shared default. #1473's validation found this tier accurate on findings the session had already self-declared, but 0/2 on independent structural inference (claims requiring cross-referencing distant transcript passages) — a failure mode assessed as capability-shaped, not purely a prompting gap.

Desired behavior:
A maintainer decision is recorded on whether the default capability tier changes. If raised: every entry point that sets the analysis model's default (the standalone observer invocation and the auto-arm hook path) uses the same new default — expressed as a capability class resolved at run time, not a pinned model ID — and the plugin's config reference documents the cost implication of the new default. If declined: the rationale is recorded on this item (e.g. cost outweighs the marginal accuracy gain, or #1473's compute-don't-assert prompt fix is judged sufficient on its own) and the item closes not planned.

Key interfaces:

• The observer's analysis-launch argument default (currently the cheap/fast capability class) — the value a maintainer decision would change.
• The auto-arm hook path that forwards the same default to the detached observer process — must stay consistent with whatever the observer itself defaults to.
• The plugin's config reference — documents the default capability tier and the per-consumer override, so the cost tradeoff is visible without reading source.

Acceptance criteria:

• A maintainer decision is recorded on this item: raise the default tier, or keep it, with rationale either way.
• If raised, the new default is identical across every entry point that sets the analysis model's default — no entry point left on the old tier.
• If raised, the config reference documents the new default and its cost implication.
• If declined, the item closes not planned with the rationale captured in a comment, so the question isn't re-litigated by a future triage pass.

Out of scope:

• The compute-don't-assert prompt rule (the structural/sequencing-claim fix) — that's session-flow/running-retro analyzer: stenographer bias — accurate on self-declared findings, 0/2 on independent inference; structural claims asserted not computed #1473, already delegable and scoped independently of this decision.
• Coverage-truncation disclosure in the ledger header — tracked under session-flow/running-retro observer: mtime-idle end detection fires mid-session (default 900s), analyzes a partial transcript, never resumes #1471, unrelated root cause.
• Changing the override mechanism (observer_analysis_model) itself — it already works; only the shared default is in question here.

Why human-gated: this is an open cost/quality policy call with no single correct answer — raising the default spends more of every non-overriding consumer's budget on every autonomous run, in exchange for catching a specific class of structural-inference error. That tradeoff is a product-intent decision, not a fact triage or an agent can resolve on the codebase's behalf.

[kyle-sexton]

Frontier disposition (drain session)

Ruling: Close as not planned.

Running-retro default analysis model tier is an operator cost/quality choice, not a fleet mandate.

Re-open trigger: When running-retro ships as autonomous acceptance evidence requiring a minimum model tier.