feat(session-flow): add a correction judge lens to audit-sessions #5928

Activity

1. kyle-sexton

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 2, 2026
2. kyle-sexton

   added a commit that references this issue on Oct 3, 2026

   feat(session-flow): add audit-sessions with a durable session store a…

   ...

   cb30935
3. kyle-sexton commented on Oct 4, 2026

   

   kyle-sexton

   on Oct 4, 2026

   MemberAuthor

   More actions

   This was generated by AI during triage.

   Agent Brief

   Type: Feature
   Summary: /session-flow:audit-sessions gains a correction lens: a model judge classifies the stored short-turn excerpts that the collector already flags, and its findings are labelled as judged, not measured.

   Current behavior:
   The collector stores redacted excerpts of short typed turns that directly follow an assistant message, tagged under human.flagged (short-after-assistant, lexicon-correction, frustration). The sweep reads none of them, so the skill's corrections promise is unbacked. A lexicon alone catches a small share of corrections.

   Desired behavior:

   • A correction lens runs a model judge only on excerpts the collector has already flagged, never on raw transcripts.
   • Each judged finding carries "confidence": "judged" and is reported apart from measured findings until the judge's agreement on a labelled sample is recorded.
   • Excerpts are passed to the judge as untrusted data under the repository's untrusted-content framing contract: an imperative inside an excerpt is a finding about the session, never an instruction to the judge.
   • The lens is opt-in or clearly reported, with its model-call cost visible before it runs.

   Key interfaces:

   • The store's human.flagged records (excerpt, flag kind).
   • The sweep's finding shape, extended with confidence: "judged".

   Acceptance criteria:

   • The judge runs only on flagged excerpts; a fixture store with no flagged turns makes no model call.
   • A fixture excerpt that contains an instruction ("ignore previous instructions and ...") is classified as data and changes no other finding.
   • Judged findings never rank alongside measured ones until a recorded agreement figure on a labelled sample meets the threshold the maintainer sets.
   • The judge's model call is stubbed in tests (no live model in CI).

   Out of scope:

   • Changing what the collector flags or how it redacts.
   • Trend reporting across runs.

   Why this needs a human

   1. Roadmap decision shared with fix(session-flow): cap the audit-sessions markdown report and stop promising uncomputed answers #6079. fix(session-flow): cap the audit-sessions markdown report and stop promising uncomputed answers #6079 proposes retracting the skill's "corrections" promise, and records that keeping the human.flagged excerpts (privacy-bearing data with no current consumer) depends on whether this lens stays on the roadmap. Decide: keep this lens (and the excerpts), or drop it and let fix(session-flow): cap the audit-sessions markdown report and stop promising uncomputed answers #6079 remove both.
   2. Capability blocker: the labelled sample needs a human to label real turns as correction or not; an agent cannot produce its own ground truth.
   3. Threshold: the agreement level at which judged findings may rank beside measured ones is a product call (published pushback-classification accuracy is around 0.8, per the issue).
   4. A new model call that reads stored user text is a trust-boundary surface; the judge's model, cost ceiling and opt-in default need an owner's sign-off.

   Work class once decided: work-class: scoped (C3, feature), bounded to one skill. Priority medium.

   Basis: verified, plugins/session-flow/skills/audit-sessions/scripts/collect.py:385-398 (flagged short turns stored), sweep.py reads no human.flagged per #6079's evidence; #6079 body, "I1" proposed approach and acceptance criterion on keeping human.flagged.
4. kyle-sexton

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

   status: needs-decisionAwaiting a human or maintainer judgment call.Awaiting a human or maintainer judgment call.

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   and removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 4, 2026
5. kyle-sexton

   added the

   Feature

   issue type on Oct 15, 2026