Repository navigation
Spectrum council reviews moments where agent work failed the user (confusion, frustration, wrong requirement, inefficiency) #52
Description
What: A Spectrum council reviews the moments where agent work failed the user (confusion, frustration, a correct execution of the wrong requirement, and visible inefficiency), using Observer's existing intervention records, and stores verdicts that Observer's recurrence report counts.
Why: a phrase list found 2 of 5 cases (PR #117, closed), and a small classifier can only flag wording. Seeing that an agent built the wrong thing well needs a reader that understands the task, which is what a council does.
So what: an agent runs one manual pilot on bestia; after the user confirms its findings are useful, the review becomes a daily scheduled run on each machine, covering all projects.
Outcome
On a schedule, a Spectrum council reads a bounded batch of recent interventions (user prompts, interrupts and rejections from agent-observer interventions) in context (agent-observer intervention). It records one verdict per moment with agent-observer verdict, classifying it as confusion, frustration, wrong_requirement, inefficiency, or fine, with the agent message or action it reacts to. It then files or updates Issues in the owning repository for problems that agent-observer recurrence shows coming back. Counts per project, host, harness and AgentsMD version show whether rule changes help.
Contribution to the Objective: reproducible real-task evidence for workflow decisions, with the improvement step left to humans and normal Issues (the Objective excludes automating improvement experiments).
Decided design (the user, 2026-10-19)
- No separate classifier model and no Cue change: the council does the judging.
- Pilot first, then schedule: one manual run must show useful findings before any recurring job exists.
- Scope after the pilot: all projects, daily.
Elon record
- Requirements and who asked: find moments where agent output failed the user, including frustration, perfect execution of the wrong requirement, and inefficiency, and use them to improve work across all projects (the user, 2026-10-11 and 2026-10-19).
- Deleted: the phrase-list detector (PR feat(analysis): detect user requests to restate an agent message (#52) #117); a trained classifier; a Cue log-only entry and retrain; a new candidate store (Observer's interventions, verdicts and recurrence already exist).
- Bottleneck: proving one council run produces findings worth acting on.
- Checked myself:
agent-observer interventions --help(prompt, interrupt and rejection moments,--unjudged,--judge);verdictandrecurrencesubcommands exist; no scheduled Spectrum task exists today in the promachos, chromeria or agent-observer projects;start_spectrumis available in T3.
Acceptance criteria
- Pilot: one Spectrum run on bestia over a bounded window (for example the last 3 days of user-created interventions, at most 50 moments) stores a verdict per moment and a short report: counts per class, the 5 strongest findings with session and event references, and proposed Issues. The user confirms whether the findings are useful.
- The pilot records its cost (runs, models, wall time) and how many moments it could not judge.
- Transcripts stay on their machine: the council runs on the machine that holds the ledger and publishes only references, counts and Issue text without quoted prompts.
- After user confirmation: a daily scheduled run per machine with a bounded batch, skipping already-judged moments, with failures visible rather than silent.
- Recurring problems become Issues in the owning repository through the normal Issue rules; nothing changes rules or code automatically.
Non-goals
Automatic rule or code changes, notifications, runtime serving, moving transcripts between machines, a trained detector.
Blockers
The scheduled run waits for the pilot and the user's confirmation. Creating the recurring schedule follows docs/adr/0001-persistent-host-automation.md in AgentsMD.
Required proof
The pilot report and verdict counts on this Issue; for the schedule, the first two scheduled runs' results.
History
PR #117 (phrase list) was closed unmerged on 2026-10-19. A small local classifier was considered the same day and dropped in favour of this council review.
Activity
lukemaj commentedon Oct 18, 2026
Design before implementation (agent, branch
feat/52-user-confusion):Method. Deterministic heuristic, no model calls, no network. A genuine user message is a candidate when all hold:
phrase(a multilingual pattern anywhere in the message, e.g. "what do you mean", "I don't understand", "explain it better/simpler/again", "in plain words", plus Polish, German, Spanish, French, Italian, Portuguese equivalents) orbare(the whole message is a short interjection such as "huh?", "what?", "bro", "??", "co?", "wie bitte?").assistant_messageevent before it with no other genuine user message in between. No preceding agent message means no finding.Where it plugs in. A new
user_confusiondetector inagent_observer/analysis.pynext tohuman_correction, marked heuristic, run bydetect_session, selectable withdiagnose --detector user_confusion. Being inDETECTORS, it appears incompare --by agentsmd|harness|project|modelwith session counts as sample sizes andunknowngroups, like the other detectors. Publish gets a flag phrase (count only).Output fields. The standard incident (
detector,heuristic: true,session,harness,project,agentsmd_version,ts,event_refs) plususer_message(native id),agent_message(event id) andsignal(phraseorbare). The summary carries no message text.Not in this change. The needs-a-look inbox flag waits for #28, so the PR will say
Refs #52. Precision and recall come from a private labeled sample of at least 100 messages from a temporary copy of the bestia ledger; only counts, the labeling rule, precision and recall will be posted.lukemaj commentedon Oct 18, 2026
Handoff: PR #117, branch
feat/52-user-confusion, head2b6d34d6a263459edad3bb1af28cdb15ac5d59bf,Refs #52.Proof on the head:
python3 -m unittest discover -s testsran 791 tests, OK.python3 -m compileall -q agent_observer tests scriptsexited 0. The newUserConfusionTesttests failed against base71bcb59and pass on the head. All fixtures are synthetic.Labeled sample: 173 real genuine prompts came from a temporary copy of the bestia ledger: all 133 non-empty prompts of 200 characters or less, plus 40 random longer ones. 5 are labeled positive. Precision is 4/4 = 1.00 and recall is 4/5 = 0.80; one clarification request about what the agent wanted done was missed. The sample and labels are private and stay outside the repository.
Labeling rule: a prompt is positive when it asks the agent to restate, clarify, simplify or elaborate something the agent itself just said. That covers a bare confusion interjection, "what do you mean", "I don't understand, explain better", "too much detail for me", "elaborate on this", and asking what the agent's instruction meant. It is negative for new questions about the world or the work, corrections and complaints, instructions, acknowledgements, probes, and prompts that only quote such wording.
Live run on the ledger copy: 18,195 sessions had genuine prompts, and the detector found 4. I spot-checked all 4: each is correct and links the right preceding agent message.
Open:
diagnoseover the whole ledger copy ran past 30 minutes and was stopped. That slowness comes from the existing detectors. The live figures come from the new detector over the same session scope.feat(analysis): detect user requests to restate an agent message (#52)
fix(analysis): keep user_confusion to wording that names a failure (#52)
lukemaj commentedon Oct 19, 2026
Round-2 handoff: PR #117, branch
feat/52-user-confusion, head103a3499e128260e18fa1c741dd814a3574cb5bb, rebased ontoorigin/main024cd97. StillRefs #52.Proof on the head:
python3 -m unittest discover -s testsran 820 tests, OK.python3 -m compileall -q agent_observer tests scriptsexited 0. The 4 round-2 tests failed before the fix.Changes this round:
hä?is fixed by NFC normalization plus umlaut folding.user_confusionis no longer also ahuman_correction.comparenotes and README say that small detector cells are a census.Labeled sample (same 173 bestia prompts and labeling rule as before):
Live run on the ledger copy: 2 findings, the same 2 messages as in the sample, both correct. This is a census, not independent evidence.
Acceptance still open:
lukemaj commentedon Oct 19, 2026
Backlog status 2026-10-19 03:40 Warsaw: PR #117 (head 103a349) is held open, not merged, for a user decision. After both reviews' precision fixes the phrase-list heuristic finds 2 of 5 labeled confusion moments (recall 0.40, 95% interval 0.12-0.77; precision 2/2) on 173 bestia prompts, and three criteria remain unmet: no non-English labeled sample (labeling on rocky/cavallo means reading prompts there), the method is a phrase list rather than something that covers varied wording, and the inbox flag waits for #28. Decision needed: merge the low-recall heuristic as a first measurable signal, or switch to a model check (costs credits and reads prompt text locally), or park #52.
[-]Detect moments where the user could not understand an agent's message[/-][+]Detect user messages where agent output failed the user (confusion or frustration), with a small local classifier[/+]lukemaj commentedon Oct 19, 2026
Rescoped on 2026-10-19 by the user's decision after design reviews by Sol and Grok: a separate small classifier (not Cue), asynchronous, covering frustration as well as confusion. PR #117 is closed unmerged. The body now holds the outcome and acceptance criteria.
lukemaj commentedon Oct 19, 2026
Breakdown constraints found on 2026-10-19: AgentsMD version identity is rejected for current direction blocks (1,781 Claude and 1,753 Codex headers; fix in #128), and user-message origin is unknown for 953 Claude, 298 Grok and 1 OpenCode messages, which affects the denominator.
[-]Detect user messages where agent output failed the user (confusion or frustration), with a small local classifier[/-][+]Spectrum council reviews moments where agent work failed the user (confusion, frustration, wrong requirement, inefficiency)[/+]lukemaj commentedon Oct 19, 2026
Rescoped again on 2026-10-19 by the user's decision: no separate classifier. A Spectrum council judges Observer's existing intervention moments, after one manual pilot on bestia.