[lukemaj]

What: Adds the user_confusion detector. It flags a short prompt that says the user did not understand the agent's last message, and links both messages by reference.
Why: each such moment shows a report that failed its reader, and nothing collected them before this change (#52).
So what: this needs re-review of the round-2 head. #52 stays open (Refs): the non-English labeled-sample criterion is not met, and the inbox flag waits for #28.

Changes

• agent_observer/analysis.py: a new heuristic detector user_confusion, run by detect_session and listed in DETECTORS, so diagnose --detector user_confusion and compare include it.
  • Method: a fixed phrase list, matched on lowercased, NFC-normalized and accent-folded text (umlauts fold to ae/oe/ue), with no model call.
  • bare: the whole message is huh/what (or que, quoi, hein, co, że co, hä) as a question or exclamation, optionally after bro/bruh/wait. bro, bruh, huh and wait what also count alone.
  • phrase: "what do you mean", "what does that mean", "what do you want me to do", "I don't understand/follow/get it" (ending the message or followed by a clause), "I'm confused", "explain it/that again/better", a whole-message "simpler?", "say that again", "come again?", "eli5", and a short list in Polish, German, Spanish, French, Italian and Portuguese.
  • Structure: the prompt is at most 200 characters, and the latest agent message comes after the previous genuine prompt.
  • Fields: the standard incident plus user_message, agent_message and signal; event_refs is [agent_event_id, user_native_id]. Findings carry no text.
• human_correction no longer also counts a prompt that user_confusion counted ("bro what?"), so one message moves one series.
• The compare notes, both views, and README.md say that a detector cell with few incidents is a census, not a version, harness, project or model effect.
• agent_observer/publish.py: counts are published as "clarification requests".
• tests/test_analysis.py: UserConfusionTest, using synthetic fixtures only.

Round-2 resolutions

1. Precision (Luna, Grok):
   • Dropped elaborate, rephrase, a leading simpler, write/say/put … better, explain better … mid-sentence, too complicated/much detail, plain/simple words, non-programmer and make it simpler.
   • Dropped the bare sorry, hmm, pardon, meaning, come, was, comment and ??. A bare interjection must be the whole message.
   • "I don't understand" now has to end the message or be followed by a clause, so "I don't understand why the test fails" does not match.
   • Grok's sentences are now negative tests.
   • Misses added: "what do you want me to do?" and "wait what".
   • Misses left out: "I still don't know whether…", "so did it ship or not?" and "which commit is on main?" read like ordinary questions about the work. "huh, the third row" is not the whole message. "tl;dr?" asks for a shorter report rather than saying the user did not understand.
2. hä?: fixed by NFC plus umlaut folding. hä?, HÄ? and decomposed hä? are tested.
3. Overlap with human_correction: fixed in code, not just documented, and tested.
4. Evidence honesty: README and this body give the real numbers with intervals, below. I did not run a labeled sample on cavallo or rocky. Labeling needs the labeler to read prompt text, and that would move transcripts off their machine. So the non-English criterion stays unmet.
5. Census note: added as described above. The cells are unchanged.
6. Rebase: rebased onto origin/main 024cd97. There was no conflict with fix: explain PR task form for empty Issue publish scopes #110 or fix: keep structured gh output free of terminal color #115. The test_diagnose_scale fixture now includes a confusion case.

Proof (head 103a3499e128260e18fa1c741dd814a3574cb5bb)

• Failing first: the 4 round-2 tests failed before the fix, each for its intended cause. Round 1's new tests failed on the original base.
• python3 -m unittest discover -s tests: Ran 820 tests ... OK. After the rebase, test_diagnose_scale, from perf: index session occurrence lookups for diagnose #119, requires its mixed fixture to trigger every detector. I added one synthetic huh? prompt after the agent message there; the assertion is unchanged.
• python3 -m compileall -q agent_observer tests scripts: exit 0.
• Labeled sample: the same 173 real bestia prompts as round 1, labeled by me with the rule on Spectrum council reviews moments where agent work failed the user (confusion, frustration, wrong requirement, inefficiency) #52. Only the 133 prompts of 200 characters or less can fire. Of the 5 positives, the detector found 2: recall 0.40, Wilson 95% interval 0.12 to 0.77. Precision is 2/2, interval 0.34 to 1.00, and there are no false positives among the 168 negatives. The 2 lost since round 1 ("elaborate on this", "too much detail for a non-programmer") are the patterns the review asked to drop.
• Live run on the ledger copy: 2 findings in 18,195 sessions with genuine prompts. These are the same 2 messages the sample counted, so this is one census, not independent evidence. Both were already spot-checked as correct with the right agent message.

Limits

• Issue criteria still open:
  • The non-English labeled sample is not met. The non-English patterns are unmeasured on real prompts.
  • "Not only a keyword list" is not met: this is a phrase list with structural checks.
  • The inbox flag waits for Needs-a-look inbox: ranked suspicious-work feed with dismiss verdicts #28.
• Small numbers: the positive count is tiny. Precision and recall here describe this machine's history only.
• publish gap: publish for this task returned nothing to publish: no sessions in scope.

Elon record

• Requirements and who asked: notice prompts where the user had to ask what an agent meant, linked by reference, and counted per project, harness and AgentsMD version (the user, Spectrum council reviews moments where agent work failed the user (confusion, frustration, wrong requirement, inefficiency) #52, 2026-10-11). Use a deterministic method with measured precision and recall (coordinator brief, 2026-10-18). Round 2 tightened precision, made the evidence honest and added the census note (Luna and Grok r1, coordinator 2026-10-19).
• Deleted: model checks, a new subsystem, a separate breakdown report, message text in findings, and the patterns that match ordinary next instructions.
• Bottleneck: real confusion prompts are rare (5 in 173), so the evidence is too thin, not the detector.
• Checked myself: ran every review sentence through _confusion_signal, re-measured on the same labeled sample, re-ran the detector on the ledger copy, and read the full test output on the exact head.

Refs #52

🤖 Generated with Claude Code

[lukemaj]

Observer publish gap: agent-observer publish --task toolboxmd/agent-observer#117 --repo toolboxmd/agent-observer --pr 117 (installed 1.2.6) returned nothing to publish: no sessions in scope, so no consumption summary is attached to this PR. This session was not assigned to that task in the ledger.

[lukemaj]

Grok roast r1 of 2b6d34d:

CHANGES NEEDED

This records a closed phrase list on short follow-ups. It does not record the moments where a report failed its reader, and the posted 4/4 and 4/5 figures do not meet #52. I ran _confusion_signal from this commit on the sentences below.

1. High. The list fires on the next instruction, and misses the failed report

A genuine prompt usually sits directly after an agent message, because the user reads the report and types what to do next. Adjacency is the common case. The phrase list then decides, and the 200-character cap keeps the short instruction while dropping the explanation of what failed.

Misses, each under 200 characters, signal none:

• "I still don't know whether you merged it or only opened the PR."
• "so did it ship or not?"
• "which commit is on main?"
• "what do you want me to do?" and "what action do you want from me?" (the shape of the one labeled miss)
• "wait what"
• "huh, the third row"
• "tl;dr?"
• "hä?" folds to "ha?" and does not match the hae token, so the German interjection written into CONFUSION_BARE_RE is a miss

Hits that are the next instruction, not a failed report, signal phrase:

• "write better commit messages next time"
• "elaborate the migration plan"
• "don't elaborate, just merge"
• "rephrase the error for the changelog"
• "simpler function names"
• "use simple words in the commit"
• "explain better how caching works"
• "too complicated, skip it"
• "ask a non-programmer to read the paragraph"
• "put it in plain text in the log"

Hits that are a different speech act, whole message, signal bare: "sorry", "hmm", "pardon", "meaning", "come", "was", "comment", "??".

The reports worth fixing are the long ones. A message that contains "I don't understand" and runs past 200 characters returns none. A bare "huh?" is kept and does not say which sentence failed. tests/test_analysis.py test_near_misses_are_not_confusion never feeds the instruction sentences above, so the suite stays green.

Cut the tokens that are ordinary follow-ups or other speech acts: elaborate, rephrase, a leading simpler, write|say|put plus better, too complicated, simple words, non-programmer / non-technical, and bare sorry, pardon, hm+, eh, meaning, comment, come, was. Keep the examples #52 named (huh?, what?, bro, bruh, what do you mean, I don't understand, explain it better / simpler). Leave the Polish, German, Spanish, French, Italian and Portuguese alternatives out of this PR until a real non-English sample exists. They are most of the regex and none of the evidence. hä? shows the bare list was not run through _fold.

2. High. The sample does not meet "at least 100, including non-English"

#52 asks for precision and recall on at least 100 real messages including non-English, and for varied wording beyond a keyword list. The head is a keyword list. The sample is 173 bestia prompts, labeled by the author, 5 positives, and no non-English human prompt. The wording-only score is 4/4 and 4/5. A Wilson 95% interval is about 0.51 to 1.00 on 4/4 and about 0.38 to 0.96 on 4/5. The labeling rule restates the patterns, so four of the five positives are the list matching itself.

The 40 longer prompts cannot be false positives: above 200 characters the gate returns none. The sample took every non-empty prompt of 200 characters or less, which is the only set that can fire, and the live run found 4. That is one census of four messages, written up once as sample precision and once as a live spot-check, unless some wording hit failed adjacency. The writeup names no such hit. bestia simply did not contain "sorry", "hmm", or "elaborate the plan". Precision on this machine's history says that, and nothing about the next machine or next week.

Synthetic tests show the regex matches strings placed next to it. They are not a non-English sample.

Say the gap in both places. In the PR and in README.md, replace "found 4 of 5 with no false positives" with the unmet clause: English only, five author labels, interval too wide to use, multilingual patterns unmeasured. On #52, leave the acceptance box open and record that this PR does not close the non-English sample or the "not only a keyword list" line. Refs #52 is the right link.

3. Medium. compare turns one interjection into a version effect

The Issue handoff table is 1 finding under AgentsMD 17.2.0 (6 sessions) and 3 under unknown (18,170), and 4 under claude (16 sessions) against 0 under grok (17,300). compare stores per_session (analysis.compare, the division by len(members)). The text view prints the count beside the session total. One "huh?" in a 6-session version is 0.167 per session against about 0.00016, a thousand-fold gap from a single bare word. At this volume the column cannot show whether a reporting rule worked.

Keep it in DETECTORS. #52 asks for counts with sample sizes, and a separate report would be a larger change. Add one sentence to the README row and to the Issue handoff: these cells are a census of 4 events, and per_session is not a version result until a version has more than a handful of findings.

The same utterance also fills human_correction. CORRECTION_RE already matches a message that opens with bro, bruh, no, or don't. One "bro" after a report increments both detectors. "no idea what you mean" and "don't understand, say it again" do too. A version comparison then moves both series from the word #52 used as its example. Say that next to the census sentence.

Cleared

Findings link the agent event and the user native id and carry no message text. The inbox flag stays with #28. An unknown AgentsMD version stays unknown. Adjacency drops "what do you mean" when another genuine prompt sits between it and the agent message, including a "thanks" followed by the question. Leave that rule. The acknowledgement miss belongs beside the 200-character cap: both delete the clarification that says what failed.

[lukemaj]

Luna review r1 of 2b6d34d:

REQUEST CHANGES

• High, false positives (agent_observer/analysis.py:38-49,62-63): _confusion_signal labels ordinary next instructions as confusion because broad tokens such as elaborate, rephrase, simpler, too complicated, plain text, and non-programmer match anywhere. The bare alternatives also classify unrelated utterances (sorry, hmm, pardon, meaning, come, was, comment) as confusion. I called _confusion_signal on these examples at the exact head and confirmed they return phrase or bare. Since this detector is intended to count moments a human could not understand the agent's message, ordinary requests like “elaborate the migration plan” and “simpler function names” inflate the signal, while bare “sorry” or “comment” has no such meaning. Adjacency does not resolve this because these are normal immediate follow-ups too.
• Medium, Unicode miss (agent_observer/analysis.py:539-540,62-63): _fold("hä?") becomes ha?, but the bare pattern contains hae, so a common German confusion interjection is missed despite being claimed as a supported language. The roast's repro is accurate.
• High, acceptance proof incomplete (README.md:100-106; Issue Spectrum council reviews moments where agent work failed the user (confusion, frustration, wrong requirement, inefficiency) #52 acceptance): the Issue requires precision/recall on at least 100 real messages including non-English. The reported 173-message sample has only five author-labeled positives and no non-English prompts; non-English behavior has synthetic tests only. The live spot-check is useful, but does not satisfy that labeled-sample criterion or establish multilingual performance. Keep Spectrum council reviews moments where agent work failed the user (confusion, frustration, wrong requirement, inefficiency) #52 open and describe the coverage as unmeasured until that evidence exists.

Cleared: findings include only references and fixed summaries, the inbox flag is correctly deferred with Refs #52, and compare retains unknown groups. The requested exact-head test suite passes; no version, manifest, or changelog change is in this diff.

[lukemaj]

Closed unmerged by the user's decision (2026-10-19). A phrase list caught 2 of 5 labelled cases, and its 2 hits would be read as the confusion rate. #52 is rescoped to a small, asynchronous classifier that runs locally; this PR's labelling rule and its 5 bestia positives become seeds for that.