Make rules reliable: turn mechanical rules into hooks, check conversation rules at reply time

What: Make AgentsMD rules hold in practice. Rules that are steps a script can do become hooks or scripts; worktree retirement moves to Chromeria (toolboxmd/chromeria#197). Rules that only the agent's judgment can meet are surfaced at the moment they apply, and the reply is checked against them.
Why: Agents read the rules and still skip them. Prose alone does not make an agent act, and right now nothing checks whether they did.
So what: An agent builds a first conversation check, measures compliance before and after, and reports here. The user reviews the rule classification and the wording.

Contribution to the Objective

"A reliable out-of-the-box system": right now the rules that govern ordinary work and conversation hold only when the agent remembers to follow them.

Evidence (2026-10-09)

Worktree retirement. finalization.md ("Retire at merge") and implementation.md (sweep merged worktrees at task start) landed in #140 on 2026-09-27.

~/dev/toolboxmd/chromeria: 18 of 28 worktrees belong to merged PRs, all created after the rule. du shows about 5.8 GB each, but pnpm's APFS clones share blocks, so all Chromeria checkouts together really use about 38 GB (measured by physical block offsets). The disk is 97 to 99% full.
Transcripts (Claude Code and Codex) after 2026-09-28 that mention gh pr merge: 37. Of those, 31 mention finalization.md, and 13 of those 31 never ran worktree remove. This is a rough text match, not verified merges. The likely failure is acting on the rule, not loading it.
When the user merges on GitHub, no agent counts as "whoever merges", so the cleanup at merge never runs.
Chromeria's own storageCleanup only scans ~/.t3/worktrees (empty) and is off by default, so it never covers these worktrees.

Conversation rules. In the session that found the above (Claude Code, Opus 5.5), the agent:

did not load operations at task start, despite the SessionStart hook telling it to;
wrote replies far over the three-sentence Communication default;
recommended a process design without an Elon record (see Elon method is not applied in conversation: give it a visible output, concrete triggers and a brief rule #198).
The user's ask: "I would like the agents to use the prose instructions when we are talking and they are not doing that."

Proposed approach

Classify every core and operations rule as one of:
- mechanical: a script can do it;
- checkable: an agent decides, a script can verify;
- judgment: prose only.
Mechanical rules become code. For worktree retirement the code lives in Chromeria: Agents get worktrees from Chromeria, and cleanup removes them at merge chromeria#197. The user runs all work through Chromeria (2026-10-09), so no AgentsMD hook. When docs: write already-decided direction changes and report the diff (#196) #197 ships, the prose it covers in finalization.md and implementation.md gets shortened to point at it.
Conversation rules apply at the moment they are needed and are checked.
- A short per-turn reminder of the conversation rules (Communication limits, Elon record before a recommendation). Candidates: UserPromptSubmit or an equivalent.
- A reply check for what is checkable: em dashes, length over the default without a reason, a recommendation without an Elon record. Candidates: a Stop hook or an equivalent that asks for a rewrite.
- Unknown: equivalent hook points on Codex, Grok and OpenCode.
Measure compliance for each converted rule before and after, with Agent Observer or the Procedure trigger audit: measure which operations procedures fire, then fix the worst #164 harness.

Acceptance criteria

The rule classification table is in this Issue, reviewed by the user.
Worktree retirement is handled by Agents get worktrees from Chromeria, and cleanup removes them at merge chromeria#197, and the matching prose is trimmed here.
A conversation reminder and reply check ship for Claude Code at minimum. Hosts without hook support are recorded.
Before and after compliance numbers for the converted rules are posted here.
Prose replaced by a mechanism is removed from the core or references.

Non-goals

Re-measuring procedure triggers (Procedure trigger audit: measure which operations procedures fire, then fix the worst #164) or rewording the Elon method (Elon method is not applied in conversation: give it a visible output, concrete triggers and a brief rule #198). This Issue reuses their evidence.
Shrinking pnpm installs in Chromeria (supportedArchitectures adds about 2.6 GB of Linux and x64 binaries per worktree). That belongs in the chromeria repo.
Deleting the existing merged worktrees. That is a separate, user-approved local cleanup.

Blockers

None known. Unknown: hook support on non-Claude hosts.

Elon record

Requirements and who asked: user (2026-10-09): instructions must work reliably, both actions like worktree cleanup and prose rules in conversation.
Deleted: an AgentsMD worktree hook (Chromeria's cleanup covers it, chromeria#197); adding more or stronger prose (already shown not to work); a separate compliance ledger; hooks for judgment-only rules.
Bottleneck: rules whose execution depends on the agent choosing to act, with nothing checking the outcome.
Checked myself: 18 merged worktrees and their sizes on disk; 37 transcripts mentioning a merge and their finalization.md and worktree remove mentions; Chromeria storageCleanup scope and defaults; my own replies in this session.

Draft for user review: rule classification

What: every core (global/AGENTS.md) and operations rule, classed as mechanical (a script can do it), checkable (the agent decides, a script can verify) or judgment (prose only), with today's mechanism.
Why: criterion 1; it decides which rules get a hook and which prose can shrink later.
So what: you review the classes and the "Proposed" column; the draft PR builds only the three conversation checks marked #201 draft.

Granularity: one row per rule in the core; one row per enforceable rule in operations (method-only procedures share a row). "Proposed" is a candidate, not scope of this PR.

Core (`global/AGENTS.md`)

#	Rule	Class	Mechanism today	Proposed
C1	Never use em dashes	checkable	none	#201 draft: Stop check + reminder
C2	Smallest-scope solution	judgment	none	prose
C3	Canonical `global/AGENTS.md` verified, reread on problem	mechanical	`bin/project-direction` hook (SessionStart, UserPromptSubmit)	keep
C4	Read private `PREFERENCES.md` in full	mechanical	same hook injects it	keep
C5	Fix behavior at its owner, not in preferences	judgment	none	prose
C6	Orient only as far as needed; take context from the brief	judgment	none	prose
C7	Do not silently widen; dissent when material	judgment	none	prose
C8	Reply at most three sentences unless asked or a risk needs more	checkable	none	#201 draft: Stop check + reminder
C9	Evidence goes in the Issue/PR, not the chat	judgment	none	prose (C8 check catches the symptom)
C10	Issue/PR opens with What / Why / So what	checkable	none	add to the `gh` create gate beside the Elon record
C11	Do not narrate process	judgment	none	prose
C12	Colleague tone: no filler, no recap, no closing offers	judgment	none	prose
C13	Keep code and errors exact; reply in the user's language	judgment	none	prose
C14	Restate plainly when the user seems confused	judgment	none	prose
C15	Briefs carry context, effort, review, proof	checkable	none	later: PreToolUse on `delegate_task`
C16	Lead with bad news; separate fact, inference, unknown; research before recommending	judgment	none	prose
C17	Before recommending, state the goal and recommend one option	checkable	none	candidate for the Stop check
C18	Never ask "can I" / "may I" for work within reach	checkable	none	candidate for the Stop check
C19	Audits and reviews stay read-only	judgment	none	prose
C20	Dirty and untracked files are user-owned	checkable	none	later: PreToolUse on `git add -A` / `git commit -a`
C21	File out-of-scope problems at once	judgment	none	prose
C22	Read ADR 0001 before persistent host automation	checkable	none	prose
C23	Elon record in every Issue and PR	mechanical	PreToolUse Elon gate blocks `gh issue/pr create`	keep
C24	Elon record before a material recommendation in conversation	checkable	none	#201 draft: Stop check + reminder
C25	Software design rules	judgment	none	prose
C26	Keep the direction triad in context	mechanical	`bin/project-direction` hook	keep
C27	Flag drift; Spec/Issue states its Objective contribution	judgment / checkable	none	contribution line could join the `gh` gate
C28	Read glossaries before naming concepts	judgment	none	prose
C29	Check status, branch, HEAD, base before mutating	checkable	none	prose
C30	Create Issues/PRs with `gh issue` / `gh pr`, not the API	checkable	none (gate only sees `gh ... create`)	later: gate `gh api` POSTs to issues/pulls
C31	Version bump before a completed commit	mechanical	`versionctl` pre-commit + commit-msg hooks, Version check CI	keep
C32	Use `operations` at task start and phase change	checkable	SessionStart pointer (`bin/operations-routing`), not checked	candidate: PreToolUse reminder on first Edit/commit without a procedure read
C33	Delegate through the routing tool	judgment	none	prose
C34	Final PR merge needs human approval	judgment	none in AgentsMD	prose
C35	Verify live state after an external mutation	judgment	none	prose
C36	One writer per branch, workspace, file set	mechanical	Chromeria worktrees	Chromeria
C37	Human Gates (release, install, credentials, money...)	judgment	host permission modes only	prose
C38	Secrets stay out of commits, Issues, logs	mechanical	none in AgentsMD	out of scope here
C39	Legacy ledgers are inputs, not trackers	judgment	none	prose
C40	Handoff records full state in the Issue/PR	judgment	none	prose

`operations`

#	Rule (owner)	Class	Mechanism today	Proposed
O1	Read the routed procedure in full before the action (`SKILL.md`)	checkable	SessionStart pointer	see C32
O2	Sweep merged worktrees at task start (`implementation.md`)	mechanical	none	toolboxmd/chromeria#197
O3	Retire worktree, branches, processes at merge (`finalization.md`)	mechanical	none	toolboxmd/chromeria#197
O4	One SemVer transition per merge unit (`implementation.md`, `delivery.md`)	mechanical	`versionctl`, Version check CI	keep
O5	WIP commits use marker + `wip:` prefix (`version-control`)	mechanical	`versionctl` commit-msg hook	keep
O6	PR body: Human summary + Elon record (`delivery.md`)	mechanical (Elon) / checkable (summary)	Elon gate	see C10
O7	Post agent cost before claiming ready (`delivery.md`)	mechanical	none	later: CI or PR hook
O8	Independent review status on the exact head (`verification.md`)	mechanical (check) / judgment (review)	convention only	later: required status check
O9	Report delivery states separately (`delivery.md`, `finalization.md`)	judgment	none	prose
O10	Website impact classification (`delivery.md`)	judgment	none	prose
O11	Read test design before editing tests (`verification.md`)	checkable	none	covered by C32 if built
O12	Never just update expected values (`verification.md`)	judgment	none	prose
O13	Scoped proof only under a trusted policy (`scoped-proof.md`)	mechanical	`bin/scoped-proof`	keep
O14	Delivery profile through the loader (`delivery-profile`)	mechanical	`bin/delivery-profile`	keep
O15	Workers never write in a main checkout (`orchestration.md`)	mechanical	live-checkout health report (#191)	keep
O16	No new-component dispatch without a Spec Issue (`orchestration.md`)	checkable	none	prose
O17	Close finished children (`orchestration.md`)	mechanical	Chromeria	Chromeria
O18	Triad changes need user decision or confirmation (`project-direction`)	judgment	none	prose
O19	"Read X in full" pointer form (`writing-for-agents`)	checkable	contract tests	keep
O20	Glossary format and ADR threshold (`domain-modeling`)	judgment	none	prose
O21	Method procedures: research, diagnosis, software design, code review, grilling, prototype, reflection, technical writing, to-spec, to-tickets, wayfinder, elon-method	judgment	none	prose

Summary

Mechanical, already code: C3, C4, C23, C26, C31, O4, O5, O13, O14, O15, O19.
Mechanical, moving to Chromeria: O2, O3 (chromeria#197).
Checkable, built in the Make rules reliable: turn mechanical rules into hooks, check conversation rules at reply time #201 draft: C1, C8, C24.
Checkable candidates for later: C10, C17, C18, C20, C30, C32, O7, O8.
Everything else stays judgment (prose).

Baseline: conversation-rule compliance before the check (criterion 4, before half)

What: in 254 real replies to the user since 8 October, no reply used an em dash, 73% ran over three sentences, and 9 of 15 recommending replies had no Elon record and no link to one.
Why: this is the "before" number for the three checks the #201 draft adds (C1, C8, C24 in the classification).
So what: the "after" half runs the same conversation-check scan over sessions once the hook is installed, after your review.

Results

Window: human prompts from 2026-10-08 15:23 to 2026-10-18 23:29 (Warsaw); Agent Observer 1.2.6 ledger after sync.

Harness	Replies	Sessions	Over 3 sentences	Median sentences	p90	Recommends, no Elon record or link
Claude Code	147	11	118 (80%)	10	27	6
Codex	79	9	47 (59%)	4	9	3
Grok Build	27	2	20 (74%)	6	15	0
OpenCode	1	1	0	1	1	0
All	254	23	185 (73%)			9

Recommendations: 15 Claude/Codex replies match the recommendation pattern; none carries all four Elon fields inline; 6 link an Issue or PR (not checked for a record there); 9 do neither.
Em dash, wider: across all 1,940 assistant messages in the Claude/Codex sessions (interim text included), 1 contains an em dash.

Method

agent-observer interventions --since 2026-10-04 --created-by all --kind prompt lists prompts; a reply is the last assistant_message after a prompt with created_by: user and before the next such prompt in the same session (the end of the user's turn, which the Stop hook sees). Every reply event came back ok from event show; 1 Grok prompt had no reply.
Each reply text goes through bin/conversation-check scan on the draft branch, the same functions the Stop hook runs:
- em dash: U+2014 outside code blocks and inline code;
- sentences: terminators . ! ? in prose, after removing code, URLs, headings, table separators and abbreviations like "e.g."; each list item and table row counts at least once;
- recommendation without record: first-person "I/we recommend, suggest, propose", "my recommendation", "Recommendation:" or "(Recommended)", with neither all four Elon field names nor an Issue/PR reference (#N or a GitHub issue/pull URL).

Limits

Length is an upper bound on violations. The rule allows more when the user asks or a risk, Human Gate or ambiguity needs it; the counter cannot tell, so 73% counts allowed long replies too. Two replies counted by hand matched the counter exactly and within one sentence; a bold-wrapped sentence can be undercounted.
Recommendation detection is narrow. Imperative advice ("Do X first") and recommendations without first-person phrasing are missed; a link to any Issue counts as a record without checking that the Issue holds one. Materiality is not judged.
Origin: only prompts Observer marks user (typed in T3 Code) count. 959 Claude and 17,653 Grok prompts of unknown origin are excluded, so plain terminal sessions may be missing; the Grok and OpenCode samples are small.
Sessions mix tasks and periods; compare before and after by the same method, not as a controlled experiment.
Raw text stays local; this comment carries counts only.

wip: join soft-wrapped lines and treat an unclosed fence as code (#201)

wip: read only plain paragraphs, count list items once (#201)

Corrected baseline (supersedes the counts in the baseline comment above)

What: the same 254 replies, rescored with the checker at 7cba1e5 (PR #235), give 184 over three sentences (72%) instead of 185; the em-dash and recommendation counts are unchanged.
Why: after two review rounds the counter reads only plain paragraph text, counts each list item as one sentence (quotes included), and skips code, quotes, tables and HTML.
So what: the "after" measurement will use this counter; nothing needs you here.

Harness	Replies	Over 3 sentences	Median	p90	Recommends, no Elon record or link
Claude Code	147	118 (80%)	8	22	6
Codex	79	46 (58%)	4	8	3
Grok Build	27	20 (74%)	6	15	0
OpenCode	1	0	1	1	0
All	254	184 (72%)			9

Same window, same reply selection and same limits as above; only the counting rules changed. Recommendation phrases are now read in paragraphs only, so one inside a list item is not counted.

State (2026-10-19 04:40 Warsaw): waiting for the user. The classification, baseline (corrected: #201 (comment)) and draft PR #235 (head 7cba1e5) are ready for review. Luna round 3 found remaining Markdown edge shapes; parser rounds stopped (#235). Decisions for the user: approve the classification and wording; answer the PR's three questions; choose a real Markdown parser dependency if the check ships.

User decisions (2026-10-19), for draft PR #235:

Drop the em dash check (0 of 254 replies had one).
Worker sessions keep the Elon record check; only the length check is skipped there, since their long reports are requested. Interactive sessions get every check.

Still open: rewrite-after-shown vs. naming findings in the next turn's reminder; whether to shorten the reminder to about 35 tokens (proposed: "Reply rules: result first, at most three sentences unless asked or a risk needs more. Give the Elon record (Requirements, Deleted, Bottleneck, Checked) before any design recommendation."); a real Markdown parser before shipping. Implementation must find a reliable way to tell a worker session from an interactive one.

feat: remind the agent of the reply rules every turn (#201)

State (2026-10-19, Warsaw): PR #235 is ready for review at head 70a8e1b (17.4.0): the Stop-hook reply check and scan are deleted; one fixed reminder now rides the existing per-prompt loader on Claude Code, Codex, Muse and OpenCode, and Grok is recorded unsupported.
Proof: full suite Ran 447 tests ... OK, versionctl tests Ran 7 tests ... OK, versionctl release-check OK; the new reminder tests fail against main's loader (details in #235).
Next: Luna reviews, then the user merges. This Issue stays open: the one-week after measurement reruns the baseline counter from git show 7cba1e5:bin/conversation_check.py (command in the PR body), and the prose trimming remains.

Merge pull request #235 from toolboxmd/feat/201-conversation-check

Navigation Menu

Repository navigation

Make rules reliable: turn mechanical rules into hooks, check conversation rules at reply time #201

Contribution to the Objective

Evidence (2026-10-09)

Proposed approach

Acceptance criteria

Non-goals

Blockers

Elon record

Draft for user review: rule classification

Core (`global/AGENTS.md`)

`operations`

Summary

Baseline: conversation-rule compliance before the check (criterion 4, before half)

Results

Method

Limits

Corrected baseline (supersedes the counts in the baseline comment above)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Make rules reliable: turn mechanical rules into hooks, check conversation rules at reply time #201

Description

Contribution to the Objective

Evidence (2026-10-09)

Proposed approach

Acceptance criteria

Non-goals

Blockers

Elon record

Activity

lukemaj commented on Oct 18, 2026

Draft for user review: rule classification

Core (global/AGENTS.md)

operations

Summary

lukemaj commented on Oct 18, 2026

Baseline: conversation-rule compliance before the check (criterion 4, before half)

Results

Method

Limits

lukemaj commented on Oct 19, 2026

Corrected baseline (supersedes the counts in the baseline comment above)

lukemaj commented on Oct 19, 2026

lukemaj commented on Oct 19, 2026

lukemaj commented on Oct 19, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Core (`global/AGENTS.md`)

`operations`