Repository navigation
Coherent agent-facing text: one inventory, a CI lint, and protocol friction as a release metric #487
Description
Agents learn the kernel's protocol only from the text it shows them: the brief, each verb's help, refusals, wake messages and kernel notes. That text is written in many places and has drifted (see #484: one verb named end while every help line says "end your turn", and one refusal message used for two opposite situations, with kernel-internal words like "leg").
The cost is measurable. In a recent study (single-agent runs on one benchmark, six author models), the share of runs that hit a protocol refusal and the refusals per run were:
| Condition | Runs hitting a refusal | Refusals per run |
|---|---|---|
| main author model, depth cap 1 | 7 / 15 | 5.1 |
| main author model, depth cap 2 | 4 / 15 | 2.7 |
| main author model, default cap | 1 / 20 | 0.2 |
| a second open model, default cap | 6 / 15 | 6.0 (never completed a submit) |
Friction is highest exactly where the agent has the fewest turns to recover, so incoherent text costs research, not just tokens.
Proposal
- One reviewable inventory. Every agent-facing string (brief sections, verb help and descriptions, refusals, wake and kernel messages) is generated from, or registered in, one module that can be printed and reviewed as a whole.
- A CI lint over that inventory:
- one term per concept (verbs, turn vs run, sleep/wake), checked against a small glossary;
- no kernel-internal vocabulary;
- every refusal names the next step.
- Friction as a release metric. The pre-release canary runs report refusals per run by author model; a release that raises it does not ship without an explanation in the release PR.
Related: #484 (end → stop), which is the first concrete fix this inventory would have caught. Planned for 0.4; sequenced with the attempt.py split so the inventory module is designed once.
Activity