Procedure trigger audit: measure which operations procedures fire, then fix the worst

What: Measure how often each operations procedure fires when it should, then fix the worst triggers, starting with wording.
Why: Most procedure triggers are judgments ("substantive prose editing", "material work", "when the action changes") one or two links below the only description the host shows, and nothing tests whether they fire.
So what: An agent builds the trigger table and the baseline from real transcripts now; rewording follows in one PR, and a hook only where rewording measurably fails.

Outcome

A trigger-coverage table for every row of skills/operations/SKILL.md, each with an observable trigger event, the file that must be opened, and its skip rate in real sessions on Claude Code, Codex, Grok and OpenCode. The worst rows get reworded; a re-measurement after a week of real work shows the change.

Evidence (2026-09-29)

Research report on building-agentskills and the AgentsMD writing guides (Opus 5.5, T3 thread "Research: building-agentskills + AgentsMD writing guides"):
- Only Project Direction loading is enforced by mechanism (hook). Every other row depends on the agent noticing a phase change.
- Rated weak: technical-writing -> references/prose.md ("when revising unclear or padded text"), writing-for-agents -> prose.md ("substantive prose editing"), Software design, Artifact placement, Reflection (friction), Finalization, Wayfinder, Repository setup, Reconciliation. Medium: Writing for agents (loses to the Implementation row on an edit), SKILL-MECHANICS.md ("before choosing frontmatter" misses editing an existing description), Verification ("claim readiness" passes unnoticed), Elon method ("material").
- AgentsMD chains references two hops deep (SKILL -> writing-for-agents -> SKILL-MECHANICS or prose); building-agentskills docs/05-authoring/line-budget.md says one level, never nested.
- Existing tests pin wording and links only (tests/test_operations_contract.py); none tests firing.
Research report on obra/superpowers (Opus 5.5, T3 thread "Research: obra/superpowers description guidance"):
- Firing comes from an always-injected bootstrap plus decision-time rationalization tables, not from descriptions. Descriptions are short (92-392 chars), start "Use when", and anchor in time ("before writing implementation code").
- Test method: naive prompts that do not name the skill; deterministic transcript checks (skill called, not called, called before the first Edit/Write); negative over-trigger scenarios; a no-guidance control; five or more runs; the models users actually run (one skill went 5/5 on Opus 4.8 to 0/5 on Opus 5).
This session: the planner skipped SKILL-MECHANICS.md and prose.md while rewriting T3 tool descriptions (Delegation tools are always loaded and name the shell CLIs they replace chromeria#66).
Elon method rarely triggers: gated on Project Direction and tied to artifacts #138 method for the Elon method: 9 of 11 Issue-creating sessions skipped it before rewording, 5 of 13 after.

Elon record

Requirements and who asked: the user (2026-09-29) asked how to verify that every AgentsMD procedure triggers when it should, naming skill mechanics and prose as examples, and agreed to measure first, reword second, and use hooks last.
Deleted: hooks as a first fix; synthetic scenario runs as a release gate (the Objective does not require them); rewriting rows that measure fine.
Bottleneck: no measurement exists, so nobody knows which procedures are dead weight.
Checked myself: read both research reports in full and the routing table rows they cite; reran the Elon method rarely triggers: gated on Project Direction and tied to artifacts #138 count for the Elon method today.

Acceptance criteria

Trigger table: every routing row has an observable trigger event and required file, retained under docs/work/.
Baseline skip rate per row from the last 14 days of real transcripts, per host where transcripts exist, with the counting script retained.
One PR rewords the worst rows: observable "before " triggers, no reference deeper than one hop where a trigger was missed.
Re-measurement one week after release, reported per row.

Non-goals

Hooks for procedures that have not failed after rewording.
Making synthetic model runs a release gate.

Blockers

None.

Required proof

The retained script and both measurements; full suite and versionctl release-check for the rewording PR.

wip: #164 trigger table and baseline skip rates

feat: name the edit in missed procedure triggers, one hop from routing (

fix: clean up trigger-test temp state, guard ignored files, commit pe…

fix: trigger-test setup removes its own partial state on failure (#164)

test: four-host trigger benchmark with the user's models (#164)

wip: #164 trigger table and baseline skip rates

feat: name the edit in missed procedure triggers, one hop from routing (

fix: clean up trigger-test temp state, guard ignored files, commit pe…

fix: trigger-test setup removes its own partial state on failure (#164)

test: four-host trigger benchmark with the user's models (#164)

feat: release procedure triggers one hop from routing as 14.2.0 (#164)

feat: name the edit in missed procedure triggers, one hop from routing (

fix: clean up trigger-test temp state, guard ignored files, commit pe…

fix: trigger-test setup removes its own partial state on failure (#164)

test: four-host trigger benchmark with the user's models (#164)

wip: a help lookup (gh pr create --help) is not the action (#164)

Round 3 result under its frozen rule, and a Grok-only rerun frozen before running (2026-10-01).

Round 3 (multiturn-r3.jsonl, instrument afac243), main / branch, file read before the action:

Claude Code: delivery 2/5 / 5/5, orchestration 4/5 / 5/5, version control 5/5 / 5/5. P1 passes.
Codex: delivery 5/5 / 5/5, version control 5/5 / 4/5. P1 passes; orchestration not measurable.
Grok: orchestration 1/5 / 5/5 and version control 5/5 / 5/5 pass. Delivery 4/5 / 3/5 fails P1 (needs 4).
P2: 0 over-triggers. P3: within 1 run everywhere (Codex implementation 5/5 / 4/5, verification 5/5 / 4/5). P4: the control read nothing.

Cause of the Grok delivery result: in 12 of Grok's 15 runs (every arm), the first match for gh pr create was gh pr create --help, a help lookup issued alongside the read. The real gh pr create came later. Both branch "misses" and main's one miss are such runs; in each, delivery.md had been read before the real call. No Claude Code or Codex record has a --help match, so their cells cannot change.

Fix at e66bd84 (multiturn.py sha256 4fe0dd26…9aa7dfa7): a shell command performs the action only if the matched command segment does not ask for --help/-h (quoted words ignored). Pinned in tests/test_trigger_harness.py. Rerun: Grok only, arms main 8cad98f / branch c0e91ab / none, 5 valid runs each, same scenario, model and P1 to P4. Its result replaces Grok's round-3 cells; records go to multiturn-r3-grok-rerun.jsonl. Round 3 stays the record for Claude Code and Codex.

docs: round-3 before/after, Grok rerun blocked by its usage balance (#…

State (2026-10-01 23:30 CEST). PR #200 head 39b71ec, v14.6.1. 403 tests OK, release-check OK. Third independent review running.

Round 3 under its frozen rule:

Claude Code: passes all three rows (delivery 2/5 to 5/5, orchestration 4/5 to 5/5).
Codex: passes delivery and version control.
Grok: passes orchestration (1/5 to 5/5) and version control. Fails delivery (3/5), because the instrument took gh pr create --help as the PR action; fixed in e66bd84.
Everywhere: P2 to P4 pass.

Details: results.

Blockers (external):

Grok: 402 Payment Required: Grok Build usage balance exhausted. The fixed Grok rerun stopped at 2 valid runs per arm, below the 3 P1 needs. Topping up the balance is the user's call; after that, rerun with go.sh-equivalent python3 multiturn.py run --host grok --model grok-4.7 --effort medium --arms main=8cad98f,branch=c0e91ab,none --runs 5.
OpenCode: the free route muse-spark-1.3-contributor-free has returned Rate limit exceeded since about 19:00 UTC. Same command with --host opencode --model opencode/muse-spark-1.3-contributor-free --jobs 3.

Still open in #164: re-measure on real sessions one week after release (measure.py --since <release date>), especially version control, which the five-turn session cannot reproduce.

Handoff (2026-10-19 00:55 Warsaw): the worker carrying #200 onto integration/backlog-2026-10-18 died when systemd-oomd killed T3's Codex scope at 00:07 (toolboxmd/chromeria#491). Its work is uncommitted in /home/luke/.t3/worktrees/agentsmd/feat-164-routing-firing-train (branch feat/164-routing-firing-train, base 2f04c90): SKILL.md rows and two test files modified, docs/work/164-trigger-audit/ files added. Nothing pushed. The frozen Grok rerun did not run; no 402 was observed. Resumes on the same worktree when Codex runs again in T3.

What: The frozen Grok-only rerun completed with 15 valid runs and passes P1 through P4.
Why: The corrected instrument can now complete the five-run-per-arm comparison that the 2026-10-01 balance error interrupted.
So what: The #200 port is being committed as an unversioned component for integration/backlog-2026-10-18; the one-week real-session follow-up remains open.

Arms unchanged: main=8cad98f, branch=c0e91ab, none. Model grok-4.7, effort medium; Grok Build 1.0.62 (1474f9cbdd91). Instrument SHA-256 remains 4fe0dd266e9c9cdbb05e8ccbdb47bbe44ad6bcb842f6bd3ed22355279aa7dfa7. Completed 2026-10-19, 00:45 CEST (Warsaw).

P1: read before the action	main	branch	none
Delivery at PR creation	5/5	5/5	0/5
Version control at commit	5/5	5/5	0/5
Orchestration at spawn	0/5	5/5	0/5

P1 passes on every reworded Grok row.
P2 passes: 0/5 early reads for each reworded file, on both main and branch.
P3 passes: research, implementation, and verification each 5/5 on both guided arms.
P4 passes: controls read no scored procedure and load no operations, 0/5 throughout.
OpenCode stays not measurable because its provider is disabled here. It was not enabled or run. Claude Code and Codex retain round-3 cells; Codex orchestration stays not measurable.

The retained batch lost its execution session after nine valid records. Those records were preserved, and the six missing slots were replaced, two per arm, within the frozen replacement allowance. A separate preliminary batch hit the canonical-checkout guard and was discarded; preliminary supervision aborts and interruption metadata are disclosed in the results. No provider funding or rate-limit error occurred in the retained batch, no balance top-up or purchase was made, and the frozen instrument was unchanged.

Records: docs/work/164-trigger-audit/multiturn-r3-grok-rerun.jsonl; dated write-up appended to results-2026-10-01.md. The historical partial rerun stays available at #200's exact SHA, rather than being pooled with this batch. These historical arms do not exercise the current integration package, and the five-turn scenario still does not establish improvement for version control, whose main arm already reads it 5/5.

The current integration helper had removed a stub constant used by the frozen non-Grok control setup. Its deterministic reproduction raised AttributeError: module 'trigger_test' has no attribute 'GROK_STUB'; restoring #200's constant makes the 37 harness tests pass, without running another model. Full repository proof will be rerun on the refreshed component commit.

What: The #200 port and frozen Grok rerun are complete locally, with P1-P4 and both required test suites passing.
Why: Delivery is blocked because ContextMD rejects the required WIP commit despite successful full reads of the exact requested procedure.
So what: Resolve toolboxmd/chromeria#501, then commit the staged component, push it, and open its PR into integration/backlog-2026-10-18; no new merge or release authority is needed or assumed here.

Canonical implementation handoff

Workspace: /home/luke/.t3/worktrees/agentsmd/feat-164-routing-firing-train.
Branch: feat/164-routing-firing-train.
Starting base: 2f04c90a926a986f82ebba00971cfb5f98ffbe01.
Refreshed integration base and current HEAD: a7cf98849cbfb793726dac6c3ce6a18656a931dd.
Task changes: nine files staged, 1,034 insertions and three deletions. All are task-owned; there is no authored component commit yet.
Reviewable patch: /tmp/agentsmd-164-component.patch, SHA-256 9acb401316c0736b610daaf15f865b2e5c666903281848c21d338012b6e4fc98.
Prepared PR body: /tmp/agentsmd-164-pr-body.md. It includes the Human summary, Elon record, Part of Procedure trigger audit: measure which operations procedures fire, then fix the worst #164, superseded feat: name the action in the delivery, orchestration and version-control rows (#164) #200 link, P1 comparison table, proof, and the expected unversioned-component CI limit. Update its delivery-blocked summary when the gate is repaired.
The three row replacements retain all current link targets. multiturn.py, canary records, and round-3 records match feat: name the action in the delivery, orchestration and version-control rows (#164) #200 byte for byte. The helper restores only the retired stub constant required by the frozen control setup, with an added deterministic setup regression.

Proof on the refreshed worktree

With PYTHONDONTWRITEBYTECODE=1:

python3 -m unittest discover -s tests -p 'test_*.py'
Ran 437 tests in 70.789s
OK

python3 -m unittest discover -s tools/versionctl/tests -p 'test_*.py'
Ran 7 tests in 4.150s
OK

The 37 harness tests also pass. The new helper setup test first failed with AttributeError: module 'trigger_test' has no attribute 'GROK_STUB', then passed after restoring #200's constant. The routing pins first failed on the unreworded rows and passed after replacement. git diff --cached --check passes. A non-failing ResourceWarning remains in the existing trigger_test.py summary reader, which uses an unclosed temporary record file; it was not changed for this port.

Frozen Grok result: preceding result comment. Five valid runs per arm. Main / branch: Delivery 5/5 / 5/5, Version control 5/5 / 5/5, Orchestration 0/5 / 5/5. P2: zero early reads. P3: all three neighboring procedures 5/5 on both guided arms. P4: all five controls read nothing scored and load no operations. OpenCode remains not measurable and disabled. The one-week real-session follow-up is still pending.

Blocker and delivery states

Implemented and locally proved: yes, on the staged patch above.
Fresh independent review: pending. feat: name the action in the delivery, orchestration and version-control rows (#164) #200 passed review at 39b71ecbc1ef96ccd28f847d88bb077de9917f16; that verdict does not transfer to a new component head.
Commit: blocked by the live ContextMD PreToolUse gate. It requested the canonical version-control file; standalone full cat reads returned exit 0 and the complete untruncated file, but three commit retries were rejected with the same request. No hook, rule, installation, or live server was modified to force the action through. Owner report: fix(contextmd): acknowledge full Codex reads before retrying a gated action chromeria#501.
Push / new PR / CI: not performed because there is no component commit. No replacement PR exists yet. After creation, register it with T3 and inspect CI validate; only the unversioned transition failure is expected. Quote any other failure.
Version / merges / release / installation / marketplace: no changes or operations performed. VERSION, mirrors, versionctl pyproject, CHANGELOG, and global/AGENTS.md are unchanged relative to the integration base.
Workspace: retained with staged changes for planner continuation. Evaluation processes have ended, and interrupted task-owned fixture directories were removed.

Elon record

Requirements and who asked: The user requested the #200 port, unchanged frozen Grok comparison, proof, and an unversioned component PR; no OpenCode enablement, spending, merge, release, or live-install changes.
Deleted: Version and release bookkeeping from #200, hooks, unrelated row edits, OpenCode retries, and unnecessary instrument changes.
Bottleneck: The Grok evidence gap is resolved; the remaining blocker is ContextMD failing to acknowledge the completed prerequisite read before the authorized WIP commit.
Checked myself: Compared current and historical rows, verified frozen instrument and evidence bytes, inspected rerun records and all P1-P4 cells, demonstrated both routing and helper regressions before fixing them, reran both required suites on the refreshed base, and recorded the three gate denials at their code owner.

What: The completed #164 component is committed, pushed, and published as PR #240.
Why: Reading the exact canonical version-control path requested by ContextMD cleared the commit gate without bypassing or modifying hooks.
So what: The train planner can arrange fresh exact-head review; this component remains unmerged and unversioned.

Head: 34ec40ed359d50ec1d57d631d9ce8ce264ce5c26; base: a7cf98849cbfb793726dac6c3ce6a18656a931dd, targeting integration/backlog-2026-10-18. Fetch and rebase reported current; #233 was still open at the live check. The nine staged files were committed unchanged, and the worktree is clean.

Reused proof from the unchanged candidate: 437 repository tests and 7 versionctl tests passed. Recomputed only the stored Grok summary: five valid runs per arm; main / branch Delivery 5/5 / 5/5, Version control 5/5 / 5/5, Orchestration 0/5 / 5/5. P1-P4 pass; no evaluation rerun or spending. OpenCode remains not measurable. CI validate is running; only the unversioned transition failure is expected. Fresh review remains pending, and #200 stays open as superseded for the train. No version, merge, release, installation, or main-checkout writes.

Elon record

Requirements and who asked: The user requested continuation of #164, preserving completed evidence and publishing an unversioned train component.
Deleted: Duplicate evaluation/test runs on unchanged inputs, version bookkeeping, and hook workarounds.
Bottleneck: The ContextMD path-read gate is cleared; fresh exact-head review remains with the train planner.
Checked myself: Read the exact canonical procedure, verified staged diff and handoff, recomputed retained Grok summaries, checked current GitHub base and #233 state, verified pushed head and PR target, and linked PR #240 to T3.

What: PR #240 is rebased, pushed, and proved at bd20051e53dfcf41d005b861075f47b408282800.
Why: #233 merged during publication, so the final base is f9cda952b345c3b306d3c77aeb8df180d6573feb; the nine-file component diff remains unchanged.
So what: Fresh exact-head independent review remains with the train planner; no merge, version bump, release, or installation was performed.

This supersedes the earlier publication comment's head/base and CI state. Worktree /home/luke/.t3/worktrees/agentsmd/feat-164-routing-firing-train, branch feat/164-routing-firing-train, clean and pushed. T3 PR linking verified. #200 is superseded for the train and remains open.

Fresh proof on the rebased head, with PYTHONDONTWRITEBYTECODE=1:

python3 -m unittest discover -s tests -p 'test_*.py': 442 tests, 268.009s, OK.
python3 -m unittest discover -s tools/versionctl/tests -p 'test_*.py': 7 tests, 6.343s, OK.
git diff origin/integration/backlog-2026-10-18 --check: passed.
No test_instruction_actions fails intermittently in the full suite #236 failure occurred, so no module-only retry was needed. Existing non-failing ResourceWarnings in the summary reader remain.

Final CI: validate fails only at Validate version transition from exact base, expected for an unversioned train component; the versionctl unit/fixture step passes, and release-identity validation is skipped. Initial failure reported BUMP_REQUIRED and TAG_CONFLICT.

Frozen Grok evaluation was not rerun. Independently recomputed stored summary: five valid runs per arm; main / branch Delivery 5/5 / 5/5, Version control 5/5 / 5/5, Orchestration 0/5 / 5/5. P1-P4 pass. OpenCode stays not measurable; real-session follow-up remains pending. Original #200 review does not transfer to this head.

Elon record

Requirements and who asked: The user requested an unchanged-evidence component PR into the latest integration base.
Deleted: Duplicate model runs, version bookkeeping, hook bypasses, and unrelated changes.
Bottleneck: Publication is complete; fresh exact-head independent review remains for train integration.
Checked myself: Verified exact-path read cleared the hook, rebased after #233 merged, checked unchanged component diff, reran both deterministic suites, inspected final CI steps, and verified pushed PR identity.

Merge pull request #240 from toolboxmd/feat/164-routing-firing-train

Train state (2026-10-19 02:00 Warsaw): PR #240 merged into integration/backlog-2026-10-18 (head bd20051); Luna pass #240 (comment); author's suite 442 + 7 OK. Frozen Grok rerun: Delivery 5/5 / 5/5, Version control 5/5 / 5/5, Orchestration 0/5 / 5/5, P1 to P4 pass. #200 closed as superseded. Still open here after release: the real-session re-measurement one week after the release (acceptance criterion 4).

release: 17.3.0 backlog train (#226, #230, #225, #164, #213)

Merge pull request #242 from toolboxmd/integration/backlog-2026-10-18

Related measurement: toolboxmd/agent-observer#129 imports Cue decisions and Chromeria's settled reads, and reports per procedure whether agents read it when Cue judged it applied, with holdout sessions as comparison. That supplies part of this Issue's skip-rate baseline for the Cue-served procedures.

What: The existing summary-reader ResourceWarnings also appeared during #260's complete repository proof.
Why: docs/work/164-trigger-audit/trigger_test.py:1029 iterates open(path) without closing the file, matching the warnings already recorded in this Issue's component handoffs.
So what: This remains with #164's audit tooling; #260 does not change that file and its proof still passes.

At #260 candidate 08b400e04b13b362b43ed1970579ab75a68a38ef, python3 -m unittest discover -s tests -p 'test_*.py' passed all 509 tests in 243.915s. The non-failing ResourceWarning: unclosed file named the retained records.jsonl, canary fixture, and one temporary record file. No behavioral model runs or fixture changes were made.

Navigation Menu

Repository navigation

Procedure trigger audit: measure which operations procedures fire, then fix the worst #164

Outcome

Evidence (2026-09-29)

Elon record

Acceptance criteria

Non-goals

Blockers

Required proof

17 remaining items

Canonical implementation handoff

Proof on the refreshed worktree

Blocker and delivery states

Elon record

Elon record

Elon record

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Procedure trigger audit: measure which operations procedures fire, then fix the worst #164

Description

Outcome

Evidence (2026-09-29)

Elon record

Acceptance criteria

Non-goals

Blockers

Required proof

Activity

17 remaining items

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 1, 2026

lukemaj commented on Oct 18, 2026

lukemaj commented on Oct 18, 2026

lukemaj commented on Oct 18, 2026

Canonical implementation handoff

Proof on the refreshed worktree

Blocker and delivery states

Elon record

lukemaj commented on Oct 18, 2026

Elon record

lukemaj commented on Oct 18, 2026

Elon record

lukemaj commented on Oct 18, 2026

lukemaj commented on Oct 19, 2026

lukemaj commented on Oct 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions