guardrails: grade every default-on guard's precision from the per-session hook event log (ADR 0003)

[kyle-sexton]

Summary

ADR 0003 says a guard earns default-on by measured precision, and until #3750 the only instrument was a manual corpus sweep. The per-session hook event log now records every guard fire (source: "envelope", status: "blocked", subject) beside every event that did not fire, so precision can be graded from real sessions: blocked-versus-fired per guard, with the subject of each block available for a true/false-positive verdict.

The operator's original ask (2026-09-04) was to "unhobble" guards and instructions that were in place for older models and no longer apply. The pipeline is the instrument; the verdicts are not yet taken. The one guard with evidence already, block-hook-bypass (zero true positives in four fires across three sessions), is owned by the PreToolUse lane and tracked in #3721; this issue is the fleet-wide pass over the rest.

Fix

1. Turn session_event_log_enabled on in the operator's own sessions for two to four weeks (the switch stays off by default for consumers).
2. From the session files, per guard: fires, blocks, and the operator's verdict on each block's subject (true positive, false positive, scope error).
3. Apply ADR 0003's clauses: a guard at zero true positives is withdrawn from default-on scope and re-filed for rescoping (clause 4), one whose oracle is wrong is deleted, one with sound precision stays. Each disposition is its own PR with the numbers in the body.

Acceptance

• A findings table (guard, sessions observed, fires, blocks, true positives, disposition) attached to this issue.
• One PR or closing note per guard with a non-"keep" disposition.

Related

• Run the ADR 0003 corpus sweep on block-hook-bypass's remaining default-on scope #3721 (block-hook-bypass corpus sweep), guardrails dispatcher is the whole per-tool-call budget miss after the hook-performance program #3685 (guardrails dispatcher budget), feat(claude-ops): per-session hook logging pipeline, default off #3750 (the instrument).
• docs/adr/0003-verification-guards-earn-default-on-by-measured-precision.md.

[kyle-sexton]

This was generated by AI during triage.

Claimed for triage by autonomous lane instance vm-triage-202609050304. Lease expires 2026-09-06T19:40:00Z. This is a triage claim only, not an execution claim; the item stays unassigned and available to the worker lane once briefed.

---

Generated by Claude Code

[kyle-sexton]

This was generated by AI during triage.

Triage verdict

Category: Task (premise confirmed). State: briefed → human-gated. Work class: read-only. Priority: medium.

Verification (2026-09-06, against current main)

The premise holds and its prerequisites are in place:

• ADR 0003 is present in the ADR set.
• The instrument from feat(claude-ops): per-session hook logging pipeline, default off #3750 is merged: the per-session hook event log ships as a claude-ops hook with its own test suite, the session_event_log_enabled switch is threaded through the setup and observability skills, and the observability skill reads the resulting session files.
• The claude-ops README documents the switch as off by default and describes the per-event line shape the grading would read.

So the instrument exists and the verdicts genuinely are not yet taken, exactly as the issue states.

Why human-gated rather than agent-ready

Two capability blockers, neither of which a brief can remove:

1. The evidence does not exist yet and cannot be manufactured. Grading requires two to four weeks of the operator's own real sessions with the switch on. Synthetic or replayed sessions would defeat the purpose: ADR 0003's clauses turn on measured precision against real work, and a guard's false-positive rate is a property of what the operator actually does, not of a fixture corpus.
2. The verdict on each block is a human judgment. Deciding whether a given block's subject was a true positive, a false positive, or a scope error is precisely the judgment ADR 0003 reserves for the operator. An agent labeling its own guard's fires would be marking its own homework.

The dispositions the grading feeds (withdrawing a guard from default-on scope, deleting one whose oracle is wrong) are also policy changes to default-on behavior for every consumer, which is maintainer territory.

What a human needs to do to start it

• Turn session_event_log_enabled on in the operator's own sessions and note the start date. The switch stays off for consumers; this is a personal-scope change only.
• Let it run the two-to-four-week window.
• Then the reduction is delegable: see below.

The delegable slice, if the operator wants it split out

Once a collection window has produced session files, the mechanical part can be filed as its own agent-ready item: a read-only reduction over the session event log that emits the findings table (guard, sessions observed, fires, blocks) with each block's subject quoted for the operator to adjudicate, leaving the true-positive column and the disposition column blank for the human. That item would be work-class: read-only and needs no repository mutation. It is deliberately not filed now, because filing it before the window opens would create an item blocked on an event that has no date.

Scope note

block-hook-bypass is explicitly out of scope here and stays tracked in #3721, which is open. This item is the fleet-wide pass over the remaining default-on guards.

Incidentally, and recorded only as an observation rather than as evidence for the grading: this triage pass was itself blocked once by block-hook-bypass while running a read-only reproduction script in a scratch directory. That is one more data point of the kind #3721 is collecting, not a verdict.

Triage claim released by vm-triage-202609050304.

---

Generated by Claude Code