Skip to content

fix(claude-config): three audit skills mandate independent verification and none states a posture for dispatch being unavailable #2279

Description

@kyle-sexton

Three claude-config audit skills mandate independent verification as their mechanism —
fresh-context, non-fork subagents that re-judge the audit's own proposals — and none of them states
what happens when that dispatch is unavailable. The degradation is left to the model's judgment, made
silently, and a run that skipped its verification phase emits a report structurally identical to one
that did not. There is also no field anywhere to record that a lane's verdicts were self-graded, and no
marketplace convention obliges a dispatch-mechanism skill to declare a degraded mode, so the gap keeps
reproducing.

This is not hypothetical. The run that produced these findings is itself the instance: its Phase C did
not execute, and the finding it emitted at warning severity was subsequently retracted twice by
follow-up passes — which is exactly the class of error Phase C exists to catch.

Rows

  • RKD-07plugins/claude-config/skills/audit-instructions/SKILL.md:289, :348-358, :368-375 — Phase B mandates "one fresh read-only subagent per surface" and Phase C mandates fresh-context non-fork verifiers; no dispatch-unavailable path exists, and the Phase D cost line does not enumerate phases that did not run.
  • CC-F2plugins/claude-config/skills/audit-prompting-postures/SKILL.md:74-76, :151-153 — Phase D mandates a verifier per "surface batch" (a term never defined) and the mandated report tail requires a coverage line and a Sources line but no verifier attestation, so a run that silently skips Phase D produces a structurally identical report.
  • F11plugins/claude-config/skills/audit-pass/SKILL.md:299-302 and reference/report-location-and-schema.md:112-119 — Phase 5's apply-verify delegates to a cross-vendor advisor "with a fresh-context (non-fork) subagent as the stated fallback", and the fallback has no fallback; the report schema's section table carries no per-lane verification-mode field.
  • RKD-08docs/conventions/ (verified absence) — no convention obliges a skill whose documented mechanism is subagent dispatch to state a posture for dispatch being unavailable.

Evidence

RKD-07 — the mechanism is mandated; the fallback is absent

plugins/claude-config/skills/audit-instructions/SKILL.md:289:

Run one **fresh read-only subagent per surface**, each sharing

SKILL.md:348-349:

Every removal or rewrite proposal is re-judged before it reaches the report. Dispatch **fresh-context,
non-fork** subagents — this is a self-grade of the audit's own proposals, so a fork that inherits the

SKILL.md:358: "A proposal the verifier defends is demoted to info or dropped, never surfaced as a
confident removal."

grep -c "cannot dispatch\|dispatch unavailable\|unavailable\|degrade" over the whole SKILL.md0.
The only dispatch-adjacent guard is a ceiling, not a floor — SKILL.md:317: "Before the total
dispatch count (lanes plus Phase C verifiers) would exceed ~20, confirm with the user."

Phase D's report contract (SKILL.md:368-375) enumerates exactly what the cost line must carry —
checks per surface, model-scoped rows skipped, per-surface token delta, and a zero-new-gates
confirmation — and does not include phases that did not run.

File this as the contract gap, not as a hidden degradation. The originating item says the skill
"says so only if the operator reads carefully". The evidence packet shows the producing run actually
did better than that: its report carried a heading "Two departures from the documented phase structure,
both material", stated "Phase B ran inline, not as one fresh read-only subagent per surface" and "Phase
C did not run… Every removal or rewrite proposal below is therefore marked unverified", and every
finding carried a literal (unverified) token. So the disclosure the item asks for was produced. The
defect is that nothing in the skill required it
— a different model, or the same model on a worse day,
discloses nothing and the report reads identical to a fully-verified one.

One hardening belongs here rather than as a separate ticket: the run that produced this degradation did
so because a session instruction barred the Agent tool and invoking the skill was not read as
requesting its documented mechanism. That instruction is machine-local and out of this repo's scope
(the operator has since amended it), but a skill may legitimately state that invocation constitutes a
request for its documented dispatch.

CC-F2 — a mandated verifier with no attestation

plugins/claude-config/skills/audit-prompting-postures/SKILL.md:74-76:

Dispatch one fresh-context, non-fork verifier per surface batch, prompted to refute each proposed
addition: "argue this component's purpose does not need this posture, or that it already carries
it." Findings a verifier refutes are dropped or demoted to `info`.

The mandated report tail, SKILL.md:151-153:

Verdicts: `MISSING` (finding, with proposed addition as a fenced diff), `PRESENT` (where it is),
`NOT-APPLICABLE` (with the failed predicate). End with a coverage line — components inventoried,
classified, unclassified — and a Sources line citing the pages fetched this run with dates.

A coverage line and a Sources line are owed; a verifier line is not. "Per surface batch" is nowhere
defined, so one verifier for everything and one verifier per component are both compliant. The skill
audits other components for grounding postures and applies neither to itself.

(One evidentiary leg is not re-verifiable from this repository: the producing run's persisted report
allegedly carried three ungrounded specifics, one manufactured by the Phase D verifier. That report is
machine state under ${CLAUDE_PLUGIN_DATA} and may have been overwritten. The structural half above
stands on the repo alone.)

F11 — the fallback has no fallback, and nothing records that it fired

plugins/claude-config/skills/audit-pass/SKILL.md:299-302:

The apply-verify step judges work this same run produced, so it is delegated: hand the applied diff
and the finding it claims to resolve — the artifact, not this run's reasoning — to a cross-vendor
advisor when one is installed and set up (the OpenAI Codex plugin, say, invoked per its own docs),
with a **fresh-context (non-fork) subagent** as the stated fallback.

reference/report-location-and-schema.md:112-119 — the schema's section table, read in full, carries
inventory, mechanical, behavioral, suppressed, delegated, skipped. There is no per-lane
verification-mode field, so findings.json cannot express "this lane's verdicts were self-graded".
Note the irony the table itself supplies at :119: skipped exists precisely because "a silent
exclusion reads as coverage" — the same argument applies to a silently-skipped verification.

Suggested shape (from the originating item; the validation pass concurs): a per-lane
verification: independent | inline-self-graded | refused on the partial and on findings.json, and
adopt the concurrent run's own improvisation — refuse and mark unverified — as the stated default
when neither an advisor nor a fresh-context subagent is available.

RKD-08 — no convention requires any of this

docs/conventions/ has no dispatch-posture entry. grep -rln "dispatch" docs/conventions/ hits
ecosystem-commands, hook-budget, hook-telemetry, loop-lane, and topic-docs — none states a
degraded-mode obligation. #1426 records that 22 skills carry a DISPATCH-DEFAULT verdict, so the
affected population is large and the gap is systemic rather than per-skill. This row may reasonably
land as an added acceptance criterion on #1426 rather than as a new convention; that is a maintainer's
call.

Adjacent, deliberately not duplicated

Provenance

Severity: MED · Provenance: mixed, stated per row. RKD-07 and RKD-08 are SELF_REPORTED — the originating item declares itself the evidence packet rather than the ticket, directs that /plugin-quality:audit be run first, and that pass never ran; RKD-07 is self-reported in the strongest sense, since the run that produced it is the run whose Phase C did not execute. CC-F2 and F11 are AUDITOR_VERIFIED (fresh-context plugin-quality:auditor passes). Every file:line and quoted line above was independently re-verified against HEAD by the validation pass.
Origin: handoff-inbox items 2026-08-10-claude-config-report-keying-and-dispatch (RKD-07, RKD-08), 20260811-021645-plugin-audit-four-components-and-guard-deadlock-ownership (CC-F2), 20260811-020411-claude-config-audit-pass-report-path-inside-scan-set (F11)
Ledgers: .work/handoff-inbox-batch-4/ledgers/I6-report-keying-dispatch.md § RKD-07, RKD-08 · .../I9-021645-four-components.md § CC-F2 · .../I8-audit-pass-report-path.md § F11
Verified against repo HEAD 9b34a82a. Line anchors re-derived at that commit — the postures and audit-pass anchors both moved under #2259 and #2234.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions