discipline:sweep-all — cost gate, relevance tie-break, membership-read spec, and missing evals

[kyle-sexton]

Summary

Four smaller runbook gaps in plugins/discipline/skills/sweep-all/SKILL.md, from the same
plugin-quality:audit post-use audit of discipline@0.9.0. None is a correctness hazard on its
own; together they are what makes the sweep unpredictable to run.

F5 (MED) — cost is prose, never a gate

The per-fork cost is documented under Gotchas ("~170K tokens" per fork; "an 8-in-scope pass in two
waves of four ran ~1.4M tokens for the audit phase alone") but no workflow step estimates or
confirms the spend. The user learns the price only by reading a Gotcha they may never open.
Proposal: make the pre-dispatch estimate a step, with a confirmation threshold.

F8 (MED) — the relevance gate has no tie-break

Situational membership is routed "from the corrector's own listing description", with no rule for
the ambiguous case. In the observed run it admitted 7 of 7 situational correctors. This is
structural, not a one-off: situational triggers are deliberately broad, so any long mixed session
admits all of them — exactly where per-fork cost peaks.

Proposal is deliberately not a rubric. Add a tie-break — "when ambiguous, exclude and report" —
and accept the trade explicitly: today's admit-by-default bias is the safe one for a correctness
skill, so the change buys predictable cost at the price of a possible missed discipline, and the
skip is reported either way (the skill already requires that).

F9 (MED) — the membership read is under-specified

"Glob the sibling corrector directories and read each one's metadata.discipline-batch" leaves out
two things that both bit in the observed run:

• No parse rule. The shipped value carries a trailing YAML comment
  (discipline-batch: core # every session makes claims that need backing). That is valid YAML
  and a real parser yields core — the frontmatter is not the defect. But the runbook does not
  say to parse it as YAML or strip the comment, so the operator improvised a POSIX sed, which
  would simply fail in a PowerShell-only session. The defect belongs to the read instruction, not
  to the correctors' metadata.
• No rule for a keyless sibling. setup and sweep-all themselves carry no
  discipline-batch key, and a naive glob of skills/* returns them. The runbook never says what
  a missing key means.

F11 (LOW) — every eval assumes forks work

All six entries in skills/sweep-all/evals/evals.json presuppose a working fork fan-out. There is
no eval for the fork-unavailable path, for the membership parse, or for a keyless sibling. (The
fork-unavailable eval lands with the preflight work; the membership ones land here.)

Source: plugin-quality:audit post-use audit of discipline@0.9.0 (F5, F8, F9, F11).

Explicitly out of scope

The audit's R7 — an artifacts-only / main-thread-sequential fallback ladder — is not proposed.
It re-creates the salience dilution the skill's declared delta exists to prevent and yields audits
weaker than the ones the skill halts to avoid. Build only if the degraded mode proves insufficient
in practice.

[kyle-sexton]

This was generated by AI during triage.

Verified — all four sub-findings confirmed, with one precision correction

F5 — cost is prose, never a gate. Confirmed. The per-fork cost figures (~170K tokens per fork; ~1.4M for an eight-member pass in two waves of four) appear only under Gotchas. No step in the sweep-all workflow estimates or confirms the spend before dispatch.

F8 — the relevance gate has no tie-break. Confirmed. Situational membership is routed "from the corrector's own listing description (its trigger phrases and 'at conversation start on …' clause), not from a guess", with a reporting requirement for skips — and no rule for the ambiguous case. There are 7 situational correctors in the shipped plugin, so the observed 7-of-7 admission is the structural maximum, not an outlier.

F9 — the membership read is under-specified. Confirmed, and the item's own concession is correct. The shipped metadata does carry a trailing YAML comment (discipline-batch: core # every session makes claims that need backing) — valid YAML that any real parser resolves to core, so the frontmatter is not the defect. The runbook says only "read each one's metadata.discipline-batch" with no parse rule, which is what invited the improvised POSIX sed. The keyless-sibling half is confirmed by enumeration: of the 16 skill directories in the plugin, exactly two carry no discipline-batch key — setup and sweep-all itself — and a naive glob of the sibling directories returns both. The runbook never says what a missing key means.

F11 — missing evals. Confirmed, with one correction. The evals file holds 6 entries and none covers the fork-unavailable path, the membership parse, or a keyless sibling. But the item overstates the premise: 5 of the 6 presuppose a working fork fan-out, not all 6. session-start-digest-no-body-loads exercises the cheap session-start digest, which by construction loads no corrector body, runs no audit, and dispatches no fork. The gap is real; the framing is one entry too broad.

Classification and routing — human-gated

Priority: medium — matches the reporter's MED/MED/MED/LOW spread. Reporter severities recorded here so an implementer can sub-sort within the band: F5 MED, F8 MED, F9 MED, F11 LOW.

Routed human-gated, keyed on the strongest surviving finding rather than the weakest. Two of the four carry a genuinely open call that is not a maintainer-vetoable implementation detail:

• F8 is a posture change, and the item itself declines to recommend it. It proposes a tie-break ("when ambiguous, exclude and report") while explicitly framing the trade rather than defaulting it: "today's admit-by-default bias is the safe one for a correctness skill, so the change buys predictable cost at the price of a possible missed discipline." Choosing predictable cost over correctness-bias coverage is product intent for a correctness skill, not a reversible default someone vetoes at merge.
• F5's confirmation threshold is a preference call. Whether to estimate before dispatch is uncontroversial; what spend requires a human confirmation is a user-facing policy number with no grounded default, and it interacts with F8 (the tie-break is precisely what bounds the spend the gate would measure).

F9 and the two membership evals are decision-free and independent — the YAML parse rule and the keyless-sibling rule stand whichever way F5 and F8 are resolved, and they do not depend on the open structural decision in #1258 either. If the human taking this wants them landed ahead of the posture call, say so on this item and they can be carved out to an autonomous-eligible follow-up in one pass. They are deliberately not split out unasked, so the four findings stay resolvable as one coherent posture decision.

Related, and worth resolving in order: #1258 (needs-human, work-class: structural) holds the open decision on whether the batch dispatches forks at all, with #1621 and #1622 blocked on it. This item is adjacent rather than a cluster member — F8 and F9 are dispatch-independent — but F5's cost model is priced in forks, so a resolution of #1258 that drops them changes the numbers a cost gate would show.

Scope note honored: the audit's R7 fallback ladder (artifacts-only / main-thread-sequential) is recorded as out of scope by the reporter and is not being re-opened here.

Role-label warning: this repository's .work-item-tracker.json has a config block with no role_labels entry, so the human-gated role resolves to the plugin default (needs-human). The label exists live in this repo, but the resolution is a default, not a declared binding.

[kyle-sexton]

Adding one item to this issue's scope, surfaced by review on #1628 and deliberately not fixed there.

The per-session subagent budget is not checked before dispatch. sweep-all now models the
concurrent limit (CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS, default 20) and knows that every
Agent-tool subagent counts toward CLAUDE_CODE_MAX_SUBAGENTS_PER_SESSION (default 200,
https://code.claude.com/docs/en/sub-agents). But the lifetime quota is not a capacity race —
waiting does not restore it, and splitting into waves does not either.

So there is a reachable state the runbook does not handle: the session has nearly exhausted its
per-session allowance (or the cap is configured low), the preflight canary spawns and passes, and
then only some of the resolved members can spawn. The pass then corrects from a partial sweep,
while the skill says a relevant corrector is never dropped.

A fix should check that the remaining per-session budget covers the canary plus the entire resolved
member set BEFORE dispatching the canary, and degrade — or tell the user a fresh session is needed —
rather than discovering the shortfall mid-fan-out.

Not fixed in #1628: that PR's scope is the inert stop-rule, the SSOT contradiction, the unenforceable
no-writes claim, and the concurrency numbers from the original audit. This is a distinct
budget-exhaustion path found while reviewing them, and it belongs with the other runbook gaps here.

[kyle-sexton]

Frontier disposition (drain session)

Ruling: Close as not planned for this drain wave.

Four sweep-all runbook gaps (cost gate, relevance tie-break, membership-read spec, missing evals) are audit friction, not correctness hazards. Structural sweep-all vocabulary and digest spec work lives on carriers #1631 and #1666; this issue is redundant with those.

Re-open trigger: #1631 and #1666 both close without addressing the four named gaps.