From 43b3d6135ba041470458092deebbd43fb7ee8728 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:15:50 -0500 Subject: [PATCH 1/2] fix(harness-config): stop endorsing a delegation throttle on an opus-4-8 audit target Row I8's delegation-throttle instance treated a throttle as the recommended shape on both opus-5 and opus-4-8 targets. The Opus 4.8 guide's "Controlling subagent spawning" steers toward more spawning, not a cap, so opus-4-8 is now unattested: no finding and no endorsement. opus-5 keeps the inert arm. Both Opus sections get pointers with an as-of date and a recheck trigger. New eval case ai-throttle-target-scope (deterministic graders), opus, --ablation none, 3 runs per pass, all VALID: before 0.67, after 1.00, negative control (worked instance deleted) 0.33. Refs: #11823, #12106 Co-Authored-By: Claude Opus 5.5 --- ...11823-audit-instructions-drift-7c3e91a4.md | 7 ++++++ .../evals/ai-throttle-target-scope/case.yaml | 18 ++++++++++++++ .../evals/ai-throttle-target-scope/prompt.md | 23 ++++++++++++++++++ .../samples/fable-flag.json | 20 ++++++++++++++++ .../samples/opus-4-8-neither.json | 24 +++++++++++++++++++ .../samples/opus-5-endorse.json | 20 ++++++++++++++++ .../audit-instructions/reference/criteria.md | 23 +++++++++++------- 7 files changed, 127 insertions(+), 8 deletions(-) create mode 100644 .changes/harness-config/test-11823-audit-instructions-drift-7c3e91a4.md create mode 100644 plugins/harness-config/evals/ai-throttle-target-scope/case.yaml create mode 100644 plugins/harness-config/evals/ai-throttle-target-scope/prompt.md create mode 100644 plugins/harness-config/evals/ai-throttle-target-scope/samples/fable-flag.json create mode 100644 plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-4-8-neither.json create mode 100644 plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-5-endorse.json diff --git a/.changes/harness-config/test-11823-audit-instructions-drift-7c3e91a4.md b/.changes/harness-config/test-11823-audit-instructions-drift-7c3e91a4.md new file mode 100644 index 00000000000..c147eee60f5 --- /dev/null +++ b/.changes/harness-config/test-11823-audit-instructions-drift-7c3e91a4.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- `audit-instructions`: on an `opus-4-8` target, row I8's delegation-throttle instance no longer endorses a throttle as the recommended shape; only `opus-5` keeps that, with pointers to both Opus guides' "Controlling subagent spawning" sections. Eval case `ai-throttle-target-scope` covers it. diff --git a/plugins/harness-config/evals/ai-throttle-target-scope/case.yaml b/plugins/harness-config/evals/ai-throttle-target-scope/case.yaml new file mode 100644 index 00000000000..3bc79d498a7 --- /dev/null +++ b/plugins/harness-config/evals/ai-throttle-target-scope/case.yaml @@ -0,0 +1,18 @@ +schema_version: "1.1" +name: ai-throttle-target-scope +graders: + - name: fable-flag + type: regex + pattern: '^[\s>*`_-]*fable-5[*`_\s]*:[*`_\s]*flag\b' + flags: 'im' + target: 'last_message' + - name: opus-5-endorse + type: regex + pattern: '^[\s>*`_-]*opus-5[*`_\s]*:[*`_\s]*endorse\b' + flags: 'im' + target: 'last_message' + - name: opus-4-8-neither + type: regex + pattern: '^[\s>*`_-]*opus-4-8[*`_\s]*:[*`_\s]*neither\b' + flags: 'im' + target: 'last_message' diff --git a/plugins/harness-config/evals/ai-throttle-target-scope/prompt.md b/plugins/harness-config/evals/ai-throttle-target-scope/prompt.md new file mode 100644 index 00000000000..1337be1b57f --- /dev/null +++ b/plugins/harness-config/evals/ai-throttle-target-scope/prompt.md @@ -0,0 +1,23 @@ +--- +description: "Medium. criteria.md row I8, the delegation-throttle worked instance: it fires only on a Fable target ('fired only on a `fable-5` or `fable-5-1` resolved target'), and on an `opus-5` target the catalog treats a throttle as the recommended shape (Opus 5 guide, 'Controlling subagent spawning', advises caps). An `opus-4-8` target gets neither: the Opus 4.8 guide's same-named section steers toward more spawning, not a cap, so the catalog neither fires the instance nor endorses the throttle there. Read-only; no audit runs. Source: criteria.md I8 base row Detect and Source bullets." +tags: [audit-instructions, outcome, model-scope] +runs: 3 +max_turns: 12 +timeout_seconds: 600 +allowed_tools: [Skill, Read, Glob, Grep] +expected_outcome: "The reply ends with fable-5: flag, opus-5: endorse, opus-4-8: neither." +--- + +Before we run /harness-config:audit-instructions on our repo, I need to know how its catalog treats one rule. Our `.claude/skills/fanout/SKILL.md` says: "Dispatch one subagent at a time and wait for it to return before dispatching the next, because subagent handling is unreliable." + +Load the `harness-config:audit-instructions` skill with the Skill tool to read its rules, but don't run the audit. Read the skill's catalog (its `reference/criteria.md`, row I8, the delegation-throttle worked instance) and tell me, for each target model, which of these the catalog does with that rule: + +- flag: reports it as a prior-model workaround +- endorse: treats the throttle as the recommended shape +- neither: no finding, and no endorsement either + +End your reply with exactly these three lines, each filled in with one of the three words: + +fable-5: +opus-5: +opus-4-8: diff --git a/plugins/harness-config/evals/ai-throttle-target-scope/samples/fable-flag.json b/plugins/harness-config/evals/ai-throttle-target-scope/samples/fable-flag.json new file mode 100644 index 00000000000..7b6a65c094f --- /dev/null +++ b/plugins/harness-config/evals/ai-throttle-target-scope/samples/fable-flag.json @@ -0,0 +1,20 @@ +{ + "pass": [ + { + "answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: neither" + }, + { + "answer": "**fable-5**: flag\n**opus-5**: endorse\n**opus-4-8**: endorse" + } + ], + "fail": [ + { + "answer": "fable-5: neither\nopus-5: neither\nopus-4-8: neither", + "why": "the Fable-scoped instance does not fire" + }, + { + "answer": "fable-5-1: flag", + "why": "names a different target" + } + ] +} diff --git a/plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-4-8-neither.json b/plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-4-8-neither.json new file mode 100644 index 00000000000..e795b664c46 --- /dev/null +++ b/plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-4-8-neither.json @@ -0,0 +1,24 @@ +{ + "pass": [ + { + "answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: neither" + }, + { + "answer": "- `fable-5`: flag\n- `opus-5`: endorse\n- `opus-4-8`: **neither**" + } + ], + "fail": [ + { + "answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: endorse", + "why": "the inert set still names opus-4-8" + }, + { + "answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: flag", + "why": "fires the Fable-only instance on an Opus target" + }, + { + "answer": "On opus-4-8 the catalog does neither flag nor endorse it.", + "why": "no answer line" + } + ] +} diff --git a/plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-5-endorse.json b/plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-5-endorse.json new file mode 100644 index 00000000000..dc37c2b7631 --- /dev/null +++ b/plugins/harness-config/evals/ai-throttle-target-scope/samples/opus-5-endorse.json @@ -0,0 +1,20 @@ +{ + "pass": [ + { + "answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: neither" + }, + { + "answer": "> `opus-5`: `endorse`" + } + ], + "fail": [ + { + "answer": "fable-5: flag\nopus-5: neither\nopus-4-8: neither", + "why": "drops the Opus 5 endorsement" + }, + { + "answer": "opus-5-5: endorse", + "why": "names a different target" + } + ] +} diff --git a/plugins/harness-config/skills/audit-instructions/reference/criteria.md b/plugins/harness-config/skills/audit-instructions/reference/criteria.md index 8184bbd1ace..d82a6c8738b 100644 --- a/plugins/harness-config/skills/audit-instructions/reference/criteria.md +++ b/plugins/harness-config/skills/audit-instructions/reference/criteria.md @@ -454,8 +454,8 @@ worked instance keeps a `fable-5` scope of its own. **Base row** ยท Unscoped. Promotion gate MET on 2026-08-08: the model-agnostic best-practices page states the claim under its all-current-models framing (section "Leverage thinking & interleaved thinking capabilities", on general instructions over prescriptive steps). **The worked instance -below keeps a `fable-5` scope of its own**, because its basis is Fable-specific and the Opus guides -disagree with it. +below keeps a `fable-5` scope of its own**, because its basis is Fable-specific and the Opus 5 guide +disagrees with it. - **Detect:** prior-model workarounds and over-prescriptive step lists: instructions enumerating behaviors a current model handles from a brief instruction, or scaffolding that pins an approach. @@ -464,10 +464,12 @@ disagree with it. concurrent workers, a one-at-a-time rule, or an instruction to block until each subagent returns before dispatching the next, where the surface's own ground for it is that subagent handling is unreliable. On a Fable target we treat a throttle resting on that premise as the generic case - with a name on it. The Fable 5 guide ("Parallel subagents") and the Opus 5 and Opus 4.8 guides - ("Controlling subagent spawning") disagree on throttling subagent dispatch, so on `opus-5` and - `opus-4-8` targets this instance is inert, not merely unattested: there we treat a throttle as - the recommended shape rather than a workaround. **A cap carrying its own + with a name on it. The Fable 5 guide ("Parallel subagents") and the Opus 5 guide ("Controlling + subagent spawning") disagree on throttling subagent dispatch, so on an `opus-5` target this + instance is inert, not merely unattested: there we treat a throttle as the recommended shape + rather than a workaround. On an `opus-4-8` target it is unattested: the instance does not fire, + and we do not endorse the throttle either, because the Opus 4.8 guide's section of that name + steers toward more spawning, not a cap. **A cap carrying its own non-model rationale is not this instance.** Reviewability of returns, rate limits, cost, or shared mutable state each justify a bound on their own terms, and that justification is the surface's to make, not this row's to override. @@ -478,8 +480,13 @@ disagree with it. - **Source:** prompting best practices, "Leverage thinking & interleaved thinking capabilities" (the gate-meeting, model-agnostic one). Convergent model guide: Fable 5, "Recommended scaffolding changes" (prior-model skills as too prescriptive). The worked instance's basis is - the same guide, "Parallel subagents"; its Opus counter-basis is the Opus 5 guide's and the Opus - 4.8 guide's "Controlling subagent spawning". Pointer for the two Fable arms: on a `fable-5` + the same guide, "Parallel subagents"; its Opus counter-basis is the Opus 5 guide's "Controlling + subagent spawning". Pointer for the inert `opus-5` arm: fetch [Opus 5 guide, "Controlling + subagent spawning"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#controlling-subagent-spawning) + live; for the unattested `opus-4-8` arm, fetch [Opus 4.8 guide, "Controlling subagent + spawning"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8#controlling-subagent-spawning) + live. As of: 2026-11-03. Recheck trigger: the Opus 5 section stops advising caps on delegation, + or the Opus 4.8 section starts advising them. Pointer for the two Fable arms: on a `fable-5` target, fetch [Fable 5 guide, "Parallel subagents"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5#parallel-subagents) live; on a `fable-5-1` target, fetch [Fable 5.1 guide, "Let the lead agent keep working while subagents run"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-1#let-the-lead-agent-keep-working-while-subagents-run) live. As of: 2026-11-01. Recheck trigger: either section is renamed or stops advising against From 04a2001a0b26ffe63a6fd89dd7b1a4a783fbb5c1 Mon Sep 17 00:00:00 2001 From: Kyle Sexton <153232337+kyle-sexton@users.noreply.github.com> Date: Tue, 3 Nov 2026 14:54:11 -0500 Subject: [PATCH 2/2] fix(harness-config): drop the restated Opus 4.8 spawning rationale The I8 opus-4-8 arm keeps its decision (no finding, no endorsement) and its live pointer; the paraphrase of the upstream guide goes, per the skill-body rule against restating volatile upstream specifics. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_015te8Cqhc6ANUKyPoYQrPgN --- .../skills/audit-instructions/reference/criteria.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/plugins/harness-config/skills/audit-instructions/reference/criteria.md b/plugins/harness-config/skills/audit-instructions/reference/criteria.md index d82a6c8738b..bb7a9b1454f 100644 --- a/plugins/harness-config/skills/audit-instructions/reference/criteria.md +++ b/plugins/harness-config/skills/audit-instructions/reference/criteria.md @@ -468,8 +468,7 @@ disagrees with it. subagent spawning") disagree on throttling subagent dispatch, so on an `opus-5` target this instance is inert, not merely unattested: there we treat a throttle as the recommended shape rather than a workaround. On an `opus-4-8` target it is unattested: the instance does not fire, - and we do not endorse the throttle either, because the Opus 4.8 guide's section of that name - steers toward more spawning, not a cap. **A cap carrying its own + and we do not endorse the throttle either. **A cap carrying its own non-model rationale is not this instance.** Reviewability of returns, rate limits, cost, or shared mutable state each justify a bound on their own terms, and that justification is the surface's to make, not this row's to override.