Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
bump: patch
---

### Changed

- `audit-instructions`: on an `opus-4-8` target, row I8's delegation-throttle instance no longer endorses a throttle as the recommended shape; only `opus-5` keeps that, with pointers to both Opus guides' "Controlling subagent spawning" sections. Eval case `ai-throttle-target-scope` covers it.
18 changes: 18 additions & 0 deletions plugins/harness-config/evals/ai-throttle-target-scope/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
schema_version: "1.1"
name: ai-throttle-target-scope
graders:
- name: fable-flag
type: regex
pattern: '^[\s>*`_-]*fable-5[*`_\s]*:[*`_\s]*flag\b'
flags: 'im'
target: 'last_message'
- name: opus-5-endorse
type: regex
pattern: '^[\s>*`_-]*opus-5[*`_\s]*:[*`_\s]*endorse\b'
flags: 'im'
target: 'last_message'
- name: opus-4-8-neither
type: regex
pattern: '^[\s>*`_-]*opus-4-8[*`_\s]*:[*`_\s]*neither\b'
flags: 'im'
target: 'last_message'
23 changes: 23 additions & 0 deletions plugins/harness-config/evals/ai-throttle-target-scope/prompt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
description: "Medium. criteria.md row I8, the delegation-throttle worked instance: it fires only on a Fable target ('fired only on a `fable-5` or `fable-5-1` resolved target'), and on an `opus-5` target the catalog treats a throttle as the recommended shape (Opus 5 guide, 'Controlling subagent spawning', advises caps). An `opus-4-8` target gets neither: the Opus 4.8 guide's same-named section steers toward more spawning, not a cap, so the catalog neither fires the instance nor endorses the throttle there. Read-only; no audit runs. Source: criteria.md I8 base row Detect and Source bullets."
tags: [audit-instructions, outcome, model-scope]
runs: 3
max_turns: 12
timeout_seconds: 600
allowed_tools: [Skill, Read, Glob, Grep]
expected_outcome: "The reply ends with fable-5: flag, opus-5: endorse, opus-4-8: neither."
---

Before we run /harness-config:audit-instructions on our repo, I need to know how its catalog treats one rule. Our `.claude/skills/fanout/SKILL.md` says: "Dispatch one subagent at a time and wait for it to return before dispatching the next, because subagent handling is unreliable."

Load the `harness-config:audit-instructions` skill with the Skill tool to read its rules, but don't run the audit. Read the skill's catalog (its `reference/criteria.md`, row I8, the delegation-throttle worked instance) and tell me, for each target model, which of these the catalog does with that rule:

- flag: reports it as a prior-model workaround
- endorse: treats the throttle as the recommended shape
- neither: no finding, and no endorsement either

End your reply with exactly these three lines, each filled in with one of the three words:

fable-5: <flag|endorse|neither>
opus-5: <flag|endorse|neither>
opus-4-8: <flag|endorse|neither>
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"pass": [
{
"answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: neither"
},
{
"answer": "**fable-5**: flag\n**opus-5**: endorse\n**opus-4-8**: endorse"
}
],
"fail": [
{
"answer": "fable-5: neither\nopus-5: neither\nopus-4-8: neither",
"why": "the Fable-scoped instance does not fire"
},
{
"answer": "fable-5-1: flag",
"why": "names a different target"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
{
"pass": [
{
"answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: neither"
},
{
"answer": "- `fable-5`: flag\n- `opus-5`: endorse\n- `opus-4-8`: **neither**"
}
],
"fail": [
{
"answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: endorse",
"why": "the inert set still names opus-4-8"
},
{
"answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: flag",
"why": "fires the Fable-only instance on an Opus target"
},
{
"answer": "On opus-4-8 the catalog does neither flag nor endorse it.",
"why": "no answer line"
}
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"pass": [
{
"answer": "fable-5: flag\nopus-5: endorse\nopus-4-8: neither"
},
{
"answer": "> `opus-5`: `endorse`"
}
],
"fail": [
{
"answer": "fable-5: flag\nopus-5: neither\nopus-4-8: neither",
"why": "drops the Opus 5 endorsement"
},
{
"answer": "opus-5-5: endorse",
"why": "names a different target"
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -454,8 +454,8 @@ worked instance keeps a `fable-5` scope of its own.
**Base row** · Unscoped. Promotion gate MET on 2026-08-08: the model-agnostic best-practices page
states the claim under its all-current-models framing (section "Leverage thinking & interleaved
thinking capabilities", on general instructions over prescriptive steps). **The worked instance
below keeps a `fable-5` scope of its own**, because its basis is Fable-specific and the Opus guides
disagree with it.
below keeps a `fable-5` scope of its own**, because its basis is Fable-specific and the Opus 5 guide
disagrees with it.

- **Detect:** prior-model workarounds and over-prescriptive step lists: instructions enumerating
behaviors a current model handles from a brief instruction, or scaffolding that pins an approach.
Expand All @@ -464,10 +464,11 @@ disagree with it.
concurrent workers, a one-at-a-time rule, or an instruction to block until each subagent returns
before dispatching the next, where the surface's own ground for it is that subagent handling is
unreliable. On a Fable target we treat a throttle resting on that premise as the generic case
with a name on it. The Fable 5 guide ("Parallel subagents") and the Opus 5 and Opus 4.8 guides
("Controlling subagent spawning") disagree on throttling subagent dispatch, so on `opus-5` and
`opus-4-8` targets this instance is inert, not merely unattested: there we treat a throttle as
the recommended shape rather than a workaround. **A cap carrying its own
with a name on it. The Fable 5 guide ("Parallel subagents") and the Opus 5 guide ("Controlling
subagent spawning") disagree on throttling subagent dispatch, so on an `opus-5` target this
instance is inert, not merely unattested: there we treat a throttle as the recommended shape
rather than a workaround. On an `opus-4-8` target it is unattested: the instance does not fire,
and we do not endorse the throttle either. **A cap carrying its own
non-model rationale is not this instance.** Reviewability of returns, rate limits, cost, or
shared mutable state each justify a bound on their own terms, and that justification is the
surface's to make, not this row's to override.
Expand All @@ -478,8 +479,13 @@ disagree with it.
- **Source:** prompting best practices, "Leverage thinking & interleaved thinking capabilities"
(the gate-meeting, model-agnostic one). Convergent model guide: Fable 5, "Recommended
scaffolding changes" (prior-model skills as too prescriptive). The worked instance's basis is
the same guide, "Parallel subagents"; its Opus counter-basis is the Opus 5 guide's and the Opus
4.8 guide's "Controlling subagent spawning". Pointer for the two Fable arms: on a `fable-5`
the same guide, "Parallel subagents"; its Opus counter-basis is the Opus 5 guide's "Controlling
subagent spawning". Pointer for the inert `opus-5` arm: fetch [Opus 5 guide, "Controlling
subagent spawning"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5#controlling-subagent-spawning)
live; for the unattested `opus-4-8` arm, fetch [Opus 4.8 guide, "Controlling subagent
spawning"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-4-8#controlling-subagent-spawning)
live. As of: 2026-11-03. Recheck trigger: the Opus 5 section stops advising caps on delegation,
or the Opus 4.8 section starts advising them. Pointer for the two Fable arms: on a `fable-5`
target, fetch [Fable 5 guide, "Parallel subagents"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5#parallel-subagents)
live; on a `fable-5-1` target, fetch [Fable 5.1 guide, "Let the lead agent keep working while subagents run"](https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5-1#let-the-lead-agent-keep-working-while-subagents-run)
live. As of: 2026-11-01. Recheck trigger: either section is renamed or stops advising against
Expand Down
Loading