Skip to content

fix(claude-config): re-arm the thinking-disable row where the model forbids it - #1917

Merged
kyle-sexton merged 3 commits into
mainfrom
docs/roster-r11-thinking
Aug 4, 2026
Merged

fix(claude-config): re-arm the thinking-disable row where the model forbids it#1917
kyle-sexton merged 3 commits into
mainfrom
docs/roster-r11-thinking

Conversation

@kyle-sexton

Copy link
Copy Markdown
Contributor

Summary

Doc-alignment roster row 11: Thinking (live page byte-identical to the archived 917-line slice — MD5 confirmed by producer and verifier independently).

Headline: the roster recorded IA-4's three harness rules as unshipped; they were in fact already covered by the I17 family — and that stale status was hiding a live defect inside the coverage. criteria 1.12.0 / claude-config 0.21.4:

  • I17 base row amended (the defect): the live thinking page states, with no effort qualifier, that Fable 5 / Mythos 5 / Mythos Preview reject thinking: {type: "disabled"} outright. The old row's remediation offered "lower the effort to high or below" — a branch that still 400s on that family — and its unscoped fence excused the unconditional-reject case. Now two Detect arms with per-arm fences and remediations; heading retitled to "…where the model forbids it"; the second arm is API-form-only (on Fable 5 the harness surfaces are documented silent no-ops and stay I17-a's; for the Mythos models the harness pages state nothing, recorded as silence — no claim either direction).
  • I17-b extended to the thinking half of mid-session churn (the page renders both dials into the prompt; switching modes or budgets starts a new cache prefix), with a deliberate, honest carve-out: a Claude Code surface prescribing a mid-session thinking toggle is out of reach — neither excused nor flagged — because harness prompt-caching docs name only model and effort as out-of-prompt cache-key settings and document a confirm dialog for effort alone; a non-vacuous re-scope trigger is recorded. Plus an I17-b/I17-c co-firing note in the house pattern.
  • IA-4 rule 2 already fully covered by I17-a (untouched); IA-5's gate (PA-AE) was answered 2026-08-02 — rules 1–3 shipped long since as I18, rules 4–8 stay deferred on recorded grounds (no instruction surface to run on; rule 8 is the same deferral row 6 recorded), nothing zero-instance shipped.

Test plan

  • Docs-only; markdownlint 0 errors; changelog parity all three modes; instruction-scan.test.sh 46/46; conflict-scan.test.sh 41/41; manifest validation pass.
  • Producer: scripted quote fidelity (9 quotes vs seven live sources, plus a negative assertion that the over-broad phrasing is gone); its own independent reviewer caught two gaps pre-handoff (I17-b/I17-c overlap note; a reach ambiguity), both fixed.
  • Orchestrator-commissioned Fable verifier: live fetches of the thinking and harness prompt-caching pages, confirmation of the old row's double defect (both readings wrong), per-arm scenario construction (including the case the old row passed as compliant), carve-out honesty, coverage spot-checks — 6/6 PASS; its two minor defects (stale instance count; Mythos over-generalization) fixed and re-checked, with the final one-line residual (a changelog echo) fixed before push.

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_019gaVX25Txd6GXdiu9HEH3X

…isable outright

The Thinking page states two restrictions in adjacent paragraphs. I17's base row
carried only the first: a thinking-disable surface paired with `xhigh` or `max`
effort on Opus 5 and later. The second has no effort qualifier at all — "Claude
Fable 5, Claude Mythos 5, and Claude Mythos Preview reject `thinking: {type:
"disabled"}`: thinking cannot be turned off on these models."

The gap was a wrong remediation, not only a missed case, so this amends the base
row rather than adding a sibling. Read as covering Fable 5, the old row fired and
handed out "lower the effort to `high` or below, or leave thinking on" — advice
whose first branch still returns a 400 there. Read as excluding it, the reject went
undetected and the row's own fence ("a thinking-disable surface named with no
effort level in reach of it") excused it. Both are now scoped to the arm that
earns them. Only the API form joins the second arm; on that family the harness
disable surfaces are silent no-ops, which is I17-a's failure and stays there.

I17-b extends from effort churn to thinking churn on the same page's cache-prefix
rule. Its harness carve-out is re-scoped to effort: the confirmation dialog is
documented for effort alone, and `code.claude.com/docs/en/prompt-caching` names
exactly two settings outside the prompt text that are still part of the cache key
— model and effort level. Left unscoped, the extended row would have asserted that
the harness warns before a thinking toggle, which nothing upstream says.

Local coverage measured rather than asserted: zero operative instances of either
arm, every occurrence being a document about the restriction.

criteria 1.11.0 -> 1.12.0; claude-config 0.21.3 -> 0.21.4.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Aug 4, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@claude

claude Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 37s —— View job


Review: fix(claude-config): re-arm the thinking-disable row where the model forbids it

Scope note: this repo has a dedicated claude-security-review.yml workflow, so per REVIEW.md this lane owns everything except security (no security findings in scope here; N/A anyway for a docs-only change).

What I checked

This PR is docs-only (criteria.md, CHANGELOG.md, plugin.json). I read the full diff against origin/main, the surrounding I17 family in criteria.md for context, and cross-checked the changelog's factual claims against the repo:

  • Base-row split into two arms (L670–L745) is internally consistent: the Must NOT flag fence ("no effort level in reach") is now correctly scoped to the Opus 5 arm only, and the second arm's Remediate correctly drops the "lower effort" branch that the changelog identifies as the actual defect (a remediation that still 400s). This matches the stated bug precisely — the fence previously excused the unconditional-reject case, and the fix scopes it rather than widening it.
  • I17-b's reach split (L770–L816) is a genuine three-way distinction (excused / out-of-reach / flagged) rather than a binary, and it's argued rather than asserted — the "out of reach, neither excused nor flagged" carve-out for a Claude Code mid-session thinking toggle is justified by what the cited prompt-caching source does and doesn't state, consistent with the doc's existing does-not-state discipline used elsewhere (e.g. the Mythos harness silence in the base row).
  • Numeric coverage claims verified against the repo, not just trusted:
    • The "six occurrences of the disable literal across four files" claim (L736–L741) — grepped and got exactly 3 in criteria.md, 1 in opus-5.md, 2 across the two CHANGELOG.md files. Matches exactly.
    • The I17-b thinking-half coverage claim's file set (catalog + two delta chapters + changelog) also checks out against budget_tokens/session-toggle occurrences in the repo.
  • No stale cross-references: grepped for the old row title ("at an effort level that forbids it") — only remaining occurrence is the changelog's deliberate before/after quote. No TOC/index exists that would need updating, and the two-arm/model-scope-annotation rationale added at the I17 header (L662–L666) correctly generalizes the existing "Detect condition, not Model scope" argument to two disjoint model families.
  • Version bumps consistent with plugin convention: criteria.md 1.11.0 → 1.12.0 and plugin.json 0.21.3 → 0.21.4 are both patch-level bumps for Changed/content-only work, matching the pattern used for the prior I8-e addition in [0.21.3]. last-updated: 2026-08-04 matches the Verified stamps added to the two amended rows.
  • I17-a/I17-c correctly left untouched — the changelog's claim that only I17 base and I17-b were re-verified live, with I17-a/I17-c keeping their prior verification stamps, matches the diff.

Findings

None. This is a well-scoped, internally consistent fix: it corrects a genuine defect (a remediation branch that still fails) rather than just adding coverage, the new arm boundaries and fences are precisely drawn, and the changelog's specific factual claims (occurrence counts, file sets) verify against the actual repository state rather than being asserted. No test-script coupling to the changed row IDs, so no test-coverage gap from this doc-only change.

@claude

claude Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 39s —— View job


Security review complete

  • Gather context (diff, changed files)
  • Review plugin.json version bump
  • Review CHANGELOG.md additions
  • Review criteria.md content changes
  • Check for injection / secrets / access-control / GitHub Actions risk
  • Post findings

No security issues found.

This PR touches exactly three files, all documentation/config metadata for the claude-config plugin:

  • plugins/claude-config/.claude-plugin/plugin.json — a one-line semver bump (0.21.30.21.4).
  • plugins/claude-config/CHANGELOG.md — prose changelog entries describing the doc changes below.
  • plugins/claude-config/skills/audit-instructions/reference/criteria.md — prose amendments to the I17 / I17-b audit-criteria rows (reference documentation consumed by an audit skill, not executable code).

None of these introduce or modify: scripts, hooks, GitHub Actions workflows or their permissions:/triggers, MCP server configs, command execution, path handling, credentials/secrets, or any data-egress surface. There's no code path here that takes untrusted input — it's markdown prose describing detection rules for a later, separate audit process. Nothing in this lane (logic flaws, authz gaps, injection, secret handling, dangerous Actions patterns) applies to a docs-only content change of this shape.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Claude has reviewed this PR 3 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 134b03a58c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread plugins/claude-config/skills/audit-instructions/reference/criteria.md Outdated
… the co-fire to accepted changes

Two Codex findings on #1917, both verified against the live pages:

- The base row routed the Fable 5 harness thinking-disable failures to
  I17-a, but no arm of I17-a detected them - an instruction presenting
  the session toggle or alwaysThinkingEnabled as turning thinking off on
  Fable 5 had no effort operand (arm 1 silent), was not the API form
  (arm 2 silent), and fell outside I17-a's MAX_THINKING_TOKENS/
  CLAUDE_CODE_DISABLE_THINKING detect. I17-a's Detect now flags those
  controls presented as working there, grounded in model configuration's
  "have no effect there".

- The I17-b/I17-c co-firing note claimed a cache restart "whether or not
  the model accepts" the change. Where I17-c condemns the control, the
  cost never materializes: a rejected request completes no turn, an
  ignored value changes no configuration. The note is now scoped to
  accepted changes, with I17-c firing alone otherwise.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@claude

claude Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 3m 21s —— View job


Review: fix(claude-config): re-arm the thinking-disable row where the model forbids it

  • Gather context (diff since last review, fix commit e6b6134)
  • Review the fix commit against the two Codex findings it claims to resolve
  • Re-check internal consistency of criteria.md after the fix
  • Check CHANGELOG.md / plugin.json for this commit
  • Post final review

Scope note: per REVIEW.md, this lane owns everything except security (dedicated claude-security-review.yml handles that; already posted, N/A for this docs-only fix commit).

What I checked

This is a re-review after commit e6b6134, which addresses the two Codex findings from the prior round. I read the full fix diff and re-read the amended I17 family in criteria.md end to end.

Both Codex findings are correctly resolved:

  • "Cover the excluded Fable harness controls" — the base row's second arm routes Fable 5 harness surfaces to I17-a (L697), but I17-a's Detect previously fired on MAX_THINKING_TOKENS=0 claims only — a session-toggle or alwaysThinkingEnabled claim with no effort operand and no MAX_THINKING_TOKENS literal fell through both arms. The fix adds a dedicated Detect clause (L756–L758) that flags exactly those two controls presented as working on Fable 5. I checked this against the source quote the base row already cited ("the session toggle, alwaysThinkingEnabled and MAX_THINKING_TOKENS=0 'have no effect there'", L695–L696) — the new clause covers precisely the two controls MAX_THINKING_TOKENS=0's own Detect didn't already reach, so the routing promise at L697 is now actually delivered. No over- or under-reach.
  • "Limit cache co-firing to accepted configuration changes" — the old co-firing note claimed a cache restart "whether or not the model accepts" the change, which was wrong for the subset of prescriptions I17-c condemns outright (rejected API request completes no turn; ignored harness value changes no configuration). The rewritten note (L798–L808) maps precisely onto I17-c's two arms (harness silent no-op, API hard 400) and correctly scopes co-firing to only the case where a surface separately prescribes an accepted mid-session change. Logically sound and consistent with I17-c's own text.

Minor finding

  • Changelog overstates I17-a's re-verification scope. CHANGELOG.md#L56-L58 says I17 base, I17-a, and I17-b "were re-verified live against their full source sets on 2026-08-04 and carry that stamp." But I17-a's own Verified line (L770–L772) doesn't carry a clean 2026-08-04 stamp the way the base row (L742) and I17-b (L822) do — it's a split stamp: "Verified 2026-08-02 against those two pages... the session-toggle and alwaysThinkingEnabled arm re-verified 2026-08-04." Only the new clause's source quote was rechecked on 2026-08-04; the original MAX_THINKING_TOKENS/CLAUDE_CODE_DISABLE_THINKING claims and their environment-variables-page citation still rest on the 2026-08-02 check. Given this doc's whole discipline is precise "Verified" stamps driving recheck triggers, the changelog's blanket "full source sets... carry that stamp" reads as a stronger claim than I17-a's row text actually supports. Low-severity — doesn't affect the audit logic, only the changelog's own accuracy about what was reconfirmed. Fix this →

Summary

Both prior Codex findings are correctly and precisely fixed — no over-broad or under-scoped detection introduced. Version bumps (criteria.md 1.11.0 → 1.12.0, plugin.json 0.21.3 → 0.21.4) were already in place before this fix commit and weren't re-bumped, which is correct since this is a same-PR fixup, not a new release. One low-severity changelog-accuracy nit above; nothing blocking.

@claude

claude Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 39s —— View job


Security review complete

  • Gather context (diff, changed files)
  • Review plugin.json version bump
  • Review CHANGELOG.md additions
  • Review criteria.md content changes
  • Check for injection / secrets / access-control / GitHub Actions risk
  • Post findings

No security issues found.

Re-reviewed against the current head (e6b6134), which includes the second commit addressing the two Codex P2 findings (I17-a's Detect gaining the Fable-5 harness-toggle case, and the I17-b/I17-c co-fire being scoped to accepted changes only). This PR still touches exactly three files, all documentation/config metadata for the claude-config plugin:

  • plugins/claude-config/.claude-plugin/plugin.json — a one-line semver bump (0.21.30.21.4).
  • plugins/claude-config/CHANGELOG.md — prose changelog entries describing the doc changes below.
  • plugins/claude-config/skills/audit-instructions/reference/criteria.md — prose amendments to the I17 / I17-a / I17-b audit-criteria rows (reference documentation consumed by an audit skill, not executable code).

None of these introduce or modify: scripts, hooks, GitHub Actions workflows or their permissions:/triggers, MCP server configs, command execution, path handling, credentials/secrets, or any data-egress surface. The only URL touched is the pre-existing platform.claude.com/docs/.../thinking Sources link, re-wrapped for its expanded parenthetical — no new external references were added. There's no code path here that takes untrusted input — it's markdown prose describing detection rules for a later, separate audit process. Nothing in this lane (logic flaws, authz gaps, injection, secret handling, dangerous Actions patterns) applies to a docs-only content change of this shape.

Re-review nit: the changelog claimed all three changed rows were
re-verified against their full source sets on 2026-08-04, but I17-a's
row text records a split stamp - only the new clause was re-checked,
the original claims keep their 2026-08-02 verification.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@claude

claude Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 1m 28s —— View job


Review: fix(claude-config): re-arm the thinking-disable row where the model forbids it

  • Gather context (diff since last review at e6b6134, latest commit 86014ba)
  • Review the latest commit against the changelog-accuracy nit from the prior round
  • Re-check internal consistency of criteria.md
  • Post final review

Scope note: per REVIEW.md, this lane owns everything except security (dedicated claude-security-review.yml already posted N/A for this docs-only PR).

What I checked

This is a re-review after commit 86014ba, which is a targeted fix for the one low-severity finding from my prior round (the changelog overstating I17-a's re-verification scope). The commit touches only CHANGELOG.md (+4/-3 lines) — criteria.md and plugin.json are unchanged from e6b6134, which is correct since the underlying rule text was already accurate; only the changelog's description of it was overstated.

The fix is precise and verified against the source it describes:

  • The changelog now says I17 base and I17-b "were re-verified live against their full source sets on 2026-08-04 and carry that stamp," while I17-a "carries a split stamp — only its new session-toggle/alwaysThinkingEnabled clause was re-verified 2026-08-04, its original claims keep their 2026-08-02 check."
  • Cross-checked directly against I17-a's own Verified line in criteria.md: L770–L772 reads "Verified 2026-08-02 against those two pages, fetched as raw markdown; the session-toggle and alwaysThinkingEnabled arm re-verified 2026-08-04 against model configuration." This is an exact match — the changelog's new wording is now a faithful paraphrase of the row's actual stamp, not a stronger claim.
  • Confirmed I17 base (L742) and I17-b (L822) both still carry clean, unsplit Verified 2026-08-04 stamps, so the changelog's blanket claim for those two rows remains accurate.
  • Full-PR diff stat against origin/main (+175/-41 across the three files) matches the PR description exactly, confirming this commit didn't touch anything outside the changelog paragraph it was meant to fix.

One cosmetic-only observation, not worth a fix: the edited paragraph now has one noticeably longer unwrapped line (L58) versus the ~90-char wrap used elsewhere in the file. MD013 (line length) is disabled repo-wide in .markdownlint-cli2.jsonc, so this doesn't fail lint and isn't a functional issue — flagging only as a minor readability nit, not a finding.

Summary

The changelog-accuracy nit from the prior round is correctly and minimally fixed — the new wording matches I17-a's actual split verification stamp word-for-word in substance. No other findings. This PR is in good shape.

@claude

claude Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Claude finished @kyle-sexton's task in 47s —— View job


Security review complete

  • Gather context (diff since last review, latest commit 86014ba)
  • Review the new commit against changed files
  • Check for injection / secrets / access-control / GitHub Actions risk
  • Post findings

No security issues found.

The latest commit (86014ba) touches exactly one file, and it's a wording-only correction:

  • plugins/claude-config/CHANGELOG.md — rewords the [0.21.4] entry so it no longer claims I17-a's full source set was re-verified on 2026-08-04. It now correctly states I17-a carries a split stamp (only the new session-toggle/alwaysThinkingEnabled clause was re-checked that date; the original MAX_THINKING_TOKENS/CLAUDE_CODE_DISABLE_THINKING claims keep their 2026-08-02 check). This directly resolves the low-severity changelog-accuracy nit flagged in the prior review round.

No scripts, hooks, workflows, MCP configs, command execution, path handling, credentials, or data-egress surfaces are touched by this commit. As with the rest of this PR, it's markdown prose describing audit-criteria documentation — nothing in this lane's scope (logic flaws, authz gaps, injection, secret handling, dangerous Actions patterns) applies.

This confirms the finding stack across the full PR (134b03ae6b613486014ba): both Codex P2 findings and the changelog-accuracy nit from prior review rounds are now resolved, with no new security-relevant surface introduced at any point.

@kyle-sexton
kyle-sexton merged commit d7e5149 into main Aug 4, 2026
32 checks passed
@kyle-sexton
kyle-sexton deleted the docs/roster-r11-thinking branch August 4, 2026 08:10
kyle-sexton added a commit that referenced this pull request Aug 4, 2026
…d where none is (#1918)

## Summary

Doc-alignment roster row 12: **Steering thinking** (live page
byte-identical to the archived 947-line slice — MD5 confirmed by
producer and verifier independently).

**criteria 1.13.0 / claude-config 0.21.5** — new sub-row **I18-a**: *a
leading thinking block treated as required where the model does not
require one*. I18 covered only what a surface does to blocks it has; the
inverse belief — that a block must be there — had no row, and its
natural remediation (fabricate one) produces exactly I18's shape 1 (a
hand-built block has no valid signature), making I18-a the upstream
cause of an I18 violation. Three detect shapes, one per consequence the
page states: reinsertion when assembling history from mixed sources;
history rewriting on resume under a different thinking configuration;
logic reading an assistant turn's first block as a thinking block.

- Honest carve-out from upstream's own text: legacy manual-budget models
*do* enforce the leading block — the finding is the missing gate, never
the mention.
- Fence: the relaxation "is about validation, not about what you should
send" — never license to drop blocks.
- Reach held to I18's line: presence-assuming read-only logic is out of
reach (neither excused nor flagged), keyed to the still-unresolved
question of what Claude Code transcripts carry, with a re-scope trigger.
- Sourcing: the Steering page is decisive (states the relaxation
operatively with its three consequences); the Thinking page carries the
pair in compressed form in its tool-use section and is cited as
corroboration — with its server-side strip claim explicitly held apart
as a different claim.
- I18 base gains only a two-sentence lead-in and a "Base row:" label;
its detect, fences, source, and stamp are untouched.

Also verified in passing (roster corrections, not in this diff): the
row's "shipped only as far as context-economy" note was stale —
PLUGIN-PHILOSOPHY's Effort tiers already carries five of the slice's
candidates citing this page; and the repo's one transcript consumer
already selects blocks by their own `type`, correct by construction.

## Test plan

- Docs-only; markdownlint 0 errors; changelog parity all three modes;
`instruction-scan.test.sh` 46/46; scripted quote fidelity 14/14 against
both live pages with control probes.
- Producer's advisor overturned an over-reach in its first draft
(extending shape 3 to read-only surfaces — unsupported); the producer
also self-diagnosed the root cause of its one false claim (a `begin
with` grep that could never match `begins with` — the
absence-through-a-blind-channel trap).
- Orchestrator-commissioned Fable verifier: 5/6 first pass — caught the
false "only source" custody claim against its own live fetch of the
sibling page — plus an ambiguous antecedent; both fixed, rebase compose
verified to preserve #1917's Sources expansion, re-check ALL PASS.
- Rebased onto post-#1917 main; stacks asserted: [0.21.5] > [0.21.4] >
[0.21.3]; criteria 1.13.0.

## Related

- No linked issue.
- Doc-alignment loop, roster row 12. Predecessors: #1908#1917.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_019gaVX25Txd6GXdiu9HEH3X

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant