fix(playbooks): resolve the max-effort contradiction, stamp benchmark figures, join the thinking-off rule - #1876
Conversation
… figures, join the thinking-off rule Three corpus-alignment fixes to locally-owned `playbooks` files. The `boris` pack's `vendor/SKILL.md` carries the same strings and is deliberately untouched — it is the verbatim upstream baseline used for drift detection, so editing it would manufacture false drift. RA-1 — this repo contradicted itself on `max` effort durability. `skills/boris/SKILL.md:119` said "max is session-only" flat and `skills/boris/reference/autonomy.md:203` said all other levels "are sticky", while `docs/PLUGIN-PHILOSOPHY.md:631` carried the `CLAUDE_CODE_EFFORT_LEVEL` exception. Verified 2026-08-02 against code.claude.com/docs/en/model-config#adjust-effort-level: PLUGIN-PHILOSOPHY is right. Both files now carry the exception. The population was enumerated by script rather than taken from the finding: three non-vendor lines, which is how autonomy.md:203 — not named in the finding — was caught. RA-4 — `skills/boris/reference/orchestration.md` restated volatile benchmark scores at §78 and §94 with no as-of date and no re-derivation event. A four-part `docs/conventions/upstream-drift` record now classifies them as launch-day history and states an observable recheck trigger. Figures unchanged: they are true of the releases they announced. RA-8 — `skills/fable-5/context/model-adaptation/opus-5.md` documented the effort-conditional 400 and the harness thinking-disable surfaces in two bullets and never joined them. A third bullet states the rule they imply, at the strength the evidence supports: it records the config-time question as untested rather than claiming the harness guards the combination, and keeps upstream's unexpanded "and later models" scope. Bumps `playbooks` 0.6.1 -> 0.6.2 with the matching CHANGELOG entry. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 37s —— View job Security review complete
Scope reviewed (via FindingsNone. This PR is pure documentation content:
No code execution paths, workflow files, hook contracts, injection surfaces, credential handling, or permission-widening changes are touched by this diff. Nothing here falls in-scope for security review. |
|
Claude finished @kyle-sexton's task in 4m 8s —— View job Review: RA-1 / RA-4 / RA-8 playbooks alignmentReviewed all 6 changed files against Verification result: every load-bearing quote checks out.
RA-4 upstream-drift record ( Population sweep verified. Confirmed no other live (non-vendor) file in the repo still states the flat "max is session-only" claim; CHANGELOG / version bump — One suggestion (non-blocking)
SummaryNo correctness, design, or convention issues found. All quoted upstream facts were independently re-verified against the live pages this session and match exactly. This is a docs-only change with no code/test surface, consistent with the "code-review lane" scope in |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 302452a689
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude has reviewed this PR 5 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
Repo-alignment finding RA-5 from the doc-corpus campaign's Phase 3a. The harness documentation keeps "built-in command" and "bundled skill" as distinct categories; `plugins/review/skills/quality-gate/context/code.md` called `/code-review` a "built-in ... bundled skill", smearing both. ## What the docs actually say Established from the raw upstream markdown before writing the correction, not from recall: - `commands.md:35` — "Most are built-in commands whose behavior is coded into the CLI" - `commands.md:37` — "**[Skill](/docs/en/skills#bundled-skills)**: a bundled skill" - `skills.md:23` — "Bundled skills are prompt-based... Most built-in commands instead execute fixed logic directly" `/code-review`, `/simplify` and `/loop` all carry the `[Skill]` marker. So the correct category is bundled skill, and the corrected text links the live `#bundled-skills` anchor. ## The CHANGELOG question, decided explicitly `plugins/review/CHANGELOG.md:272` carries the same smear. It is **left as written**. Released changelog entries are history, and this repo already set that precedent in its own 0.15.3 entry: "Those entries are left as written — history is corrected forward, not rewritten." The forward correction is this entry. ## A method note worth carrying The verifier re-grounded against raw markdown via `curl` rather than a summarizing fetch — and that mattered. WebFetch's summarizer twice reported that `/simplify` does not appear in the commands reference, which would have produced a wrong FAIL. The raw file carries it at line 130. That is the same failure mode the knowledge plugin's new absence-fetch rule exists to prevent: a truncating or summarizing channel cannot manufacture a presence, only a false absence. ## Verification Independently verified with the implementer's rationale withheld. Every quoted upstream string checked verbatim against the live pages; every quoted historical changelog string checked against the repo's own history. Three surveyed-and-excluded sites are recorded in the review so a later round does not re-litigate them: the vendored Boris copy (three sites, one a literal quotation whose correction would falsify the quote) and `plugins/debugging/CHANGELOG.md:32`, both covered by the same never-rewrite-history policy. No linked issue ## Related - Phase 3a of the doc-corpus application campaign; this repo conforms before any check ships to consumers. - Siblings: #1875 (knowledge, pipeline amendments), #1876 (playbooks, effort/benchmark/thinking alignment). --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ord claims The RA-4 upstream-drift record added in 302452a said each benchmark figure's basis is "the launch announcement cited in its own section", but neither §78 nor §94 carried a URL — the record's own basis leg fell short of `docs/conventions/upstream-drift/README.md`'s bar ("the official page URL (with anchor where one exists)"). Flagged as P2 by an external reviewer and independently by this branch's verifier: a record whose purpose is to make a volatile claim re-checkable described an attribution that did not exist. Both announcements were fetched with raw `curl` and confirmed to carry the exact figures: - <https://www.anthropic.com/news/claude-opus-4-8> — HTTP 200, 208,141 bytes as fetched, `<title>Introducing Claude Opus 4.8`, dated May 28, 2026. The capabilities table (2600x1392 PNG, the first `<figure>` under `<h2 id="opus-48s-capabilities">`) reads SWE-Bench Pro 64.3% (4.7) → 69.2% (4.8) and Terminal-Bench 2.1 66.1% → 74.6% — §78's two figures. - <https://www.anthropic.com/news/claude-fable-5-mythos-5> — HTTP 200, 346,462 bytes as fetched, `<title>Claude Fable 5 and Claude Mythos 5`, dated Jun 9, 2026. The capabilities table (2600x2870 PNG, alt "Benchmark table showing Claude Fable and Mythos compared to other leading models", under `<h2 id="evaluating-claude-fable-5-and-claude-mythos-5">`) reads SWE-Bench Pro 80.3% → 69.2%, FrontierCode (Diamond) xhigh 29.3% → 13.4%, GDPval-AA 1932 → 1890, OSWorld-Verified 85.0% → 83.4% — §94's four figures. Its methodology caption independently corroborates §94's starred-benchmark sentence: the starred rows are exactly cybersecurity, biology, Terminal-Bench 2.1, HLE, and HealthBench, and the caption gives the same blocking-safeguard fallback reason. Both pages publish the figures only inside that table image. Literal "69.2" occurs zero times in the Opus page's HTML, and the Fable page's apparent "80.3"/"29.3" hits are a Sanity image-palette `population` value and an SVG path. The header record now states this, so a re-checker who greps the fetched HTML does not read a correct citation as broken. anthropic.com serves no raw-markdown channel — `.md`, `/index.md`, and `/llms.txt` all 404 — so full-page `curl` plus reading the table image is the ceiling here. Population enumerated by grep over non-vendor `plugins/playbooks/skills/` for the benchmark names: exactly two figure-bearing lines (`orchestration.md` §78 and §94) and one citation claim (`orchestration.md:6`); no other non-vendor file in the plugin carries a benchmark figure or a claim of this shape. `skills/boris/vendor/SKILL.md` carries the same figures and stays untouched — it is the verbatim upstream baseline used for drift detection. The 0.6.2 CHANGELOG entry is unreleased on this branch (origin/main is 0.6.1), so it is amended in place to describe what ships rather than taking a second bump. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 2m 22s —— View job Code review complete
Scope reviewed ( Codex's P2 finding — resolvedCodex flagged that the RA-4 drift record's claim ("basis is the launch announcement cited in its own section") wasn't backed by an actual citation. That's fixed in the second commit (92762fb): Correctness checks
One pre-existing nit, still unaddressed
SummaryNo blocking correctness, architecture, or convention issues. Both prior review-pass findings (Codex's missing citations, the SKILL.md row-length nit) have been checked against the current head — the citation gap is fixed, the row-length nit remains as a cosmetic carryover. Docs-only change, consistent with |
|
Claude finished @kyle-sexton's task in 2m 12s —— View job Security review complete
Scope reviewed (diff between the PR's base commit FindingsNone. This diff is pure documentation/prose content:
No workflow files, hook contracts, credential handling, injection surfaces (command/SQL/path/template), or permission-widening changes are touched. The external links added ( Note: Codex flagged a content-accuracy point (citation specificity in the benchmark drift record) — that's a documentation-accuracy concern, not a security issue, so it's out of scope for this security-only lane. |
The Opus 5 thinking-off x effort rule claimed `"and later models" is
upstream's own scope wording`. It is not. Re-verified this session by
fetching all three plausible cited pages through the raw-markdown
channel:
- platform.claude.com/.../whats-new-opus-5.md — 16380 bytes, 352 lines
- code.claude.com/docs/en/model-config.md — 83644 bytes, 732 lines
- platform.claude.com/.../prompting-claude-opus-5.md — 11225 bytes, 105 lines
`and later models` occurs zero times on any of them. What
whats-new-opus-5.md actually says, at line 237, is "This is generally
available behavior on Claude Opus 5 onward". The bullet's own paraphrase
("on Opus 5 and later") was never the problem and is unchanged; only the
meta-claim about whose wording it is changes, to quote the substring the
page really carries.
The CHANGELOG's 0.6.2 entry repeated the same false attribution
verbatim, so it is corrected in place. No second version bump: 0.6.2 is
unreleased and what ships is unchanged.
Same defect class as the drift record this PR already fixed — prose
asserting an attribution its source does not carry. Swept every added
line in the branch diff for the class; the other attribution claims all
check out against their sources (model-config's five quoted strings, both
launch-announcement URLs and anchor ids, and the claim that neither
announcement publishes its benchmark figures as page text).
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 50s —— View job Security review complete
Scope reviewed: diff between merge-base This trigger fired on a
Both edits are pure prose corrections with no change to frontmatter, executable content, permissions, or structure. FindingsNone. As with the prior two security-review passes on this PR, the full diff remains pure documentation content:
No workflow files, hook contracts, credential handling, injection surfaces (command/SQL/path/template), or permission-widening changes are touched by this diff. Nothing in this PR falls in-scope for security review. |
|
Claude finished @kyle-sexton's task in 1m 12s —— View job Code review complete
Scope reviewed: this pass covers what changed since the last code-review pass (head New commit (
|
Same sweep, same class. The 0.6.2 entry asserted that `skills/boris/vendor/SKILL.md` carries the max-effort session-only claim "at 3 lines" and the benchmark figures "at 14 lines". Neither count reproduces under a stated rule: grep -n -E "session-only|Max applies only" .../vendor/SKILL.md -> 2 grep -n -E "SWE-Bench|Terminal-Bench|GDPval|FrontierCode|OSWorld" -> 13 Both numbers can be recovered only by picking a counting rule after seeing the target, which is the same defect this branch is here to stop: a specific asserted in prose that its source does not carry. The counts are not load-bearing — the point is that the vendored upstream baseline is deliberately untouched so drift detection stays honest — so they are removed rather than replaced with a number chosen to fit. No version bump: 0.6.2 is unreleased and what ships is unchanged. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 41s —— View job Security review complete
Scope reviewed: diff between merge-base This trigger fired on a
FindingsNone. As with every prior security-review pass on this PR, the full diff remains pure documentation content:
No workflow files, hook contracts, credential handling, injection surfaces (command/SQL/path/template), or permission-widening changes are touched by this diff. Nothing in this PR falls in-scope for security review. |
|
Claude finished @kyle-sexton's task in 47s —— View job Code review complete
Scope reviewed: this pass covers what changed since the last code-review pass (head New commit (
|
… the user-invoked-only /verify (#1878) Repo-alignment finding RA-10 from the doc-corpus campaign's Phase 3a. **The finding as filed was refused; a different, real defect was found underneath it and fixed.** ## The refusal RA-10 claimed the `>=2.1.145` gate on the bundled `/verify` note was stale. It is not. Live `curl` of the raw skills reference (HTTP 200, 73,907 bytes, 933 lines — non-truncated, and the length is recorded because a rendered fetch of a long page can only manufacture a false absence) still states: "All three skills require Claude Code v2.1.145 or later" for `/run`, `/verify`, `/run-skill-generator`. The triage conflated two orthogonal axes. `2.1.145` is the **availability floor**. `2.1.215` changed the **invoker**: "others, including `/verify` and `/code-review`, run only when you invoke them... Before v2.1.215, Claude could also run `/verify` and `/code-review` on its own." Nothing about an invocation restriction makes a version floor stale. ## The real defect, in the opposite direction The triage asked whether the repo *claims* auto-invocation, and correctly found it does not. But the repo did something stronger: it **instructed the agent to invoke `/verify`** — a surface the model may no longer invoke. Four live assertion sites now state the opposite: `/verify` is surfaced as a suggestion for the *user* to run, never delegated to. `/run` is deliberately untouched — upstream restricts only `/verify` and `/code-review`, and `/run` survives as supplementary and agent-invocable at all nine live sites. ## The eval gap The first pass left `confirm/evals/evals.json` encoding the removed delegation, so the skill would have shipped **failing its own graded rubric**. Caught by an independent verifier on population completeness — the campaign's most repeated error. Eval 9 is now entailed clause-by-clause by the shipped skill text instead of contradicting it, and its id/name/prompt are unchanged because `/run` delegation genuinely remains. Changelog entries for already-shipped versions are deliberately left as written: a released entry is an immutable record of what that version did. ## Verification Two independent verification rounds, rationale withheld. The second re-enumerated from scratch rather than inheriting the first's list: 50 `/verify` lines repo-wide, 34 unrelated, 12 changelog text, 4 live assertion sites — all four corrected, zero remaining. The `agent-invocable` wording now shipping inside a graded rubric was grounded empirically (this session's skills listing exposes `run` while bundled `verify` and `code-review` are absent) rather than on upstream's non-exhaustive "including". CI gates were run directly rather than trusted from the commit message, including the markdownlint pass that the skill-check harness had skipped. No linked issue ## Related - Phase 3a of the doc-corpus application campaign; this repo conforms before any check ships to consumers. - Siblings: #1875 (knowledge, pipeline amendments), #1876 (playbooks), #1877 (review, merged). --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Fixed by adding the real URLs, not by dropping "cited" — both launch announcements were fetched and confirmed to carry the exact figures, so the convention's stated bar is met rather than the claim weakened.
One disclosure the citation needs in order to survive a re-check. Both pages publish these figures only inside a table image, never in page text. Literal Population. Grep over non-vendor Also found while sweeping, and fixed in a follow-up commit: the same defect class on a different leg of this PR. Observed and deliberately not addressed: §94 also restates volatile pricing and specs with no citation. The drift record's scope sentence is explicitly benchmark figures, so that is a separate finding rather than a reason to widen this diff. |
The thinking-off x effort rule added to `fable-5`'s `opus-5.md` carried two
defects of the classes this PR exists to fix.
Self-contradiction, same PR: the rule enumerated `effortLevel` among the
surfaces that set "`xhigh` or `max`" effort, while this PR's first commit
records the opposite for `boris` §72 — the persisted setting does not accept
`max`. Read distributively, the way the rule's own framing tells an auditor to
read it, that sent a reader hunting settings files for `effortLevel: max`, a
literal that cannot exist. Re-verified against the current raw markdown of
`code.claude.com/docs/en/model-config`, "Set the effort level": `effortLevel`
takes `low`, `medium`, `high`, or `xhigh`, and `max` and `ultracode` "are not
accepted here"; `code.claude.com/docs/en/settings` states the same accepted
list independently. `effortLevel` now reads "which takes `xhigh` but not
`max`", which keeps the rule at full strength — `effortLevel: xhigh` paired
with a disable surface is still the defect it describes.
The other three effort surfaces were checked the same way and all carry `max`:
`CLAUDE_CODE_EFFORT_LEVEL` (its one durable route), `--effort`, and skill and
subagent `effort` frontmatter, per the frontmatter references in
`code.claude.com/docs/en/skills` and `.../sub-agents`.
Invalid literal in a rule about readable literals: the API disable example was
written `thinking: {type: "disabled"}` with the key unquoted, which no JSON
parser accepts. Upstream writes it `thinking: {"type": "disabled"}`
(`platform.claude.com/docs/en/about-claude/models/whats-new-opus-5`,
"Disabling thinking requires effort `high` or below"; the same page quotes
`thinking: {"type": "adaptive"}` the same way). The rule's entire selling point
is that the hazard is statically checkable by reading configuration literals,
so shipping an unparseable one as its own example undercut the property it
claims.
While in that sentence: "both operands are literals in settings files" was
overbroad against its own enumeration — `--effort` is a launch flag and
skill/subagent `effort` is markdown frontmatter, neither of which lives in a
settings file. Now "configuration literals", true of all four surfaces, with
the closing "wherever such configuration is audited" moved with it.
The 0.6.2 CHANGELOG entry is amended in place rather than bumped: 0.6.2 is
unreleased, so what ships is the corrected text, and the entry restated the
overbroad settings-file phrasing it would otherwise have carried one file over.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 48s —— View job Security review complete
Scope reviewed: diff between merge-base This trigger fired on a
Both edits are pure prose/wording corrections — no change to frontmatter, executable content, permissions, or structure. FindingsNone. As with every prior security-review pass on this PR, the full diff remains pure documentation content:
No workflow files, hook contracts, credential handling, injection surfaces (command/SQL/path/template), or permission-widening changes are touched by this diff. Nothing in this PR falls in-scope for security review. |
|
Claude finished @kyle-sexton's task in 1m 34s —— View job Code review complete
Scope reviewed: this pass covers what changed since the last code-review pass (head New commit (
|
…nnel extraction artifacts (#1879) Second batch of evidence-forced docpage-digest amendments. `knowledge` 0.10.15 -> 0.10.16. Two rules ship; six items are deliberately held, each with its reason stated. ## What ships - **PA-Y — the near-miss disclosure rule, written into the profile.** It previously lived only inside Ruling 1 Decision A and was re-derived per unit, which is why undisclosed near-misses are the largest MINOR class in the slice that measured it. A harness page covering the same subject without stating the claim's specific rule is a near-miss: the tag survives, and the row must disclose it by name and line. - **PA-Z — blog-channel extraction artifacts** (collapsed H1, split reading-time) with reconstruction guidance. Its deferral condition was met: a second `claude.com/blog` run reproduced both exactly, re-verified byte-exact in both slices. The guidance leads with the canonical URL slug and states what a slug **loses** — word boundaries recover, punctuation and casing do not — so a slug-recovered title is labelled reconstructed. - **PA-AG — verified and closed, no edit.** The `cc-applicable`/`mixed` boundary rule already in the profile matches what its source asked for. ## What is held, and why — two different reasons **Five held because the campaign's judgment file holds them.** PA-I (J-6), PA-Q (J-7), PA-N (J-8), PA-M (J-12) and PA-AD (J-14) were shipped in the first draft of this branch and then withdrawn. The triage marks them `evidence-forced`; `PHASE2-JUDGMENT-AMENDMENTS` writes them up as judgment calls under a header stating "None applied." Two records disagreed, and the judgment file governs on its own reasoning: - J-14 says outright it is "held here only because no defect was demonstrated, so the bar for the applied subset is not met." - J-7 says it "should be decided **after** Decision-A ordering" — an escalated, owner-reserved question. Shipping it would have pre-empted a decision the owner kept for himself. They are now full rows in the owner-decision artifact, each restating the triage-vs-judgment conflict rather than resolving it silently. Between the two readings they would otherwise have landed in neither place. **Three held because they change untracked instruments.** PA-B's second half, PA-P and PA-AK all modify scripts under the memory-tier `.work/` root, which is never committed and is not part of the shipped plugin. ## Verification Independently verified twice. The first round FAILED on the five judgment-held rules. The second confirmed the split is clean: `SKILL.md` is byte-identical to its 0.10.15 state, grep finds no residue of the withdrawn rules, and PA-Y reads as a whole rule rather than half of the graded pair it was originally merged into. The implementer also ran a second pass over the whole *file* rather than the diff, and caught three defects in its own first commit — including a new rule that contradicted one shipped in #1875, which is exactly the two-rules-for-one-boundary drift this pipeline exists to prevent. No linked issue ## Related - Follows #1875 (`knowledge` 0.10.15), which shipped the first evidence-forced batch. - Phase 3a self-alignment merged in #1876, #1877, #1878; this repo conforms before any of it ships outward. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…urfaces (#1880) Applies the evidence-forced half of the RA-9 extend-or-cite pass to instruction-audit catalog row **I10** (reasoning-echo). `claude-config` 0.17.0 -> 0.18.0; `criteria.md` 1.4.0 -> 1.5.0. ## What this does NOT do, and why that is the point **I10 keeps `Model scope: fable-5`. It is not promoted, and `mythos-5` is not added.** The campaign triage ranked this promotion its **#1 item**, on the theory that the extended-thinking platform page is a second, model-independent source that would satisfy I10's promotion gate. It is not, and the page's own structure is what settles it: - `:874` — the H2 reads "Thinking output on Claude Fable 5 **and Claude Mythos 5**" - `:876` — names **both** models for the adjacent raw-chain-of-thought property - `:887` — names **only** Claude Fable 5 for `stop_details.category: "reasoning_extraction"` Eleven lines apart, in one section. The page had a sentence-adjacent opportunity to widen the refusal and declined it. That is deliberate scoping, not loose phrasing. Adding `mythos-5` would fabricate scope from a claim about a different property — exactly what the catalog's own model-scoping block warns against. That reasoning is now recorded **in the row itself**, so it is not re-litigated a fourth time. ## What it does - **A corroborating second Source** — the extended-thinking page, independent of the Fable 5 model guide already cited. - **A concretized Remediate line.** It previously said, abstractly, to read structured `thinking` blocks or use a send-to-user tool. It now names the actual surfaces: `Ctrl+O` verbose mode and `showThinkingSummaries: true`, and `display: "summarized"`. **Each surface is attributed to the page that actually states it.** The pass this work came from asserted all three were on the thinking page; two of them are not — `Ctrl+O` and `showThinkingSummaries` are stated at `model-config:532`, and only `display: "summarized"` is on the thinking page. Both pages are therefore added to `## Sources`, because `criteria.md` carries its own invariant that "the trigger set is the source set — naming a subset would leave the harness-behavior rows depending on pages nothing watches." ## Verification Independently verified by a second model, with the implementer's rationale withheld. Both pages re-fetched raw (`model-config.md` 83,644 bytes; `thinking.md` 52,769 bytes, byte-identical to the frozen snapshot), all three surface names confirmed verbatim, and the promotion-gate facts re-confirmed at both snapshot and live bytes. **Self-fire check.** Because a catalog row that fires on this repo would break the campaign's own rule against shipping a consumer check we fail: the scanner was run against both trees. I10 candidates in `criteria.md` go 2 -> 4, and all four are inert — `criteria.md` is a skill reference file, outside the audited population (`CLAUDE.md` / `rules/` / `skills/` / `agents/` / `output-styles/` under the user and project roots). No new class of self-hit is introduced. Detection is untouched: `instruction-scan.test.sh` reports 46/46, and this edit changes Source and Remediate only. ## Held back deliberately The **I8-b promotion**, whose gate genuinely IS met by verbatim two-guide convergence, is not here. Promoting it makes the row fire on this repo's own `plugins/review/context/severity.md` — the same work as triage row RA-2, an open owner decision. Shipping the promotion first would make the next audit run flag this repository. No linked issue ## Related - Phase 3b of the doc-corpus campaign; Phase 3a self-alignment merged in #1875, #1876, #1877, #1878, #1879. - Source pass: `RA9-EXTEND-OR-CITE-PASS-2026-08-02.md` (23 rows adjudicated; the triage had checked 6). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y.md enumerable tier criteria (#1881) Promotes instruction-audit catalog row **I8-b** (conservative-reporting detection) to unscoped and gives `plugins/review/context/severity.md` enumerable tier criteria. `claude-config` 0.18.0 -> 0.19.0; `review` 0.15.5 -> 0.16.0; `criteria.md` 1.5.0 -> 1.6.0. ## The promotion I8-b's gate is MET on its second arm (multiple model guides converge): the Sonnet 5 prompting guide states all three trigger phrases verbatim in one sentence (`source.md:140`, "Code review harnesses"), converging with the Opus 5 guide. Annotated in row I7's met-gate precedent form; the Sonnet 5 guide URL added to `## Sources` per the trigger-set-is-the-source-set invariant. **Source attribution corrected while citing:** "don't nitpick" appears nowhere in the Opus 5 guide (`grep -cin nitpick` -> 0); that guide states only the other two phrases (its line 20). The Sonnet 5 guide is the phrase's only cited home. This strengthens the convergence gate — three phrases now each attributed to a page that actually contains them. ## #1880's "Held back deliberately" premise was wrong That PR deferred this promotion because unscoping I8-b would allegedly make it fire on `severity.md`. It does not, on three independently verified grounds: 1. **Zero scanner candidates** — the real scanner over `severity.md` emits one I6 row and no I8-b; the I8-b ERE greps 0 on both trees. 2. **I8-b's own carve-out** (`criteria.md:252-255`) excludes "severity-based routing where everything is still reported somewhere" — severity.md classifies findings and withholds none. 3. **Outside the audited population** — SKILL.md Phase A inventories CLAUDE.md / rules/ / skills/ / agents/ / output-styles/ under user and project roots plus hook text; `plugins/review/context/` is none of those (same result #1880 recorded for `criteria.md` itself). The promotion could have shipped alone. The severity.md work ships here anyway, **re-founded on its own source**: nine lines below the three-phrase line, the same Sonnet 5 guide says to "be concrete about where the bar is rather than using qualitative terms like `important`" (`source.md:150`) — and `important` is one of severity.md's own tier names. That is triage row RA-2, a distinct claim from the one I8-b cites. ## The severity rewrite Each tier now carries a decidable test instead of a qualitative label; no finding changes tier. Guards added where the criterion-stating change could have silently re-tiered: - The P1-P5 fold explicitly takes precedence for P-scored findings (otherwise every P3 security finding would have read into the new CRITICAL test). - CRITICAL's subsequent-change limb reads "otherwise-correct change", so cascade architecture violations (break a *correct* future change) stay CRITICAL while code duplication (bites only through an *incomplete* future edit) stays IMPORTANT. All eleven tier examples adjudicated against the new tests — twice, independently. ## Verification Independently verified by a second model with the implementer's rationale withheld, across two rounds. Scanner counts replayed from git refs both rounds: origin/main 23 rows / 6 files, all fenced; working tree 28 / 7, every addition this branch's own quoting. Planted-positive check: a constructed three-phrase file emitted I8-b on all three lines in the same invocation where severity.md emitted zero. Scanner-vs-git-grep equivalence proven on identical row sets. `instruction-scan.test.sh` 46/46; markdownlint 0 errors. **Known citation defect in an immutable commit message:** `3603c8a9b8` says "eight lines later (`source.md:149`)". Both figures are wrong — the correct citation is `source.md:150`, ten lines after the three-phrase line at `source.md:140`. No tracked file carries the wrong number; recorded here rather than rewriting pushed history. No linked issue ## Related - Phase 3b of the doc-corpus campaign; Phase 3a merged in #1875, #1876, #1877, #1878, #1879, #1880. - Reverses #1880's "Held back deliberately" reasoning with evidence (above). - Triage rows: RA-2 (severity criteria), RA-9/Q6 (I8-b promotion), owner decision Q9 (bundling). --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…9, I20 (#1882) Ships the consumer-facing payload of the doc-corpus campaign: four new rows in the `claude-config:audit-instructions` catalog. `claude-config` 0.19.0 -> 0.20.0; `criteria.md` 1.6.0 -> 1.7.0. ## The rows - **I17** (base + I17-a + I17-b) — the thinking-disable x `xhigh`/`max` effort hazard. Stated at full strength: a per-request 400, real and **unguarded** (EC-9 replayed live 400s in the local OTEL store at `cc-logs.json:65283/:65291`, `model=claude-opus-5`; no documented pre-request guard). Statically checkable against settings files. Fences: `effortLevel` accepts `xhigh` but not `max` (auditors are not sent hunting a literal that cannot exist); `ultracode` covered explicitly ("sends `xhigh` to the model" — matches neither literal yet triggers the hazard). - **I18** — thinking blocks altered on the way back to the model: signature preservation, the `block.type == "thinking"` type-filter smell, within-turn echo integrity. Scope extends through the Agent SDK (owner decision Q3) and — per EC-2's structural corpus test (every thinking block in 3.6 GB carries `signature`, zero without) — local transcript handling and transcript-parsing tooling. - **I19** — restated external benchmark figure with no recheck trigger. Ships at **OPINION tier, off by default**: no official page states the requirement, and claiming ANTHROPIC-DOCS with `Source: none` would break the catalog's own authority invariant. The worked example is this repo's own RA-4 fix (PR #1876) and `docs/conventions/upstream-drift/README.md`. - **I20** — prefilled assistant response (model-delta standing row; RA-6 replayed `prefill` at 0 on this repo, expected hit rate near zero, cheap to carry). ## Disputes carried, resolved with evidence - **`redacted_thinking`:** the brief forbade any clause; literal obedience was impossible — the type-filter rule's decisive upstream sentence (`thinking.md:867`) states its failure mode *as* silently dropping `redacted_thinking` blocks. Shipped: the upstream sentence verbatim, plus a Must-NOT-flag fencing out any clause premised on those blocks being present locally (EC-2 found zero structural instances in 3.6 GB). - **IA-4's thinking-churn half dropped:** `prompt-caching.md` contains zero occurrences of "thinking" (replayed live twice) — a thinking-toggle cache-invalidation claim would have been a fabricated citation. I17-b covers effort churn only, quoting the page verbatim including the harness's own confirm-before-applying surface. - **Brief/source count discrepancy, recorded:** the brief cites 136,295 thinking blocks; EC-2's own record says 136,176. No shipped row depends on either (a local count is not a consumer fact). ## Verification Independently verified by a second model with the implementer's rationale withheld — two independent full passes (the first died with the session; the second re-derived everything from bytes and live raw fetches: 52,769 B `thinking.md`, 29,777 B `prompt-caching.md`). - **Standing gate (never ship a check this repo fails):** zero adjudicated findings across all four rows on the merged tree, twice replayed. Every candidate adjudicated by name: `opus-5.md:125` (model-adaptation delta, fenced), `orchestration.md` x3 (carry the RA-4 four-part drift record), `action-quality.md:9` (pointer, no figure). The vendor fence is load-bearing: unfenced, I19 fires 15 times, 11 inside `boris/vendor/` — the verbatim upstream baseline kept for byte-drift detection. - **Planted positives:** 7/8 greppable shapes fire on a constructed file; I18 shape 1 (signature dropped on reconstruction) has no greppable literal and is validated by reasoned read — recorded as a residual regression gap, not rounded up. - Post-review corrections shipped as follow-up commits (append-only): two quote defects found by review (a silently converted comma, a wrong section attribution), one truncation-marker fix (`c375d90739`), and the merge-reconciliation with #1881 (versions renumbered; `plugin.json`/frontmatter auto-merged "clean and wrong" on identical strings and were corrected by hand). - **Known inaccuracy in an immutable commit message:** `c375d90739`'s body claims the unmarked truncation "is the misquote-without-ellipsis defect the criteria file itself flags elsewhere" — `criteria.md` carries no quote-fidelity criterion; the classification came from the independent verifier, not the file. Recorded here rather than rewriting pushed history (same handling as #1881's citation defect). - `instruction-scan.test.sh` 46/46; markdownlint 0 errors; changelog parity/bump/order pass; `check-skill.sh` PASS. No linked issue ## Related - Phase 3b consumer payload of the doc-corpus campaign; sibling merges #1875-#1881. - Owner decisions Q3 (Agent SDK scope), Q4/IA-12 (agnosticism routing test), Q5. - EC-2 and EC-9 empirical records: `.work/_harness-snapshots-2026-07-31/EMPIRICAL-CHECKS-RESULTS-2026-08-02.md` (memory tier). - Follow-ups filed from this lane: mechanized pre-scan for I17-I20 (gate sweep is currently a scratchpad script), post-merge version-literal assertion for the silent auto-merge trap. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 participant
Three repo-alignment findings from the doc-corpus campaign's Phase 3a — making this repo conform to the digested Anthropic documentation before any of it ships outward.
playbooks0.6.1 -> 0.6.2.RA-1 — the repo contradicted itself on
maxeffort durabilityboris/SKILL.mdstated flatly thatmaxeffort is session-only;docs/PLUGIN-PHILOSOPHY.mdcarried theCLAUDE_CODE_EFFORT_LEVELexception. Both describe a fact a user could act on.Resolved against the live page, fetched this session rather than recalled — model-config states verbatim that
max"applies to the current session only, except when set through theCLAUDE_CODE_EFFORT_LEVELenvironment variable", and thatmaxis "not accepted" in the persistedeffortLevelsetting.Population was enumerated, not taken from the finding. The finding named one site; the sweep found a second undocumented one at
boris/reference/autonomy.md:203("Max applies only to current session"), which is also fixed.RA-4 — volatile benchmark figures carried no re-check trigger
boris/reference/orchestration.mdcites GDPval-AA, Terminal-Bench 2.1 and SWE-Bench Pro figures with nothing recording when they were last true. Both carriers now sit under a four-part upstream-drift record (claim, cited page, as-of date, recheck trigger) per this repo's owndocs/conventions/upstream-drift/README.md. Figures are unchanged — they are true of the releases they announced, and the convention's prefer-the-pointer rule argues against restating a fresh snapshot.RA-8 — two halves of one checkable rule, never joined
fable-5/context/model-adaptation/opus-5.mddocumented the effort-conditional thinking-off 400 and theMAX_THINKING_TOKENS=0exception as separate bullets. They are now stated as the single statically-checkable rule: a config pairing a thinking-disable surface withxhigh/maxeffort on Opus 5+ produces a per-request 400.Deliberately stated at evidence strength: it does not claim the harness guards the combination. Whether it does is an unrun empirical check, and the section records the config-time question as untested.
Verification
Independently verified with the implementer's rationale withheld. The hard block held — zero vendored files in the diff, though
boris/vendor/SKILL.mdcarries the same strings at three sites and is hand-copy-prohibited. Every load-bearing quoted string re-fetched from the live upstream pages by the verifier.The RA-1 check instrument was validated against four trees including two that must fail; the first-pass version would have false-passed by largely measuring its own fix.
No linked issue
Related