feat(claude-config): audit the model-configuration settings the harness accepts but declines to honor - #1923
Conversation
…reached The `audit` skill advertises settings-file correctness against current official docs, and categories A-G reach schema, permissions, MCP servers, hooks, plugins, environment variables, and the skill-listing budget. None reached the model-configuration keys, so `effortLevel`, `fallbackModel`, `availableModels`, and `enforceAvailableModels` went unaudited inside the skill's own stated scope. Category H adds four rows sharing one shape: each names a value the harness accepts into the file and then does not apply as its author expects. `effortLevel: max` is not the level that persists; a fourth `fallbackModel` entry risks being ignored; a specific model listed beside its own family wildcard narrows the allowlist to that one version; `enforceAvailableModels` without a non-empty `availableModels` does nothing at all. Phase 2's enumeration, `validation-categories.md`, and a mandatory Phase 3 model-config fetch move with it, so the skill's own flow reaches the category. Three rows are `warning`; the `enforceAvailableModels` pairing is `error`, because this skill's own severity guide rates an enforcement bypass that way and that is what the finding is - an administrator who set the key believes the Default option is constrained when it is not. That row reports the pairing the page states, not the placement: both keys belong in the highest-precedence managed source, and no file this skill reads can decide whether they are there. How loudly each surfaces differs, so the rows say so individually rather than the section claiming a blanket silence - false for the allowlist row, where a narrowed alias shows a substitution notice naming both models. Two rows also have an authoring-time path: the declared schema constrains `effortLevel` by `enum` and `fallbackModel` by `maxItems`. They stay because the schema is advisory and the harness reads a file that violates it, and where the two authorities disagree they are reported separately - `maxItems` caps raw array length while the page caps after deduplication, so a four-entry chain holding one duplicate fails the schema and satisfies the harness. Rows were admitted on one test - the key must appear in a file this skill actually opens. That excluded the managed-source placement rule, which no local file can decide. `modelOverrides` key validation is excluded separately and stated as a note: validating a key means resolving it against Models overview, whose own restatement deferral governs new consumers of its facts. The separate frontmatter-`effort` lint stays deferred and untouched. Category H differs from it on three counts - a different key, a different file, and a host that already exists - which is what that deferral permits. The rows restate the session-only effort semantics rather than citing this repo's philosophy doc, because this checklist ships into consumer repositories where that doc is absent and the citation would resolve to nothing. They also quote the page verbatim where A-F restate their sources inline; that departure is deliberate, and the Phase 3.3 fetch is what keeps the quotes honest. Also fixes a false claim the new category exposed: `I17`'s `effortLevel: max` carve-out justified itself with the settings schema making the value "unreachable there". The schema is advisory, the value is writable, and the harness reads a file that violates it - which is why category H checks for it. The carve-out is unchanged and still correct; it now rests on the editor-catches-it reason and points at the category owning the file-level check. The loop-lane convention's model-alias claim carried a 2026-07-23 verification stamp predating Opus 5 GA and v2.1.219. Re-verified against today's bytes and holding, so the date refreshes in place per that convention's own rule that a no-drift firing takes no changelog entry and no bump. Verified against https://code.claude.com/docs/en/model-config fetched as raw markdown, 82,975 B, MD5 5c9e06c53a3dcd6defe81b0590923a93, and against https://json.schemastore.org/claude-code-settings.json. Every quoted literal and both schema constraints matched by script, with near-miss control probes. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Claude finished @kyle-sexton's task in 5m 17s —— View job Review: PR #1923 — audit the model-configuration settings the harness accepts but declines to honorScope reviewed: all 7 changed files ( Fresh-docs check: Since this PR's new Category H rows lean heavily on exact upstream wording, I independently fetched Cross-checked version/changelog consistency: Findings1. (Moderate) Category H's
It's also worth noting these four values aren't secrets (unlike the 2. (Nit) Garbled sentence in the Category H intro.
reads as two verbs back-to-back ("checks for constrains") — a word/clause got dropped. The parallel sentence in Not flagged, but noted
|
|
Claude finished @kyle-sexton's task in 1m 13s —— View job Security Review — PR #1923
No security issues found. All 7 changed files are documentation/prose and metadata:
None of this touches This PR is a low-risk, docs/skill-content-only change with no security-relevant surface. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 66a85aab78
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Claude has reviewed this PR 2 times. The lane skips further automatic reviews after 5; deleting this comment resets the count. |
…te the enforce check Two defects in the category H that 66a85aa added. The safe-read route could not surface the keys. Category H reads `settings.json` and `~/.claude/settings.json` directly, but the safe-read rule routes `settings.local.json` through `check-structure.sh`, and that helper emitted only environment, permission, and plugin counts. A category H key living only in the local file was therefore invisible, and its defect silently missed - the exact failure the category exists to catch. Counts could not have closed the gap: the allowlist wildcard rule turns on which family each entry names, and the fallback cap turns on entry order. So the helper now reports the four keys by value for that file. They are configuration identifiers - level names, model names, a boolean - not credentials, so the env-value and secret-field guard is untouched, and new assertions prove env values and env key names still never appear in the output. `unset` and `(empty list)` are reported distinctly because they are different findings. Existing output lines are unchanged and the new ones appended; the script's own suite went from 10 checks to 24, all passing, shellcheck clean. The enforcement-bypass row fired on the wrong condition. It keyed on `enforceAvailableModels` being present, so `{"enforceAvailableModels": false}` with no allowlist - someone turning enforcement off deliberately - would have been reported as an `error`. It now requires the flag to be `true` AND the list unset or empty, gating on the value rather than the key. Severity stays `error` for the gated case, which is a real bypass: the administrator believes the Default option is constrained when it is not. Checklist and `validation-categories.md` both name what the helper emits, so the local-file route is documented where an auditor will look rather than implied. Quote fidelity re-run against the same capture, unchanged: all six literals matched, all three near-miss controls absent. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
Claude finished @kyle-sexton's task in 4m 30s —— View job Review: PR #1923 (updated — re-review after synchronize)
Scope reviewed: the diff since the prior review, i.e. commit Prior findings — statusBoth my earlier Moderate finding and the two Codex P2 comments described the same underlying gap, and both are now fixed:
Traced the new jq carefully for edge cases ( Test coverage: new Case 5 (values present, effort/fallback/allowlist/enforce all surfaced, env still counted not dumped, no secret or env-key-name leak) and Case 6 (absent-vs-empty distinction) both use the file's existing Still open (non-blocking)
Not flagged, but noted
Both defects raised in the original review (and matched independently by Codex) are resolved correctly, with test coverage added for the new behavior. Only the pre-existing wording nit remains open. |
|
Claude finished @kyle-sexton's task in 2m 16s —— View job Security Review — PR #1923 (re-run on synchronize)
No security issues found. This job re-ran because the PR was updated with a new commit (
Focused on the two script files as the only ones with executable logic:
The two Codex P2 comments from the prior round (settings.local.json coverage gap, and gating the enforcement check on |
…and its fences (#1924) ## Summary Doc-alignment roster row 241: **Choosing a Claude model and effort level in Claude Code** (claude.com blog) — the post the owner's model-routing lane and two main-tree surfaces cite for the model-vs-effort dial discrimination. First custody baseline taken (article-body-only hash, so site chrome can't fake drift). The roster's premise was corrected en route: the post was *not* "digested nowhere" — planning's session-config already carried a faithful two-knob digest; what it lacked was custody and the post's own qualifiers. **planning 0.27.3** — the "Two orthogonal knobs" section gains: - Its citation, led by the strongest ground available: model-config's own delegation sentence ("For guidance on which model and effort level fit different kinds of work, see [the post] on the blog") — the reference page points *at* the post; the narrowed negative is the supporting ground: no reference page states the try-versus-know **diagnostic** (choosing-a-model *orders* the levers — "Tuning effort is often a better lever than switching models" — and the effort page pairs effort against prompting; ordering a lever is not diagnosing which failure you have). - The post's **context-first prior step** neither knob had: on a wrong answer, examine the provided context before touching a dial — sharpened here because the step is this skill's own product (the Brief *is* the context fix). - Two missing fences: the discriminator is "a starting point, not a hard rule" (the post's figure caption, disclosed as such), and raising effort is sharpest below the model's default. - The ambiguity signal: ambiguity the rounds could not retire argues up; a Brief precise enough to execute from argues down. Both pre-existing main-tree citations verified holding against the live post. Owner-facing findings routed to the dotfiles ledger (not this repo's to edit): the CLAUDE.md lane's dropped "clearly tried" (making the two branches overlap), the Fable-reservation grounds contradiction, the missing context-first step — with one earlier finding partially withdrawn after the verifier surfaced the choosing-a-model source. ## Test plan - Docs-only; markdownlint 0 errors; check-skill PASS identical to base; scripted quote fidelity across four live sources (post, effort, model-config, choosing-a-model) — after the producer caught and fixed its own circular self-match, all spans verbatim with one disclosed editorial elision. - Independent fresh-context Fable verifier (rationale withheld, 8 binary criteria incl. an adversarial test of the load-bearing negative): 7/8 PASS; its one FAIL (the negative stated too broadly) fixed with its prescribed wording, the fix cross-verified verbatim against both live pages before amending. ## Related - No linked issue. - Doc-alignment loop, roster row 241. Predecessors: #1908–#1920, #1922, #1923. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019gaVX25Txd6GXdiu9HEH3X Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
Doc-alignment roster row 18 — the priority head's final row: Model configuration (code.claude.com/docs/en/model-config), the corpus's most-cited harness page, first formally captured here (82,975 B, MD5
5c9e06c5…) with a semantic-stability proof: normalized token streams of the live page and the frozen 2026-07-31 snapshot are identical (10,222 tokens each, control-probed) — retroactively confirming seven earlier rows' model-config-dependent work ran against an unchanged page.claude-config 0.21.7 / criteria 1.15.0:
effortLevel: max/ultracode(not accepted there),fallbackModelbeyond three after dedup (with the raw-maxItems:3schema test reported separately — the two tests genuinely diverge, verified empirically), a specific entry disabling its family wildcard (including the Mantle-ID and custom-model-option arms), andenforceAvailableModelsagainst an unset/empty list (rated error per the skill's own enforcement-bypass rubric). Visibility stated per row from the page's own text (row 3's substitution notice; row 2's documented silence). Category H is wired into the skill's phase flow: "Eight categories", a Category H section, and a MANDATORY Phase 3.3 live model-config fetch gating every H finding.settings.local.jsonrouted through the safe-read path.effortLevel: max"unreachable" in settings — false (JSON Schema is advisory; the harness reads a violating file). The directive is unchanged; the justification now states the true reason and points at the sibling that owns the file-level check, making the two halves of one plugin cohere.Recorded upstream: schemastore's
effortLeveldescription omits Opus 5/Sonnet 5 from its effort-support list — stale against the live page; their fix, flagged because this PR now cites that schema.Test plan
instruction-scan.test.sh46/46;conflict-scan.test.sh41/41; scripted quote fidelity (6 literals exact, 3 near-miss control probes absent); schema constraints script-verified including the no-uniqueItemsfact the divergence rests on.Related
🤖 Generated with Claude Code
https://claude.ai/code/session_019gaVX25Txd6GXdiu9HEH3X