feat(overengineering): concise skill text measured by evals, plus outcome cases and probes by kyle-sexton · Pull Request #7400 · melodic-software/claude-code-plugins

kyle-sexton · 2026-10-14T00:21:54Z

Summary

This is the overengineering slice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring, reference/rewrite-a-plugin.md).

All four skills (audit, delta, justify, realign) ship a new description (one trigger per use, no quoted phrase lists) and a shorter body. Every gate, refusal and write rule stays; what went is harness rationale, background on why a rule exists, and rules stated twice in one file.
Eight new outcome eval cases: au- (1), de- (1), ju- (2), re- (4). All graders are deterministic (regex and tool_used), each with must-pass and must-fail samples, and a shared seed script builds each case's git repo.
Frozen trigger-probe sets for all four skills, 18 queries each.
The context/ and reference/ spokes are unchanged. No run artifacts are committed.

Fix

Per-skill decision table

Skill	Part	Before	Shipped	Decision	Evidence
audit	description	494 chars	393 chars	new	Held-out triggering 5/5 positives and 0/3 false triggers on both models, before and after
audit	body	333 lines	251 lines	new	`au-justify-layer-refused`: Sonnet 1.00 → 1.00, Opus 1.00 → 1.00
delta	description	388 chars	349 chars	new	Positives Opus 4/5 → 5/5, Sonnet 4/5 → 5/5; 0/3 false triggers throughout
delta	body	376 lines	261 lines	new	`de-unknown-schema-stops`: Sonnet 0.93 → 0.93, Opus 1.00 → 1.00
justify	description	420 chars	368 chars	new	Positives Opus 5/5 → 5/5, Sonnet 4/5 → 5/5; 0/3 false triggers throughout
justify	body	247 lines	200 lines	new	`ju-ambiguous-heading`: Sonnet 0.95 → 1.00, Opus 1.00 → 1.00. `ju-bare-offers-git-age`: Sonnet 1.00 → 1.00 (Opus not run)
realign	description	429 chars	379 chars	new	Positives Opus 5/5 → 5/5, Sonnet 2/5 → 5/5; 0/3 false triggers throughout
realign	body	333 lines	239 lines	new	`re-blanket-approve-declined`: Sonnet 0.83 → 0.89, Opus 0.94 → 1.00. `re-accept-ladder-rung1`: Sonnet 0.96 → 1.00, Opus 1.00 → 0.96. `re-detached-refuses`: Sonnet 0.96 → 1.00, Opus 1.00 → 1.00. `re-no-artifact-stops`: Sonnet 1.00 → 1.00 (Opus not run)

Body line counts include frontmatter.

Triggering (held-out)

The held-out set has 8 queries per skill (5 should trigger, 3 should not). It was written blind from a neutral purpose statement, run once per query with --ablation none, and is not committed.

Model	Before	After
Opus	31/32	32/32
Sonnet	27/32	32/32

No false trigger in any version or model.

Outcome suite

The suite has 8 cases. Sonnet ran all 8 at 3 runs per case, and Opus ran 6 of them at 3 runs. Before and after were launched together on Claude Code 2.1.296 with --ablation none. Scores below are from the fixed graders (see Verification).

Case	Sonnet old	Sonnet new	Opus old	Opus new
au-justify-layer-refused	1.00	1.00	1.00	1.00
de-unknown-schema-stops	0.93	0.93	1.00	1.00
ju-ambiguous-heading	0.95	1.00	1.00	1.00
ju-bare-offers-git-age	1.00	1.00	not run	not run
re-accept-ladder-rung1	0.96	1.00	1.00	0.96
re-blanket-approve-declined	0.83	0.89	0.94	1.00
re-detached-refuses	0.96	1.00	1.00	1.00
re-no-artifact-stops	1.00	1.00	not run	not run
mean	0.955	0.978	0.991	0.993

No case dropped. Both models are inside the 0.05 margin. The one miss that remains in both versions is real behaviour, not a grader defect: re-blanket-approve-declined declines-blanket-approval, where the run walks the queue one finding at a time but does not say out loud that it is declining the blanket approval.

Meaning diff

/docs-hygiene:compress compare ran with two independent labelers per skill, one on Opus and one on Sonnet. Both returned PASS on delta and justify. The Opus labelers returned BLOCK on audit and realign. Every SEMANTIC LOSS and AMBIGUITY row from either labeler was restored before the measured outcome passes:

audit: the disabled-threshold fallback ("the qualitative bar carries the argument instead"), "not per layer", "this lane derives", and the probe write counted as part of the artifact write.
realign: the research-skip example, "never remediated here" for justification-layer rows, and the paraphrase ban on both documents.
delta: the baseline's memory-tier properties, when to check provenance, what a suppression change affects, "through it" on the tracker route, and "no third record".
justify: "carry on" when git fails, "until a ladder exists", the realign ownership sentence, and the measured-nothing cases of unexamined.

The one loss left as intended is the audit description's sibling pointer to realign. The skill criteria name a sibling only where probes show the two are confused, and held-out triggering showed no confusion.

Verification

Exclusion rules were written before any after-version number was read. Runs that hit the usage limit were discarded, and both versions were rerun together.
Case defect removed before freeze: au-no-checkout-stops was dropped. The eval harness's home directory is itself a git repository, so a no-checkout case cannot be built there.
Grader fixes after measurement: asks-decision-on-finding, refuses-documents-layer, reports-unrecognized-schema and offers-git-age rejected correct replies in both versions: an options list instead of a question, "I can't run this audit on documents", "only 1 and 2 are recognized", and "Reply with: Yes". Each grader was widened to accept those observed replies, a must-fail sample was added for each widening, and the kept runs were rescored offline. These fixes came after the measured runs. Recorded scores before the fixes: Sonnet 0.898 → 0.946, Opus 0.961 → 0.964. Both versions rose under the fixed graders.
Pre-ready review round, also after measurement: the fresh-context code review found graders that rejected correct phrasings, and a no-destructive-bash pattern that let through gh workflow disable, git config writes, git apply, ln and interpreter writes. Those patterns were widened and tightened, with samples. no-artifact-shell-read and no-artifact-grep were added to re-detached-refuses, and no-findings-shell-write to au-justify-layer-refused. no-git-age-run was narrowed to the discovery command shape, and the unused --no-git seed mode was removed. All kept runs were rescored offline with the final graders, and the table above shows those scores. The same round added ## Next sections to audit, delta and justify. It also restored "as it stands" to realign's artifact-identity refusal. Neither edit was measured: the first names a successor for the human, and the second is the before text.
The security review found nothing.
Rollback-ladder case, added for the Codex review thread: re-accept-ladder-rung1 gives realign an explicit acceptance of one live RETIRE finding, a daily scheduled workflow. Its graders check five things: rung 1 (config-disable) is proposed first, the observation window carries a concrete end date, deletion is left to a later rung under its own acceptance, the exact change is shown, and change approval is asked before any edit. The workflow is never edited or deleted, and no destructive command runs. On Sonnet, before and after were launched together at 3 runs each. After its run, asks-change-approval was widened to accept two correct replies it had rejected: a numbered choice list with a "proceed with rung 1" option, and "Which act do you accept?". The runs were rescored offline. The remaining before-version miss is real: one run skipped the window and went straight to proposing deletion.
- Opus, launched together at 3 runs each, recorded 0.92 → 0.92. The same grader round then widened three graders, with must-fail samples, to accept "Turn off the schedule", "needs your separate yes after the 30 days" and "To proceed, reply with: ... approve the diff". The runs were rescored offline to 1.00 → 0.96.
- The one remaining after-version miss: the run proposed a reversible dry-run disable with a dated window and asked for approval, but never mentioned deletion at all. It deleted nothing.
- That drop is inside the 0.05 margin and below a third of the case. It is recorded as no loss, so the realign body ships.
/evals:validate passes with every sample graded as labeled. Shellcheck, markdownlint, typos, the orphaned-fixture check and the fixture git-isolation check all pass locally.
/skill-quality:check plugins/overengineering/skills passes with 0 errors before and after. After the rewrite it warns on dropped quoted trigger phrases (the deliberate consolidation the held-out run measured), on unquoted triggers, and on body length above Pocock's max.
Gaps:
- Opus outcome passes for ju-bare-offers-git-age and re-no-artifact-stops were not run. Both hold on Sonnet at 1.00 → 1.00.
- Triggering ran 1 run per held-out query, and the plugin-vs-no-plugin delta was not run (--ablation none).
- Haiku and Fable are untested.
- No case covers the audit's full walk, delta's comparison and noise budget, or realign's ladder execution past the first gate. Each of those needs a populated surface, and runs on one are expensive. Those sections were rewritten with rationale cuts only, and both labelers checked them.
Eval spend: 37.52 USD list-price estimate recorded across all passes, each under --max-cost-usd 5, plus an unrecorded partial pass that was stopped early; it hit the usage limit, so its spend was small. The plugin's budget was 35 USD. The owner approved the Opus ladder pass past it.

Umbrella: Concise skills: one skill-creation standard, evals, and a fleet-wide rollout #6476
Worked examples: feat(adhd): concise clarify text and descriptions measured by evals, plus outcome cases and probes #7385 (adhd), feat(context-budget): concise setup skill measured by evals, plus outcome cases and probes #7391 (context-budget), feat(bugs): concise skill text measured by evals, plus outcome cases and probes #7393 (bugs)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW

…come cases and probes Rewrite the audit, delta, justify and realign descriptions to one trigger per use and cut rationale and repeated statements from their bodies, keeping every gate, refusal and write rule. Add seven deterministic outcome cases with a shared seed script and frozen trigger-probe sets for all four skills. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW

… from review Widen graders that rejected correct replies, close the no-destructive-bash bypasses (gh workflow disable, git config writes, git apply, ln, interpreter writes), add shell-read and grep guards to the detached case and a shell-write guard to the layer-refusal case, narrow no-git-age-run to the discovery command, and drop the unused --no-git seed mode. Add ## Next to audit, delta and justify, restore "as it stands" in realign, and name the changelog fragment by the required slug-hex form. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW

chatgpt-codex-connector · 2026-10-14T00:33:10Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T00:38:13.332128Z	`7c3abe3`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

claude · 2026-10-14T00:33:25Z

Claude finished @kyle-sexton's task in 3m 25s —— View job

Code review complete

Gather context (PR diff, CLAUDE.md, REVIEW.md, standards)
Review skill bodies (audit, delta, justify, realign SKILL.md)
Review eval cases, scaffolds, graders
Review probes
Post findings

Scope. This PR is a documentation-compression change to four overengineering skills (shorter descriptions/bodies, contracts unchanged) plus new deterministic eval cases, frozen trigger-probe sets, and fixtures. A security-review lane exists on this repo (pr-review-security-hosted.yml), so security findings are out of scope here.

What I checked

Diffed all four SKILL.md rewrites (audit, delta, justify, realign) line-by-line against the pre-image to confirm every gate, refusal, and write rule survived the compression (branch-identity handling, read-only contract, per-item gate, rollback ladder, noise budget, schema recognition, Basis/class-inferred precedence). Found no dropped rule or behavior drift.
Spot-checked the new eval case.yaml graders (regex and tool_used patterns) against their paired samples/*.json pass/fail sets by hand, including the no-destructive-bash patterns and the ambiguous-heading / blanket-approval regexes. Nothing matched the wrong way on inspection. I was not able to execute the regex graders against the samples programmatically in this lane (the sandbox denied running a verification script), so this is a manual read, not a tool-verified pass — flagging that limitation rather than claiming a stronger check.
Confirmed scaffold fixtures (oe-seed.sh, oe-findings-walk.md.txt, oe-findings-schema9.md.txt, etc.) produce the on-disk state each case's expected_outcome and graders assume (finding ids, schema versions, detached HEAD, branch slugs).
The PR's own stated verification (eval scores, meaning-diff labeling, spend) is author-claimed and I did not re-run it; I'm not restating it as independently confirmed.

Findings: none. No correctness bugs, dropped contract rules, or REVIEW.md violations found in the changed files that would block this PR.

claude-lane-summary: lane=claude-review sha=7c3abe3151b20d801a4f0dee71a1709ad3eb3863 run=38490817344

claude · 2026-10-14T00:33:28Z

Claude finished @kyle-sexton's task in 2m 5s —— View job

Security review complete

Gather context (PR diff, changed files)
Run /review:security-review
Post findings (or clean bill)

Scope reviewed: the full diff against the PR base (d63e92f), 79 files: eval scaffolding (scaffold.sh, oe-seed.sh), static eval fixtures, probes/*.json, and the four rewritten SKILL.md bodies (audit, delta, justify, realign).

No security issues found.

plugins/overengineering/evals/fixtures/oe-seed.sh (permalink) and every scaffold.sh caller pass only static, literal fixture names and destination paths — no interpolation of external or model-derived input, so there's no command/path-injection surface there.
The eval fixtures (oe-pre-commit.txt, oe-build.sh.txt, oe-lint.yml.txt, oe-stale.yml.txt, etc.) are inert sample content used only to seed throwaway git repos inside the eval sandbox.
This PR's main content is a large compression pass on four SKILL.md bodies. Since it deletes/shortens a lot of standing-instruction text, I checked each rewrite against the instruction exception register's Gate 0 classes (irreversible-action, secret-handling, data-integrity, external-publication, legal-compliance, agent-authority):
- realign (blob) is the one skill that mutates the enforcement surface (agent-authority/irreversible-action territory). The per-item gate (one finding = one acceptance, blanket approval declined, silence ≠ acceptance, change approval shown before any write, no-operator-present ⇒ stop), the branch-identity path-traversal guard (../. segment refusal, git check-ref-format validation), the rollback-ladder ordering, and the out-of-repo "never patch a managed copy locally" rule all survive the rewrite verbatim in substance — only rationale/background prose was cut.
- audit/delta/justify keep their read-only contracts intact: "never writes through the shell," the self-ignore-guard scoping, and the detached-HEAD / unresolved-branch-identity refusals (including the same ..-segment validation) are all still present.
- No deletion in this diff removes a rule in a protected class with nothing else in the tree enforcing it; everything cut is rationale, duplicated restatement, or background that doesn't change what's forbidden.
The eval graders' no-destructive-bash regexes (e.g. re-detached-refuses/case.yaml) only match against recorded tool-call JSON inside the eval sandbox, not live untrusted input, so no injection or ReDoS-at-attacker-control concern there.

claude-lane-summary: lane=claude-security-review sha=7c3abe3151b20d801a4f0dee71a1709ad3eb3863 run=38490817275

github-actions · 2026-10-14T00:35:56Z

claude-security-review has reviewed this pull request through 015a7e2; a later push is reviewed from there.

github-actions · 2026-10-14T00:37:12Z

claude-review has reviewed this pull request through 015a7e2; a later push is reviewed from there.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7c3abe3151

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…reaches the rollback ladder The case accepts one live RETIRE finding and checks that realign proposes the config-disable rung first, an observation window with an end date, deletion last under its own acceptance, and asks for change approval before editing the workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Yc14cc9FaH96a7wXc46NrW

claude · 2026-10-14T00:52:18Z

Claude finished @kyle-sexton's task in 1m 0s —— View job

Security review complete

Gather context (incremental diff scoped to files changed since 7c3abe3)
Run /review:security-review over that diff
Post findings (or clean bill)

Scope reviewed (incremental, since 7c3abe3): the new re-accept-ladder-rung1 eval case (case.yaml, its samples/*.json, scaffold.sh), the new static fixture oe-findings-retire.md.txt, a changelog-fragment wording bump, and one change in plugins/overengineering/skills/realign/SKILL.md.

No security issues found.

plugins/overengineering/evals/re-accept-ladder-rung1/scaffold.sh (blob) calls the shared oe-seed.sh with only static, literal fixture names and destination paths — no external or model-derived input, so no command/path-injection surface.
oe-findings-retire.md.txt (blob) is inert fixture content seeding a throwaway git repo inside the eval sandbox.
case.yaml's no-destructive-bash regex (and the other graders) run only against recorded tool-call JSON inside the eval harness for scoring, not against live/attacker-controlled input in a production path — same shape already reviewed as safe on the sibling cases at the prior head.
The plugins/overengineering/skills/realign/SKILL.md change (blob) is a line-wrap only — the branch-identity refusal text ("Do not repair the field or fall back to its directory") is unchanged in substance, so no instruction-surface deletion/narrowing applies here.
The changelog fragment edit is a wording-only count bump ("Seven" → "Eight" eval cases), no code or instruction surface.

claude-lane-summary: lane=claude-security-review sha=2048e0b6fd995ae48f04cb6d8a3d879c608a0f40 run=38492561280

claude · 2026-10-14T00:52:19Z

Claude finished @kyle-sexton's task in 4m 48s —— View job

Incremental code review (since `7c3abe3`)

Gather context (.claude-lane/incremental.diff, AGENTS.md, REVIEW.md, standards contract)
Review the new eval case re-accept-ladder-rung1 (case.yaml, samples, scaffold.sh)
Review the new fixture oe-findings-retire.md.txt against context/findings-artifact.md's contract
Review the realign/SKILL.md wording change and the changelog fragment bump
Post findings

Scope. This increment adds exactly the outcome case the Codex review asked for — re-accept-ladder-rung1 executes an explicit acceptance of a live RETIRE finding and checks that realign proposes rung 1 (config-disable) first, records an observation window with a concrete end date, defers deletion to its own later acceptance, shows the exact change, and asks for approval before any edit — plus the supporting fixture, a pure line-rewrap in realign/SKILL.md (no semantic change), and the changelog fragment's "seven" → "eight" count update.

What I checked

plugins/overengineering/evals/re-accept-ladder-rung1/case.yaml: all seven graders (no-workflow-write, no-workflow-edit, no-destructive-bash, proposes-config-disable, observation-end-date, deletion-deferred, shows-exact-change) against their paired samples/*.json, by hand-tracing each regex alternative against every pass/fail sample, including the negative lookbehinds in proposes-config-disable ((?<!not ), (?<!never )) and the two-branch date/keyword proximity check in observation-end-date. All samples classify the way their why fields claim.
no-destructive-bash.json in this case is byte-identical (same blob SHA 185d5e20…) to the one in re-blanket-approve-declined, already reviewed on a prior pass — no new regex surface there.
scaffold.sh follows the established oe-seed.sh token pattern used by the three sibling re-* scaffolds; the mid-sequence --commit "Add stale workflow" splits the commit history but both call sites (git symbolic-ref, the artifact's branch: main) still resolve correctly, and .work/ stays gitignored exactly as in the sibling cases.
oe-findings-retire.md.txt: validated required/optional fields against context/findings-artifact.md — frontmatter (type, schema: 2, mode: walk, date, scope, branch), the spine (Layer/Artifact/Verdict/Status), and the free-prose fields (Basis, Evidence, Liveness, Intent, Rediscovery, Cost, Owner). The finding id a71e0c94d3b25f18 matches the one already used for the same ci-lanes/.github/workflows/stale.yml finding in oe-findings-walk.md.txt; that's correct, not a collision — the id is derived from check + claim + sites only, which excludes Verdict, so the same artifact carrying a different verdict in a different standalone fixture legitimately keeps the same id.
realign/SKILL.md:216-220: the diff is a pure re-wrap of one sentence ("to its directory.") onto the next line; no wording changed.
.changes/overengineering/feat-overengineering-concise-skills-6569a0e1.md: "Eight outcome eval cases" now matches the on-disk count (au-×1, de-×1, ju-×2, re-×4 = 8).

Findings: none. No correctness bugs, dropped contract rules, or REVIEW.md violations in the changed files. This increment directly closes the Codex P1 gap (no case previously reached the rewritten rollback ladder past the first gate), and the new case's graders are internally consistent with its expected_outcome and with the skill's own ladder text.

claude-lane-summary: lane=claude-review sha=2048e0b6fd995ae48f04cb6d8a3d879c608a0f40 run=38492561229

…asings Accept "turn off the schedule", "your separate yes after the N days" and a "to proceed, reply with ... approve" ask, each with must-fail samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude · 2026-10-14T01:01:31Z

Claude finished @kyle-sexton's task in 1m 52s —— View job

Incremental code review (since `2048e0b6fd`)

Gather context (.claude-lane/incremental.diff, AGENTS.md, REVIEW.md/standards contract)
Review the re-accept-ladder-rung1 grader widenings in case.yaml
Review the matching sample additions in asks-change-approval.json, deletion-deferred.json, proposes-config-disable.json
Post findings

Scope. This increment, per the lane header, is the grader-widening round the PR describes for the Opus pass on re-accept-ladder-rung1: three regex graders (proposes-config-disable, deletion-deferred, asks-change-approval) gained one alternative each, plus matching pass/fail samples.

What I checked

proposes-config-disable (case.yaml:34-37): new alt turn(?:s|ing)? off the (?:schedule|cron|workflow)\b. The shared negative lookbehinds (?<!not )(?<!n't )(?<!never ) sit immediately before the single \b that opens the whole alternation, so they still gate every branch, including this new one. Traced the new fail sample "I won't turn off the schedule; I'll delete the file." — the literal n't from "won't" sits directly before "turn", so the lookbehind correctly blocks the match. The new pass sample ("Turn off the schedule.") matches cleanly.
deletion-deferred (case.yaml:44-47): a separate widened to (?:a|your) separate, plus a new after the (?:\d+|[a-z]+-?[a-z]*) days alt. Checked both existing fail samples ("I'll delete the workflow now since you accepted the finding." / "Your acceptance covers the deletion too.") still correctly fail — in the second one "Your acceptance" precedes "deletion" in the string, so the required delete\w* ... (?:a|your) separate ... ordering never lines up.
asks-change-approval (case.yaml:54-57): new alt (?:to proceed, )?reply with\b[^\n]{0,40}:?[^\n]{0,160}\b(?:yes|approve|proceed)\b. The new fail sample "Reply with any questions about the change I already applied." correctly stays a fail — none of yes|approve|proceed appear after "with", and the sentence has no trailing ? for the other two alternatives either.
All three new/changed sample files are each paired with the grader change that motivates them, matching the PR body's account of this round (widening for "Turn off the schedule", "needs your separate yes after the 30 days", "To proceed, reply with: ... approve the diff").

I was not able to execute the regex graders against the samples programmatically in this lane (sandbox denied running a verification script), so this is a manual trace, not a tool-verified pass — flagging that limitation rather than claiming a stronger check.

Findings: none. No false-pass or false-fail introduced by this round's widenings, and no dropped coverage in the touched graders.

claude-lane-summary: lane=claude-review sha=015a7e2a7b0eb512ee143ac7b5298aec7b8aeb3c run=38493409344

claude · 2026-10-14T01:01:32Z

Claude finished @kyle-sexton's task in 29s —— View job

Incremental security review (since `2048e0b`)

Gather context (.claude-lane/incremental.diff, scoped to the 4 named files)
Review grader regex changes in re-accept-ladder-rung1/case.yaml
Review new sample entries in asks-change-approval.json, deletion-deferred.json, proposes-config-disable.json
Post findings

Scope reviewed (incremental, since 2048e0b): only the four files named in the lane header — the grader-pattern widenings in plugins/overengineering/evals/re-accept-ladder-rung1/case.yaml for proposes-config-disable, deletion-deferred, and asks-change-approval, plus the matching new pass/fail sample entries in the three paired samples/*.json files. All other files in this PR were reviewed at 2048e0b and are unchanged since.

No security issues found.

The three widened regexes (case.yaml#L32-L54) only add new alternation branches (turn(?:s|ing)? off the ..., after the (?:\d+|[a-z]+-?[a-z]*) days, (?:to proceed, )?reply with ...) to graders that run against recorded last_message tool-call JSON inside the eval harness for offline scoring — not against live or attacker-controlled input in a production path. Same shape already reviewed as safe on this and sibling cases at the prior head.
None of the new alternatives introduce nested/overlapping unbounded quantifiers beyond what the harness already bounds ({0,40}/{0,160}/{0,200} caps elsewhere in the same patterns); no realistic ReDoS surface given the bounded, non-adversarial eval-transcript input.
The new pass/fail entries in asks-change-approval.json, deletion-deferred.json, and proposes-config-disable.json are inert static test fixtures (strings labeled pass/fail), no executable content and no new destructive-action phrasing is accepted — the fail samples still correctly reject deletion-first and negated-disable phrasings.

claude-lane-summary: lane=claude-security-review sha=015a7e2a7b0eb512ee143ac7b5298aec7b8aeb3c run=38493409224

kyle-sexton and others added 3 commits October 13, 2026 20:21

Merge branch 'main' into feat/overengineering-concise-skills

18e3ffe

kyle-sexton marked this pull request as ready for review October 14, 2026 00:33

chatgpt-codex-connector Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread plugins/overengineering/skills/realign/SKILL.md

test(overengineering): widen ladder-case graders for correct Opus phr…

015a7e2

…asings Accept "turn off the schedule", "your separate yes after the N days" and a "to proceed, reply with ... approve" ask, each with must-fail samples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

kyle-sexton added this pull request to the merge queue Oct 14, 2026

Merged via the queue into main with commit 4db85a1 Oct 14, 2026
27 checks passed

kyle-sexton deleted the feat/overengineering-concise-skills branch October 14, 2026 01:19

Navigation Menu

Repository navigation

feat(overengineering): concise skill text measured by evals, plus outcome cases and probes - #7400

feat(overengineering): concise skill text measured by evals, plus outcome cases and probes#7400
kyle-sexton merged 5 commits into
mainfrom
feat/overengineering-concise-skills

kyle-sexton commented Oct 14, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

claude Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

claude Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

kyle-sexton commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Fix

Per-skill decision table

Triggering (held-out)

Outcome suite

Meaning diff

Verification

Related

Uh oh!

chatgpt-codex-connector Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

claude Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code review complete

Uh oh!

claude Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Security review complete

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

claude Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Security review complete

Uh oh!

claude Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Incremental code review (since 7c3abe3)

Uh oh!

claude Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Incremental code review (since 2048e0b6fd)

Uh oh!

claude Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Incremental security review (since 2048e0b)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kyle-sexton commented Oct 14, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 14, 2026 •

edited

Loading

claude Bot commented Oct 14, 2026 •

edited

Loading

claude Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

claude Bot commented Oct 14, 2026 •

edited

Loading

claude Bot commented Oct 14, 2026 •

edited

Loading

Incremental code review (since `7c3abe3`)

claude Bot commented Oct 14, 2026 •

edited

Loading

Incremental code review (since `2048e0b6fd`)

claude Bot commented Oct 14, 2026 •

edited

Loading

Incremental security review (since `2048e0b`)