Repository navigation
feat(session-flow): concise skill text measured by evals, plus outcome cases and probes - #7442
Conversation
…e cases and probes Tighter bodies and shorter descriptions for thirteen session-flow skills, each kept only with before/after eval evidence on Opus and Sonnet. Adds sixteen outcome cases (19 with the three hardened recall cases) and frozen trigger-probe sets for seventeen skills. Refs #6549 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ff eval The fh-confirms-before-resume fixture carries a CANARY-1234 marker labelled as a deploy token so the redaction rule applies; gitleaks' generic-api-key rule matches the assignment shape. Refs #6549 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Claude finished @kyle-sexton's task in 2m 42s —— View job Security review complete
Scope. Diffed Instruction-surface deletions. This PR's stated intent is to cut rationale/history prose while keeping every gate, stop, command, and report line. I checked each removed
None of these removals drop an enforced control — they're compressions of duplicate prose, consistent with the PR's own claim and the meaning-diff labeler's PASS result. Eval fixtures / secrets. The new Result: no security findings. This PR is text-only (skill prose trims) plus synthetic eval fixtures; no new code paths, auth, data handling, or Actions config changed. claude-lane-summary: lane=claude-security-review sha=f9472e9d43765ce68e0ad7619e8e1d76cf4d68e4 run=38517529849 |
|
Claude finished @kyle-sexton's task in 5m 9s —— View job Review complete
Scope: the What I checked:
No findings to report. The known grader limitations ( claude-lane-summary: lane=claude-review sha=f9472e9d43765ce68e0ad7619e8e1d76cf4d68e4 run=38517529812 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9472e9d43
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
claude-security-review has reviewed this pull request through 3a99a74; a later push is reviewed from there. |
|
claude-review has reviewed this pull request through 3a99a74; a later push is reviewed from there. |
…vailable recall source recall-asks-before-other-projects adds a tool_used grader that fails a Read of another project's transcript under .claude/projects. recall-reports-tried-kept-reverted splits unavailable-source-named into unavailable-prs-named and unavailable-tracker-named, so naming one unavailable source no longer covers the other, and an outcome word in a neighbouring sentence no longer counts. Both land with must-pass and must-fail samples. The ho-stops-after-resume-prompt fixture comment now reads "next:" in place of a TODO marker, which the repository's comment-marker scan rejects. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Gtv6du4Bzd9KmWtHnjc78i
|
Claude finished @kyle-sexton's task in 1m 31s —— View job Security review complete
Scope. Per the lane's incremental-review instructions, I reviewed only what changed since this lane's last review at
What changed:
Instruction-surface lens. None of these changes delete, narrow, or soften a standing instruction ( Verified by hand:
Result: no security findings. This increment is test/eval-only (fixture comment rename, grader-regex tightening, new grader samples) — no production code path, auth/access-control logic, data handling, or Actions config changed. The grader changes strengthen (not weaken) detection of the cross-project transcript consent boundary the claude-lane-summary: lane=claude-security-review sha=3a99a74c6cbcb618e91f954056303c7dfad03ce9 run=38574602857 |
|
Claude finished @kyle-sexton's task in 3m 1s —— View job Review complete
Scope (incremental, reviewed only what changed since What I checked:
No findings. claude-lane-summary: lane=claude-review sha=3a99a74c6cbcb618e91f954056303c7dfad03ce9 run=38574602839 |
1 participant
Refs: #6549
Refs: #6476
Summary
This is the
session-flowslice of the concise-skills rollout, run through the per-plugin runbook (/playbooks:skill-authoring). The protocol: before and after on Opus and Sonnet, outcome cases at 3 runs with deterministic graders, held-out triggering at 1 run per query,--ablation none, and--max-cost-usd 5on every pass.clean-stop,find-handoff,handoff,keep-going,orchestrate,orient,reanchor,recall,reconcile,retro,running-retro,show-options,workflow. Total SKILL.md words across the plugin drop 41,956 to 40,049 (-4.5%). The cuts are rationale, history notes and lines repeated elsewhere in the same skill. Every gate, stop, command and report line stays.audit-frictionandcheck(not rewritten);audit-sessionsandcontinue-in-background(the rewrite was within 1% of base);setupandtidy-work(user-invoked, and no outcome case reaches them).evals/, plus the three existingrecall-*cases moved to deterministic graders only. Every grader has must-pass and must-fail samples.probes/.Fix
Per-skill decision table
Sizes are description characters, then SKILL.md lines and words. Outcome scores are means over 3 runs, before → after.
af-: S 0.78 → 0.67, O 0.67 → 0.67 on identical text (grader defect, see gaps)as-: 1.00 → 1.00 on both modelsck-: 1.00 → 1.00 on both modelscs-: S 0.93 → 1.00, O 1.00 → 1.00cb-: 1.00 → 1.00 on both modelsfh-: S 0.67 → 0.72, O 0.83 → 0.83ho-: 1.00 → 1.00 on both modelskg-: S 1.00 → 0.93, O 1.00 → 1.00oc-: S 1.00 → 1.00, O 0.83 → 1.00or-: S 0.83 → 0.61 with the "What this skill does NOT do" section cut (one run re-ran a background job, two did not route recovery tokeep-going). Restored verbatim; rerun with a fresh before control: S 0.83 → 0.83, O 0.94 → 1.00ra-: 1.00 → 1.00 on both modelsrecall-asks: 1.00 → 1.00 both;recall-reports: S 1.00 → 0.83, O 0.92 → 1.00;recall-states: S 0.89 → 1.00, O 0.67 → 0.89rc-: 1.00 → 1.00 on both modelsre-: S 0.40 → 0.40, O 0.53 → 0.47 (both arms fail the same two graders every run, see gaps)rr-: S 0.71 → 0.71, O 1.00 → 0.96so-: S 1.00 → 1.00, O 0.75 → 0.75wf-: S 0.67 → 0.67, O 1.00 → 0.78 (traces read: both failing after runs namesession-flow:clean-stopwithout the leading slash, so the measured grader misses them; behaviour matches)Meaning diff
A fresh-context labeler (
docs-hygiene:compare-labeler, Opus) compared each rewritten skill directory against base text extracted fromorigin/main. All 17 rewritten skills came back PASS: no SEMANTIC LOSS, AMBIGUITY or EMBEDDED INSTRUCTION rows. RELOCATED rows point toreference/gather.md,orient/reference/native-recap.md,retro/reference/native-insights.mdandorchestrate/context/gotchas.md, each checked this session.Verification
All runs on Claude Code 2.1.299, with before and after launched together. Eval spend (list-price estimate): triggering 43.47 USD, outcome 52.10 USD.
Outcome suite (3 runs each)
Sonnet's
as-andcb-after arms measured the rewrites that were then reverted; Opus ran after the reverts, so its after arm for those two cases is the shipped (base) text.Before measurement, every grader was audited for negations before a name, single matches where several facts are required, missing shell-write bypass forms (
cp,mv,install,dd,sed -i, redirects, pipe-to-shell), forbidden reads, stray "no X" matches, and patterns too narrow for either version's wording. Each fix landed with must-pass and must-fail samples, and the suite was frozen (sha256 digest1f9b14e16614) before any run.validate-cases.pyexits 0.After measurement, review tightened two
recall-*graders:recall-asksaddsno-cross-project-read-tool(aReadof another project's transcript), andrecall-reportssplitsunavailable-source-namedintounavailable-prs-namedandunavailable-tracker-named, so each unavailable source needs its own outcome. Theho-fixture'sTODOcomment becamenext:for the comment-marker scan. The new suite digest is92329b997f4c. The recall rows above were measured under the earlier graders; they are not re-measured yet (see Gaps).Triggering (held-out)
6 queries per skill (4 should trigger, 2 should not), written blind from a neutral purpose statement, 1 run per query, not committed.
Misses:
continue-in-backgroundpos-val-01 (Opus before, Sonnet both arms),running-retropos-val-04 (Sonnet before),workflowpos-val-02 (Sonnet after) and pos-val-04 (Sonnet both arms).Gaps
af-rejects-bad-tokenmisses "--apply-fixesisn't an argument this skill accepts" (audit-friction text is identical in both arms);wf-names-clean-stopmisses a baresession-flow:clean-stop;or-cites-ttl-decisionmisses some 90s/300s phrasings. These would be fixed in a follow-up so the frozen suite stays as measured.recall-*graders need a before/after re-run on Opus and Sonnet. It is blocked on this host:claude plugin evalrefuses Bash-granting cases because~/.docker/contextsis now a symbolic link, and the earlier traces were already removed, so no offline rescore is possible.re-andfh-fail the confirm-before-acting graders in both arms on most runs: the models act before asking under the case's pressure. The rewrite does not change that.Local checks
/skill-quality:checkon the plugin: 19 passed, 0 failed.typos,markdownlint-cli2,lychee --offline,check-changelog-fragments.sh, and gitleaks (with three commit-less.gitleaksignoreentries for the CANARY deploy-token fixture) are clean.Related
🤖 Generated with Claude Code