Repository navigation
test(harness-memory): freeze description-trigger probes for stateless and audit - #11933
Conversation
…sured Held-out trigger passes on Opus (16 passes, 192 with-arm runs, 3 runs per query, Claude Code 2.1.316) compared main's descriptions with shorter ones (stateless 620 -> 456 chars, audit 444 -> 383). Neither ships. Each skill has INVALID negative passes: runs that never called Skill were denied Globs of the real home directory, before the decision point. The plan routes that to "INVALID from the eval sandbox itself (#11727): that skill is unmeasured, its description stays main". Bodies and descriptions are byte-identical to main; this commit adds only the changelog fragment (bump: none) for the frozen probe sets. | skill | split | before | after | verdict | |-----------|---------|---------------|---------------|------------------| | stateless | pos | 24/24 fired | 24/24 fired | | | stateless | neg | 24/24 quiet | 24/24 quiet | unmeasured, main | | audit | pos | 24/24 fired | 24/24 fired | | | audit | neg | 21/24 quiet | 21/24 quiet | unmeasured, main | Per query, every pos-val-01..08 fired 3/3 in both versions for both skills. Every neg query held 0/3 in both versions, except audit neg-val-04, which fired 3/3 in both. No query changed majority. Invalid runs (plan rule: denial before the decision), 12 of 192: - stateless neg-val-04: 3 before, 2 after; neg-val-06: 1 before - audit neg-val-03: 3 before, 3 after run-validity 0.13.1 (raw, before the plan's post-decision override): 8 positive passes VALID with warnings, 8 negative passes INVALID. The post-decision denial rule changed 99 runs' verdicts (stateless 43, audit 56). skill-quality check: 2 passed, 0 failed, 4 warnings (same as before). Refs: #6583 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
claude-security-review reviewed Security review: incremental (e5129d0 → 191119e)No security issues found. I read every hunk of the incremental diff in both files that changed.
These files are frozen test data for description-trigger probes. They run nothing, grant no tools, hold no tokens or secrets, and change no trust control, merge hold, manifest or instruction surface. Neither file matches a trusted-path glob. Removing names from a probe's competitor list widens nothing. Under skip-gate item 2, this change has no security-relevant surface. The PR context file contains no embedded text telling the lane to pass the PR, skip a check, drop a finding or post somewhere else. critical-findings: 0 |
|
claude-review reviewed Incremental review (e5129d0 → 191119e): no findingsScope: I read every hunk of
What changed: Each probe's Checked against criteria:
Closing issue #6583: Its Done-when items are about the merged PR body ("a per-skill decision table", "any skill left unmeasured, and why, is named"). The current PR body meets both, and this incremental change touches neither. Not verified: I had no shell, so I did not run |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e5129d05f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Both probe files listed competitors from other plugins, against the same-plugin rule in plugins/skill-quality/probes/template.md. A parallel rewrite in another plugin could then move these scores. Each file now names only its sibling skill. No description changed, and the held-out passes loaded only this plugin, so no decision moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #6583
Summary
Concise-skills rollout for
harness-memory, descriptions only (scope recorded on #6476: #6476 (comment)). Freezes description-trigger probes for both model-invoked skills. Both shorter descriptions were measured but stay at main: the pre-recorded rule leaves them unmeasured.Fix
plugins/harness-memory/probes/{stateless,audit}.json: frozen probes.bump: none).Verification
Held-out trigger probes (16 per skill, written blind, not committed), Opus model and judge, 3 runs per case, decision rule recorded before any pass.
.claudepaths, the eval's own temp root) before any Skill call in negative runs. The plan routes sandbox-caused invalidity to "unmeasured, keep main" (fix(evals): stop outcome passes failing validity on a required user-scope read and on forked skills reporting 0 turns #11727), so neither rewrite ships./skill-quality:check plugins/harness-memory/skills: 2 passed, 0 failed (warnings unchanged from main).Related
🤖 Generated with Claude Code