feat(claude-memory): /init-then-prune eval fixture graded against C1/C2/C5 (#2989)

[kyle-sexton]

Closes #2989

Summary

Adds the /init-then-prune eval fixture to claude-memory:audit's eval suite (AI Hero course lane 9, #2911): a static bloated CLAUDE.md in the shape /init produces, graded against expected audit findings — including the discrimination the C5 navigation-pointer carve-out (#2987 / PR #3009) introduced.

Fix

• New fixture plugins/claude-memory/skills/audit/evals/fixtures/init-bloated-claude-md.md — a fictional TypeScript-monorepo CLAUDE.md with 212 visible lines: a file-by-file Repository Structure inventory and per-file Package Details prose (C5 codebase-description material), restated standard conventions in Code Style Guidelines and Git Workflow (C2 deletion-test material), a copied Express Middleware Reference section (C5 framework-doc material) — plus deliberate KEEP content so the eval grades discrimination rather than blanket flagging: a curated runbook pointer (docs/runbooks/2025-11-retry-storm.md, "Where to Look First") that must survive under C5's navigation-pointer carve-out, and a non-obvious pnpm build:proto first-run gotcha that must not be flagged. A static fixture was chosen over live /init generation for determinism (lane 9 decision, docs/upstream/aihero-steering-lanes.md).
• New eval case init-shaped-bloat-graded-with-c5-carve-out (id 12) in evals/evals.json, referencing the fixture via files[]. Expectations pin C1 (FAIL over the 200-visible-line target), C2 (standard-convention lines flagged, grouped by section per C2's collapse rule), C5 (inventory + framework-doc flagging), and the two KEEP discriminators above.
• Version bump claude-memory 0.10.0 → 0.11.0 with the matching CHANGELOG entry.

Verification

• check-jsonschema --schemafile plugins/skill-quality/reference/evals.schema.json plugins/claude-memory/skills/audit/evals/evals.json — ok
• bash plugins/skill-quality/scripts/check-evals-quality.sh …/evals.json — PASS (2 pre-existing WARNs on cases 3 and 11, untouched by this PR; the new case carries files[] so Q4 prose-path does not fire)
• bash scripts/check-orphaned-fixtures.sh --check — no orphans (fixture consumed via files[])
• bash scripts/validate-plugins.sh — all manifests + catalog validated; generate-catalog.mjs --check and generate-cheatsheet.mjs --check in sync
• bash scripts/check-changelog-parity.sh --check / --check-bump origin/main / --check-order — all green
• bash scripts/check-changed-skills.sh origin/main — CHECK-SKILL audit: PASS
• npx markdownlint-cli2 on the fixture and CHANGELOG — 0 issues
• Fixture visible-line count verified: 212 non-empty lines (> 200, C1 FAIL threshold with margin)
• The live model-run of the suite (claude plugin eval) is not executable in this container; the repo's CI eval gates (schema + quality lint) are the checks exercised here and both pass

Related

• Refs claude-memory: aihero course lane 9: steering-surface validations (navigation pointers, harness claims, auto-memory, /init-then-prune eval) #2911 (lane 9 specification)
• Refs claude-memory: patch audit criteria for navigation pointers — C5 carve-out, C7 stale-highway note, C3 nested CLAUDE.md + @-mention rows, design-smell remediation #2987 (C5 navigation-pointer carve-out this fixture's expectations honor; merged via PR feat(claude-memory): navigation-pointer audit criteria — C5 carve-out, C7 note, C3 rows, C5 fix patterns #3009)

🤖 Generated with Claude Code

https://claude.ai/code/session_01QbfCrj3X9FfGL7VRZYmrn4

---

Generated by Claude Code

[claude]

Claude finished @kyle-sexton's task in 3m 11s —— View job

---

Code review complete

• Load /review:code-review skill instructions
• Gather PR context (diff, CLAUDE.md/AGENTS.md, criteria.md, evals schema, precedent fixtures)
• Review changed files
• Post findings

Summary

Reviewed against the CLAUDE.md/AGENTS.md-linked repo conventions and the /review:code-review high-signal bar (correctness/maintainability only; no security lane applies to this PR).

Verified independently (not just author-claimed):

• Fixture visible-line count: grep -c "." on init-bloated-claude-md.md → 212 non-empty lines, correctly exceeding the C1 200-line FAIL threshold with margin.
• No lines over 100 chars in the fixture (markdownlint line-length would pass).
• The "files": ["evals/fixtures/init-bloated-claude-md.md"] path convention (relative to the skill dir, not evals.json's own dir) matches every other multi-fixture case in the repo (domain-driven-design:curate-language, mcp-tools:audit, testing:audit, etc.) — correct.
• evals.json structurally matches the schema's rich form (id, name, prompt, expected_output, files, expectations) and case 12 slots in cleanly after case 11.
• CHANGELOG.md's ## [0.11.0] entry format (no date header, - **\skill`: title (refs).**bullet shape) matches the## [0.10.0]entry immediately below it, and theplugin.json0.10.0 → 0.11.0 minor bump is correct semver for anAdded`-only change.
• Read reference/criteria.md C1/C2/C5 in full and cross-checked each against the fixture: the Repository Structure inventory, Package Details prose, and Express Middleware Reference section are genuine C5 targets; the Code Style Guidelines (camelCase/const/async-await) and Git Workflow bullets are genuine C2 standard-convention targets; the "Where to Look First" runbook pointer and the pnpm build:proto gotcha are legitimate KEEP discriminators under C5's carve-out and C2's gotcha exception, respectively.

One finding posted inline on CHANGELOG.md:19 — the new entry cites docs/upstream/aihero-steering-lanes.md as the source of the "static fixture over live /init" decision, but that file was dissolved into docs/upstream/aihero-course.md on 2026-08-17 (per that file's own line 268-269) and no longer exists standalone in the repo. Minor, but it's a dead reference introduced by this PR, in a PR whose entire payload is a fixture for testing staleness detection.

Not independently run (outside this lane's allowed tools — Read/Glob/Grep/gh pr * only, no general Bash): check-jsonschema, check-evals-quality.sh, check-orphaned-fixtures.sh, validate-plugins.sh, check-changelog-parity.sh, check-changed-skills.sh, markdownlint-cli2. These are author-claimed and unverified by me; the static checks I could do by hand (line count, line length, JSON structure, path convention) came out clean.

[claude]

Claude finished @kyle-sexton's task in 41s —— View job

---

Security review — skipped (no security-relevant surface)

• Gather context (PR feat(claude-memory): /init-then-prune eval fixture graded against C1/C2/C5 (#2989) #3013 — adds eval fixture for /init-then-prune graded against C1/C2/C5)
• Run /review:security-review per custom instructions
• Post findings

Per /review:security-review's skip gate, this PR is stopped early: the diff has no security-relevant surface.

All four changed files are content/config for the claude-memory:audit eval suite:

• plugins/claude-memory/.claude-plugin/plugin.json — version bump only (0.10.0 → 0.11.0)
• plugins/claude-memory/CHANGELOG.md — changelog prose
• plugins/claude-memory/skills/audit/evals/evals.json — a new static eval case (prompt + expectations)
• plugins/claude-memory/skills/audit/evals/fixtures/init-bloated-claude-md.md — a static, fictional CLAUDE.md fixture (plain prose describing an invented "Harbormaster" repo) used only as eval input text

None of these introduce executable code, GitHub Actions workflow changes, permission changes, secret/credential handling, or any injection surface (command, SQL, path, template). The fixture is inert text consumed as a documentation string by the audit skill's grading harness — it is not parsed as code or executed. No findings.

[github-actions]

Last security-reviewed head: 010b25de83a358e04540c928115cc8f57aa4bce1. On the next push, the relevance gate compares only the commits since this SHA; delete this comment to force a full re-review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b99a703382

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Claude has reviewed this PR 2 times. The lane skips further automatic reviews after 5; deleting this comment resets the count.

[claude]

Claude finished @kyle-sexton's task in 1m 19s —— View job

---

Code review complete

• Load /review:code-review skill instructions
• Gather PR context (diff at new head 010b25d, prior review findings, CLAUDE.md/AGENTS.md conventions)
• Review changed files (focus: commit 010b25d, the fix pushed since the last review round)
• Post findings

Summary

This push (010b25d) is a direct fix-up in response to the two findings raised on the prior head (b99a703): Codex's P2 (case 12 audited a fictional repo the fixture doesn't ship, so C7/C9 would misfire against it) and my own note about a stale docs/upstream/aihero-steering-lanes.md citation. I re-reviewed both changed hunks against their source findings and the current repo state.

Verified independently:

• Codex's C7/C9 scope finding — correctly fixed. evals.json case 12's prompt now explicitly restricts grading to "C1, C2, and C5 only" and states why (fixture ships without the fictional repo it describes, so C7 file-reference currency and C9 command validation can't run meaningfully). A new expectation (line 153) pins that no C7/C9 findings are reported against the fictional paths/commands. This directly closes the collision Codex identified between a full per-check audit and the KEEP discriminators.
• My stale-doc-reference finding — correctly fixed. CHANGELOG.md:19 now cites docs/upstream/aihero-course.md, "Lane 9: steering validations" instead of the dissolved aihero-steering-lanes.md. I confirmed this section exists at docs/upstream/aihero-course.md:328, and line 341 of that file explicitly records "Static fixture chosen over live /init generation (determinism)" — the citation now points at real, accurate content.
• Reviewed the rest of the case-12 diff (expectation wording tightened to "NOT flagged by C5" / "NOT flagged by C2" instead of the prior unscoped phrasing) — consistent with the new C1/C2/C5-only scope, no contradictions introduced.
• evals.json structure around case 12 remains schema-shaped (id, name, prompt, expected_output, files, expectations); nothing else in the file changed.

Not independently run (this lane's tools are Read/Glob/Grep/gh pr * only, no general Bash): check-jsonschema, check-evals-quality.sh, markdownlint-cli2. Author-claimed in the PR description as passing; not reverified by me on this push, but nothing in this diff's shape would newly break them.

No new findings on this push — both prior findings are resolved correctly and no regressions were introduced.

[claude]

Claude finished @kyle-sexton's task in 39s —— View job

---

Security review — skipped (no security-relevant surface)

• Gather context (PR feat(claude-memory): /init-then-prune eval fixture graded against C1/C2/C5 (#2989) #3013 sync event, HEAD 010b25d)
• Invoke /review:security-review and follow its instructions exactly
• Post findings

Per /review:security-review's skip gate, this PR is stopped early: the diff since the last reviewed head has no security-relevant surface.

The only commit since the last security review (b99a703 → 010b25d) touches two files, both non-executable:

• plugins/claude-memory/CHANGELOG.md — corrects a doc citation (docs/upstream/aihero-steering-lanes.md → docs/upstream/aihero-course.md), addressing the earlier Claude review comment.
• plugins/claude-memory/skills/audit/evals/evals.json — narrows eval case 12's prompt/expected_output/expectations to explicitly scope grading to C1/C2/C5 and disclaim C7/C9, addressing the Codex review comment about the fixture shipping without its fictional repository.

Both are prose/config edits to an eval fixture's grading text — no executable code, no GitHub Actions workflow changes, no permission changes, no secret/credential handling, and no injection surface (command, SQL, path, template). This mirrors the prior full-PR review of b99a703, which likewise found no security-relevant surface across all four changed files (plugin.json version bump, CHANGELOG prose, evals.json eval case, and a static inert fixture). No findings.