testing: evals grade against the private source monorepo's layout and house test stack (+ ghost testing.md ref)

[kyle-sexton]

Plugin: testing · Category: hardcoded opinion / source-repo coupling (1) — highest-severity finding for this plugin
Source: work-readiness sweep (read-only audit vs docs/PLUGIN-PHILOSOPHY.md + docs/MIGRATION-PLAYBOOK.md)

Findings

• plugins/testing/skills/diagnose/evals/evals.json:6-7 — expected_output asserts "identifies the MonolithApi.Tests collection fixture issue — … [Collection(MonolithApiTestFixture.CollectionName)]"
• plugins/testing/skills/write/evals/evals.json:12-13 — "integration tests for a new Platform.Messaging library … co-located Platform.Messaging.Tests/ in libs/dotnet/"
• plugins/testing/skills/write/evals/evals.json:7 — "places in co-located {Project}.Tests/ directory. References testing.md for framework conventions (xUnit v3, Shouldly)"

Why it breaks agnosticism

MonolithApi, MonolithApiTestFixture.CollectionName, Platform.Messaging, and libs/dotnet/ are the source monorepo's actual project names and directory layout; xUnit v3 + Shouldly are that shop's frameworks. The eval expected_output asserts these as CORRECT behavior — the fixture meant to prove repo-agnosticism instead grades against one private repo's layout. Violates PLUGIN-PHILOSOPHY "Design boundary" (no repository names / undocumented consumer layout) and migration-gate "de-couple from the source repo". Bonus defect: evals.json:7 references testing.md, a ghost file — this plugin ships write.md/organize.md; the ref survived from the source plugin.

Fix direction

Rewrite eval prompts/expected_output to neutral phrasing grading the DECISION (co-locate vs centralize, fixture-collision recognition) without naming any project, path, or framework — or supply a fixture repo whose conventions the skill reads (seam 3) so expected_output cites "the project's documented naming/location". Drop or repoint the ghost testing.md reference.

[kyle-sexton]

This was generated by AI during triage.

Triage → T2 planning complete · confidence: high. Leak confined to 2 eval files; ghost-ref fix verified unambiguous; sibling evals (plan, run-e2e) grepped clean. Each rewrite preserves the exact behavioral assertion. Labeling status: ready.

Neutralize eval fixtures so they grade the DECISION, not the private monorepo

Scope: skills/diagnose/evals/evals.json, skills/write/evals/evals.json. Sibling evals (skills/plan/evals/evals.json, skills/run-e2e/evals/evals.json) grepped for the leaked tokens — clean, no change.

Root cause

Two eval fixtures grade output against the org's private monorepo — project names (MonolithApi, MonolithApiTestFixture, Platform.Messaging), layout (libs/dotnet/), house stack (xUnit v3, Shouldly) — plus a ghost file (testing.md) that doesn't ship. This is the exact coupling these evals should prove absent: the shipped context files defer framework/naming/location to "the consuming project's testing conventions." The evals regressed from an earlier private version and now assert the opposite of the skill's contract.

Discriminating principle (per token)

Strip a token if it (a) identifies the private monorepo — a project/type name or path — or (b) asserts a specific stack as the mandated convention. Keep public ecosystem vocabulary, hedged as illustrative ("e.g.", "or equivalent"), when it's a legitimate recognition target.

Mirrors the shipped context files: investigate.md "WebApplicationFactory (or equivalent)"; organize.md "(e.g. {Project}.Tests co-located, in .NET)". So xUnit v3, Shouldly goes (asserted as mandate); a hedged WebApplicationFactory stays (recognition target the plugin itself names illustratively).

Fix 1 — skills/diagnose/evals/evals.json, eval id 1 (expected_output)

Remove MonolithApi.Tests, MonolithApiTestFixture.CollectionName. Keep hedged WebApplicationFactory. New assertion: "identifies a shared-state collection-fixture collision — a test class standing up its own instance of the shared application fixture (e.g. a per-class WebApplicationFactory) instead of joining the shared collection fixture the project's conventions prescribe. Explains the process-global singleton root cause. Does NOT suggest retrying." Prompt unchanged (realistic user input, no private identifier).

Fix 2 — skills/write/evals/evals.json, eval id 1 (+ ghost-ref)

Remove ghost testing.md, xUnit v3, Shouldly. New assertion: TDD + unit-test classification + name pattern + co-located {Project}.Tests, then "Defers framework and naming to the project's documented testing conventions rather than hardcoding a stack."
Ghost-ref: flip, don't repoint. testing.md doesn't ship; write.md deliberately ships no framework catalog (it defers). Repointing to write.md would be wrong — and the original clause is worse than a dangling pointer: if it resolved it would reward the skill for hardcoding the house stack, the exact behavior the plugin exists to avoid. So the assertion is inverted to grade deferral.

Fix 3 — skills/write/evals/evals.json, eval id 2

Prompt: "integration tests for a new Platform.Messaging library?" → "…for a new internal library?". expected_output: remove Platform.Messaging, Platform.Messaging.Tests/, libs/dotnet/; replace with "a co-located test project following the project's unit-test-project convention ({Project}.Tests)".

Acceptance criteria

• Both files still parse as valid JSON.
• grep -E 'MonolithApi|Platform\.Messaging|libs/dotnet|testing\.md|Shouldly|xUnit v3' over both files → zero matches.
• Each eval still routes to the same context/*.md and asserts its behavioral contract (fixture-collision recognition; TDD + unit-classification + convention-deferral; testable-vs-contract placement).
• No new bundled "framework conventions" file introduced — fix is deferral to the consuming project.
• Sibling evals (plan, run-e2e) unchanged (already clean).

Notes

• The one .NET mechanism kept (WebApplicationFactory / process-global singleton / collection fixture) is a public ASP.NET Core pattern already named illustratively in investigate.md — survives as texture while the graded target becomes "the project's conventions." No test intent lost.
• diagnose chore: initialize marketplace scaffold #1 prompt left concrete (dotnet test / "Bootstrap logger is frozen") — realistic, no private identifier, matches a symptom row in investigate.md. Only the grader (expected_output) needed neutralizing. A fully agnostic prompt is optional stylistic tightening, not a correctness gap.