education:explain / skill-quality: evals.json warm-context cases untestable under the fresh-session recipe (PR #739 follow-up)

[kyle-sexton]

This was generated by AI during triage (#657 sweep housekeeping, T8h batch conversion).

Folded from a #657 line (checked off there, this issue is the target).

Problem

Evals #1/#4/#6 in plugins/education/skills/explain/evals/evals.json depend on warm conversational context (a prior assistant message, or an in-context referent), but the manual eval recipe pastes each prompt into a fresh session (docs/MIGRATION-PLAYBOOK.md:351) and the evals schema (plugins/skill-quality/reference/evals.schema.json) has no conversation-history fixture field.

Sharpest case: eval #1 (empty-argument-anaphora-default) and eval #7 (cold-start-empty-argument-asks-what) carry byte-identical prompts ("I don't get it.") with contradictory expected outputs. Under the fresh-session recipe, #1 collapses into #7's cold-start condition and cannot pass as written. #4 ("this proof" with empty files) and #6 ("wait, what does this actually do?") likewise have no referent to resolve, so the skill's own cold-start rule fires instead of the warm-context behavior each case means to exercise. validate-evals is structure-only, so it stays green over this — the gap is invisible to automated checks.

Decision needed

1. Add a conversation-history fixture mechanism to the evals schema upstream in skill-quality (plugins/skill-quality/reference/evals.schema.json + whatever runner/recipe reads it) — covers every skill's warm-context cases, not just education:explain's. Larger, cross-plugin change.
2. Restructure the three cases to be self-contained — embedding prior context into the prompt itself changes what eval chore: initialize marketplace scaffold #1 asserts (it stops testing anaphora-with-warm-context and starts testing something else), or merging/removing a case drops real warm-path coverage entirely.

This is genuinely open design space (not a single maintainer-vetoable default): option 1 is the more correct fix but is a skill-quality-owned schema change with blast radius beyond this one plugin; option 2 is scoped to education:explain but knowingly degrades what the evals actually cover. A maintainer should pick the scope before either is implemented.

Origin: PR #739 review threads (evals.json lines 7, 43, 67).

[kyle-sexton]

This was generated by AI during triage.

Problem

Evals #1 and #6 (and, more weakly, #4) in plugins/education/skills/explain/evals/evals.json assert warm-context behaviour — resolving an empty or anaphoric argument against a prior turn — but the only way to exercise evals today is to paste each prompt into a fresh session (docs/MIGRATION-PLAYBOOK.md:351), where no prior turn exists. The evals schema (plugins/skill-quality/reference/evals.schema.json) has no conversation-history fixture field and actively rejects one (additionalProperties: false at both object levels, lines 7 and 24), so #1 and #6 cannot be reproduced as written and the gap is invisible to automated checks.

What the harness actually does today

Three gates, per docs/MIGRATION-PLAYBOOK.md:336-355:

1. Presence — a file-exists check for evals/evals.json.
2. Schema — /skill-quality:skill-quality validate-evals validates structure against the bundled schema. It does not run the cases and treats an absent file as non-failing.
3. Exercise — a manual, human-judgment pass: paste each prompt into a fresh session and read the output against expected_output/expectations. Cases with a files list need those fixtures present; there is no mechanism for prior conversation turns. An automated runner is deferred (MIGRATION-PLAYBOOK.md:354), so nothing executes these cases programmatically today.

So validate-evals stays green over the defect — it is structure-only — and the only consumer check that would catch it is the manual fresh-session pass, which is precisely the recipe that cannot represent warm context.

Affected cases (verified against the file)

• #1 empty-argument-anaphora-default and #7 cold-start-empty-argument-asks-what carry byte-identical prompts ("I don't get it.") with contradictory expected outputs (resolve-against-prior-message vs. ask "explain what?"). In a fresh session #1 collapses into #7's condition and cannot pass as written. This is the sharpest case and is a genuine defect independent of the broader schema question.
• #6 auto-invokes-on-colloquial-trigger ("wait, what does this actually do?") — the anaphor this has no referent in a fresh session; the skill's cold-start rule fires instead of the warm-path behaviour the case means to test.
• #4 feynman-gap-check-surfaces-gap is weaker: its prompt names its subject (the Atiyah-Singer index theorem), so a fresh session can attempt it and hit the gap check. It is arguably testable as-is; I would not group it with #1/#6.

External facts: none relied on. #739 is merged; the review threads that flagged this were on its evals.json. No automated eval runner exists in plugins/skill-quality/ (verified: no runner/conversation references).

Options

A — Add an optional conversation-history/context fixture field to the evals schema, upstream in skill-quality. [RECOMMENDED]
Add an optional field (e.g. context / history) to $defs.eval_case, teach the manual recipe to seed it, and have the deferred runner honour it when it lands.

• Why first: fixes the root cause for every skill's warm-context cases, not just education:explain; keeps the cases testing what they mean to test. An optional additive field is backward-compatible — existing evals stay valid.
• Tradeoffs / blast radius: it is a skill-quality-owned schema/API change (requires widening additionalProperties: false at evals.schema.json:24), plus a recipe update in MIGRATION-PLAYBOOK.md, plus the deferred runner must implement it. Cross-plugin surface.
• Reversibility: high at introduction (optional field, no forced migration); medium once skills adopt it, since removal then needs cross-plugin coordination.

B — Mark the warm-context cases as manual-only / context-dependent and scope them out of the fresh-session recipe.
Add an explicit marker (needs a small schema field too, e.g. requires_context: true) or a documented convention, and exclude such cases from the fresh-session pass.

• Tradeoffs: smallest honest change; documents the limitation without pretending the cases are testable. But it labels the gap rather than closing it — warm-path behaviour stays unverified until the runner supports context.
• Reversibility: high — a marker/note is trivially removable.

C — Restructure the three cases in education:explain to be self-contained.
Embed the prior turn into the prompt string (or otherwise remove the context dependency); scoped to this one plugin, no schema change.

• Tradeoffs: changes what the cases assert. #1's entire point is "empty argument resolves to the assistant's own prior message"; folding context into the prompt makes it a different test and drops real warm-path coverage. Resolving the #1/#7 collision this way means #1 no longer tests the bare "I don't get it." warm case.
• Reversibility: medium — prompt edits revert cleanly in git, but the lost coverage is a silent semantic regression.

D — Sequence: fix the #1/#7 collision now, defer the general mechanism.
Disambiguate the byte-identical #1/#7 pair immediately (a real defect regardless of the schema decision), and file the general warm-context fixture mechanism (Option A) as a tracked skill-quality item.

• Tradeoffs: delivers the cheapest correctness win without pre-committing the cross-plugin schema scope; leaves #6 still untestable in the interim.
• Reversibility: high.

Recommendation

A, as the root-cause fix that generalises across skills — but this is a skill-quality-owned schema decision with blast radius beyond education:explain, and the item is needs-human + wayfind: design, so a maintainer should pick the scope. If A is not greenlit now, D captures the one unambiguous defect (the #1/#7 byte-identical collision) cheaply while A is decided.

[kyle-sexton]

Frontier disposition (drain session)

Ruling: Close as not planned.

Warm-context evals.json cases are documented as untestable under the fresh-session recipe; that limitation is accepted for education:explain.

Re-open trigger: If education:explain becomes autonomous acceptance evidence requiring warm-context eval coverage.