Repository navigation
calibrate-judge build: a sample holding --- makes its case fail to load and go untested #6547
Description
Observed
In the code-tidying pilot (#6539), calibrate-judge.py build wrote 5 calibration cases that claude plugin eval refused with invalid YAML frontmatter: Unexpected EOF. Each sample held a markdown table rule (---). prompt_md() puts the sample in one double-quoted append_system_prompt scalar via yaml_quoted(), which escapes backslash, quote, newline, tab and carriage return, but not -. The CLI's frontmatter reader ended the frontmatter at the --- inside the quoted value.
Those 5 cases were silently untested: score lists a sample with no reproduced run as untested, but a case that never loads is easy to miss in a long run. Observed at Claude Code 2.1.289 on 2026-10-07.
Workaround used
Escaping each - in the sample (a YAML double-quoted escape that parses to the same string) let the cases load and calibrate.
Fix wanted
yaml_quoted()escapes so that no---run survives in the emitted line, within the subsetvalidate-cases.pyreads.- A fixture sample holding a table rule, with a test that
build's output loads invalidate-cases.pyand round-trips to the original sample.
Refs: #6476
Generated by Claude Code
Activity
kyle-sexton commentedon Oct 11, 2026
Needs a design decision before anyone implements this, so I'm adding
needs-human.Three independent reviewers checked this against
origin/maintoday (2026-10-11). Two of the three said not to implement it as the issue describes, because no escape works for both readers:\-is not valid YAML, andvalidate-cases.pyrejects it.\\ \" \/ \n \t \r, so\u002D(or\u002d) is rejected too.Decision needed: pick one.
\uXXXXand add a round-trip test. This changes the contract between the build script and the validator.---out of the frontmatter, for example into a sidecar file the case points at.No branch was created; nothing is in flight for this issue.
kyle-sexton commentedon Oct 13, 2026
Decision recorded. The owner approved applying independently verified recommendations on 2026-10-12.
Basis: https://code.claude.com/docs/en/plugin-evals (case.yaml section: append_system_prompt goes under execution:, fetched 2026-10-12); plugins/evals/skills/validate/scripts/validate-cases.py:200-204,259; plugins/evals/skills/plugin-eval/scripts/calibrate-judge.py:163-173,288-295
calibrate-judge build puts each sample in prompt.md frontmatter, and the CLI ends frontmatter at a --- inside the quoted value, so those cases never load. Reviewers found no escape both readers accept and asked the owner to pick a design.
Verification notes:
Verified by an independent fresh-context check (confirmed, risk low): gh issue view 6547 (owner comment 2026-10-11 with three options); plugins/evals/skills/validate/scripts/validate-cases.py:200-204 (ESCAPES) and :259-260 (whole-line --- only); plugins/evals/skills/plugin-eval/scripts/calibrate-judge.py:163-173 yaml_quoted, :288-299 prompt_md; validate-cases.py handles case.yaml execution fields (lines 110-120, 672-699); https://code.claude.com/docs/en/plugin-evals fetched 2026-10-12: case.yaml execution: holds append_system_prompt; prompt.md frontmatter overrides; git log origin/main on both scripts; gh pr list search 6547
Admission to an agent lane stays with the work-class rules and owner ratification.
kyle-sexton commentedon Oct 13, 2026
Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.