Repository navigation
fix(evals): flag regex graders that differ under JavaScript in /evals:validate #8385
Description
Problem
/evals:validate grades a case's regex graders against its samples with Python's re (plugins/evals/skills/validate/scripts/validate-cases.py), but claude plugin eval runs the same patterns with a JavaScript engine. Tokens that Python supports and JavaScript does not pass validation and then behave differently in the paid run.
Observed 2026-10-18 on the #8331 branch: a grader pattern starting with \A sorted every sample correctly under the validator, yet in the paid run JavaScript read \A as a literal A (and throws "Invalid escape" under the u flag), so the grader failed every with-plugin run. It cost a paid run and a diagnosis round.
Outcome wanted
The validator flags any regex grader pattern that does not compile, or compiles differently, under JavaScript semantics, before any money is spent.
Done when
-
validate-cases.pyreports FAIL for a regex grader using\A,\Z,\z, possessive quantifiers, atomic groups, or other Python-only syntax, naming the grader and the token. - Where node is available, the validator also runs each regex grader's samples through JavaScript's
RegExpand fails when the JavaScript result differs from the expected sort. - A test case for each token.
Related
- feat(evals): run plugin-eval suites on Windows, WSL, cloud sessions and GitHub Actions #8378: making plugin evals run in every environment.
- feat(planning): carry universal, overridable design defaults through planning output #8326: the effort where this showed up.
Activity
github-actions commentedon Oct 19, 2026
Understood: /evals:validate checks regex graders with Python's
re, butclaude plugin evaluses a JavaScript engine, so Python-only tokens (\A,\Z,\z, possessive quantifiers, atomic groups) pass validation and then fail in the paid run. You want validate-cases.py to flag those tokens by grader and token name, and, where node is available, to run each grader's samples through JavaScript RegExp and compare them to the expected sort, with a test per token. The scope and done-when list are clear. No duplicates found in my search.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Oct 19, 2026
The work-resolve-issue run that
agent-readystarted failed before it reported, or the token broker refused it because its workflow files differ from main's. work-unblock removed and addedagent-readyback to start a fresh run on main's workflow files. It does this once per version of those files, so if this run fails the same way, a person looks at it.melodic-automation-lanes commentedon Oct 19, 2026
The work-resolve-issue lane stopped partway through this issue and needs a person.
Why, in the run's own words:
What the run says it did not do:
Partial work is on the branch
work-resolve-issue/issue-8385-39307036662-1. A retry continues from it; delete the branch first to start again from the default branch.Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/39307036662
To retry, remove
needs-humanand addagent-readyagain.