Repository navigation
Conversation
Implements the deterministic foundation + runner architecture from PLAN-e2e-testing-harness.md (Phase 1 + the Phase 2 Claude Code adapter), under tests/e2e/. - Runner CLI (run-e2e) with scenario/vendor/model/fixture/runs filtering, git-clone isolation per run, per-scenario maxWallSec timeout, and skipped-never-passed handling for missing vendor CLIs. - Vendor adapters: real claude-code (JSON usage), best-effort codex (usage flagged pending the Phase 0 spike), and a no-spend mock for free plumbing self-tests. Canned-answer injection for headless runs. - Deterministic graders: git-tree-healthy, artifacts-spec/plan/bootstrap (dependency-free minimal JSON-Schema validator), app-tests-pass, bug-fixed, script-output, sync-convergence, implicit answers-lint, plus a strictly-advisory LLM judge. - Telemetry (per-step token breakdown, N-run aggregation) + version baseline regression flags + REPORT.md/summary.json generator. - Scenarios S0-S4, medplum-provider fixture (fixture.json + on-demand vendor.mjs, three states, Medplum test-pack wiring), seeded task inputs with canned answers, and a passing L1 install/uninstall smoke. - npm scripts (test:e2e:l1, e2e, e2e:list); gitignore results/ and vendored fixture states. Verified without spend: L1 smoke passes against the real CLI; a full mock run passes all graders through telemetry + report; schema validator, answers-lint, and baseline flags check out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the cheap layers (PLAN §3/§11) into GitHub Actions so they gate every PR: syntax-check of harness modules, L1 install/uninstall smoke (npm run test:e2e:l1), and a harness self-check that every scenario spec parses and the run matrix builds (--list + --dry-run --vendor mock). No models, no paid APIs, no network. The paid L2 agentic matrix stays manual/human-triggered — never on push/PR/schedule. L0 package unit tests (feat/write-unit-tests) plug into the same job when they land. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the deterministic foundation + runner architecture from
PLAN-e2e-testing-harness.md— Phase 1 (fixtures, deterministic graders, runner skeleton, L1 smoke) plus the Phase 2 Claude Code adapter — undertests/e2e/. Nothing here spends on paid APIs by default; the paid L2 matrix stays manual/human-triggered as the PLAN requires.What's included
runner/run-e2e.mjs) —--scenario/--vendor/--model/--fixture/--runsfiltering (individual runnability is first-class; no filter = full matrix),--list,--dry-run,--judge,--save-baseline. Per-run git-clone isolation, per-scenariomaxWallSectimeout, and missing vendor CLIs reported skipped, never passed.claude-code(claude -p --output-format json, parsesusage+ cost), best-effortcodex(usage flagged pending the Phase 0 spike), and a no-spendmockfor free plumbing self-tests. All inject canned answers so headless runs never stall on an interview.git-tree-healthy(conventional commits, scoping, conflict/secret scan, stacked-branch base resolution),artifacts-spec/plan/bootstrap(validated against the repo's realschemas/*.jsonvia a dependency-free minimal validator),app-tests-pass/bug-fixed/script-output,sync-convergence(the §14 oracle), implicitanswers-lint, and a strictly-advisory LLM judge.REPORT.md+summary.json.medplum-providerfixture (fixture.json+ on-demandvendor.mjsbuilding three states + wiring the Medplum test pack), seeded task inputs with canned answers, and a passing L1 install/uninstall byte-clean smoke.test:e2e:l1,e2e,e2e:list);.gitignoreforresults/and vendored fixture states.Verified without spend
node tests/e2e/l1/smoke.test.mjs— passes against the real CLI.mockscenario run passes all 5 graders → telemetry → report.answers-lint, and baseline regression flags unit-checked.Deferred (per the PLAN — needs human action / paid runs)
vendor.mjs(setupstream.pinnedCommitfirst) to materializestates/.Fixes #22