Skip to content

chore(e2e): end-to-end testing & benchmarking harness - #23

Open
hugobessa wants to merge 3 commits into
mainfrom
chore/e2e-testing-harness
Open

hugobessa wants to merge 3 commits into
mainfrom
chore/e2e-testing-harness

Conversation

@hugobessa

@hugobessa hugobessa commented Aug 2, 2026 •

Copy link
Copy Markdown
Contributor

Implements the deterministic foundation + runner architecture from PLAN-e2e-testing-harness.md — Phase 1 (fixtures, deterministic graders, runner skeleton, L1 smoke) plus the Phase 2 Claude Code adapter — under tests/e2e/. Nothing here spends on paid APIs by default; the paid L2 matrix stays manual/human-triggered as the PLAN requires.

What's included

  • Runner CLI (runner/run-e2e.mjs) — --scenario/--vendor/--model/--fixture/--runs filtering (individual runnability is first-class; no filter = full matrix), --list, --dry-run, --judge, --save-baseline. Per-run git-clone isolation, per-scenario maxWallSec timeout, and missing vendor CLIs reported skipped, never passed.
  • Vendor adapters — real claude-code (claude -p --output-format json, parses usage + cost), best-effort codex (usage flagged pending the Phase 0 spike), and a no-spend mock for free plumbing self-tests. All inject canned answers so headless runs never stall on an interview.
  • Deterministic graders — git-tree-healthy (conventional commits, scoping, conflict/secret scan, stacked-branch base resolution), artifacts-spec/plan/bootstrap (validated against the repo's real schemas/*.json via a dependency-free minimal validator), app-tests-pass / bug-fixed / script-output, sync-convergence (the §14 oracle), implicit answers-lint, and a strictly-advisory LLM judge.
  • Telemetry + reporting — normalized per-run record with per-step token breakdown, N-run aggregation (mean/median/p95/variance), version-baseline regression flags (pass-rate drop, >20% token/cost), and REPORT.md + summary.json.
  • Scenarios S0–S4, the medplum-provider fixture (fixture.json + on-demand vendor.mjs building three states + wiring the Medplum test pack), seeded task inputs with canned answers, and a passing L1 install/uninstall byte-clean smoke.
  • npm scripts (test:e2e:l1, e2e, e2e:list); .gitignore for results/ and vendored fixture states.

Verified without spend

  • node tests/e2e/l1/smoke.test.mjs — passes against the real CLI.
  • A full mock scenario run passes all 5 graders → telemetry → report.
  • Minimal schema validator, answers-lint, and baseline regression flags unit-checked.

Deferred (per the PLAN — needs human action / paid runs)

  • Fixture app content isn't committed — run vendor.mjs (set upstream.pinnedCommit first) to materialize states/.
  • Phase 0 Codex usage spike; multi-model matrix + second fixture (Phase 3); per-release baseline capture (Phase 4).
  • Actual paid L2 matrix runs — manual/human-triggered by design.

Fixes #22

hugobessa and others added 3 commits August 1, 2026 21:12
Implements the deterministic foundation + runner architecture from
PLAN-e2e-testing-harness.md (Phase 1 + the Phase 2 Claude Code adapter),
under tests/e2e/.

- Runner CLI (run-e2e) with scenario/vendor/model/fixture/runs filtering,
  git-clone isolation per run, per-scenario maxWallSec timeout, and
  skipped-never-passed handling for missing vendor CLIs.
- Vendor adapters: real claude-code (JSON usage), best-effort codex
  (usage flagged pending the Phase 0 spike), and a no-spend mock for
  free plumbing self-tests. Canned-answer injection for headless runs.
- Deterministic graders: git-tree-healthy, artifacts-spec/plan/bootstrap
  (dependency-free minimal JSON-Schema validator), app-tests-pass,
  bug-fixed, script-output, sync-convergence, implicit answers-lint,
  plus a strictly-advisory LLM judge.
- Telemetry (per-step token breakdown, N-run aggregation) + version
  baseline regression flags + REPORT.md/summary.json generator.
- Scenarios S0-S4, medplum-provider fixture (fixture.json + on-demand
  vendor.mjs, three states, Medplum test-pack wiring), seeded task
  inputs with canned answers, and a passing L1 install/uninstall smoke.
- npm scripts (test:e2e:l1, e2e, e2e:list); gitignore results/ and
  vendored fixture states.

Verified without spend: L1 smoke passes against the real CLI; a full
mock run passes all graders through telemetry + report; schema
validator, answers-lint, and baseline flags check out.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the cheap layers (PLAN §3/§11) into GitHub Actions so they gate
every PR: syntax-check of harness modules, L1 install/uninstall smoke
(npm run test:e2e:l1), and a harness self-check that every scenario
spec parses and the run matrix builds (--list + --dry-run --vendor mock).
No models, no paid APIs, no network. The paid L2 agentic matrix stays
manual/human-triggered — never on push/PR/schedule.

L0 package unit tests (feat/write-unit-tests) plug into the same job when
they land.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a way to test the skill generation and execution

1 participant