Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/e2e-checks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: e2e-checks

# Free, deterministic layers of the e2e harness — run on EVERY PR (PLAN §3/§11:
# "L0/L1 (free, deterministic) stay on every PR; only L2 is gated behind manual
# invocation"). No models, no paid APIs, no network — just the CLI + harness
# plumbing. The paid L2 agentic matrix is manual/human-triggered and lives
# elsewhere (never on push/PR/schedule).

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

jobs:
static:
name: L0/L1 static + smoke
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: '20'

# L0 (partial): syntax-check every harness module so a broken import or
# parse error fails fast and cheap.
- name: Syntax-check harness modules
run: |
set -e
for f in $(find tests/e2e -name '*.mjs'); do
node --check "$f"
done
echo "all harness modules parse"

# L1 — install/bootstrap smoke: run the CLI against a scratch project and
# assert placement + idempotency + byte-clean uninstall.
- name: L1 install/uninstall smoke
run: npm run test:e2e:l1

# Harness self-check: every scenario spec parses, the fixture wiring loads,
# and the run matrix builds — without spending anything. Catches a broken
# scenario JSONC or adapter registration before a paid L2 run ever hits it.
- name: Harness self-check (scenarios parse + matrix builds)
run: |
node tests/e2e/runner/run-e2e.mjs --list
node tests/e2e/runner/run-e2e.mjs --vendor mock --dry-run

# NOTE: L0 package unit tests (CLI file-placement/marker semantics,
# JSON-Schema validation of sample payloads, validate-skill-md lint) plug
# in here once feat/write-unit-tests lands — add them as further steps in
# this job so they gate every PR too.
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ logs/
coverage/
.nyc_output/

# e2e harness — raw run output (regenerated each run) and vendored fixture apps
# (large, external — built on demand by each fixture's vendor.mjs, never committed)
results/
tests/e2e/fixtures/*/states/
tests/e2e/fixtures/*/.upstream-cache/

# vinta-ai-maestro run state — journal, transcripts, gate logs. Kept inside the
# project (never a global cache) because it contains repository contents
# verbatim; see packages/vinta-ai-maestro/SPEC.md § Security and data handling.
Expand Down
293 changes: 293 additions & 0 deletions PLAN-e2e-testing-harness.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
},
"scripts": {
"check-ai-models": "node scripts/check-ai-models.mjs",
"test:e2e:l1": "node tests/e2e/l1/smoke.test.mjs",
"e2e": "node tests/e2e/runner/run-e2e.mjs",
"e2e:list": "node tests/e2e/runner/run-e2e.mjs --list"
"validate-schemas": "node scripts/validate-schemas.mjs",
"test": "node --test tests/*.test.mjs"
},
Expand Down
121 changes: 121 additions & 0 deletions tests/e2e/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# E2E testing & benchmarking harness

Drives **real agents** through the canonical vinta-ai-workflows skills against
seeded fixture projects, then grades the *outcome* (feature works, bug fixed,
script runs) and the *shape of the work* (git tree healthy, artifacts
schema-valid) across the **vendor × model matrix**, capturing token / cost /
wall-time telemetry per run.

See [`../../PLAN-e2e-testing-harness.md`](../../PLAN-e2e-testing-harness.md) for
the full design. This README is the operator's guide.

> **L2 is manual, human-triggered — never on a cron.** You run it before a
> release or to investigate a change. Every scenario is runnable individually,
> and every skill invocation is pre-answered so no run stalls on an interview.

## Layers

| Layer | What | Cost | When |
|---|---|---|---|
| **L0** | Package static tests (CLI unit, schema validation, `validate-skill-md`) | free | every PR |
| **L1** | Install/uninstall smoke (`l1/smoke.test.mjs`) | free | every PR |
| **L2** | Agentic scenario E2E (S0–S4, real models) | paid | manual |

## Quick start

```bash
# free — no models:
node tests/e2e/l1/smoke.test.mjs # L1 install/uninstall smoke
node tests/e2e/runner/run-e2e.mjs --list # what's available
node tests/e2e/runner/run-e2e.mjs -s S2-feature -v mock --dry-run # preview matrix
node tests/e2e/runner/run-e2e.mjs -s S2-feature -v mock # self-test plumbing (mock adapter)

# paid — real agent (needs the fixture vendored + a vendor CLI/key):
node tests/e2e/fixtures/medplum-provider/vendor.mjs # one-time: build fixture states
node tests/e2e/runner/run-e2e.mjs -s S3-bug -v claude-code -m claude-opus-4-8 -n 1
```

No `--scenario`/`--vendor` filter ⇒ the **full matrix** (a deliberate pre-release
sweep). Individual runnability is the default operating mode.

## Runner CLI

```
run-e2e [filters] [options]

Filters:
-s, --scenario <ids> S0-bootstrap | S1-sync | S2-feature | S3-bug | S4-script
-v, --vendor <names> claude-code | codex | mock
-m, --model <ids> pin model(s); default per-vendor if omitted
-f, --fixture <name> restrict to scenarios using this fixture

Options:
-n, --runs <N> repeat each cell N times for a pass-rate signal (default 1)
--judge <dims> advisory LLM-judge dims (spec-fidelity, plan-quality, ...)
--results-dir <d> output root (default <repo>/results, gitignored)
--keep keep isolated fixture copies for debugging
--dry-run print the matrix; run nothing
--list list scenarios / vendors / graders
--save-baseline write this run's aggregate as the version baseline
--no-fail-on-regression
```

A vendor whose CLI/key is missing is reported **skipped**, never passed.

## Scenarios (`scenarios/`)

| Id | State | Skills | Deterministic graders |
|---|---|---|---|
| **S0-bootstrap** | `clean` | `/vinta-bootstrap-ai-tools` | artifacts-bootstrap, git-tree-healthy |
| **S1-sync** | `stale` | `/vinta-sync-ai-tools` | sync-convergence, git-tree-healthy |
| **S2-feature** | `current` | create-spec → plan-feature → implement-plan | artifacts-spec/plan, app-tests-pass, git-tree-healthy |
| **S3-bug** | `current` | `/systematic-debugging` | bug-fixed, git-tree-healthy |
| **S4-script** | `current` | `/add-one-off-script` | script-output, git-tree-healthy |

`answers-lint` is applied implicitly to every scenario (flags a run where the
agent asked an unanswered question — i.e. the canned answers drifted).

## Graders (`graders/`)

Deterministic graders are **authoritative** (they gate pass/fail). The LLM-judge
(`--judge`) is **strictly advisory** — a quality/trend signal in the report,
never a gate. See `graders/index.mjs` for the registry.

## Telemetry & baselines

Each run writes `results/<version>/<vendor>/<model>/<scenario>/run-N.json`
(gitignored). Aggregates roll up into `results/<version>/summary.json` +
`REPORT.md`. `--save-baseline` freezes the aggregate under `baselines/<version>.json`
(committed); subsequent runs diff against the previous baseline and flag
pass-rate drops or >20% token/cost increases.

## Fixtures (`fixtures/`)

`medplum-provider` is the primary fixture — a real React + Vite + TS Medplum app
whose tests run **offline against `MockClient`**, so `app-tests-pass` needs no
live server. The app itself is **not committed** (it's large + external); build
its `clean` / `stale` / `current` states on demand:

```bash
node tests/e2e/fixtures/medplum-provider/vendor.mjs # all three states
node tests/e2e/fixtures/medplum-provider/vendor.mjs --state clean
```

⚠ Before first vendoring, set `upstream.pinnedCommit` in
`fixtures/medplum-provider/fixture.json` to a real SHA.

Task inputs and their **canned answers** live in
`fixtures/medplum-provider/tasks/` and are part of the fixture ground truth.

## Status / roadmap

This is the deterministic **foundation + runner** (PLAN Phase 1 + the Phase 2
Claude Code adapter). Still open, per the PLAN:

- **Phase 0 spike** — confirm Codex headless parity + usage reporting (§12/§14).
The Codex adapter is wired but usage is best-effort until the spike resolves.
- **Fixture app content** — the planted bug test, the feature scaffolding, and
the seeded MockClient bundles are materialized by `vendor.mjs` + a first real
bootstrap; the harness reports clearly when a state isn't vendored yet.
- **Second fixture** (`node-ts-cli`) and **multi-model matrix** — Phase 3.
- **Baseline capture per release** — Phase 4.
23 changes: 23 additions & 0 deletions tests/e2e/baselines/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Baselines

Per-version aggregated E2E results, **committed** so runs can flag regressions
between releases (PLAN §8).

Each file is `<version>.json` written by `run-e2e --save-baseline`:

```jsonc
{
"version": "0.4.0",
"cells": [
{ "scenario": "S2-feature", "vendor": "claude-code", "model": "claude-opus-4-8",
"runs": 3, "passRate": 1.0, "tokens": { "mean": ..., "median": ..., "p95": ... },
"costUsd": { ... }, "wallSec": { ... } }
]
}
```

A run compares its aggregate against the most recent *previous* baseline and
flags: pass-rate drops, or token/cost increases beyond 20% per cell.

Capture a baseline right before cutting a release, from a full-matrix sweep with
`--runs >= 3` (so the pass-rate + variance are meaningful).
52 changes: 52 additions & 0 deletions tests/e2e/fixtures/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Fixtures

Seeded, reproducible repos the scenarios run against. Each fixture is a real app
(not a toy) so the workflows are exercised on representative code.

## Contents of a fixture

```
<fixture>/
fixture.json # metadata: stack, run/test commands, per-state git refs,
# the version `stale/` was bootstrapped from
vendor.mjs # builds the clean/stale/current states on demand
tasks/ # seeded task inputs + their canned answers (ground truth)
feature.md feature.answers.md
bug.md bug.answers.md
script.md script.answers.md
bootstrap.md bootstrap.answers.md
sync.md sync.answers.md
states/ # (gitignored) built by vendor.mjs — each is its own git repo
clean/ stale/ current/
```

## Three committed states (PLAN §4)

- **`clean/`** — app with no `ai-tools/` / `AGENTS.md` / vendor dirs → drives **S0**.
- **`stale/`** — app bootstrapped against an **older** package version → drives **S1**.
- **`current/`** — app bootstrapped against the **current** version → drives **S2–S4**.

States are **not committed** to this repo — they're built by `vendor.mjs` and
gitignored. Each state is materialized as its own git repo so the runner can
clone it into an isolated copy and the git-tree grader has a real base to diff.

## Design rules

- Fixture states are **frozen** (pinned upstream commit + committed state).
- Every run operates on an **isolated copy** — never the vendored fixture in place.
- Every task has a **machine-checkable** success signal so grading isn't subjective.

## Maintenance

- Re-base `stale/` to "current minus one meaningful change" each release so the
sync scenario keeps testing a real upgrade (see `fixture.json.maintenance`).
- Keep the pinned upstream commit current enough that the app still builds on the
supported Node version.

## `medplum-provider` (primary)

Real React + Vite + TS Medplum app; tests run offline against `MockClient`.
Requirements captured in `medplum-provider/fixture.json.medplum`: wire the two
Vitest setup files, keep `@medplum/*` versions matched, Node ≥ 22.4. A live
Medplum stack (Docker) is opt-in and off by default — reserve it for contracts
`MockClient` can't prove.
57 changes: 57 additions & 0 deletions tests/e2e/fixtures/medplum-provider/fixture.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"name": "medplum-provider",
"description": "PRIMARY fixture — the Medplum provider example app (React + Vite + TS on @medplum/core / @medplum/react, React Router, Mantine). Tests run fully offline against MockClient, so app-tests-pass needs no live Medplum server.",
"upstream": {
"repo": "https://github.com/medplum/medplum",
"subdir": "examples/medplum-provider",
"pinnedCommit": "REPLACE_WITH_PINNED_SHA",
"note": "Vendored + pinned by vendor.mjs. Keep @medplum/{core,definitions,mock,fhirtypes} on matching versions; Node >= 22.4 (or a ws dev dep) for the WebSocket polyfill @medplum/core v5 needs at module load."
},
"project": {
"name": "medplum-provider",
"default_branch": "main",
"code_host": "github",
"stack_summary": "React + Vite + TypeScript on @medplum/core / @medplum/react, React Router, Mantine; tests via Vitest + MockClient"
},
"commands": {
"install": "npm ci",
"test": "npx vitest run",
"run": "npm run dev",
"ai_plans_dir": "ai-plans",
"scripts_dir": "scripts"
},
"medplum": {
"vitestSetup": {
"globalSetup": "test.globalSetup.ts",
"setupFiles": "test.setup.ts",
"note": "Both must be registered in vitest.config.* — globalSetup indexes FHIR bundles once and shares globalSchema.types to workers; setupFiles assigns the shared index at module scope. Without them, filtered searchResources silently return nothing."
},
"liveStack": {
"optIn": true,
"default": false,
"note": "Off by default. For server-only behavior (access policies, subscriptions, $everything, bots, GraphQL/transaction edges) bring up the full stack via Docker and run graders as integration checks. Slower + nondeterministic — reserve for contracts MockClient can't prove."
}
},
"states": {
"clean": {
"path": "states/clean",
"ref": "main",
"description": "App with NO ai-tools/, AGENTS.md, or vendor dirs. Drives S0 — Bootstrap."
},
"stale": {
"path": "states/stale",
"ref": "main",
"bootstrappedFrom": "0.3.0",
"description": "App bootstrapped against an OLDER vinta-ai-workflows version (deliberately behind current). Drives S1 — Sync."
},
"current": {
"path": "states/current",
"ref": "main",
"bootstrappedFrom": "0.4.0",
"description": "App bootstrapped against the CURRENT version, harness fully in place. Drives S2–S4."
}
},
"maintenance": {
"staleCadence": "Re-base `stale/` to 'current minus one meaningful change' each release so S1 keeps exercising a real upgrade rather than a no-op. Cheapest: regenerate from clean/ by bootstrapping with a pinned older package version (see states.stale.bootstrappedFrom)."
}
}
20 changes: 20 additions & 0 deletions tests/e2e/fixtures/medplum-provider/tasks/bootstrap.answers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Canned answers — bootstrap (vinta-bootstrap-ai-tools interview)

Answer every interview question with the following; do not ask for anything else.

- **Project name**: medplum-provider
- **Default branch**: main
- **Code host**: github
- **Stack**: React + Vite + TypeScript on @medplum/core / @medplum/react, React
Router, Mantine; tests via Vitest + MockClient.
- **Medplum stack applies?**: Yes — the app consumes @medplum/react. It has no
`bots/` dir and no tenant compartmenting, so apply only the SDK-consumer parts;
skip bot/access-policy/notification categories.
- **ai-plans dir**: `ai-plans`
- **Vendors to wire**: claude-code (primary). Others optional.
- **Testing pack**: Medplum (`MockClient` + the two Vitest setup files), on top
of the Vitest runner pack.
- **Keep a hand-tuned region marker** in AGENTS.md: include the literal comment
`<!-- hand-tuned:keep -->` so sync preservation can be tested later.
- **Commits**: conventional; separate the AGENTS.md, config, and skills/agents
commits logically.
10 changes: 10 additions & 0 deletions tests/e2e/fixtures/medplum-provider/tasks/bootstrap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Bootstrap request — Wire AI tooling into this repo

This is a clean medplum-provider checkout with no `ai-tools/`, `AGENTS.md`, or
vendor dirs. Run `/vinta-bootstrap-ai-tools` to analyze the codebase and produce
the harness: `AGENTS.md`, derived skills + sub-agents, per-vendor wiring, and the
`.vinta-ai-workflows.yaml` config.

Success = `AGENTS.md` reflects this stack (React + Vite + TS on Medplum), the
generated sub-agents and config are schema-valid, and everything lands as clean,
conventionally-scoped commits.
15 changes: 15 additions & 0 deletions tests/e2e/fixtures/medplum-provider/tasks/bug.answers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Canned answers — bug (systematic-debugging interview)

Answer every question with the following; do not ask for anything else.

- **Repro command**: `npx vitest run src/pages/patient/allergies/AllergyPanel.test.tsx`
- **Expected**: the created allergy's substance is returned when reading the
patient's allergies.
- **Actual**: read returns nothing — the written `AllergyIntolerance.patient`
reference is malformed.
- **Constraint**: fix the write path under `src/pages/patient/allergies/`. Do NOT
modify or weaken the test to make it pass.
- **No regressions**: the full `npx vitest run` suite must stay green.
- **Root cause over patch**: correct the reference construction, don't special-case
the test input.
- **Commit**: a single `fix(...)` conventional commit touching the create path.
Loading
Loading