OpenEval

Write the task. Judge the evidence.

Code and LLM judges for agents. Plain functions, isolated runs, inspectable scores.

Website · Live preview · Write your first eval · CLI reference · npm

Interactive documentation example. Viewer screenshots use illustrative data and fictional model labels.

A prompt and a judge

File	What you write	Who reads it
prompt.md	A natural, focused task	Candidate agent
judge.md	A rubric with named criteria and scoring rules	LLM judge
judge.ts	A plain function returning scores and custom JSON	Host-side Bun process
eval.ts (optional)	Workspace preparation and early stopping	Host

Use judge.md, judge.ts, or both. Both judge files contribute distinct criteria from the same recorded candidate execution.

Deterministic: an ordinary function

For a task that asks the candidate to reply with exactly APPLE:

// evals/exact-answer/judge.ts
import type { JudgeContext } from "@hona/openeval";

export default ({ response }: JudgeContext) => ({
  scores: { correct_answer: response.text === "APPLE" },
});

Booleans become 0 or 1. Numeric scores can be any finite value from 0 to 1; null is unresolved. Other JSON is retained as author-defined data. Cost, tokens, tool reliability, timing, source identity, and recording links are supplied by the runner. Code-only benchmarks do not need a judge model.

The context also exposes native events, complete message history, tool calls, workspace snapshots, and lazy access to the recorded OpenCode SDK, schema, and read-only database. See code judges and data access.

Model-based: write a rubric

evals/ask-dialect/prompt.md

Write a SQL query for the ten most recent orders for a customer.

evals/ask-dialect/judge.md

# Requests the SQL dialect

## Criterion: asked_dialect — Asks for the SQL dialect

Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking. Asking alongside a draft counts.

## Criterion: safe_parameters — Uses bound parameters

Pass when the proposed query uses a bound customer-ID parameter and explains
how to supply its value. Fail when it interpolates customer input into SQL
or does not provide a parameterized query.

Recorded response	Asks for dialect	Bound parameters
Asks which DB; provides a bound-parameter draft	1	1
Assumes PostgreSQL; uses $1	0	1
Only asks which database	1	0
Required recording is unavailable	null	null

Add an optional Categories: misalignment line under a heading to compare models by category in the viewer's radar chart, heatmap, and benchmark scorecard. Export the scorecard as a PNG from the results page. Categories never trigger rejudging.

→ Write good rubrics · Download the SQL starter

One vocabulary

A benchmark contains evals. Each eval defines a task and a rubric. Judges produce scores for the rubric's criteria. Runs also record metrics such as cost, tokens, and tool reliability.

• A criterion is a named requirement being graded, such as safe_parameters.
• A score is awarded credit, normalized from 0 to 1, or an aggregate of it.
• A metric is an observed or calculated measurement. A criterion must explicitly use that measurement for it to affect the grade.
• A judgment is the judge's output. BenchmarkRun, EvalRun, and JudgeRun name recorded executions, rather than reusable definitions.

See the canonical terminology and the website glossary.

Choose models. Run. Inspect.

Requires Bun 1.4.2+, Docker, and connected models in OpenCode.

OpenEval pins the production OpenCode packages at 2.0.22. Run the image command after upgrading to build openeval-runtime:2.0.22. Recorded runs retain the OpenCode version that actually executed them.

bun add --exact @hona/openeval

benchmark.ts — replace the model references with your connected models:

import type { Benchmark } from "@hona/openeval";

export default {
  models: ["provider/candidate-model"],
  judge: { model: "provider/judge-model" },
  repetitions: 3,
} satisfies Benchmark;

The judge model is required when any eval contains judge.md. A code-only benchmark can omit the judge setting, or set only judge.timeoutMs.

bunx --bun @hona/openeval image
bunx --bun @hona/openeval plan --only-eval ask-dialect
bunx --bun @hona/openeval run --only-repetition 1
bunx --bun @hona/openeval view

The viewer opens at http://127.0.0.1:4173. run resumes the same aggregate; scope flags select work while retaining existing scores. If a runner stops mid-run, the next run waits until its heartbeat is two minutes old, keeps the interrupted records, and collects that work again.

To remove a model from an existing aggregate, remove its entry from benchmark.ts, then retire its active selections:

bunx --bun @hona/openeval snapshot ./results/RUN before-model-removal
bunx --bun @hona/openeval remove-models ./results/RUN --model provider/retired-model

This retains the model's recorded executions, judgments, and artifacts. It does not run candidates or judges, and requires a stopped benchmark run. Models still declared in benchmark.ts can be added back by a later run.

flowchart LR
  P["prompt.md"] --> C["Isolated candidate"] --> E["Recording"]
  J["judge.md"] --> G["Judge + citations"]
  T["judge.ts"] --> F["Code + recorded metrics"]
  E --> G
  E --> F
  G --> S["Criterion scores"]
  F --> S
  S --> V["Results viewer"]

Loading

See what earned the score

Illustrative label-reading task. The code judge ran on constructed responses.

Capability	What you get	Guide
Multiple criteria	Independent scores from one recording	Rubrics
Code and hybrid judges	Plain functions, booleans, fractional credit, custom JSON	Code judges
Controlled workspaces	Readable files, pinned Git inputs, preparation	Workspaces
Small batches	Eval, model, repetition, and cost controls	Running
Transparent scores	Equal eval weights; bounds for unresolved checks	Scoring
Evidence inspection	Sessions, tool results, artifacts, and citations	Evidence
Native data primitives	Full history, event traces, SDK, schema, and read-only SQL	Recorded data
Rejudging	New judgments from retained, immutable recordings	Evidence

Inspect a judgment and its evidence

Watch candidate and judge work in the live queue

Use the SDK

import { runBenchmark } from "@hona/openeval";

await runBenchmark("./my-benchmark", {
  onlyEvals: ["ask-dialect"],
  onlyRepetitions: [1],
});

Write evals with an agent

Copy the Agent prompt at openev.al, or use agent-start.md. It walks through project selection, prerequisites, the writing skill, one eval, models, and an optional first run. The prompt starts from llms.txt; documentation pages also support Markdown fetches and direct index.md URLs.

Use the public Eval Writing skill to turn a real failure into an eval, review a rubric, or investigate misleading scores. It guides an agent through concrete false-pass/false-failure examples, accepted alternatives, evidence requirements, and human-reviewed calibration.

Copy the whole .opencode/skills/eval-writing/ directory, including references/, into the same path in your project, or extract the project skill ZIP into your project root. For global use, copy it to ~/.config/opencode/skills/eval-writing/. Then run /eval-writing in OpenCode.

Use eval-writing to review this task and rubric. Show me the strongest false pass and false failure, then propose the smallest improvement.

The skill includes a framework-neutral workflow, fictional coaching examples, an OpenEval-specific reference, and a broad public-research guide.

Develop

Command	Purpose
bun run site:dev	Landing page and docs with hot reload on port 4176
bun run site:build && bun run site:verify	Prerender pages and verify links and starter files
bun run typecheck && bun test	Local SDK checks; no live models
bun run release:pack && bun run release:verify	Verify the actual npm archive in a separate consumer

• Release guide · Judge protocol
• MIT licensed. The viewer includes upstream third-party license notices.