lucebench

Endpoint capability gate for OpenAI-compatible chat-completion servers — and Anthropic-messages tool scenarios. Ten evaluation areas, progressive snapshot levels, vendored fixtures, machine-readable reports.

Distribution, import, primary CLI, and GitHub repo are all lucebench (no hyphen). Hyphenated console scripts (luce-bench, …) remain as aliases.

Who this is for

Audience	Job
Inference server authors	Post-restart / post-PR smoke → level1 capability gate
Lab / machine operators	Named snapshots comparable across hosts and models
Agent-runtime owners	Cheap preflight before expensive agent/terminal-bench runs

Not this tool: research leaderboard campaigns (use lm-evaluation-harness / OpenCompass), app/prompt/RAG QA (Promptfoo, DeepEval), or full agent loops (fizeau, terminal-bench). lucebench scores the endpoint.

Areas: smoke, ds4-eval, gsm8k, truthfulqa-mc1, hellaswag, code, longctx, agent, agent_recorded, forge.

Install

uvx lucebench --base-url http://127.0.0.1:8000   # one-shot, no venv pollution
uv add lucebench                                   # uv-managed project
pip install lucebench                              # plain pip
pip install 'lucebench[dev]'                       # + pytest, ruff for contributors

# Unreleased main (or a PR branch)
uvx --from "git+https://github.com/easel/lucebench@main" \
  lucebench --base-url http://127.0.0.1:8000

PyPI: https://pypi.org/project/lucebench/ · latest tag v0.3.3.

anthropic is a hard runtime dependency (forge + default judge path). The legacy [forge] extra still resolves as an empty alias.

Every run prints a version banner first — [lucebench] vX.Y.Z — so stale caches are easy to spot.

Quick start

# Smoke (3 cases) — default when --areas is omitted
uvx lucebench --base-url http://127.0.0.1:8000

# Progressive post-change gate (host + levels)
uvx lucebench snapshot --level level1 \
  --base-url http://127.0.0.1:8000 --name my-machine

# OpenRouter
export OPENROUTER_API_KEY=sk-or-...
uvx lucebench --base-url https://openrouter.ai/api \
  --model qwen/qwen3.6-27b --auth-env OPENROUTER_API_KEY

# Multi-area sweep (see "areas all" policy below)
uvx lucebench --areas all --name my-machine \
  --base-url http://127.0.0.1:8000

--areas all policy

--areas all runs: smoke, ds4-eval, gsm8k, truthfulqa-mc1, hellaswag, code, longctx, agent, forge.

It does not include agent_recorded or agent_recorded_v1 (side effects: optional service restart, LLM judge cost). Run those explicitly:

lucebench --areas agent_recorded --base-url http://127.0.0.1:8000

snapshot --level level3 does include agent_recorded by design (baseline-grade full suite). See docs/helix/02-design/contracts/CONTRACT-001-cli.md.

More examples

# Single case, json-out for downstream analysis
lucebench --areas ds4-eval --case-id aime2025-02 \
  --base-url http://localhost:8080 --json-out /tmp/aime02.json

# Limit each area to N questions
lucebench --areas all --name quick --questions 2 \
  --base-url http://localhost:8080

# Parallel against a stateless gateway (skip on single-GPU local servers)
lucebench --areas ds4-eval --base-url https://openrouter.ai/api \
  --model openai/gpt-5.4 --auth-env OPENROUTER_API_KEY --parallel 8

# Single-case multi-mode reasoning probe
lucebench-probe --case-id aime2025-02 \
  --url http://localhost:8080 --out-dir ./probes/my-model

A sweep writes per-area JSON and a combined _summary.md under ./snapshots/<name>/. Envelope fields and headline metrics: docs/helix/02-design/contracts/CONTRACT-002-result-envelope.md and docs/metrics.md.

What's benchmarked

Area	Cases	Grader	Source
smoke	3 (arithmetic, capital, sequence)	case-insensitive substring	own — default sanity check
ds4-eval	92 (GPQA Diamond, SuperGPQA, AIME2025, COMPSEC)	strict Answer: X extract	antirez/ds4 (MIT)
gsm8k	100 (test split sample, seed 42)	#### N marker, last-number fallback	openai/gsm8k (MIT)
truthfulqa-mc1	100 (validation split sample, seed 42)	MC letter extract (2–13 choices)	truthful_qa (Apache-2.0)
hellaswag	100 (validation split sample, seed 42)	MC letter extract (A–D endings)	Rowan/hellaswag (MIT)
code	10 (mid-function completion)	ast.parse(prompt + completion)	openai/human-eval (MIT) port
longctx	6 frontiers (2k → 64k tokens)	^Risk: prefix check	own ports
agent	N codex-style prompts paired with coding tasks	code-fence / json-tool / apply_patch detect	own ports
agent_recorded	48 recorded Claude Code + Codex session prefixes replayed turn-by-turn	LLM judge on final turn + per-turn cache metrics	own — mined via scripts/extract-agentic-fixture.py
forge	7+ tool-calling scenarios	error_type == None	antoinezambelli/forge 0.7.1 (MIT)

Each row in the result carries:

• pass (bool), graded (full grader output)
• wall_seconds, http_status, error
• prompt_tokens, completion_tokens, timings (when surfaced by the server)
• content, reasoning_content, finish_reason, finish_details

The default sampling shape is send-nothing-explicit — the server applies its own defaults. Sampling flags are omitted unless the user sets them.

Snapshot levels

Level	Intent	Areas (caps)
level0	Alive?	smoke
level1	Post-change gate	smoke; code×5; gsm8k×5; agent×2; longctx×1
level2	Broader capability	smoke, code, gsm8k, truthfulqa-mc1, hellaswag, agent, longctx (full)
level3	Baseline-grade	level2 + ds4-eval, agent_recorded, forge

lucebench snapshot --level level1 --base-url http://127.0.0.1:8000

Programmatic use

from lucebench.areas import ds4_eval
from lucebench.runner import run_case

cases = ds4_eval.load_ds4_eval_cases()
case = next(c for c in cases if c["id"] == "aime2025-02")

row = run_case(
    url="http://localhost:8080",
    case=case,
    model="my-model",
    think=True,
)
graded = ds4_eval.grade_case(case, row)
print(graded["pass"], graded["given"], "/", graded["correct"])

Library imports used by external sweeps are provisional until a library Contract lands; prefer the CLI for stable automation.

Attribution

This project redistributes evaluation fixtures from upstream MIT- licensed projects. See NOTICE for full attribution; in short:

• ds4-eval cases — antirez/ds4, MIT
• GSM8K cases — openai/gsm8k, MIT
• TruthfulQA MC1 cases — truthful_qa, Apache-2.0
• HellaSwag cases — Rowan/hellaswag, MIT
• HumanEval prompts — openai/human-eval, MIT
• forge eval scenarios — antoinezambelli/forge, MIT

The lucebench code itself is Apache-2.0.

Product docs

Governed product docs live under docs/helix/ (vision, competitive analysis, PRD, contracts). Alignment for external release: docs/helix/06-iterate/alignment-external-release.md.

Contributing

git clone https://github.com/easel/lucebench
cd lucebench
uv sync --extra dev
uv run pytest
uv run ruff check src tests scripts

CI runs the same matrix on Python 3.10–3.13 + a wheel-build check that verifies fixtures are bundled.