crucible

crucible is an evaluation harness for LLM and RAG systems, reporting recall@k, MRR, nDCG, and token-F1 against your own ground-truth sets.

You already have the outputs — a ranked list of doc ids from your retriever, an answer from your generator. crucible tells you whether they're any good, and whether they got better or worse than last week. Hand it an eval file (inputs, retrieved docs, generated answers, ground truth) and it hands back a scored table plus a JSON report, with pass/fail thresholds so it drops straight into CI. It never calls OpenAI or Anthropic: it scores outputs you feed it, so it runs fully offline and produces the same numbers every time.

_{A real crucible run over an example eval set — deterministic metrics, threshold gate, exit code for CI.}

crucible in action

cargo run --release --bin crucible -- run examples/qa_eval.json

The eval file is JSON or YAML (crucible sniffs which). A case supplies whichever fields the metrics you care about need — retrieval-only, answer-only, or both:

{
  "name": "rag-qa-smoke",
  "k": 3,
  "thresholds": { "recall_at_k": 0.6, "mrr": 0.6, "token_f1": 0.35, "keyword_hit": 0.8 },
  "cases": [
    {
      "id": "capital-france",
      "input": "What is the capital of France?",
      "retrieved": ["wiki:paris", "wiki:france", "wiki:lyon"],
      "relevant": [{ "id": "wiki:paris", "gain": 3.0 }, { "id": "wiki:france", "gain": 1.0 }],
      "answer": "The capital of France is Paris.",
      "expected": "Paris",
      "keywords": ["paris"]
    }
  ]
}

Relevance can be a bare id (binary, gain 1.0) or { "id": ..., "gain": ... } for graded nDCG. That suite prints this (also available as --format json or --format markdown):

CRUCIBLE — rag-qa-smoke
4 cases · k=3

  case                 recall@k   prec@k      mrr   ndcg@k    exact       f1   kw_hit
  -----------------------------------------------------------------------------------
  capital-france          1.000    0.667    1.000    1.000    0.000    0.286    1.000
  speed-of-light          1.000    0.333    1.000    1.000    0.000    0.588    1.000
  tallest-mountain        1.000    0.333    0.500    0.631    0.000    0.364    1.000
  author-1984             1.000    0.333    0.500    0.631    0.000    0.400    1.000
  -----------------------------------------------------------------------------------
  AGGREGATE               1.000    0.417    0.750    0.815    0.000    0.409    1.000

  thresholds
    PASS recall@k     required 0.600, got 1.000
    PASS mrr          required 0.600, got 0.750
    PASS f1           required 0.350, got 0.409
    PASS kw_hit       required 0.800, got 1.000

  PASSED

Notice exact is 0.000 everywhere while kw_hit is 1.000: the answers are full sentences ("The capital of France is Paris.") and the references are terse ("Paris"), so exact-match rejects them and keyword-hit accepts them. That contrast is the point — different metrics catch different failures, and a report that only showed one would lie to you. The CLI exits non-zero when any threshold fails, so crucible run suite.json in CI turns a quality regression into a red build.

Metrics

Given a ranked list of ids and which ones are relevant:

• recall@k — of all the relevant docs, how many made the top-k?
• precision@k — of the top-k returned, how many were relevant?
• MRR — how high was the first relevant doc? (1/rank, averaged)
• nDCG@k — are the good docs near the top? (supports graded relevance)

Given a generated answer and a reference:

• exact_match — does the answer match the reference (case/punctuation-normalized)?
• token_f1 — how much do the answer's words overlap the reference's? (SQuAD-style)
• keyword_hit — did the answer contain the facts it had to?

Every metric is implemented from scratch in plain Rust and checked against hand-computed values in the tests — MRR of a known ranking is 0.5, nDCG of a single hit at rank 2 is 1/log2(3), F1 of the quick brown fox vs the brown fox is 6/7. No metric ships without a test that pins its number. Absent inputs mean absent scores, not zeros: a retrieval-only case gets no F1, and the aggregate skips it rather than dragging the mean to zero. Set a threshold on a metric no case exercised and the run fails loudly instead of passing by default.

Installation

cargo install crucible-eval    # installs the `crucible` binary

Or cargo add crucible-eval to use it as a library.

Usage

use crucible::{run, Suite};
use crucible::retrieval::{recall_at_k, reciprocal_rank};
use crucible::textmatch::token_f1;
use std::collections::HashSet;

// Score a whole suite:
let suite = Suite::parse(std::fs::read_to_string("suite.json")?.as_str())?;
let report = run(&suite);
println!("{}", report.to_json());

// Or reach for a single metric — ids are generic over any Eq + Hash type:
let relevant: HashSet<&str> = ["d1", "d2"].into_iter().collect();
assert_eq!(recall_at_k(&["d3", "d1", "d5", "d2"], &relevant, 2), 0.5);
assert_eq!(reciprocal_rank(&["d3", "d1"], &relevant), 0.5);
assert_eq!(token_f1("the quick brown fox", "the brown fox"), 6.0 / 7.0);

The metric functions are standalone and allocation-light; the runner just orchestrates them over a parsed suite. Use whichever layer fits.

Run the tests:

cargo test          # unit + integration + doc tests
cargo clippy --all-targets -- -D warnings
cargo run --example score_qa

How it works

crucible is provider-agnostic by construction: no HTTP client, no API keys, no network. You capture your system's outputs and crucible grades them — which is what keeps the numbers reproducible and the crate auditable in one sitting. Wiring up a live provider (call a model, capture its output into a case, then score) is a documented extension point, not a hidden dependency. Parsing leans on serde / serde_json / serde_yaml and the CLI on clap; the metrics themselves are pure std.

License

MIT — see LICENSE.