Repository navigation
Public benchmark suite: Nimble vs Jev on 13 human-labeled datasets (vibe-coded, offered as-is) - #2
Conversation
895cdd0 to
7158518
Compare
EntelligenceAI PR SummaryAdds a public benchmark framework covering 13 human-labeled datasets, including dataset converters, reproducible manifests, Nimble and Jev evaluation runners, comparison/statistical reporting, documentation, and extensive tests. It also extends the CUDA scorer with configurable dtype, attention, and host-memory offloading, and adds support for merging against a locally supplied base checkpoint. Review Scorecard
Issues found:
Not safe to merge — the benchmark framework is well organized and supported by reproducible manifests, runners, and tests, but it contains multiple correctness and recovery defects in evaluation and dataset-ingestion paths, including the high-blast-radius Need to merge before these are addressed? Anyone with write access can comment Evaluated against
Files requiring special attention
|
WalkthroughThis PR introduces a reproducible public-benchmark suite spanning 13 human-labeled datasets and Choice, Noul, and Score tasks. It adds dataset adapters, manifest-driven conversion and validation, local and Jev evaluation workflows, comparison and suite reporting, CUDA offload support, and comprehensive network-free and end-to-end tests. ChangesFiles
Document the public benchmark suite, its 13 external datasets, dataset selection and licensing, conversion and integrity guarantees, evaluation and resume workflows, Jev comparison, metrics, limitations, reproducibility requirements, and offline validation commands. The README also adds the guide to the documentation navigation. Files
Add reproducibility manifests for the public subsets, including provenance, licenses, source and dataset hashes, deterministic selection metadata, family integrity, filtering and conversion statistics, label distributions, review status, and complete selected-ID inventories. Files
Implement the shared public-benchmark conversion pipeline and record utilities. The tooling discovers and validates source modules, normalizes Choice, Noul, and Score records, prevents reference leakage, preserves Unicode safely in JSONL, applies tokenizer-based length filtering, selects complete families deterministically or from manifests, hashes sources, validates immutable outputs, and emits reproducibility metadata and statistics. Files
Add source adapters for Aegis 2.0, BoolQ, Civil Comments, HelpSteer2, MASSIVE, MultiNLI, PAWS, PubMedQA, SQuAD 2.0, SummEval, and VitaminC. Each adapter defines provenance and task criteria, reads supported upstream formats, validates and skips malformed records, assigns stable IDs and family metadata, and constructs standardized records while excluding answer or reference leakage from model inputs. Files
Add local and Jev public-benchmark evaluators with input/reference validation, optional candidate shuffling, MLX and CUDA or concurrent HTTP inference, retries, failure handling, provenance, streaming output, interruption recovery, fingerprint checks, and resumable runs. Add run comparison and suite aggregation with accuracy, confidence intervals, calibration, distribution and score metrics, family completeness, domain and type breakdowns, McNemar analysis, multilingual MASSIVE comparisons, JSON artifacts, and Markdown reports. Files
Extend CUDA scoring for CUDA and ROCm devices, configurable dtypes and attention implementations, Accelerate device maps, host offloading, GPU memory budgets, and offloaded output-head projection with runtime placement metadata. Add support for merging adapters against a validated local base snapshot and include the resolved base path in manifests and READY-file reuse checks. Files
Add coverage for offloaded CUDA heads, public record conversion and manifest selection, deterministic family-preserving sampling, JSONL safety, source-module contracts, all dataset adapters, input/reference separation, nested SQuAD and MASSIVE inputs, local and Jev evaluation, retries and malformed responses, resume behavior, comparison statistics, family completeness, skipped runs, multilingual aggregation, and Markdown reporting. |
![entelligence-ai-pr-reviews[bot]](https://avatars.githubusercontent.com/in/932812?s=60&v=4){kind=small}
| "count": 249, | ||
| "families": 125, |
There was a problem hiding this comment.
Two-response families cannot yield 249 records across 125 families, and the committed comparison confirms one size-1 family. That partial family is included in family-complete metrics as a valid one-row family, so the published benchmark violates its whole-family claim and skews that metric.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
Fix docs/assets/public-benchmarks/subsets/helpsteer2-manifest.json and the generation inputs so length filtering cannot leave a partial prompt family: drop an entire two-response family when either response exceeds the token limit, or retain both responses. Regenerate the selected dataset, hashes, comparison outputs, and suite aggregates so count, IDs, and family metrics agree.
| contains U+2029) and would corrupt the parse. | ||
| """ | ||
| with open(path, encoding="utf-8", newline="\n") as stream: | ||
| rows = [json.loads(line) for line in stream if line.strip()] |
There was a problem hiding this comment.
Valid non-object JSONL values such as null are returned unchanged, then passed to converters that call .get, causing an uncaught AttributeError. The CLI catches only ValueError/OSError, so one malformed row produces a traceback instead of a controlled input error.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/datasets/public_records.py around line 106, reject decoded JSON values that are not dictionaries with the same controlled ValueError contract used for malformed input. Ensure callers cannot pass null, arrays, strings, or numbers to public source converters and that the CLI reports the row/input error rather than producing an uncaught AttributeError.
| def family_key(prompt): | ||
| return "helpsteer2-" + hashlib.sha256(prompt.encode()).hexdigest()[:16] |
There was a problem hiding this comment.
Because build() drops over-budget records individually before family selection, a prompt with one over-budget response leaves its sibling as a singleton despite this family key. The manifest proves it: 249 selected records across 125 families after 4 drops, so one claimed two-response family is incomplete.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/datasets/public_sources/helpsteer2.py and the associated build flow, preserve family completeness after token filtering. If any response for a prompt exceeds the input budget, exclude the entire prompt family before selection, then regenerate the HelpSteer2 manifest and benchmark outputs.
| if other["reference"]["target"] != first["reference"]["target"] or other["type"] != first["type"]: | ||
| raise ValueError(f"{row_id}: reference or type differs between {names[0]} and {name}") |
There was a problem hiding this comment.
join_runs() only checks reference.target and type, so same-ID rows with different domain or family are accepted. The comparison then uses different populations, and report_markdown() can raise KeyError when it indexes every run using the first run's domains instead of rejecting the mismatch.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/compare_public.py at lines 59-60, make join_runs validate all record identity fields required for comparison, including domain and family (and any other record/reference fields whose mismatch would change metrics), before compare or report generation proceeds. Add a clear ValueError for mismatched rows.
| ids_manifest = None | ||
| if ids_from is not None: | ||
| ids_manifest = json.loads(Path(ids_from).read_text()) | ||
| selected = select_ids(converted, ids_manifest["ids"]) |
There was a problem hiding this comment.
A valid-but-malformed manifest such as {} or [] reaches ids_manifest["ids"] and raises KeyError or TypeError. main() catches only ValueError/OSError, so a bad --ids-from file produces an unhandled traceback instead of a controlled CLI error.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/datasets/public_benchmarks.py at lines 167-168, validate that the decoded --ids-from manifest is a mapping with a valid nonempty/list-shaped `ids` field before passing it to select_ids, and convert malformed shapes into ValueError so main's existing parser.error path handles them.
| if len({r["id"] for r in records}) != len(records): | ||
| raise ValueError("Dataset must have unique IDs") |
There was a problem hiding this comment.
run() accepts an empty JSONL because it checks only duplicate IDs. It then builds the scorer and calls summarize_rows([]), whose ECE raises ValueError: Calibration needs at least one scored record; the CLI crashes instead of rejecting the empty dataset early.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/evaluate_public.py:281, add an explicit nonempty-dataset validation immediately after parsing records and before creating the output directory or scorer, using the same clear ValueError contract as evaluate_records.
| for raw in rows: | ||
| record = spec.record(raw, subset) |
There was a problem hiding this comment.
build passes every decoded row directly to spec.record; a JSON null or a row missing required fields can raise AttributeError or KeyError. main catches neither, so malformed downloaded data aborts with an unhandled traceback instead of the controlled conversion error path.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/datasets/public_benchmarks.py around lines 150-151, ensure malformed decoded source rows or record-converter KeyError/TypeError/AttributeError failures are converted to a controlled ValueError with dataset/row context, so the CLI's existing parser.error handling reports the failure without a traceback.
| device_map=device_map, max_gpu_memory=max_gpu_memory, dtype=dtype, attention=attention) | ||
| load_seconds = time.perf_counter() - started | ||
| manifest["runtime"] = getattr(scorer, "runtime", None) | ||
| manifest_path.write_text(json.dumps(manifest, indent=2, allow_nan=False) + "\n") |
There was a problem hiding this comment.
The manifest is persisted with direct Path.write_text; an interruption during its truncate/write can leave invalid JSON. On the next resume, the code unconditionally parses that file, so a run interrupted while starting or scoring cannot resume despite the documented resume contract.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/evaluate_public.py:326, persist the manifest through a temporary file in the same directory, flush/sync as appropriate, and atomically replace manifest.json only after the complete JSON is written. Apply the same crash-safe pattern to summary.json at line 343.
7158518 to
aa9de78
Compare
| rows = [json.loads(line) for line in Path(path).read_text(encoding="utf-8").split("\n") if line.strip()] | ||
| if not rows or len({r["id"] for r in rows}) != len(rows): |
There was a problem hiding this comment.
This reader silently drops blank records and raises an unhandled TypeError for valid non-object JSON such as null; it also treats a malformed torn final append as fatal. Comparing such files can hide records or abort instead of preserving the valid prefix and reporting completed-file corruption.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/compare_public.py around lines 40-41, replace the list-comprehension JSONL loader with a line-by-line parser that preserves newline termination, skips only an undecodable/incomplete final record, raises a clear corruption ValueError for newline-terminated malformed records, internal blank/corrupt lines, and non-object JSON values, and then enforces unique string IDs.
| API_KEY_VARIABLE = "TYPESAFE_API_KEY" | ||
| DEFAULT_MODEL = "jev-1.13.0" | ||
| RETRYABLE_STATUSES = (429, 500, 502, 503, 504, 529) | ||
| FORBIDDEN_KEYS = {"reference", "teacher", "teacher_targets"} |
There was a problem hiding this comment.
The guard recognizes only reference, teacher, and teacher_targets, while request_for forwards the entire state. A supplied record with input.state.target, answers, or labels therefore passes and sends reference-like material to TypeSafe, despite the shared public-record contract forbidding those keys.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/evaluate_public_jev.py:25 and 64-88, use the shared public-record forbidden-input contract or an equivalent complete recursive allowlist so target, labels, answers, score, distribution, and all other reference-bearing keys cannot appear anywhere in the state or questions sent to TypeSafe.
| rows_path.write_text("".join(json.dumps(r, ensure_ascii=False, allow_nan=False) + "\n" for r in rows), | ||
| encoding="utf-8") |
There was a problem hiding this comment.
When retryable error rows are removed, write_text replaces rows.jsonl directly. An interruption or disk-write failure during this rewrite can truncate or partially overwrite previously completed rows, destroying the checkpoint that resume depends on.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/evaluate_public_jev.py:299-303, replace the direct rows_path.write_text checkpoint rewrite with an atomic temporary-file write followed by os.replace, preserving the previous rows.jsonl if writing or replacement fails.
| assert (saved['adapter_sha256'] == adapter_hash and saved['base_revision'] == contract['revision'] | ||
| and saved.get('base_path') == str(base_path)) |
There was a problem hiding this comment.
READY.json files created by the previous version contain adapter_sha256 and base_revision but no base_path. The new comparison therefore fails, and the subsequent existing-output check raises FileExistsError, so a valid old merge cannot be reused or rebuilt in place.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/scoring/merge_local_adapter.py lines 37-38, maintain backward-compatible reuse for READY.json files produced before base_path was added, or provide an explicit migration path that updates the manifest before the existing-output guard rejects the directory.
aa9de78 to
22cbe0e
Compare
The retained 324-example holdout is synthetic, model-checked, and drawn from six source families. These converters turn public, human-labeled datasets into the existing evaluation record contract so the scorer can be measured against annotations it had no part in creating. public_benchmarks.py discovers one module per dataset under nimble/datasets/public_sources/. Each module declares its provenance and license, reads its downloaded file, and converts one raw row into a record; returning None skips the row. Selection is seeded and label-blind, draws whole families so contrastive siblings never split across a subset, and can instead take the exact ids of another build (--ids-from) to evaluate a second locale on the same utterances. Each build writes a manifest with the pinned source, license, both checksums, seed, counts, and the selected ids. public_records.py holds the shared record construction. It infers noul and score questions from the criteria shape, keeps every reference key out of the model input, and validates an optional human label distribution against the candidate keys. Sources: VitaminC contrastive verification; MASSIVE scenario routing for en-US and de-DE; BoolQ; SQuAD 2.0 answerability with the answer text kept out of the prompt; PAWS; MultiNLI with the five annotator votes as a distribution; Civil Comments with rater fractions as a distribution; Nemotron Content Safety V2 prompt safety; HelpSteer2 helpfulness and SummEval relevance and consistency as ordered rubrics; PubMedQA. JSONL readers split on newline only: abstracts and prompts contain U+2028 and U+2029, which str.splitlines() treats as line breaks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The 9B BF16 checkpoint needs about 18 GB, more than a 16 GiB GPU holds. CudaCandidateScorer gains optional device_map and max_gpu_memory so that accelerate keeps the layers that fit on the GPU and streams the rest from host memory, a dtype override for GPUs without native BF16 matrix support, and an attention override. The default path is unchanged. Under a device map the output head may be offloaded, in which case its parameter is a meta tensor and the values live in the accelerate hook. The candidate projection now reads the physical weight through that hook and moves the selected rows to the hidden state's device. Each result carries a runtime block with the dtype, placement, device, and library versions so an evaluation manifest can record how it was produced. merge_local_adapter accepts --base for a base snapshot outside the Hugging Face hub cache layout. Verified on an AMD Radeon RX 6800 XT (gfx1030) with torch 2.13.0+rocm7.1, where BF16 matrix multiplication is emulated at about half FP32 throughput but correct; with 13 GiB resident, 22 of 37 layers sit on the GPU. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every existing evaluation runner requires a saved Jev annotation on each record. Public datasets carry human labels and no such annotation, so evaluate_public scores records with only their input, reusing adapt_input, assess, and summarize unchanged. It adds expected calibration error with the binning established in evaluate_banking77, Jensen-Shannon divergence and total variation against a human label distribution where a record carries one, a --shuffle-seed control that permutes candidate order so A-Z position bias can be measured, the offload options of the CUDA scorer, and --resume with invariant checks against the saved manifest. evaluate_public_jev scores the same records through the TypeSafe API with the standard library only. Requests carry each record's state and questions and nothing else; reference labels are joined after inference. Runs append as they go and resume by skipping completed ids, retry on 429 and 5xx with exponential backoff, and abort on rejected credentials. compare_public joins two runs by id and reports accuracy with Wilson intervals, ECE, Brier, family-complete accuracy across contrastive siblings, agreement, and an exact McNemar test on discordant records. summarize_public_suite aggregates every subset into one report with macro and micro averages per run and per task type, and joins the MASSIVE en-US and de-DE runs on their shared ids for a paired multilingual comparison. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
docs/PUBLIC_BENCHMARKS.md describes the thirteen subsets, their sources, licenses, release years, selection rules, prompt wording, and build and evaluation commands, then reports Bespoke-Nimble-9B and Jev 1.13.0 on the same 3,880 human-labeled records. Macro accuracy is 74.8% for Nimble and 76.0% for Jev; micro accuracy is 75.9% and 77.3%. Jev leads on the boolean subsets and slightly on choice; Nimble leads on the three rubric subsets. Gaps that an exact McNemar test separates at p < 0.05 are Civil Comments, PAWS, MASSIVE de-DE, and VitaminC in Jev's favour and SummEval relevance in Nimble's. On the same 350 utterances, switching from English to German costs Nimble 3.5 points and Jev 0.5. Jev has lower expected calibration error on eleven subsets. The guide states what limits these numbers: small seeded subsets, unknown contamination for both models, few families behind the SummEval and SQuAD subsets, class skew, unfitted temperatures, and Jev's two-decimal probability rounding, which makes its NLL incomparable. It also lists the candidate datasets rejected for license, availability, or label provenance. docs/assets/public-benchmarks/ keeps each subset's manifest with its ids and checksums, each subset's comparison, and the suite aggregate, so the evaluation is reproducible without the local data directory. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
22cbe0e to
4b64434
Compare
| "note": "Validation split; helpfulness 0-4 scored as level index 0-4. Two responses per prompt share a family. correctness, coherence, complexity, and verbosity are kept in reference only.", | ||
| "release_year": 2024, | ||
| "source": "data/public/raw/helpsteer2/validation.jsonl", | ||
| "source_sha256": "1f5f121d1fddb540b24498ba38bfe132d87f7e2e9bac891464b32d30965884e7", |
There was a problem hiding this comment.
The recorded source checksum does not match the bytes currently served by the manifest's unpinned source URL, so following the documented download cannot reproduce or verify this benchmark source.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In docs/assets/public-benchmarks/subsets/helpsteer2-manifest.json, replace the floating source URL at line 4 with the exact upstream revision used to generate the data, or regenerate the source and selected dataset from the current URL. Update line 9 to the SHA-256 of that exact source and verify the manifest's dataset checksum and IDs against it.
|
|
||
| def record(raw, subset=""): | ||
| """Build one record; None skips non-human labels, redacted prompts, and empty prompts.""" | ||
| if raw.get("prompt_label_source") != "human" or raw.get("reconstruction_id_if_redacted") is not None: |
There was a problem hiding this comment.
A JSONL line such as null reaches this method because read_jsonl accepts arbitrary decoded JSON values, and build() passes each value directly to record(). This line then raises AttributeError instead of a controlled corruption error, aborting conversion with a traceback.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
At nimble/datasets/public_sources/aegis2.py:46, ensure non-dictionary decoded rows cannot reach `.get()` and produce AttributeError. Prefer enforcing the repository-wide JSONL contract in nimble/datasets/public_records.py:101-105 by validating each decoded value as a dictionary and raising the established clear corruption ValueError; keep direct calls to aegis2.record safe as well.
| return [json.loads(line) for line in archive.extractfile(members[0]).read().decode().split("\n") | ||
| if line.strip()] |
There was a problem hiding this comment.
The comprehension drops every blank physical line and unconditionally parses the remainder, so an internal blank is silently accepted while a malformed unterminated final record raises JSONDecodeError instead of being treated as an interrupted append. Corrupt tar JSONL can therefore be silently altered or abort conversion.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/datasets/public_sources/massive.py lines 57-58, parse tar-member JSONL while preserving physical newline termination: ignore only a malformed undecodable final line that is genuinely unterminated, raise for malformed newline-terminated records and internal blank/corrupt lines, reject non-object JSON values, and preserve valid records.
|
|
||
| def record(raw, subset=""): | ||
| """Build one PAWS record, or None when the label is not 0/1 or a sentence is missing.""" | ||
| if raw.get("label") not in (0, 1) or type(raw.get("label")) is bool: |
There was a problem hiding this comment.
A JSONL line containing null, an array, or another non-mapping reaches this converter through read_jsonl/build, and line 28 calls .get, aborting the benchmark with AttributeError instead of handling the invalid row.
| if raw.get("label") not in (0, 1) or type(raw.get("label")) is bool: | |
| if not isinstance(raw, dict) or raw.get("label") not in (0, 1) or type(raw.get("label")) is bool: |
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/datasets/public_sources/paws.py:28, guard record() against non-dictionary raw rows before calling .get. Return None for null, arrays, and other non-mapping JSON values while preserving the existing label validation for dictionaries; add the guard in the same condition.
|
|
||
| def pairwise(left, right, ids): | ||
| """Agreement and discordant correctness between two runs on the same records.""" | ||
| agree = sum(left[i]["student"]["prediction"] == right[i]["student"]["prediction"] for i in ids) |
There was a problem hiding this comment.
When both saved runs contain error rows, their predictions are both None, so this comparison counts simultaneous failures as prediction agreement. The paired agreement metric is therefore inflated by provider failures.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
In nimble/evaluation/compare_public.py at line 102, ensure prediction_agreement does not count invalid/error rows as agreeing predictions. Exclude records containing an error from the agreement numerator, while preserving the existing correctness and McNemar accounting for invalid responses.
| torn = lines[-1] != "" | ||
| rows = [] | ||
| for number, line in enumerate(lines, 1): | ||
| if not line.strip(): |
There was a problem hiding this comment.
completed_rows skips every blank line, including internal ones, so corrupted rows.jsonl is accepted instead of raising. This silently hides a malformed or missing checkpoint record during resume.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
Update nimble/evaluation/evaluate_public.py at line 202 so only the permitted trailing condition is ignored; internal blank JSONL lines must raise the same corruption error as other completed-record corruption while preserving final unterminated-tail handling.
| if not line.strip(): | ||
| continue | ||
| try: | ||
| rows.append(json.loads(line)) |
There was a problem hiding this comment.
json.loads results are appended without checking they are dictionaries; a valid null or array then reaches line 213 and raises an unhelpful TypeError instead of the corruption ValueError used for malformed records.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
Update nimble/evaluation/evaluate_public.py at line 205 to require every decoded saved row to be a dictionary before appending it, and raise the existing clear corruption ValueError for null, arrays, and other non-object JSON values.
| rows_path.write_text("".join(json.dumps(r, ensure_ascii=False, allow_nan=False) + "\n" for r in rows), | ||
| encoding="utf-8") |
There was a problem hiding this comment.
When a torn tail is found, write_text replaces rows.jsonl directly. A crash or write failure during recovery can truncate the valid prefix and lose previously completed rows, defeating resume.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
Update nimble/evaluation/evaluate_public.py at lines 211-212 so torn-tail recovery writes the canonical rows to a temporary sibling file, flushes and closes it, then atomically replaces rows.jsonl only after the complete rewrite succeeds.
| scorer = scorer_factory() if scorer_factory is not None else build_scorer( | ||
| backend, model_path, model_id, revision, temperature, max_input_tokens, | ||
| device_map=device_map, max_gpu_memory=max_gpu_memory, dtype=dtype, attention=attention) |
There was a problem hiding this comment.
Resume still constructs the model even when pending is empty. If interruption occurs after the last row is flushed but before summary creation, finalization unnecessarily requires the GPU or MLX stack and can fail despite all results being present.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
Update nimble/evaluation/evaluate_public.py at lines 269-282 so scorer construction occurs only when pending contains records; when all rows are already saved, finalize the report without requiring a model or backend.
| rows = sorted(previous + new_rows, key=lambda r: order[r["id"]]) | ||
| report = {**manifest, "created_utc": datetime.now(timezone.utc).isoformat(), | ||
| "load_seconds": load_seconds, "resumed_rows": len(previous), **summarize_rows(rows)} | ||
| summary_path.write_text(json.dumps(report, indent=2, allow_nan=False) + "\n", encoding="utf-8") |
There was a problem hiding this comment.
summary.json is the completion marker, but write_text creates it non-atomically. A crash during the write leaves a partial file, and the next resume immediately rejects any existing summary at line 251.
Prompt to fix with AI
Copy this prompt into your AI coding assistant to fix this issue.
Update nimble/evaluation/evaluate_public.py at line 287 to write summary.json to a temporary sibling and atomically replace the completion marker only after the full JSON document is written, so partial summaries remain resumable.
|
Wow this is amazing. Thanks! |
What this is, up front
This is a completely vibe-coded benchmark. I pointed Claude Code at the repo, asked for a broader public test of Nimble against Jev, and let it research, build, run, and document the whole thing with agents. I reviewed the results and the shape of the code, not every line. I'm offering it in case it is useful to you; if it isn't, closing this without comment is a perfectly good outcome.
What it adds
Nimble's published evaluation is 324 synthetic, model-checked examples from six source families. This PR adds an evaluation path for public, human-labeled datasets, and runs Bespoke-Nimble-9B and Jev 1.13.0 on the same records across 13 subsets (3,880 records) chosen to mirror what people use Jev for: intent routing (English and German on the same utterances), yes/no over a passage, RAG answerability, adversarial paraphrase, entailment, comment moderation, prompt safety, medical QA, contrastive fact verification, and three ordered rubrics.
nimble/datasets/public_sources/— one small converter module per dataset, auto-discovered; seeded, label-blind subset selection that keeps contrastive families together; provenance, license, and checksums in every manifest. Every dataset was verified reachable and commercially licensed before inclusion; the rejects and why are listed in the guide.nimble/evaluation/evaluate_public.py— evaluation without a Jevteacherannotation (every existing runner requires one), plus ECE, distribution metrics where a dataset publishes human label distributions, and a--shuffle-seedcontrol for A–Z position bias.evaluate_public_jev.py,compare_public.py,summarize_public_suite.py— the Jev side, per-subset comparison (Wilson CIs, family-complete accuracy, exact McNemar), and the suite table.CudaCandidateScorergains optional accelerate offload (--device-map auto --max-gpu-memory …) so the 9B checkpoint runs on a 16 GiB GPU; the default path is unchanged. That is how the numbers below were produced (an AMD RX 6800 XT via ROCm, BF16 emulated).merge_local_adaptertakes--basefor a local base directory.docs/PUBLIC_BENCHMARKS.mdanddocs/assets/public-benchmarks/— the guide, every subset's ids and checksums, and the comparison outputs.Only four pre-existing files change (
cuda_scorer.py,merge_local_adapter.py, its test, and a table row plus one sentence in the README). No new dependencies. Data and run outputs stay gitignored.Results (Nimble vs Jev, macro accuracy 74.8% vs 76.0%)
Jev has lower ECE on 11 of 13 subsets. On the rubric rows, expected-score error is at parity. Switching the same 350 utterances to German costs Nimble 3.5 points and Jev 0.5. Full tables, prompt wording, and caveats (small n, few families in SummEval/SQuAD, contamination unknown for both models, Jev's two-decimal probability rounding) are in the guide.
Things you may want to change or reject
instructions/criteriawording per dataset was written once by the agent from your TypeSafe guidance and never tuned; you will likely have opinions.cuda_scorer.py; if you'd rather keep that module untouched, the runner works with the existing default path on a big enough GPU and the offload commit can be dropped independently.unittestfiles against synthetic fixtures; the real-data checks were done by hand and are described in the guide, not automated.🤖 Generated with Claude Code