[aviggiano]

Summary

• adds ultrafuzz json validate with strict shared Draft 2020-12 validation, immutable byte snapshots, schema digests, bundle identity, and bounded diagnostics
• replaces generic topology JSON contracts with complete typed schemas and central producer-side prompt validation commands
• introduces current-only runtime, eval, and Modal durable document schemas, readers, writers, semantic gates, and no-repair behavior
• defines seven closed current-only benchmark-analysis input/output contracts with exact identity, join, duplicate-chain, classification, and credit gates
• registers artifact, eval, Modal, reference-cache, and topology schemas with pinned per-schema and bundle identities
• makes Modal input, result, checkpoint, restore, execution-dependency, and worker-error boundaries use registered strict contracts
• rejects malformed-present or partial durable handoffs and recovery evidence without conversion or synthesis
• removes automatic invalid-judge-output repair conversations; judge output must be one exact strict JSON object matching the advertised schema
• removes historical artifact aliases, normalization, synthesis, malformed-as-missing paths, and workspace-mirror promotion
• preserves intentional custom Smithers package extension points while keeping root fields, required pins, and entry types strict
• serializes telemetry cursor drains across reload, callbacks, and durable commit, with fatal cursor failures and stable reporter idempotency keys
• replaces the open v1 event envelope with a closed v2 union covering all 23 production event variants and exact Smithers 0.32 status/state enums

Current checkpoint

This is intentionally a draft and work is continuing. Latest remote checkpoint: c22f2ca.

Passed through published checkpoints:

• portable public-eval timestamp and Modal URL JSON Schema/Zod parity: focused Modal 20/20, focused eval 7/7, full eval 308/308, builds/schema registries/typechecks, lint, formatting, and diff checks

• NanoEval timestamp JSON Schema/Zod parity: EVMBench tests 58/58, package build/typecheck, focused lint, formatting, and diff checks

• root formatting, lint, and workspace build through the pre-event-v2 checkpoint

• artifact event-v2 and strict-journal tests: 129/129, plus artifact typecheck and schema registry/snapshot checks

• prompt tests: 50/50

• topology tests: 39/39 and focused dashboard API tests: 10/10 after malformed-present topology/state/report hardening

• full eval typecheck and tests: 244/244 after strict judge handoffs and closed event-v2 telemetry/recovery migration

• telemetry/reporter focused tests: 26/26, plus focused lint, formatting, and diff checks

• Modal public diagnostics tests: 21/21 and public-worker tests: 40/40 after strict current-only matrix, summary, journal, and run-state migration

• earlier Modal build, typecheck, schema registry, and node-provider tests: 70/70

• benchmark launch and collect workflows through the pre-event-v2 checkpoint

• runtime build/typecheck and test compilation after the event-v2 migration; 11/11 strict Smithers/workflow-sync cases, 7 resume/preflight cases, and the broader 47-case runtime selection pass across the focused runs

• cross-process eval journal contention under normal and one-core-constrained execution; its CI-only harness timeout now defers to the production readiness/lock deadlines

• benchmark-analysis CLI typecheck, eval schema-registry build, eval contract/durable tests 13/13, benchmark CLI tests 13/13, and documentation checks

• report-bundle focused tests 13/13 after strict event-journal validation and exact immutable-byte capture

• current-main integration: workspace build, lint, and formatting; artifacts 131/131; prompts 50/50; generated-workflow verifier 48/48; Modal runner 59/59; eval-history validation/rendering for 126 current v6 observations

• reference-cache schema registry/read/write tests 9/9, focused CLI validator tests 6/6, references and CLI typechecks, focused lint, formatting, and diff checks

• EVMBench strict-contract typecheck, lint, formatting, build, schema-copy parity, built-registry/profile smoke, catalog digest verification, and tests 38/38

• focused strict Smithers CLI lifecycle/inspect tests 3/3, eval-status tests 3/3, CLI validator tests 7/7, and runtime reference-expectation tests 2/2

• resolved-config tests 46/46 (including exact nonduplicated diagnostics for seven provider-specific semantic failures), full CLI tests 66/66 (including composed-registry tests 8/8), config/CLI builds and typechecks, Modal config tests 6/6, runtime publication checks, and focused lint/format checks

• exact Smithers status-envelope health tests 2/2 plus runtime typecheck, focused lint, formatting, and diff checks

• run-state v4 provenance parity: artifact tests 132/132, focused Modal tests 42/42, runtime typecheck, schema snapshots, lint, formatting, and diff checks

• strict Modal recovery-summary parsing: focused lifecycle tests 7/7, Modal typecheck, lint, formatting, and diff checks

• closed dashboard HTTP/SSE/audit and runtime materialize/clean audit contracts: dashboard build/tests 18/18, runtime focused tests 11/11, runtime typecheck, complete current $ref audit, schema registries, serialized-byte checks, persisted run-document checks, lint, formatting, and diff checks

• exact Smithers lifecycle/inspection contracts: runtime lifecycle suite 32/32 plus 2/2 new negative matrix cases, focused CLI lifecycle cases 3/3, and runtime/CLI typechecks

• run-state v4 dynamic-lineage parity and eval integration: artifact tests 132/132, eval and CLI typechecks, focused lint, formatting, and diff checks

• closed CLI result v2/operator-input contracts: 60/60 CLI tests across the source checkpoint, contract/validator integration 14/14, focused end-to-end command flow, EVMBench parity 38/38, CLI build, docs check, lint, formatting, and diff checks

• closed config profile helpers: config tests 46/46 with exact nonduplicated field diagnostics, config build/typecheck, lint, formatting, and diff checks

• artifact-manifest v3 provenance: artifact tests 134/134, schema snapshots, artifacts/runtime/evals typechecks, runtime producer paths 2/2, runtime artifact gates 69/69, eval telemetry 20/20, docs, lint, formatting, and diff checks

• strict analysis-bundle reads: artifact tests 135/135, duplicate-key/no-mutation regression, lint, formatting, and diff checks

• strict init/workflow-link recovery evidence: focused runtime tests 3/3, runtime build/test compilation, lint, formatting, and diff checks

• exact workflow-control state/timestamp evidence: runtime typecheck, focused workflow-control tests 9/9, lint, formatting, and diff checks

• strict Modal worker-result collection: focused worker-result/runner tests 85/85, Modal typecheck, lint, formatting, and diff checks

• closed dashboard JSON-seam audit: dashboard tests 19/19, dashboard typecheck, lint, formatting, and diff checks

• eval-suite v2/run v3 contract source and schema checkpoint: evals typecheck, eval schema-registry build (including separated owner/cross-registry ID collision checks), and diff checks; fixture/parity/docs and broad-test follow-up remain in progress

• strict smoke soft-fail diagnostics: full CI-script suite 55/55 (including focused diagnostics cases 11/11), eval/dependency builds, lint, formatting, and diff checks

• canonical eval-suite/review fixtures: eval runner and scoring tests 60/60, Modal public-bundle tests 20/20, and diff checks

• eval-suite JSON Schema/Zod parity and strict negatives: suite tests 51/51, eval typecheck/build/schema registry, docs, lint, formatting, and diff checks; broader eval run 280/282 with the known private-ground-truth fixture and history SVG baseline follow-ups remaining

• downstream eval-contract fixture cleanup: Modal typecheck and targeted Modal tests 114/114 (public diagnostics 21/21), plus full built CLI tests 74/74

• eval strict-suite formatting repair: full workspace formatting check, focused lint, and diff checks; no semantic changes

• obsolete loose eval helper cleanup: full workspace lint and formatting plus diff checks

• strict Modal private model-work evidence: full Modal suite 388/388 across 29 files, Modal typecheck, lint, formatting, and diff checks

• validator failure/ref hardening: artifact tests 136/136, full CLI tests 75/75 plus focused post-rebase 9/9, artifact/CLI typechecks, lint, formatting, and diff checks

• scoped CLI opaque-JSON type repair: CLI dependency typecheck, CLI build/schema registry, lint, formatting, and diff checks

• strict Modal recovery probes: full Modal suite 386/386 across 29 files, Modal typecheck, lint, formatting, and diff checks

• registered public-eval-diagnostics v2 whole-document contract: focused eval tests 15/15, focused Modal tests 145/145, eval build/typecheck, schema-registry verification, docs, lint, formatting, and diff checks

• strict Modal launch-state persistence: focused launch/runner tests 88/88, full source-branch Modal suite 389/389, Modal typecheck, lint, formatting, and diff checks

• strict Modal recovery-state persistence: focused aggregate recovery test 17/17, full source-branch Modal suite 462/462, Modal typecheck, lint, formatting, and diff checks

• strict Modal worker-lineage persistence: focused aggregate test 7/7, full source-branch Modal suite 463/463, Modal typecheck, lint, formatting, and diff checks

• trusted local/Modal validator startup identity chain: Modal startup/provider 72/72, generated-workflow/trusted-CLI 53/53, workflow compilation 2/2, Modal/runtime typechecks, lint, formatting, and diff checks

• registered report-bundle-manifest v2: focused registry/report-bundle tests 14/14, CLI build/schema registry, docs, lint, formatting, and diff checks

• strict Modal pinned-source proof persistence: focused aggregate test 8/8, full source-branch Modal suite 464/464, Modal typecheck, lint, formatting, and diff checks

• strict Modal worker-result publication: focused aggregate test 26/26, full source-branch Modal suite 467/467, Modal typecheck, lint, formatting, and diff checks

• registered benchmark cohort and lanes policy documents: focused eval tests 13/13, eval build/typecheck, composed CLI registry build, real CLI validation of all three checked-in documents, docs, lint, formatting, and diff checks

• strict Modal recovery-lifecycle collection: focused aggregate tests 71/71, full source-branch Modal suite 467/467, Modal typecheck, lint, formatting, and diff checks

• strict eval-report reads: eval durable tests 9/9, focused CLI regression 1/1, eval/CLI schema-registry builds, lint, formatting, and diff checks

• strict Modal public-diagnostics collection/publication: focused aggregate tests 85/85, full source-branch Modal suite 467/467, Modal typecheck, lint, formatting, and diff checks

• strict Modal public-bundle collection: immutable bounded strict parsing shared with local reads, duplicate-key rejection, focused public-bundle/runner tests 84/84, full Modal suite 467/467, Modal typecheck, lint, formatting, and diff checks

• strict Modal worker model identity: workers derive the model from authenticated lineage instead of independent environment JSON; focused 72/72, full Modal suite 468/468, Modal typecheck, lint, formatting, and diff checks

• registered Modal benchmark-control manifest v1: producer validates serialized bytes, publication/cleanup strict-parse immutable bytes, duplicate keys and historical/missing identities fail closed; Modal contract/config tests 25/25, CI handoff tests 20/20, Modal typecheck/build, composed CLI registry build, lint, formatting, and diff checks

• registered release-validation report v2: exact current-only command-result schema, serialized-byte validation, no redundant failure projection, named reconciliation/path/identity gates; artifact tests 138/138, schema snapshots, artifact typecheck/build, composed CLI registry build, lint, formatting, and diff checks

• registered runtime durable-document contracts: cloud generation, invariant baseline/snapshot/handoff, Smithers submission/reset, workflow control/link/dependencies, and workspace-patch baseline/preparation now use strict schemas, shared codecs, and semantic gates; runtime build/typecheck and focused tests 89/89

• registered current-only eval recovery-equivalence, eval-status, ground-truth, and longitudinal-history contracts: full eval 295/295, eval typecheck/build/schema registry, Modal ground-truth/public-worker 44/44, Modal typecheck/lint, plus focused formatting and diff checks

• registered Modal benchmark-config v2: all formerly defaulted fields are required, v1 and consumer judge fallbacks are removed, public targets are explicit, strict registered loading/fingerprinting and nontransforming Zod parity are enforced; full Modal suite 470/470, Modal build/typecheck/schema registry, lint, formatting, and diff checks

• Modal public-worker cohort snapshot reuse: the registered strict cohort is read once and the exact validated object supplies EVMbench ground-truth provenance; Modal typecheck/build, public-worker tests 40/40, lint, formatting, and diff checks

• strict Modal public failure-diagnostic parsing: producer stdout and collected base64 diagnostics reject duplicate keys and invalid strict JSON; focused Modal tests 104/104, Modal typecheck, lint, formatting, and diff checks

• registered standalone terminal-disposition contract and strict semantic evidence: artifact snapshots/typecheck, runtime and Modal typechecks, artifact fixtures 9/9, runtime artifact gates 69/69, Modal terminal/strict-reader tests 20/20, lint and diff checks

• reconciled current main final-report coverage work without restoring historical compatibility or report mutation: pure authoritative coverage derivation targets properties v2 / implemented-properties v3, the pure projector targets report v2, artifact tests 139/139, runtime artifact gates 69/69, projector tests 7/7, artifacts/runtime typechecks, schema snapshots, lint, formatting, and diff checks

• strict benchmark cleanup/launch CI readers: all CI scripts 56/56 and full Modal 470/470 pre-rebase; post-rebase focused CI 27/27, Modal schema/semantic/config 29/29, Modal build/typecheck, lint, formatting, and diff checks

• strict Modal workflow-control seal reads: registered runtime v2 parsing replaces subset/manual parsing; Modal typecheck, node-provider 71/72 before the expected diagnostic update, corrected focused regressions 2/2, formatting and diff checks

• registered eval history automatic-publication plan and publication-generation contracts, strict pending-history/publication readers, strict Braintrust/HTTP JSON, and removed generic eval JSON cast helpers: pre-rebase full eval 303/303; final-base focused eval 42/42, publication CI 22/22, eval build/typecheck/schema registry, docs, lint, formatting, and diff checks

• strict external CI JSON boundaries: GitHub event/jobs/artifact envelopes are bounded and duplicate-key-safe before narrow projection, fetched responses validate bytes instead of using response.json(), and BENCHMARK_MODELS_JSON no longer treats whitespace as absence; focused qualification 7/7 and Modal CI-config 18/18 plus lint, formatting, and docs checks

• strict package/schema metadata reads and non-JSON cloning: CLI package discovery, validator dependency identity, and every package schema verifier now use bounded duplicate-key-safe parsing; prompt rename cloning uses structuredClone; full dependency/CLI build, CLI 77/77, prompts 50/50, lint, formatting, and diff checks

• strict eval-history publication inputs: embedded matrix, diagnostics, and summary documents require canonical base64, bounded duplicate-key-safe JSON, and their registered current eval contracts; producer benchmark configs use the canonical Modal reader; post-rebase eval/dependency builds, focused publication 16/16, lint, formatting, and diff checks, with the broader source checkpoint at CI 59/59 and Modal 474/474

• strict Modal security-sensitive JSON: launch-lock metadata, Kimi credential files and OAuth responses, judge credentials, and embedded credential staging use bounded duplicate-key-safe immutable reads; malformed-present evidence is terminal and Kimi default/trim normalization is removed; focused 134/134, full source Modal 502/502, build/typecheck, lint, formatting, and diff checks

• removed eval score-row compatibility aliases: runtime_seconds and cost_estimate are rejected in favor of the sole canonical efficiency block; analysis consumes that block directly and the three newly published history observations are current v6/report-only; full eval 303/303, eval build/typecheck/schema registry, docs, lint, formatting, and diff checks

• closed generated Smithers workflow input: exact local/cloud union, required current run/schema/task fields, bounded recursive RFC 8259 operator value, and no loose objects, unknown values, defaults, casts, or stripping; runtime build/test compilation, generated verifier 52/52, focused runtime 7/7, lint, formatting, and diff checks

• strict browser JSON boundaries: the shared duplicate-key-safe parser is browser-safe; every successful dashboard HTTP response and all SSE strings are bounded and strictly parsed before current identity/shape checks, with value-redacted failures and no frontend response.json() or raw JSON.parse; artifacts 139/139, dashboard 23/23, full dashboard typecheck, Vite/server build, docs, lint, formatting, and diff checks

• strict package-manager manifest projection: all five runtime Smithers package.json reads use bounded immutable duplicate-key-safe parsing and project only consumed package-manager-owned fields; focused package-manifest regression, runtime test compilation, lint, formatting, and diff checks

• strict config semantic helpers: agent-auth and triage helper schemas reject unknown fields with field-specific diagnostics instead of Zod stripping them; focused config 31/31, config build/typecheck/schema registry, lint, formatting, and diff checks

• hardened package projection and remaining workflow output Zod schemas: manifest maps preserve adversarial keys without prototype mutation, empty bin is invalid, and generated task/preparation/verification outputs are strict objects; runtime test compilation, focused manifest and generated/start-run tests, lint, formatting, and diff checks

• retained artifact Zod parity: analysis-bundle, invariant-ledger, invariant-source-proof, and workspace-patch parsers now match their canonical JSON Schemas on portable structure; projected uniqueness, ordering, joins, and disjointness run through named gates with exact diagnostics; a 693-mutation differential found zero mismatches; artifacts 142/142, schema snapshots, downstream runtime build/typecheck, lint, formatting, and diff checks

• repaired the extracted workspace-patch supersession harness after strict present-path/snapshot helpers changed: the test now executes the same causal-ENOENT, immutable-byte, UTF-8, and strict-JSON helpers as the generated workflow; focused 12/12, runtime/dependency build and test compilation, lint, formatting, and diff checks

• preserved preliminary severity provenance through final-report validation: severity_guess remains the agent-authored upstream estimate while final severity remains matrix-derived; runtime build/typecheck, projector 7/7, lint, formatting, and diff checks

• canonical dashboard browser wire validation: build-time standalone validators come from the composed registered schemas; successful/error HTTP, SSE, and outgoing request documents validate completely; request identity cannot be overridden; the browser bundle rejects executable CommonJS loads; dashboard 24/24, dashboard/server/browser typechecks, lint, formatting, and diff checks

• removed final-report and scoring compatibility processing: report validation now asserts IDs/order/titles/severity/provenance directly and renders the unchanged validated report, preserves preliminary and lifecycle severity fields, and eval scoring accepts only canonical evidence fields; final-report 8/8, full eval 304/304, runtime/eval typechecks, lint, formatting, and diff checks

• strict generated DeepSeek/Kimi adapter documents: bounded duplicate-key-safe JSON, immutable credential snapshots, exact current token/session/wire/state/usage contracts, causal-ENOENT-only absence, no alias conversion, and atomic validated credential publication; focused generated adapter tests 17/17, runtime build/typecheck/test compilation, lint, formatting, and diff checks

• removed the analysis-bundle recovery-summary compatibility omission: every current data kind must be included or explicitly omitted exactly once; focused analysis-bundle 8/8, artifact build/typecheck, lint, formatting, and diff checks

Current migration boundary:

• run state is now current-only v4 with closed run/task/aggregate/reference/blocker provenance, canonical output_contracts, exact task identities, and no generic JSON bags or workflow-state aliases

• the breaking event-record v2 artifact contract is published for review

• runtime event-v2 producers and workflow synchronization now compile against the closed union; the local runtime build/typecheck is green and the previous remote build boundary has been repaired

• Modal model-work corroboration now uses canonical eval readers: malformed-present summaries cannot fall back, invalid journal rows cannot be filtered, and only exact absence selects runs.jsonl

• strict JSONL snapshots now classify dangling symlinks and other present-but-unreadable journal paths as invalid rather than absent

• public eval diagnostics now have a registered closed v2 whole-document JSON Schema, a non-transforming parity-audited Zod parser, and named semantic gates for lineage, ordering, byte limits, readiness, and projected summaries; they require exact matrix/record joins and canonical run state, synthesize no rows or statuses, and fall back to the journal only on literal summary absence

• eval judging now rejects fenced JSON, duplicate keys, extra fields, missing/unknown IDs, and invalid immutable report bytes without retry or synthesized finding identities

• the obsolete repair-named event-journal API is gone; the only remaining read path validates and rejects torn or malformed bytes

• benchmark-analysis readers and writers now consume the published current-only schemas and semantic gates with exact archive paths, versions, joins, cumulative accounting, and validated output manifests

• dashboard flow/report discovery no longer replaces malformed topology or present-but-unreadable run evidence with empty/missing views

• eval telemetry and recovery now canonical-assert complete event-v2 histories with expected run identity, exact ordering/uniqueness, and real positive attempt IDs; malformed complete rows are terminal

• resume and synchronization require pinned Smithers 0.32 full-output evidence, exact run/runState/node enums and joins, and no legacy alias, state synonym, failed-task synthesis, or quota resubmission fallback

• report bundles validate any present event journal as the closed v2 history for the expected run and archive the exact validated bytes; invalid, torn, foreign, symlinked, or nonregular journals are terminal

• state export now derives counts from canonical replay, requires exact Smithers full-output/status envelopes, validates every envelope, metadata block, nested row, enum, and count without filtering or bare-data fallback, and fails closed on malformed, foreign, duplicate, symlinked, or nonregular evidence

• origin/main through a42790e is integrated; compatible source-span, Modal pinned-image, submodule hydration, retry-prompt, and published-history changes are retained without restoring legacy I/O normalization or v1 readers

• finding evidence source spans use closed structural schemas plus named host semantic gates for descending, unordered, overlapping, or nondisjoint ranges

• reference cache manifests now use only ultrafuzz.reference-cache-manifest.v1, are registered in the trusted CLI bundle, reject the legacy "1.0" spelling, execute named path uniqueness/ordering gates, and validate exact serialized bytes before publication; materialized run reference manifests are likewise shape-validated before write

• dashboard HTTP responses, SSE events, and audit records now use registered closed schemas with frontend wire validation; runtime materialize/clean audit journals validate complete immutable history before append

• lifecycle inspection now accepts only exact pinned Smithers command and stream envelopes; malformed rows, duplicate keys, ANSI/control adaptation, synthesized unknown states, aliases, extra fields, and partial objects are terminal

• eval dynamic-node lineage now consumes a named, non-empty source_node_id member of the closed run-state v4 provenance contract rather than indexing a generic provenance object

• every CLI JSON result now uses the command-discriminated current-only ultrafuzz.cli.result.v2 envelope; run accepts only mutually exclusive --input-json or --input-file, validates immutable bounded operator input, and has no legacy --input fallback

• the generic artifacts JsonValue export remains removed; intentionally opaque RFC 8259 values are scoped locally to CLI operator workflow input and redacted third-party tool envelopes

• CLI validation composes artifact, config, dashboard, eval, EVMBench, Modal, reference, runtime, and topology schema owners; public inspect state has an exact projection that structurally excludes the private execution snapshot

• validator invocation/internal/worker failures deterministically use setup-error exit 2; encoded absolute, Windows, file, traversal, and unsupported URI refs are rejected, repeated refs reuse one bounded snapshot, and truncated failures always retain a redacted diagnostic

• config model-profile helpers no longer use z.unknown() or passthrough schemas and preserve one ordered field-specific diagnostic per invalid field

• artifact manifests are current-only v3 with exactly two provenance metadata variants: pinned-reference identity/revision/expectations or Smithers concrete-node identity; v2, generic, combined, partial, and extra metadata are rejected without conversion

• analysis-bundle documents now use the shared bounded immutable strict parser, rejecting duplicate keys, invalid UTF-8, symlinks, nonregular files, and read-time mutation without rewriting bytes

• generated-agent recovery markers and authenticated workflow-link journals now use strict bounded parsing; malformed-present markers/journals are terminal and byte-preserving, and the former torn-marker auto-repair path is removed

• workflow control accepts only canonical Smithers node/run state enums, maps only current waiting states, and rejects malformed-present control timestamps instead of normalizing aliases or substituting clock values

• Modal worker-result collection validates emitted and persisted v2 bytes, treats only causal ENOENT as absent, rejects malformed/wrong-attempt/current-run evidence, and no longer retries invalid statuses or accepts historical worker statuses

• Modal recovery probes use the pinned strict run-state v4/run-plan v2 readers, treat only underlying ENOENT as absence, reject malformed remote output, and accept no historical logical-ID aliases or synthesized defaults

• Modal launch-state reads, writes, runner inspection, termination, and recovery now use the registered current-only v3 byte/schema/semantic path; duplicate keys and malformed-present evidence are terminal, and the sole writer uses the hardened canonical atomic write path

• Modal recovery-state reads/writes use the registered canonical current-only contract, only causal nested ENOENT means absent, and launch plus failure-state persistence errors are aggregated instead of silently discarding the persistence failure

• Modal worker-lineage durable state uses canonical strict read/write with a retained non-transforming Zod parity check; duplicate keys, nested read failures, and attempted overwrite are terminal

• every generated task verifies all five planned schema-binding fields against the sealed artifact registry before CLI preflight and model work; local and Modal preflights parse strict JSON, reject duplicate keys and shadow binaries, and Modal puts /usr/local/bin first on child PATH

• report ZIPs contain only the closed registered ultrafuzz.report-bundle-manifest.v2 index; the unregistered ultrafuzz.report_bundle.v1 spelling has no reader or conversion path

• Modal pinned-source proof and worker-result publication use registered canonical strict I/O; duplicate-key persisted evidence is terminal and the second private status/result writer is removed

• both benchmark cohort families now have registered whole-document schemas, and lane policy is current-only v2 with explicit trials_per_variant; the prior Zod default and v1 acceptance are removed, with nonportable cohort/policy integrity in named gates

• Modal recovery-lifecycle synthesis and collection both execute the registered schema and semantic gates; collected duplicate-key or otherwise malformed lifecycle bytes are terminal before retained Zod and secret checks

• eval report now uses the registered score-summary reader and requires real nonempty UTF-8 Markdown; schema/semantic diagnostics include bounded actionable paths instead of a generic cast or empty-text fallback

• Modal public-eval-diagnostics collection strict-parses immutable bytes before canonical schema/Zod/semantic checks, and publication uses the shared durable writer; duplicate-key documents cannot pass through either boundary

• private model-work inspection uses strict planned-graph v3/run-state v4 readers, rejects malformed-present and graph/state node mismatch, treats only ENOENT as unavailable, and durably records a conservative flag before surfacing evidence corruption

• dashboard speculative artifact-preview, transcript, run-health, and open command-argument shapes are removed; frontend contracts reuse canonical event/finding/run-state/topology types, with only the bounded opaque operator run.workflowInput seam retained intentionally

• eval suites now have a registered whole-document v2 schema and exact private/public-full/public-smoke workflow-input arms; the bounded operator object arm forbids reserved benchmark keys, no-op model/variant/metric fields are removed, active judge timeout_seconds remains strictly typed, durable diagnostics are closed, and review findings use canonical finding-v2

• benchmark smoke soft-fail readiness treats only literal diagnostics absence as unknown; any malformed-present, invalid UTF-8, duplicate-key, oversized, nonregular, symlinked, or canonical-shape-invalid evidence is terminal and never downgraded to an allowed feature-branch fallback

• the unused dashboard renamedFrom compatibility response and prompt-rename conversion/writer path are removed; create-node mutations preflight their strict audit history

• eval history, ground truth, recovery equivalence, and status are registered current-only contracts with exact serialized-byte validation, semantic gates, bounded diagnostics, and exact Modal public/private conversion arms; obsolete generic eval-result helpers are removed

• Modal benchmark configuration is current-only v2 with required fields and explicit public targets; no v1 reader, defaulting, or judge-configuration fallback remains

• public Modal preparation no longer raw-rereads the EVMbench cohort after canonical validation; ground-truth cloning consumes the same validated in-memory cohort snapshot

• privacy-safe public failure diagnostics now use the shared duplicate-key-safe parser on both the worker stdout and collected-log boundaries

• terminal disposition is a registered parity-audited root contract; workflow-sync/Modal marker readers validate canonical bytes and semantic context, malformed-present context fails closed, and neutral property projections are limited to explicitly unplanned cases

• current main through 556b5cc is integrated; its authoritative coverage derivation and CommonMark-safe projector are retained as pure validation primitives, while historical report readers, post-agent rewrites, manifest resealing, and synthesized recovery remain excluded

• benchmark cleanup and launch consume registered strict control/config readers; obsolete threat-model mode, execution/dry-run/targets compatibility fields, and their workflow/publication branches are removed, with duplicate-key and old-field regressions

• Modal execution-snapshot closure derives only from a complete registered workflow-control-integrity v2 seal and rejects duplicate keys, schema drift, generation mismatch, and path rebinding

• eval history publication plans/generations are registered current-only documents with semantic gates; pending-history merge and CAS publication consume strict typed readers, malformed provider JSON is terminal, and the exported generic eval JSON cast helpers are gone

• authoritative final-report context is supplied before generation and the post-agent projector is validation-only: current properties v2, implemented-properties v3, and report v2 outputs must equal the canonical projection; agent-owned files are never rewritten; focused projector and generated-workflow tests 72/72, runtime and dependency builds plus test compilation, lint, formatting, and diff checks

• Modal smoke checkpoint and completion evidence are registered current-only documents; workers validate the durable checkpoint and atomically publish validated bytes, while the controller validates one immutable byte snapshot; duplicate keys, unknown fields, historical/missing identities, and cross-document identities are terminal; Modal build, typecheck, registry verification, full suite 478/478, focused post-change tests 21/21, lint, formatting, and diff checks

• generated task verification now executes registered document and contextual semantic gates before publishing bytes or writing the trusted marker; schema-valid semantic failures leave no publication/marker and cannot reach descendants; focused verifier 55/55, runtime build and start-run integration, lint, formatting, and diff checks

• retained EVMBench and Modal Zod parsers now have exact portable structural parity with canonical JSON Schema; nonportable timeout, aggregate, and projected-identity rules execute as named semantic gates after strict JSON → Ajv → Zod; EVMBench 45/45, Modal 511/511, builds/typechecks/lint/format/diff checks

• final-report implementation coverage is required and fully typed: tracked reports carry all compared arrays, topologies without the track carry the exact current not-planned object, and omission/former unavailable strings fail; prompts and canonical Markdown are aligned; artifacts 143/143, focused runtime 63/63 plus coverage gate, eval 61/61, Modal bundle 20/20, prompts 50/50, schema snapshots/lint/format/diff checks

• named review gates now execute before publication: strategy detections reject duplicate composite hit identities; severity classification enforces Impact × Likelihood matrix equality and exact upstream triaged-field preservation using authenticated ancestor context in both generated and controller validators; artifacts 143/143, generated verifier 56/56, focused host integration, runtime builds, lint/format/diff checks

• bounded smoke validation now includes a single-attempt agent node that deliberately writes an invalid findings draft, requires the producer CLI to return exit 1 with JSON_SCHEMA_VIOLATION, authors the canonical [] correction in the same session, and requires exit 0 before returning; topology 39/39, prompts 50/50, prompt typecheck, formatting, and diff checks

• CLI report/report-bundle and dashboard report/findings consumers now require successful current verifier/controller authority, exact planned contract/schema/digest bindings, immutable publication bytes, current prerequisite and semantic/context gates, and a canonical report JSON/Markdown pair; focused runtime 3/3, dashboard 24/24, and the full CLI dependency/build chain pass; legacy unauthoritative CLI fixtures and a missing-authority classification regression are the immediate follow-up

• strict release-regression fixtures now match the current contracts without production fallbacks: all 12 previously named failures are corrected, and the extracted workspace-patch harness executes the real bounded immutable snapshot, strict UTF-8/duplicate-key JSON, path-safety, mutation, and causal-ENOENT helpers; artifact gates 69/69, focused runtime 3/3, replay 19/19, replay plus supersession 31/31, runtime typecheck, lint, formatting, and diff checks

• eval scoring and telemetry plus Modal public publication now consume only immutable verified final-report snapshots with exact run/path/contract/schema/digest authority; telemetry explicitly skips unverified mutation, and remote bundles enforce report schema, document semantic gates, and canonical JSON/Markdown parity; eval 306/306, effective Modal 514/514, eval/Modal typechecks, lint, formatting, and diff checks

• CLI report and bundle regressions now build exact two-node sealed verifier/controller authority rather than unauthoritative files; report JSON/Markdown bytes and SHA-256 remain unchanged through report reads plus ordinary, forced, and custom ZIP bundles; missing authority after recorded success is invalid, never absence; focused CLI report 23/23 and verified-output 4/4 with builds/test typechecks/diff checks

• remote CI formatting repair for the verified-authority diagnostic: targeted Prettier check and diff check pass; the canonical branch is republished for the release-gate rerun

• current origin/main integration through 12d716e: pinned-submodule expectations/snapshots are registered strict contracts; Smithers task manifests, agent-source proofs, and Modal pinned-source proofs carry required typed submodule evidence; no repair/conversion path was introduced; artifacts 143/143, focused runtime 63/63, workflow compilation 2/2, Modal 514/514, eval 307/307, CLI validation of checked-in history, workspace build/typecheck/lint/format/diff checks

• registered Modal public-benchmark-bundle v5: canonical Draft 2020-12 schema, composed CLI identity, Ajv-first parser, named uniqueness/publication-closure gates, and retained canonical report JSON/Markdown authority; full merged-base Modal suite 516/516

• nested pinned-submodule symlink ownership uses path-component depth, with a strict nested-repository escape regression; focused checks pass and the full runtime gate is running; rollback regression formatting is aligned with the workspace formatter

• canonical Modal state parsing: launch state, recovery lifecycle, and recovery state now consume registered schemas plus exported named semantic helpers; standalone Zod acceptance and worker-status v2 conversion are removed; unknown in-memory values use Ajv first and return the identical reference without cloning, stripping, defaulting, or repair; isolated merged-base build/typecheck, focused 97/97, full Modal 516/516, lint, and format checks pass

• exact prompt-path binding: run planning no longer falls back from a topology-declared prompt path to a same-ID catalog entry at another path; runtime/dependency builds, test compilation, focused 6/6, lint, formatting, and diff checks pass

• registered eval LLM-judge result v1: provider structured output projects directly from the canonical registered schema, host parsing is strict JSON → Ajv → the named candidate-alias context gate, schema_version is required, and the parallel Zod authority/versionless legacy object are removed; eval 307/307, eval build/typecheck, composed CLI build, docs, lint, formatting, and diff checks pass

• registered pinned NanoEval final-report and JSONL-record contracts: every recorder variant is closed and Ajv-first, retained Zod covers the same portable shape, metric relationships execute through named gates, extra.data is the sole explicitly opaque recursively typed RFC 8259 seam, and normalized EVMBench safe integers are aligned; EVMBench 56/56, package typecheck/build, composed CLI registry build, lint, formatting, and diff checks pass

• schema-specific instance byte policy: the 64 MiB default is explicit on every registry entry and included in bundle identity, while only registered Modal public-benchmark-bundle v5 permits 256 MiB; schema selection precedes the immutable instance read, the exact limit reaches async/sync workers and Modal readers/serializers, and external schemas remain capped; artifacts 144/144, Modal document boundaries 11/11, EVMBench 56/56, composed CLI build, targeted >64 MiB CLI regression, formatting, and diff checks pass

• closed retained artifact JSON Schema/Zod parity gaps: generated-test paths and implementation selections are portable and exact, run-state integers are safe, timestamps/UUIDs share canonical lexical rules, Unicode limits count code points in both validators, and node-map key equality is exclusively the existing named gate; artifacts 150/150, artifact build/typecheck/schema snapshots, runtime typecheck/dependency build, composed CLI build, lint, formatting, and diff checks pass

• shared registered EVMBench CLI result contract: the CLI-owned init/run/resume/status/report definitions are referenced directly, the duplicate Zod authority and stale validated-agent-report value are removed, expected-command matching is a named context gate, and invalid/open envelopes fail closed; EVMBench 57/57 plus typecheck, CLI 78/78 plus build, lint, formatting, JSON, and diff checks pass

• pinned submodule closure now spans local and Modal cloud handoffs: cloud tasks carry the exact expectation, sealed dependency tree, pinned commit/branch, isolated shallow Git baseline, hydration, and final verification; drift is terminal and no field repair or compatibility conversion exists; artifacts 150/150, Modal node-provider 74/74, focused runtime 62/62 across the two files after supplying the host-only npm prerequisite, runtime/Modal typechecks, lint, formatting, and diff checks pass

• full Modal release-gate closure: the private-model-work regression now asserts the canonical named run-state node-map diagnostic instead of a stale wording fragment; the focused case is 6/6 and the full Modal suite is 521/521 with lint, formatting, and diff checks passing

• eval history generation is now exact and nonconverting: producer timestamps must already be canonical UTC .sssZ instants and GitHub repository URLs must already omit a trailing slash; the old toISOString() and slash-stripping normalization paths are removed, patterns are shared across JSON Schema/Zod, and impossible dates remain a named semantic failure; evals 308/308, composed CLI/dependency build, lint, formatting, and diff checks pass

• eval analysis bundles no longer normalize or discard source timestamps: valid offset forms survive exactly, extrema are selected by instant without representation changes, invalid-present timestamps fail, and reversed launcher/aggregate completion evidence is terminal instead of omitted; evals 309/309, composed CLI/dependency build, lint, formatting, and diff checks pass

• eval reporter graph telemetry now consumes only the registered strict planned-graph document and its semantic gates; only an initial causal ENOENT is absence, while present/malformed/disappearing graphs fail without skipped nodes, synthesized logical IDs/kinds, or filtered dependencies; evals 312/312, composed CLI/dependency build, lint, formatting, and diff checks pass

• exact registered validator-preflight success envelope: artifacts-owned whole-envelope JSON Schema, CLI $ref reuse, strict JSON → registered Ajv → named identity gate across local runtime, generated Smithers, and Modal; artifacts 184/184, full CLI 81/81, runtime main stage 532 pass/20 skip plus generated adapters 18/18 and the npm-qualified pinned closure 1/1, full Modal 525/525, runtime/Modal typechecks, CLI dependency build, schema registry, workspace lint/format, and diff checks

• registered analysis-bundle manifest and six formerly internal payload documents: complete closed JSON Schemas, shared registered Ajv-first readers/writers, retained nontransforming Zod parity, and named reconciliation/order/coverage gates; artifacts 189/189 plus build/schema registry, evals 312/312, Modal 525/525, workspace format/lint/build/typecheck and diff checks pass

• exact aggregate PATH=/tmp/ultrafuzz-npm.ED7ruT/node_modules/.bin:$PATH pnpm test at 0b72159: CI scripts 59/59, security 8/8, prompts 50/50, artifacts 189/189, config 47/47, references 9/9, EVMBench 58/58, topology 39/39, runtime 532 pass/20 intentional skips plus generated adapters 18/18, evals 312/312, dashboard 24/24, Modal 525/525, and CLI 81/81

• registered current-only runtime adapter recovery marker: exact closed schema and schema-specific byte limit, strict JSON → runtime registry Ajv parsing, exact serialized-byte validation before durable publication, and target binding; focused crash recovery 2/2, runtime document/audit contracts 5/5, runtime dependency build/schema registry/test compilation, lint, and diff checks pass

• terminal/nonretryable post-agent output-validation failures: reproducible missing or schema-invalid authored output settles as a genuine failed task without resume --retry-failed or Modal replacement/recovery; authored bytes are preserved and infrastructure recovery remains enabled; Modal 527/527, focused runtime regression, runtime/Modal typechecks, and diff checks pass

• declaration-only semantic artifact resolution by exact sealed attempt, producer-declared contract/path, and authenticated artifact directory: stale loop siblings and conventional filenames are excluded; direct/transitive modes, duplicate/rebinding failures, and immutable results are covered; Runtime dependency/build/schema-registry/test compilation, focused 6/6, lint, formatting, and diff checks pass; isolated draft feat(runtime): resolve semantic artifacts by declarations #484

• current origin/main integration through 4abae99: upstream attempt-ledger deferral, bounded Kimi probe retry, and 138 current eval-history observations are retained; producer-free report rewriting, historical compatibility, workspace-mirror repair, and other conversion paths remain excluded; focused runtime merge 3/3, generated strictness 4/4, evals 312/312, history/chart verification, dependency-closure builds, test compilation, workspace lint/format, and diff checks pass; isolated draft merge(main): retain strict JSON contracts #485 records the diagnosed environment-only full-runtime cases

• reviewable stacked property-contract checkpoint 7d786eb: host and generated consumers resolve properties, implementations, references, lenses, triage, campaigns, and report coverage only through exact authenticated declared ancestor contracts and paths; reverse lens coverage is driven by the sealed plan and cannot be hidden by omitted state/catalog rows; nodes carrying multiple property-role contracts execute every applicable gate; typed projection context, stale-attempt exclusion, custom paths, invariant-suite handoffs, and unrelated-node exclusion are covered; artifacts 197/197 and Runtime artifact-gates/generated-verifier/verified-output 153/153 pass with Runtime build/typecheck/test compilation, lint, formatting, and diff checks; draft fix: consolidate authenticated JSON artifact authority #481 is published and CI is running

• reviewable stacked generated-tests v3 checkpoint 7580be0: draft feat: type generated-test support bundles #486 publishes the required explicit generated_tests plus support_files contract and initial producer/consumer migration; size/digest binding and safe-path tightening are still being completed and will be pushed as the next coherent checkpoint

Still in progress:

• review and promote draft fix: consolidate authenticated JSON artifact authority #481 after remote CI and diff inspection
• cross-artifact differential, triage, lifecycle, strategy, and final-report preservation gates
• canonical property-campaign execution/coverage and generated-test support-file producer shapes
• exhaustive acceptance audit, full release qualification, representative local and Modal smoke runs, and final independent review

Tracks #439.

[aviggiano]

I don't see Validation command anywhere

[aviggiano]

Keeping this draft open as deferred scope. Issue #439 is labeled help wanted and its maintainer deferral comment explicitly excludes it from the #413 shipping goal. The branch and commits are preserved for separately authorized future ownership; this PR is intentionally not being reviewed, fixed, or merged in the current workstream.

[aviggiano]

it looks like there's been a regression here

[aviggiano]

Code review of exact head 1cbf53e380952c20abfac82ae6dce2a300a0fc4a.

I found the following blockers and correctness issues:

1. [P1] Preserve JSON numeric semantics before Ajv validation

   parseNumber() converts every lexeme with Number(matched), which can round an invalid value into a schema-valid one before Ajv sees it. A schema requiring type: integer and maximum: 9007199254740991 incorrectly accepts the JSON instance 9007199254740991.4; ultrafuzz json validate exits 0 because the parser turns it into the integer 9007199254740991. Since the CLI and host share this parser, this can falsely authenticate invalid artifacts at both boundaries. Please retain lossless number semantics, or reject lexemes that cannot be represented without changing the schema-relevant value, and add fractional/large-number regressions.

2. [P1] Build the strict-JSON dependency before eval-history qualification

   qualify-modal-benchmark-publication.mjs now statically imports packages/artifacts/dist/index.js, but the qualify_modal_benchmark job only checks out the repository, sets up Node, and invokes the script. dist/ is ignored and absent from a clean checkout. Reproducing from git archive HEAD exits immediately with ERR_MODULE_NOT_FOUND before any eligibility output is written, so no successful Modal benchmark can reach automatic history publication. Install/build the required package in this job or keep the qualifier self-contained.

3. [P1] Update the runtime gate suite for canonical report IDs

   The new gate now requires M-01 and its prefixed title, but the positive fixture in final report gates preserve severity records... still emits the upstream ID/title, while its negative case now uses the newly correct M-01 ID. pnpm --filter @ultrafuzz/runtime test fails at this head; the isolated test is 1/1 failing with the canonical-ID/title diagnostics. Redesign both halves of the regression around the new presentation identity.

4. [P1] Tell the final-report producer how to retain the upstream property finding ID

   The prompt now requires every production issue to be renamed to H-01/M-01/L-01, and canonical projection requires property_provenance.finding_id to use that report ID. However, campaign backend lookup can recover the original campaign ID only through property_provenance.source_finding_id, which the prompt never names and the schema marks optional. Following the prompt with { finding_id: "H-01", fuzzer_backend: "recon" } deterministically fails PROPERTY_REPORT_FUZZER_BACKEND_MISMATCH; using the original ID instead fails canonical projection. Explicitly require the authenticated upstream ID in source_finding_id whenever presentation renumbers it, and add a prompt/conformance regression.

5. [P2] Resolve external references under nested $id scopes

   rewriteAndLoadReferences() always resolves against path.dirname(referringPath), ignoring Draft 2020-12 base-URI changes introduced by nested $id. In a root schema with $defs.scoped = { "$id": "sub/", "$ref": "child.json" }, the command loads root-level child.json instead of sub/child.json; with different const values in those files it returns valid for the wrong instance. Track the current schema resource/base URI while walking and resolve each reference using Draft 2020-12 URI rules before applying the local-root safety check.

6. [P2] Do not interpret data inside const/enum as subschemas

   The same walker recurses through every nested object, including instance-valued keywords. A valid schema containing const: { "$ref": "missing.json" } and a matching instance should pass, but the command returns setup-error after trying to open missing.json. Reference discovery needs to traverse only schema-valued keyword positions.

Validation at this SHA:

• @ultrafuzz/artifacts: 239/239 passed
• @ultrafuzz/prompts: 72/72 passed
• dashboard: 35/35 passed
• focused eval/history/publication/recovery: 123/123 passed
• formatting, lint, and git diff --check: passed
• @ultrafuzz/runtime: failed as described in finding 3 (isolated repro confirmed)

The PR was still a conflicting draft, with remote release and Modal collection checks in progress, when this review was posted.

[aviggiano]

Addressed all six blockers from the exact-head review on final pushed head 66d59f9e. This head also merges current main at 98f22bee.

• Strict JSON rejects value-changing numeric lexemes before Ajv, including fractional-to-integer, out-of-range integer, decimal-rounding, and underflow cases. It also has positive regressions for exactly representable large positive and negative integers whose Number#toString spelling is scientific notation.
• Eval-history qualification performs a frozen install and builds @ultrafuzz/artifacts plus dependencies before importing the qualifier. An exact staged-index clean checkout independently passed frozen install, dependency build, and qualifier import.
• Runtime report fixtures use canonical presentation IDs/titles and reject the outdated upstream presentation identity.
• Renumbered authenticated property findings require property_provenance.source_finding_id; prompt anchors and runtime regressions cover missing, mismatched, and valid provenance.
• External references resolve relative to nested Draft 2020-12 $id scopes while retaining local-root containment.
• Reference discovery walks schema-valued keyword positions only, so const and enum instance data containing $ref are not loaded as schemas.

Final local validation on 66d59f9e:

• focused and package suites: prompts 73/73; artifacts 241/241; config 54/54; topology 51/51; evals 392/392; Modal 554/554; CLI 149/149; CI scripts 62/62
• full runtime: 706 passed, 21 skipped, 0 failed; generated-agent Bun lane 19/19
• pnpm validate:release: all 13 required commands passed, including docs, security, package suites, benchmark history, and workspace typecheck
• workspace format, ESLint, build, typecheck, and staged diff checks: passed
• clean-checkout qualifier proof: frozen install, artifacts dependency build, and qualifier import passed

Exact-head GitHub evidence: draft/build gates passed install, formatting, lint, and build; both Socket security checks passed; the draft release gate passed and the full release-validation lane was skipped by draft policy. Semgrep passed with zero annotations. PR #457 remains open, draft, and unmerged. No review thread was resolved.

[greptile-apps]

Too many files changed for review (647 files, 500 file limit).

[aviggiano]

Superseded by merged PRs #558 and #563. The strict producer-visible contracts and follow-up CI fixes are now on main at ee933f2, with core CI, Modal benchmarks, and downstream eval-history publication all green.