Remove Braintrust reporting and implicit judge gateway

docs/cli.md

@@ -33,14 +33,13 @@ accept `--json` and emit the `ultrafuzz.cli.result.v2` envelope.
 | `clean <run-id>`          | Remove selected generated paths after confirmation and path checks.                                                       |
 | `dashboard`               | Serve the local loopback dashboard and API.                                                                               |
 | `eval plan`               | Dry-run an eval suite matrix without launching workflows.                                                                 |
-| `eval run`                | Launch runs for an eval suite matrix and stream node telemetry to the configured provider.                                |
+| `eval run`                | Launch runs for an eval suite matrix and watch local progress.                                                            |
 | `eval status <id>`        | Show disclosure-safe node progress and ETA for every row in an eval matrix.                                               |
 | `eval score <id>`         | Score finished eval run reports against external ground truth, optionally with `--llm-judge`.                             |
 | `eval report <id>`        | Show the scored eval run variant ranking.                                                                                 |
 | `eval compare <id>`       | Compare scored eval variants against a `--baseline` variant.                                                              |
 | `eval analyze <type>`     | Generate private offline benchmark analysis from a finalized handoff ZIP.                                                 |
 | `eval history [id]`       | Validate/render public eval history, or append one complete scored run.                                                   |
-| `eval publish <id>`       | Replay a recorded eval run's node telemetry to a provider post hoc.                                                       |
 
 The dashboard/API is a local operator surface over product state, not a
 workflow-engine API.
@@ -176,15 +175,15 @@ explicit `--copy` selections for files you have reviewed.
 
 ## Eval Commands
 
-`eval plan | run | status | score | report | compare | bundle | analyze | history | publish` drive eval suites that
+`eval plan | run | status | score | report | compare | bundle | analyze | history` drive eval suites that
 benchmark the pipeline against targets with known ground-truth bugs. The suite
 YAML (default from `[eval].eval_config`, overridable with `--suite`) defines
-the experiment; the `ultrafuzz.toml` `[eval]` section binds the reporting
-provider (`braintrust | none`) and credential env-var names.
+the experiment; the `ultrafuzz.toml` `[eval]` section provides the suite and
+ground-truth paths. Only local reporting (`provider = "none"`) is supported.
 Common flags:
 
 - `--suite <suite-yaml-path>` (plan, run)
-- `--provider <name>` (plan, run, publish)
+- `--provider none` (plan, run)
 - `--target-root <path>` (plan, run)
 - `--ground-truth-root <external-path>` (plan, run)
 - `--row <row-id>` (run, repeatable)
@@ -201,7 +200,6 @@ Common flags:
 - `--history <path>` and `--charts <directory>` (history)
 - `--publication-url <url>` (history source bundle location, required when appending)
 - `--check` (history validation without writes)
-- `--resume` (publish)
 
 Artifacts land under `.ultrafuzz/evals/runs/<eval-run-id>/`. See
 [Eval Suites](reference/evals.md) and the

docs/config.md

@@ -465,11 +465,9 @@ before launch.
 
 ## Eval suites
 
-The `[eval]` section binds eval suites to a reporting provider per
-environment. It holds the default suite path, the machine-specific
-`ground_truth_root` (which must resolve outside the repository), the active
-`provider` (`braintrust` or `none`), and per-provider
-`[eval.providers.<name>]` connection profiles containing env-var _names_ only
-(matching the `[agents.CodexAgent] api_key_env` pattern). The committable
-experiment definition lives in the eval YAML referenced by `eval_config`. See
-[docs/reference/evals.md](reference/evals.md).
+The `[eval]` section holds the default suite path and the machine-specific
+`ground_truth_root`, which must resolve outside the repository. Reporting uses
+`provider = "none"`; no external reporter is installed. The committable
+experiment definition lives in the eval YAML referenced by `eval_config`.
+Optional LLM judging requires its own explicitly configured HTTPS endpoint and
+judge credential. See [Eval Suites](reference/evals.md).

docs/how-to/run-evals-on-modal.md

@@ -1,8 +1,8 @@
 # Run evals on Modal
 
-Ultrafuzz can run long evaluation rows in Modal sandboxes while reporting live
-telemetry and scores to Braintrust. The runner uses the Modal TypeScript SDK;
-Python is not required.
+Ultrafuzz can run long evaluation rows in Modal sandboxes while retaining
+telemetry, scores, and reports in the run workspace. The runner uses the Modal
+TypeScript SDK; Python is not required.
 
 ## Security and storage boundaries
 
@@ -23,7 +23,7 @@ Non-secret run state is stored under `/data/<run-id>/<model>/workspace` on a
 private Modal Volume. This preserves Ultrafuzz state, generated tests, reports,
 and Smithers workspaces when a sandbox exits. Each sandbox has a 24-hour
 timeout; eval watching stops two hours earlier so terminal persistence, scoring,
-publishing, and volume flushing retain a bounded completion window.
+and volume flushing retain a bounded completion window.
 
 Benchmark sandboxes reserve 16 physical CPU cores (32 vCPUs) and 32 GiB of
 memory, with a 64 GiB memory limit. The Modal benchmark worker override
@@ -87,15 +87,15 @@ refresh-token rotation remains single-writer.
 
 ## Create a private runtime config
 
-The v2 document is an exact contract: every operational value is explicit.
+The v3 document is an exact contract: every operational value is explicit.
 The loader does not add models, loop counts, image names, credential names,
-timeouts, reporting settings, or target selections. Configure each model row you
+timeouts, judge settings, or target selections. Configure each model row you
 intend to run; the public benchmark config generator sets `loops = 1` for its
 smoke and full lanes.
 
 ```json
 {
-  "schema_version": "ultrafuzz.modal.benchmark.v2",
+  "schema_version": "ultrafuzz.modal.benchmark.v3",
   "run_id": "example-run",
   "app_name": "ultrafuzz-evals",
   "image_name": "ultrafuzz-security-runner:latest",
@@ -112,15 +112,10 @@ smoke and full lanes.
   "benchmark_execution": {
     "excluded_node_ids": []
   },
-  "eval_reporting": {
-    "provider": "braintrust"
-  },
-  "braintrust": {
-    "project": "private-evals",
-    "api_key_env": "BRAINTRUST_API_KEY",
-    "judge_api_key_env": "OPENAI_API_KEY",
-    "judge_url": "https://api.openai.com/v1/chat/completions",
-    "judge_credential_ttl_seconds": 57600
+  "judge": {
+    "api_key_env": "OPENAI_API_KEY",
+    "url": "https://api.openai.com/v1/chat/completions",
+    "credential_ttl_seconds": 57600
   },
   "node_timeout_seconds": 7200,
   "loops": 3,
@@ -172,9 +167,29 @@ inside the sandbox, receives the target binding above, and preserves it in the
 converted scorer input. A count mismatch fails before evaluation starts.
 
 For judges that require short-lived credentials, configure an HTTPS
-`braintrust.judge_credential_endpoint`. The worker requests a model-scoped
+`judge.credential_endpoint`. The worker requests a model-scoped
 credential in memory immediately before scoring and never persists it.
 
+### Migrate old benchmark configurations
+
+Fresh launches require `ultrafuzz.modal.benchmark.v3`. Regenerate public pair
+configs with the current generator. For a fresh private run, replace the old
+`braintrust` object with `judge`, map `judge_api_key_env` to `api_key_env`,
+`judge_url` to `url`, `judge_credential_endpoint` to `credential_endpoint`, and
+`judge_credential_ttl_seconds` to `credential_ttl_seconds`. Remove the old
+reporting `project`, `api_key_env`, and top-level `eval_reporting` settings.
+The new `judge.api_key_env` names the judge credential, not the retired
+reporting credential.
+
+Do not rewrite a saved configuration for an existing launch: its exact bytes
+are part of the launch identity. Current config-based launch, resume,
+collection, and cleanup reject old v2 documents. Existing sandboxes can still
+be stopped using their launch state without loading an old config:
+
+```bash
+ultrafuzz-modal terminate --state /path/to/launch-state.json
+```
+
 ### Curated private lanes
 
 `benchmark_execution.excluded_node_ids` prunes named nodes from the production
@@ -223,26 +238,26 @@ pnpm --filter @ultrafuzz/modal... build
 candidate="$(git rev-parse HEAD)"
 generation="$(date +%s)-1"
 control=".ultrafuzz/modal/public-$generation"
-node scripts/ci/verify-cohort-reachability.mjs --lane smoke
+mode=smoke
+node scripts/ci/verify-cohort-reachability.mjs --lane "$mode"
 node scripts/ci/prepare-modal-benchmarks.mjs \
   "$candidate" https://github.com/monad-developers/ultrafuzz \
-  "$generation" "$control" smoke
-node scripts/ci/validate-modal-benchmark-launch.mjs "$control/manifest.json" . smoke
+  "$generation" "$control" "$mode"
+node scripts/ci/validate-modal-benchmark-launch.mjs "$control/manifest.json" . "$mode"
 ```
 
 Use a fresh positive `number-attempt` generation ID for each plan; it is a
 local lineage identifier and does not require a GitHub run. Resolve any
-reachability failure before paying for an image or sandbox. Select `full` in
-all three commands only when intending the complete four-provider cohort.
+reachability failure before paying for an image or sandbox. Set `mode=full`
+only when intending the complete four-provider cohort.
 Review `manifest.json` and every generated pair config before proceeding.
 An explicit `BENCHMARK_MODELS_JSON` can override the runner selection while
 the validator enforces the lane's provider count and target policy.
 
 Export `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`, and the selected pair's provider
 key on the launcher host. Every public pair also needs `OPENAI_API_KEY` for
-its judge. Public rows score local artifacts and do not require
-`BRAINTRUST_API_KEY`. Do not add these credentials to Actions secrets or
-commit them with a config.
+its explicitly configured judge. Reporting requires no additional credential.
+Do not add these credentials to Actions secrets or commit them with a config.
 
 For the single-pair smoke plan, the following commands start paid compute:
 
@@ -269,12 +284,44 @@ pnpm exec ultrafuzz-modal status --state "$state"
 If a launch did not persist its state, the retained
 `prepare-modal-benchmark-cleanup.mjs` and `terminate-modal-benchmark.sh` helpers
 can validate a preserved plan and terminate its exact build and pair scopes.
-Use trusted tooling and confirm uncertain remote cleanup before retiring
-credentials. Keep cleanup access available until the detached work is stopped.
+Run the following from the same clean, reviewed candidate checkout with its
+built tooling. Reuse the original `candidate`, `generation`, `mode`, and
+`control` values recorded at launch; do not derive them from an untrusted
+manifest. The validator requires the checkout to match that candidate and
+checks every pair against its benchmark policy before writing the cleanup list:
+
+```bash
+cleanup_pairs="$control/cleanup-pairs.tsv"
+node scripts/ci/prepare-modal-benchmark-cleanup.mjs \
+  "$control/manifest.json" "$cleanup_pairs" "$candidate" \
+  https://github.com/monad-developers/ultrafuzz "$generation" "$mode" . &&
+  bash scripts/ci/terminate-modal-benchmark.sh \
+    "$control" "$control" "$cleanup_pairs" "$generation" true .
+```
+
+Use a new cleanup-list filename if it already exists; validation refuses to
+overwrite it. The two `control` arguments are the plan and state directories.
+`true` tells the helper to use any preserved state files first; missing state
+files do not prevent cleanup by the validated build and pair scopes. Both
+cleanup passes run even when an individual termination is uncertain, and any
+uncertainty makes the helper fail. Confirm remote cleanup before retiring
+credentials, and keep cleanup access available until detached work is stopped.
+
+After a pair completes, collect and extract its validated public bundle using
+the exact config and state selected for that pair:
+
+```bash
+results="$control/results"
+model_slug="$(jq -er '.models[0].slug' "$config")"
+pnpm exec ultrafuzz-modal collect \
+  --state "$state" --public-results --config "$config" --output "$results" &&
+  pnpm exec ultrafuzz-modal unpack-public \
+    --bundle "$results/$model_slug/public-results.json" \
+    --output "$control/unpacked/$model_slug"
+```
 
-After a pair completes, collect its validated public bundle with `collect
---public-results --config <path>` and extract it with `unpack-public`. Preserve
-all scored rows and candidate/target lineage. A complete generation can then
+Collection writes each model's bundle under its slug in the results directory.
+Preserve all scored rows and candidate/target lineage. A complete generation can then
 be appended locally with [manual history publication](run-evals.md#publish-longitudinal-history),
 and the reviewed history/chart changes submitted through a normal pull
 request. Existing charts stay unchanged until such a publication.
@@ -435,8 +482,8 @@ identities, terminal states, report-presence flags, diagnostic codes, exact
 launch lineage, and a bounded failed-node projection (node ID, status, timeout
 flag, and allowlisted failure category or code). It never contains messages,
 paths, findings, or provider output.
-This lets failed Actions runs publish useful lifecycle evidence without
-repeating paid model work. Investigate arbitrary sensitive run data on the
+This preserves useful lifecycle evidence from failed runs without repeating
+paid model work. Investigate arbitrary sensitive run data on the
 private volume under the repository's normal access controls. Collection validates each contract and
 generic log line before writing locally and refuses pre-hardening or malformed
 volume artifacts.

docs/how-to/run-evals.md

@@ -2,8 +2,8 @@
 
 Eval suites benchmark the Ultrafuzz pipeline against targets with known
 ground-truth bugs and rank prompt or topology variants by precision, recall,
-and F1. The full loop is `plan → run → score → report → compare`, with an
-optional `publish` step that mirrors node telemetry to an eval cloud provider.
+and F1. The full loop is `plan → run → score → report → compare`. Evaluation
+records, telemetry, scores, and reports stay in local artifacts.
 
 ## Configure The Suite
 
@@ -14,15 +14,14 @@ Two files split the configuration:
   counts, grading metrics, and the telemetry policy. It never names a
   provider, an endpoint, or an env var.
 - The `ultrafuzz.toml` `[eval]` section is per-environment: the default suite
-  path, the machine-specific `ground_truth_root`, the active `provider`
-  (`braintrust | none`), and `[eval.providers.<name>]` profiles
-  holding credential env-var _names_.
+  path and the machine-specific `ground_truth_root`. `provider = "none"` is
+  the only supported reporting selection; no external reporter is installed.
 
 ```toml
 [eval]
 eval_config = ".ultrafuzz/evals/bug-finding.yml"
 ground_truth_root = "/secure/eval-ground-truth"
-provider = "none" # keep the local loop offline; switch when publishing
+provider = "none" # evaluation reporting stays local
 ```
 
 Ground truth must live outside the repository; suite targets reference
@@ -52,8 +51,8 @@ ultrafuzz eval plan --target-root /path/to/target-checkouts
 `plan` validates the config and suite and prints the target × variant × trial
 matrix without launching anything. Local target checkouts are expected under
 `--target-root`, one directory per target id, at the suite's pinned git refs;
-pass `--skip-target-validation` to skip the ref check. An unknown provider or
-missing provider profile fails here, before any run is launched.
+pass `--skip-target-validation` to skip the ref check. A reporting provider
+other than `none` fails here, before any run is launched.
 
 Ordinary eval YAML requires an explicit `trials_per_variant`. The public
 benchmark lane manifests under `benchmarks/` default it to `1` when omitted.
@@ -69,9 +68,8 @@ ultrafuzz eval run --row <row-id> --no-watch
 ```
 
 `run` launches an Ultrafuzz run per matrix row (or a `--row` selection),
-polls the runs to a terminal state, and streams node telemetry to the active
-provider when the suite enables it. `--no-watch` launches detached without
-polling or streaming. Artifacts accumulate under
+polls the runs to a terminal state, and records local telemetry when the suite
+enables it. `--no-watch` launches detached without polling. Artifacts accumulate under
 `.ultrafuzz/evals/runs/<eval-run-id>/`.
 
 ## Score And Compare
@@ -94,36 +92,31 @@ variant IDs; an explicit `--allow-incompatible` waiver preserves and reports
 any differences.
 Deterministic scoring requires neither a provider nor provider credentials.
 
-The optional gateway judge has a separate credential and consent boundary:
+The optional LLM judge requires an explicit endpoint, dedicated credential,
+and consent for private data:
 
 ```bash
+export ULTRAFUZZ_EVAL_JUDGE_URL=https://your-judge.example/v1/chat/completions
 export ULTRAFUZZ_EVAL_JUDGE_API_KEY=...
 # Required only when scoring a target marked sensitivity: private:
 export ULTRAFUZZ_EVAL_JUDGE_ALLOW_PRIVATE_DATA=true
 ultrafuzz eval score <eval-run-id> --llm-judge
 ```
 
-An optional `ULTRAFUZZ_EVAL_JUDGE_URL` must be HTTPS without embedded
-credentials. Judge and reporter requests do not follow redirects.
+`ULTRAFUZZ_EVAL_JUDGE_URL` must be HTTPS without embedded credentials. Judge
+requests do not follow redirects. There is no default gateway and no fallback
+to a reporter or model-provider credential. LLM judging is an explicit paid
+operation; deterministic scoring needs neither setting.
 
-## Publish Telemetry (Optional)
+## Migrate retired reporting settings
 
-```bash
-ultrafuzz eval publish <eval-run-id> --provider braintrust
-ultrafuzz eval publish <eval-run-id> --resume
-```
-
-`publish` replays a recorded eval run's journals from offset 0 and
-reconstructs the entire node trace (spans, heartbeats, manifests, artifacts,
-scores) on the provider after the fact — useful for CI runs that executed
-with reporting off or for backfilling a newly added provider. `--resume`
-continues from the persisted publish cursor instead of replaying. Missing
-credential env vars fail only at this step. Reporter failures degrade to
-warnings and never fail an eval run.
-
-For sensitive targets, the suite's `reporting.artifacts` policy defaults to
-`manifest-only`: providers see the DAG, timings, and file names/hashes while
-payloads stay on disk.
+Braintrust reporting and `eval publish` have been removed. Delete
+`[eval.providers.braintrust]` from project configuration and set
+`[eval].provider = "none"`. Remove any `ULTRAFUZZ_EVAL_PROVIDER=braintrust`
+override; `--provider none` can explicitly override old environment or project
+settings. Legacy connection metadata can still be read from saved
+configuration, but it cannot enable a reporter or cause credential access.
+Local `eval history`, report, compare, and bundle commands remain available.
 
 The pinned Ultrafuzz-bench and EVMBench targets are public open-source benchmark
 fixtures. Their Modal lane explicitly publishes the allowlisted `report.md`,
@@ -221,7 +214,7 @@ immutable identity, not the later documentation commit. The ordinary CI
 without launching models or Modal compute.
 
 The eval summary and comparison record Ultrafuzz runner tokens and runner cost
-with explicit completeness. Judge usage in Braintrust and sandbox spend in
-Modal remain separate provider-side records keyed by the immutable run IDs; use
-those three sources together for the offline frequency/cost review rather than
-treating the runner ledger as total spend.
+with explicit completeness. Keep judge-service and Modal billing records
+separate and correlate them with the run window, model, or explicit provider
+metadata. Judge requests do not automatically attach an eval run ID. Use these
+records together for cost review; the runner ledger alone is not total spend.