[aviggiano]

Fixes #1133

The codebase still contained a Braintrust HTTP reporter, credential forwarding, remote telemetry publication, and an implicit Braintrust gateway for optional LLM judging after the paid-CI cleanup. This removes those integrations and the eval publish CLI/library API. Local eval history, durable artifacts, reports, deterministic scoring, comparison, bundles, and generic telemetry observers remain available.

Reporting accepts only none; unsupported providers fail before credential access, network activity, or run-state creation. Historical provider metadata can still decode but cannot activate a reporter, and an explicit --provider none can override legacy settings. Optional LLM judging now requires both ULTRAFUZZ_EVAL_JUDGE_URL and its dedicated judge key, preserving HTTPS validation, redirect rejection, bounded responses, and private-data consent. There is no implicit replacement service.

Migration: Modal benchmark configuration is now ultrafuzz.modal.benchmark.v3, with a vendor-neutral judge object and no reporting credentials or provider switch. Regenerate configurations for new runs using the documented migration. Existing v2 configurations are rejected rather than rewritten; existing sandboxes can still be terminated using their launch state alone. Updated manual benchmark generators, collection tooling, schemas, fixtures, defaults, and documentation follow this contract.

Validation passed without paid inference or cloud workloads:

• 421 eval tests, 678 Modal tests, 94 configuration tests, 27 affected CLI tests, and 102 CI-script tests; affected TypeScript checks and CLI test compilation.
• Workspace build, formatting, changed-code strict lint, Knip, size limits, documentation checks, and Markdown lint.
• Installed-package validation for all 12 packages. Seeded stale compiled reporting files were removed by the build, and the installed-payload regression confirms the retired modules are absent.
• Parallel implementation followed by independent review; no outstanding findings.

The repository's BRAINTRUST_API_KEY Actions secret was already removed; provider-side key revocation remains an account-owner operation.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding previous finding or newly identified actionable defect.

Findings

1.  Provider documentation is stale ▶

Fix with agent prompt

### Issue 1
docs/reference/cli.md:undefined-637
Only `none` is now supported, but this reference still says provider bindings and credential names configure a reporter and that `eval run` streams telemetry to that provider. This describes a configuration path that no longer exists and can mislead users. Update the prose and command summary to describe local telemetry only; `docs/cli.md:36` contains the same stale wording.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Removes Braintrust reporting, credential forwarding, remote telemetry publication, and the eval publish surface while preserving local evaluation artifacts and generic telemetry observers.

• Restricts evaluation reporting to the local none provider and keeps historical provider metadata inert.
• Requires an explicit HTTPS judge URL and dedicated credential for optional LLM judging.
• Migrates Modal benchmark configuration to the vendor-neutral v3 judge contract.
• Updates schemas, CLI contracts, runtime behavior, tests, packaging validation, and documentation.
• The latest documentation changes resolve the previous stale-provider finding.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Eval plan] --> B[Eval run]
  B --> C[Local journals and artifacts]
  C --> D[Deterministic scoring]
  C --> E[Local reports and comparisons]
  C --> F[Generic telemetry observers]
  D -->|Optional LLM judge| G{Explicit judge URL and key configured?}
  G -->|Yes| H[HTTPS judge endpoint]
  G -->|No| I[Fail closed]
  C -. no built-in reporter .-> J[No remote Braintrust publication]

Loading

_{Reviews (4) · Last reviewed commit: "Remove obsolete eval publishing guidance"}

[greptile-apps]

Provider documentation is stale

Only none is now supported, but this reference still says provider bindings and credential names configure a reporter and that eval run streams telemetry to that provider. This describes a configuration path that no longer exists and can mislead users. Update the prose and command summary to describe local telemetry only; docs/cli.md:36 contains the same stale wording.

Prompt To Fix With AI

This is a comment left during a code review.
Path: docs/reference/cli.md
Line: 637

Comment:
**Provider documentation is stale**

Only `none` is now supported, but this reference still says provider bindings and credential names configure a reporter and that `eval run` streams telemetry to that provider. This describes a configuration path that no longer exists and can mislead users. Update the prose and command summary to describe local telemetry only; `docs/cli.md:36` contains the same stale wording.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!