Repository navigation
Remove Braintrust reporting and implicit judge gateway integration #1133
Description
Remove the remaining Braintrust integration from Ultrafuzz after the paid-CI credential cleanup in #1129. The integration still includes a custom HTTP reporter, post-hoc telemetry publication, provider configuration and credential forwarding, and an implicit Braintrust gateway used by optional LLM judging.
The desired behavior is local evaluation/reporting without a Braintrust account, key, endpoint, or upload path. Optional LLM judging remains available only with an explicitly configured endpoint and judge credential; do not replace Braintrust with another implicit paid service.
- Delete the Braintrust reporter/client, factory registration, exports, credential lookup, and trusted-endpoint environment handling. Reject retired or unsupported reporting providers before any credential access or network request.
- Remove the now-unused
eval publishcommand, its CLI result contract, and its provider-backed replay implementation. Retain localeval historypublication, durable journals, reports, scoring, comparison, bundles, and generic telemetry code that still has live consumers. - Remove Braintrust profiles from packaged defaults, editable repository config, fresh initialization, schemas/examples, and user-facing help. Update config/schema fixtures and contract digests consistently. Historical data must never silently reactivate an external reporter.
- Remove Braintrust credentials and reporting commands from Modal workers, launch/resume paths, and generated benchmark configurations. Keep manual benchmark execution and explicitly configured judging functional.
- Replace the misleading required
braintrustModal configuration object with a vendor-neutraljudgeobject containing only judge settings. Remove telemetry project/key fields and the private benchmark reporting-provider switch. Version the changed benchmark configuration contract and clearly reject obsolete configs; document regeneration/migration rather than silently reinterpreting a sealed launch configuration. - Remove the default
gateway.braintrust.devjudge endpoint. Optional LLM judging must require an explicitULTRAFUZZ_EVAL_JUDGE_URLand its dedicated judge credential. Preserve private-data consent, HTTPS/endpoint validation, redirect rejection, and bounded response handling. Deterministic scoring stays local by default. - Update retained benchmark generation, validation, collection/history tooling and their fixtures for the new contract. Keep security checks applicable to the retained local/manual paths.
- Remove obsolete documentation and dependencies, if any. Residual Braintrust mentions should be limited to intentional migration guidance or regression cases proving retired settings/credentials cannot enable access.
- Add meaningful offline regressions proving local evals and reports work without Braintrust, retired providers fail before key/network access, and no implicit judge gateway exists. Run affected package/CLI/schema tests, build, strict lint, docs, dead-code, and packaging checks without paid inference or cloud workloads.
Acceptance: no runnable Braintrust client, default gateway, credential-forwarding path, or active configuration remains. Local evaluation workflows and deliberately configured judge calls continue to work. The separate PR must include Fixes for this issue.
Implementation will use a separate worktree and PR on top of #1132, keeping the Braintrust removal diff separate from the existing CI and audit-profile changes. BRAINTRUST_API_KEY has already been removed from this repository's Actions secrets; provider-side credential revocation remains an account-owner operation.
Activity