Skip to content

Remove Braintrust reporting and implicit judge gateway integration #1133

Closed

Description

@aviggiano

Remove the remaining Braintrust integration from Ultrafuzz after the paid-CI credential cleanup in #1129. The integration still includes a custom HTTP reporter, post-hoc telemetry publication, provider configuration and credential forwarding, and an implicit Braintrust gateway used by optional LLM judging.

The desired behavior is local evaluation/reporting without a Braintrust account, key, endpoint, or upload path. Optional LLM judging remains available only with an explicitly configured endpoint and judge credential; do not replace Braintrust with another implicit paid service.

  • Delete the Braintrust reporter/client, factory registration, exports, credential lookup, and trusted-endpoint environment handling. Reject retired or unsupported reporting providers before any credential access or network request.
  • Remove the now-unused eval publish command, its CLI result contract, and its provider-backed replay implementation. Retain local eval history publication, durable journals, reports, scoring, comparison, bundles, and generic telemetry code that still has live consumers.
  • Remove Braintrust profiles from packaged defaults, editable repository config, fresh initialization, schemas/examples, and user-facing help. Update config/schema fixtures and contract digests consistently. Historical data must never silently reactivate an external reporter.
  • Remove Braintrust credentials and reporting commands from Modal workers, launch/resume paths, and generated benchmark configurations. Keep manual benchmark execution and explicitly configured judging functional.
  • Replace the misleading required braintrust Modal configuration object with a vendor-neutral judge object containing only judge settings. Remove telemetry project/key fields and the private benchmark reporting-provider switch. Version the changed benchmark configuration contract and clearly reject obsolete configs; document regeneration/migration rather than silently reinterpreting a sealed launch configuration.
  • Remove the default gateway.braintrust.dev judge endpoint. Optional LLM judging must require an explicit ULTRAFUZZ_EVAL_JUDGE_URL and its dedicated judge credential. Preserve private-data consent, HTTPS/endpoint validation, redirect rejection, and bounded response handling. Deterministic scoring stays local by default.
  • Update retained benchmark generation, validation, collection/history tooling and their fixtures for the new contract. Keep security checks applicable to the retained local/manual paths.
  • Remove obsolete documentation and dependencies, if any. Residual Braintrust mentions should be limited to intentional migration guidance or regression cases proving retired settings/credentials cannot enable access.
  • Add meaningful offline regressions proving local evals and reports work without Braintrust, retired providers fail before key/network access, and no implicit judge gateway exists. Run affected package/CLI/schema tests, build, strict lint, docs, dead-code, and packaging checks without paid inference or cloud workloads.

Acceptance: no runnable Braintrust client, default gateway, credential-forwarding path, or active configuration remains. Local evaluation workflows and deliberately configured judge calls continue to work. The separate PR must include Fixes for this issue.

Implementation will use a separate worktree and PR on top of #1132, keeping the Braintrust removal diff separate from the existing CI and audit-profile changes. BRAINTRUST_API_KEY has already been removed from this repository's Actions secrets; provider-side credential revocation remains an account-owner operation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions