Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,10 @@
- Fixes durable resume hydration so it runs after the engine resets stale in-progress attempts; upgrading alone would have restored a task as finished while the durable record said pending (#274).
- A partially installed workflow-runner dependency tree now reinstalls itself instead of failing every later resume of a durable run (#274).
- `ultrafuzz doctor` now reports a posture for every workflow-engine compatibility patch instead of only the two CLI patches, and its registry check survives the upstream package rename so the upgrade signal stays alive (#274).
- Adds first-class DeepSeek V4 Pro support through DeepSeek's official Claude
Code endpoint, including cache-aware token telemetry, first-party pricing,
Modal benchmark plumbing, and public benchmark coverage (#163).
- Adds first-class DeepSeek V4 Pro support through DeepSeek's
Anthropic-compatible Claude Code endpoint, including cache-aware token
telemetry, first-party pricing, Modal benchmark plumbing, and public benchmark
coverage (#163).
- Fixes Kimi token accounting so UltrafuzzBench publishes tokens and an API-comparison cost for Kimi runs, and pins Kimi model pricing to the Moonshot provider (#162).

## v0.0.4
Expand Down
23 changes: 21 additions & 2 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,15 @@ at a specific Codex config directory.
### OpenRouter through Codex

The v0.1.0 Codex workflow can use an OpenRouter catalogue model through an
isolated Codex provider configuration. Select the catalogue ID and name the
same credential variable in Ultrafuzz that the provider configuration reads:
isolated Codex provider configuration. This is a compatibility pairing: the
Codex CLI harness pointed at a gateway rather than at OpenAI. Two separate
things are true of it. The pairing is not selected by default — the shipped root
config points `[models.default]` at `CodexAgent` against OpenAI, and reaching
OpenRouter means editing `[models.default]` and `[agents.CodexAgent]` as shown
below; there is no OpenRouter profile or agent ref to select and no `--agent`
value that reaches it. Separately, once OpenRouter is the provider, Codex CLI is
the harness this release ships for it. Select the catalogue ID and name the same
credential variable in Ultrafuzz that the provider configuration reads:

```toml
[models.default]
Expand Down Expand Up @@ -205,6 +212,18 @@ auth = "api-key"
api_key_env = "DEEPSEEK_API_KEY"
```

This is a compatibility pairing: it runs the Claude Code harness against
DeepSeek's Anthropic-compatible endpoint, not DeepSeek's first-party coding
agent. The profile is opt-in and non-default: `[models.default]` stays on
`CodexAgent`, and this profile is selected per node or group in
`.ultrafuzz/topology.yml` (`model_profiles = ["deepseek"]`). `--agent` and
`--model` override fields of the default profile rather than selecting a profile
by id, so `ultrafuzz run --agent DeepSeekAgent` swaps only the default profile's
agent and carries neither this profile's `deepseek-v4-pro` nor its
`reasoning = "max"`. Add `--model deepseek-v4-pro` to pin the model on such a
run; an `--agent` override clears reasoning, so no `--effort` is passed and the
Claude Code CLI default applies.

DeepSeek V4 Pro is API-key only. The adapter runs the installed Claude Code CLI
against DeepSeek's documented Anthropic-compatible endpoint,
`https://api.deepseek.com/anthropic`, using `ANTHROPIC_AUTH_TOKEN`; it clears
Expand Down
5 changes: 4 additions & 1 deletion packages/runtime/src/init.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,10 @@ const AGENT_TEMPLATES = [
file: "deepseek.ts",
template: "smithers/agents/deepseek.tsx",
ref: "DeepSeekAgent",
stockSha256: new Set(["c23a03c84e2f62d2e6b23ee7b27b1464a633fe20bb5b91c34d2c93d37dcf7e35"])
stockSha256: new Set([
"c23a03c84e2f62d2e6b23ee7b27b1464a633fe20bb5b91c34d2c93d37dcf7e35",
"65bf43f333cbced8ff0157e942d3c78267d8245d6c0041e053c8577a463e7407"
])
},
{
file: "kimi.ts",
Expand Down
9 changes: 5 additions & 4 deletions packages/runtime/src/templates/smithers/agents/deepseek.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,11 @@ const DEEPSEEK_REASONING_EFFORTS = ["low", "high", "max"] as const;
const DEEPSEEK_CLAUDE_CONFIG_DIR = ".ultrafuzz/deepseek-claude";

/**
* DeepSeek's supported coding-agent integration is Claude Code over its
* Anthropic-compatible endpoint. Keep it as a distinct factory so credentials,
* model selection, telemetry semantics, and pricing provenance never inherit
* Anthropic defaults accidentally.
* Runs the Claude Code harness against DeepSeek's Anthropic-compatible
* endpoint: a compatibility pairing, not DeepSeek's first-party coding agent.
* Keep it as a distinct factory so credentials, model selection, telemetry
* semantics, and pricing provenance never inherit Anthropic defaults
* accidentally.
*/
export function createDeepSeekAgent(options: DeepSeekTaskOptions = {}): SmithersClaudeCodeAgent {
const reasoningEffort = deepSeekReasoningEffort(options.reasoningEffort);
Expand Down
7 changes: 4 additions & 3 deletions ultrafuzz.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,9 @@ agent = "KimiAgent"
model = "kimi-k3"
reasoning = "max"

# Opt-in DeepSeek V4 profile. DeepSeek's official coding-agent integration
# runs Claude Code against its Anthropic-compatible API endpoint.
# Opt-in DeepSeek V4 profile. This runs the Claude Code harness against
# DeepSeek's Anthropic-compatible API endpoint: a compatibility pairing, not
# DeepSeek's first-party coding agent.
[models.deepseek]
agent = "DeepSeekAgent"
model = "deepseek-v4-pro"
Expand Down Expand Up @@ -82,7 +83,7 @@ auth = "subscription"

[agents.DeepSeekAgent]
# DeepSeek V4 is API-key only. The generated adapter reads DEEPSEEK_API_KEY and
# routes Claude Code to DeepSeek's official Anthropic-compatible endpoint.
# routes the Claude Code harness to DeepSeek's Anthropic-compatible endpoint.
auth = "api-key"
api_key_env = "DEEPSEEK_API_KEY"

Expand Down