Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,48 @@ Set `auth = "api-key"` to bill through an OpenAI API key read from
from `CODEX_HOME/auth.json`; optional `config_dir` points one generated agent
at a specific Codex config directory.

## Claude agent

`ultrafuzz init` also generates a `ClaudeAgent`, backed by the Claude Code CLI
(`claude --print`). It is available as an opt-in profile; the default profile
stays on `CodexAgent`:

```toml
[models.claude]
agent = "ClaudeAgent"
model = "claude-opus-4-8"

[agents.ClaudeAgent]
auth = "subscription"
```

Select it per node or group in `.ultrafuzz/topology.yml`
(`model_profiles = ["claude"]`). `--agent` and `--model` override fields of the
default profile rather than selecting a profile by id, so a one-off Claude run
is `ultrafuzz run --agent ClaudeAgent`; add `--model claude-sonnet-5` to pin a
model, otherwise the Claude CLI default is used.

Set `auth = "subscription"` to run against your logged-in Claude Code CLI
session with no API key — `ClaudeAgent` clears `ANTHROPIC_API_KEY` so the
subscription is used; optional `config_dir` sets an isolated `CLAUDE_CONFIG_DIR`
for running multiple Claude subscriptions side by side. Set `auth = "api-key"`
to bill against the Anthropic API using the key read from `api_key_env`
(default `ANTHROPIC_API_KEY`). Both modes drive the `claude` CLI, so it must be
installed either way; `auth` only changes how that CLI authenticates.

Unlike `CodexAgent`, the Claude Code CLI exposes no reasoning-effort control, so
a `reasoning` value on a Claude profile is ignored. Pin the model through the
profile's `model` field rather than under `[agents.ClaudeAgent]`.

`ClaudeAgent` runs every task with Claude Code's permission checks bypassed
(`--permission-mode bypassPermissions`): the agent has unattended access to the
filesystem and shell inside its worktree, because no operator is present to
answer a permission prompt. This follows `permissions.trust_model =
"skip-permissions"`, the only trust model ultrafuzz accepts, and matches how
`CodexAgent` already runs. It is fixed rather than configurable per agent — a
project that needs stricter behaviour must edit the generated
`.smithers/agents/claude.ts`.

Default triage requires quorum `3` from a panel size of `4`:

```toml
Expand Down
27 changes: 27 additions & 0 deletions packages/config/src/model-profiles.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@ import { z, type ZodIssue } from "zod/v4";
import { DEFAULT_MODEL_PROFILE_ID, synthesizeDefaultModelProfile } from "./defaults.js";
import { diagnostic, type ConfigDiagnostic, type ResolvedConfig } from "./types.js";

export interface DefaultProfileOverrides {
agent?: string;
model?: string;
}

const MODEL_TIMEOUT_SECONDS = 86_400;
const PROFILE_ID_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/;
const SAFE_AGENT_REF_PATTERN = /^[A-Za-z_][A-Za-z0-9_.:-]{0,127}$/;
Expand Down Expand Up @@ -89,6 +94,28 @@ export function validProfileId(id: string): boolean {
return profileIdSchema.safeParse(id).success;
}

export function applyDefaultProfileOverrides(config: ResolvedConfig, overrides: DefaultProfileOverrides): void {
if (overrides.agent === undefined && overrides.model === undefined) {
return;
}
const profile = config.models.profiles[config.models.default];
if (profile === undefined) {
return;
}
// A profile's model and reasoning are chosen for its agent, so switching the
// agent without naming a model must not hand one backend's model to another.
if (overrides.agent !== undefined && overrides.agent !== profile.agent && overrides.model === undefined) {
delete profile.model;
delete profile.reasoning;
}
Comment on lines +107 to +110

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Stale reasoning leaks when both --agent and --model are provided together

The clearing block is only triggered when overrides.model === undefined. When a user runs --agent ClaudeAgent --model claude-sonnet-5, the condition evaluates to false (because overrides.model !== undefined), so profile.reasoning is not deleted. The resulting profile keeps the old agent's reasoning value (e.g. "xhigh" from the Codex default), which flows through to task metadata as reasoningEffort: "xhigh" on a ClaudeAgent task — contradicting the invariant the PR establishes. The test for the pinned case (--agent ClaudeAgent --model claude-sonnet-5) does not include reasoningEffort in its type assertion, so this case is currently untested. The fix is to drop the && overrides.model === undefined guard so that reasoning is always cleared on any agent switch, regardless of whether a model is also supplied.

Suggested change
if (overrides.agent !== undefined && overrides.agent !== profile.agent && overrides.model === undefined) {
delete profile.model;
delete profile.reasoning;
}
// A profile's model and reasoning are chosen for its agent, so switching the
// agent must not hand one backend's model or reasoning to another.
if (overrides.agent !== undefined && overrides.agent !== profile.agent) {
delete profile.model;
delete profile.reasoning;
}
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/config/src/model-profiles.ts
Line: 107-110

Comment:
**Stale `reasoning` leaks when both `--agent` and `--model` are provided together**

The clearing block is only triggered when `overrides.model === undefined`. When a user runs `--agent ClaudeAgent --model claude-sonnet-5`, the condition evaluates to false (because `overrides.model !== undefined`), so `profile.reasoning` is not deleted. The resulting profile keeps the old agent's reasoning value (e.g. `"xhigh"` from the Codex default), which flows through to task metadata as `reasoningEffort: "xhigh"` on a ClaudeAgent task — contradicting the invariant the PR establishes. The test for the pinned case (`--agent ClaudeAgent --model claude-sonnet-5`) does not include `reasoningEffort` in its type assertion, so this case is currently untested. The fix is to drop the `&& overrides.model === undefined` guard so that `reasoning` is always cleared on any agent switch, regardless of whether a model is also supplied.

```suggestion
  // A profile's model and reasoning are chosen for its agent, so switching the
  // agent must not hand one backend's model or reasoning to another.
  if (overrides.agent !== undefined && overrides.agent !== profile.agent) {
    delete profile.model;
    delete profile.reasoning;
  }
```

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code

if (overrides.agent !== undefined) {
profile.agent = overrides.agent;
}
if (overrides.model !== undefined) {
profile.model = overrides.model;
}
}

function schemaIssues(schema: z.ZodType, value: unknown): ZodIssue[] {
const parsed = schema.safeParse(value);
return parsed.success ? [] : parsed.error.issues;
Expand Down
71 changes: 62 additions & 9 deletions packages/runtime/src/init.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@ import { builtInPromptRelativePaths, scaffoldPrompts } from "@ultrafuzz/prompts"
import { defaultReferenceCatalogYaml } from "@ultrafuzz/references";
import { loadRuntimeTemplate } from "./runtime-template.js";
import { renderSmithersPackageJson } from "./smithers-package.js";
import type { InitProjectInput, InitProjectResult } from "./types.js";
import type { InitProjectInput, InitProjectResult, RuntimeDiagnostic } from "./types.js";
import { configDiagnostics, runtimeFailure, runtimeResult, toProjectRelative } from "./utils.js";

const DEFAULT_TOPOLOGY = loadDefaultTopology();

const AGENT_REGISTRY_FILE = ".smithers/agents/index.ts";
const AGENT_TEMPLATES = [
{ file: "claude.ts", template: "smithers/agents/claude.tsx", ref: "ClaudeAgent" },
{ file: "codex.ts", template: "smithers/agents/codex.tsx", ref: "CodexAgent" }
] as const;

export function initProject(input: InitProjectInput) {
const projectRoot = path.resolve(input.projectRoot);
const created: string[] = [];
Expand Down Expand Up @@ -104,13 +110,24 @@ export function initProject(input: InitProjectInput) {
);
writeProjectFile(
projectRoot,
".smithers/agents/codex.ts",
loadRuntimeTemplate("smithers/agents/codex.tsx"),
".smithers/agents/toml.ts",
loadRuntimeTemplate("smithers/agents/toml.tsx"),
input.force === true,
created,
preserved,
overwritten
);
for (const agent of AGENT_TEMPLATES) {
writeProjectFile(
projectRoot,
`.smithers/agents/${agent.file}`,
loadRuntimeTemplate(agent.template),
input.force === true,
created,
preserved,
overwritten
);
}
} catch {
return runtimeFailure<InitProjectResult>([
{
Expand Down Expand Up @@ -152,12 +169,48 @@ export function initProject(input: InitProjectInput) {
preserved.push(toProjectRelative(projectRoot, absolutePath));
}

return runtimeResult(true, {
project_root: projectRoot,
created: publicInitPaths(created),
preserved: publicInitPaths(preserved),
overwritten: publicInitPaths(overwritten)
});
return runtimeResult(
true,
{
project_root: projectRoot,
created: publicInitPaths(created),
preserved: publicInitPaths(preserved),
overwritten: publicInitPaths(overwritten)
},
staleAgentRegistryDiagnostics(projectRoot)
);
}

// init preserves project-owned files, so a project scaffolded before an agent
// was added keeps its old registry: the new adapter lands on disk but nothing
// exports it, and the agent is only rejected later, at launch. Report it here
// instead of leaving the mismatch silent.
function staleAgentRegistryDiagnostics(projectRoot: string): RuntimeDiagnostic[] {
const registryPath = path.join(projectRoot, AGENT_REGISTRY_FILE);
if (!fs.existsSync(registryPath)) {
return [];
}
const registryText = fs.readFileSync(registryPath, "utf8");
return AGENT_TEMPLATES.filter(
(agent) =>
fs.existsSync(path.join(projectRoot, ".smithers", "agents", agent.file)) &&
!registersAgentFactory(registryText, agent.ref)
).map((agent) => ({
code: "INIT_AGENT_REGISTRY_STALE",
message: `${AGENT_REGISTRY_FILE} does not register ${agent.ref} in agentFactories, so runs cannot select it; rerun ultrafuzz init --force to regenerate the registry, or add the entry by hand`,
severity: "warning" as const,
source: "runtime",
path: AGENT_REGISTRY_FILE
}));
}

// Generated adapters export only their factory, so agentFactories is the sole
// path that resolves them. Merely naming the agent elsewhere in the registry --
// an `export { ClaudeAgent }` left over from an older adapter, say -- does not
// make it selectable, so match the factory entry rather than the bare name.
function registersAgentFactory(registryText: string, ref: string): boolean {
const factories = /agentFactories\s*=\s*\{([^}]*)\}/u.exec(registryText);
return factories === null ? false : new RegExp(`\\b${ref}\\s*:`, "u").test(factories[1] ?? "");
}

function writeProjectFile(
Expand Down
20 changes: 6 additions & 14 deletions packages/runtime/src/plan-run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,11 @@ import {
type NodeStateInput,
type RunLayout
} from "@ultrafuzz/artifacts";
import { redactResolvedConfig, serializeRedactedResolvedConfigToml } from "@ultrafuzz/config";
import {
applyDefaultProfileOverrides,
redactResolvedConfig,
serializeRedactedResolvedConfigToml
} from "@ultrafuzz/config";
import {
loadPromptCatalog,
renderPrompt,
Expand Down Expand Up @@ -398,19 +402,7 @@ function renderPromptsForPlan(input: {
}

function applyWorkflowRunOverrides(config: PlanRunValue["resolved_config"], input: PlanRunInput): void {
if (input.agent === undefined && input.model === undefined) {
return;
}
const profile = config.models.profiles[config.models.default];
if (profile === undefined) {
return;
}
if (input.agent !== undefined) {
profile.agent = input.agent;
}
if (input.model !== undefined) {
profile.model = input.model;
}
applyDefaultProfileOverrides(config, { agent: input.agent, model: input.model });
}

function promptLogicalNodes(graph: PlannedGraph, layout: PlanRunValue["layout"]): PromptGraphNode[] {
Expand Down
64 changes: 64 additions & 0 deletions packages/runtime/src/templates/smithers/agents/claude.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
import { readFileSync } from "node:fs";
import path from "node:path";
import { ClaudeCodeAgent as SmithersClaudeCodeAgent } from "smithers-orchestrator";
import { readStringTable, stringField } from "./toml";

type ClaudeAuthConfig = { auth?: string; api_key_env?: string; config_dir?: string };
type ClaudeAuthOptions = { apiKey?: string; configDir?: string };
// reasoningEffort is part of the shared agent factory contract but has no
// ClaudeCodeAgent equivalent, so it is accepted and ignored.
export type ClaudeTaskOptions = { model?: string; reasoningEffort?: string };

export function createClaudeAgent(options: ClaudeTaskOptions = {}): SmithersClaudeCodeAgent {
return new SmithersClaudeCodeAgent({
...(options.model === undefined ? {} : { model: options.model }),
// Every task runs with Claude Code's permission checks off: agents work
// unattended in a throwaway worktree, so there is nobody to answer a
// prompt. This mirrors permissions.trust_model = "skip-permissions" in
// ultrafuzz.toml, which is the only trust model ultrafuzz accepts, and is
// deliberately not configurable per agent -- edit this generated file if a
// project needs otherwise.
permissionMode: "bypassPermissions",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 bypassPermissions is hardcoded with no user-visible documentation

permissionMode: "bypassPermissions" is always set unconditionally, meaning the Claude Code CLI runs with unrestricted access to the filesystem and shell for every task. The template comment above ClaudeTaskOptions notes that reasoningEffort is silently ignored, but there is no parallel note explaining that permission mode is fixed and not user-configurable. A user inspecting only the ultrafuzz.toml [agents.ClaudeAgent] section (which exposes only auth, api_key_env, and config_dir) has no indication that all permission checks are bypassed.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/runtime/src/templates/smithers/agents/claude.tsx
Line: 14

Comment:
**`bypassPermissions` is hardcoded with no user-visible documentation**

`permissionMode: "bypassPermissions"` is always set unconditionally, meaning the Claude Code CLI runs with unrestricted access to the filesystem and shell for every task. The template comment above `ClaudeTaskOptions` notes that `reasoningEffort` is silently ignored, but there is no parallel note explaining that permission mode is fixed and not user-configurable. A user inspecting only the `ultrafuzz.toml` [agents.ClaudeAgent] section (which exposes only `auth`, `api_key_env`, and `config_dir`) has no indication that all permission checks are bypassed.

How can I resolve this? If you propose a fix, please make it concise.

Fix in Claude Code

...claudeAuthOptions()
});
}

function claudeAuthOptions(): ClaudeAuthOptions {
const config = readClaudeAuthConfig();
const auth = config.auth ?? "subscription";
if (auth === "api-key") {
return { apiKey: requiredEnv(config.api_key_env ?? "ANTHROPIC_API_KEY") };
}
if (auth === "subscription") {
// ClaudeCodeAgent clears ANTHROPIC_API_KEY itself so the logged-in
// Claude subscription (`claude -p`) is used; we only forward an
// isolated config directory when one is configured.
return config.config_dir === undefined ? {} : { configDir: resolveConfigDir(config.config_dir) };
}
throw new Error(`unsupported ClaudeAgent auth mode in ultrafuzz.toml: ${auth}`);
}

function readClaudeAuthConfig(): ClaudeAuthConfig {
const configPath = path.join(process.cwd(), "ultrafuzz.toml");
const claude = readStringTable(readFileSync(configPath, "utf8"), "agents.ClaudeAgent");
return {
auth: stringField(claude, "auth"),
api_key_env: stringField(claude, "api_key_env"),
config_dir: stringField(claude, "config_dir")
};
}

function requiredEnv(name: string): string {
const value = process.env[name];
if (value === undefined || value.trim() === "") {
throw new Error(`agents.ClaudeAgent auth is api-key, but ${name} is not set`);
}
return value;
}

function resolveConfigDir(value: string): string {
if (value.trim() === "") {
throw new Error("agents.ClaudeAgent.config_dir cannot be empty");
}
return path.isAbsolute(value) ? value : path.resolve(process.cwd(), value);
}
35 changes: 1 addition & 34 deletions packages/runtime/src/templates/smithers/agents/codex.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import { readFileSync } from "node:fs";
import path from "node:path";
import { CodexAgent as SmithersCodexAgent } from "smithers-orchestrator";
import { readStringTable, stringField } from "./toml";

type CodexAuthConfig = { auth?: string; api_key_env?: string; config_dir?: string };
type CodexAuthOptions = { apiKey?: string; configDir?: string; env?: Record<string, string> };
Expand All @@ -15,8 +16,6 @@ export function createCodexAgent(options: CodexTaskOptions = {}): SmithersCodexA
});
}

export const CodexAgent = createCodexAgent();

function codexAuthOptions(): CodexAuthOptions {
const config = readCodexAuthConfig();
const auth = config.auth ?? "subscription";
Expand All @@ -42,38 +41,6 @@ function readCodexAuthConfig(): CodexAuthConfig {
};
}

function readStringTable(text: string, tableName: string): Record<string, string> {
const fields: Record<string, string> = {};
let inTable = false;
for (const rawLine of text.split(/\r?\n/u)) {
const line = rawLine.trim();
if (line === "" || line.startsWith("#")) {
continue;
}
const table = /^\[([^\]]+)\]$/u.exec(line);
if (table) {
inTable = table[1]?.trim() === tableName;
continue;
}
if (!inTable) {
continue;
}
const assignment = /^([A-Za-z0-9_-]+)\s*=\s*"((?:\\.|[^"\\])*)"\s*(?:#.*)?$/u.exec(line);
if (assignment?.[1] && assignment[2] !== undefined) {
fields[assignment[1]] = JSON.parse(`"${assignment[2]}"`) as string;
}
}
return fields;
}

function stringField(table: Record<string, string>, key: string): string | undefined {
const value = table[key];
if (value === undefined) {
return undefined;
}
return value;
}

function requiredEnv(name: string): string {
const value = process.env[name];
if (value === undefined || value.trim() === "") {
Expand Down
10 changes: 8 additions & 2 deletions packages/runtime/src/templates/smithers/agents/index.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
import { createClaudeAgent } from "./claude";
import { createCodexAgent } from "./codex";

export { CodexAgent, createCodexAgent } from "./codex";
export const agentFactories = { CodexAgent: createCodexAgent };
export { createClaudeAgent } from "./claude";
export { createCodexAgent } from "./codex";

// Agents are constructed per task from the selected model profile, never at
// import time: an agent's auth is only read when that agent is actually used,
// so a project running one backend does not need the other's credentials.
export const agentFactories = { ClaudeAgent: createClaudeAgent, CodexAgent: createCodexAgent };
Loading
Loading