Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
- Working budget before finalization: {{working_budget_seconds}} seconds.
- Clock: `date -u +%s` once at start = START; working deadline START+{{working_budget_seconds}}, hard deadline START+{{timeout_seconds}}.
- Re-run `date -u +%s` before each expensive step; compare, never estimate.
- Keep the session handle returned by a long-running command and poll that handle until completion. Do not relaunch the command to check progress.
- When a command is coalesced, poll its original native session or the reported `ultrafuzz command-hook --status HANDLE`. Use a unique first-line `# ultrafuzz-command-key: NAME` comment only when you deliberately need another concurrent invocation.
- Stop starting new delegated or tool work when the finalization reserve begins.
- During the reserve, write and validate every required artifact, marking unfinished work blocked instead of omitting outputs.
- Crossing the hard deadline kills this node with no output at all.
6 changes: 5 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Ultrafuzz CLI

Every command accepts `--project <path>`. Commands that support automation
Public commands accept `--project <path>`. Commands that support automation
accept `--json` and emit the `ultrafuzz.cli.result.v2` envelope.

| Command | Purpose |
Expand Down Expand Up @@ -44,6 +44,10 @@ accept `--json` and emit the `ultrafuzz.cli.result.v2` envelope.
The dashboard/API is a local operator surface over product state, not a
workflow-engine API.

The hidden `command-hook` integration consumes provider hook JSON, not the
public CLI envelope. Its `--status HANDLE` mode inspects an invocation within a
guarded node attempt; see the [internal command hook reference](reference/cli.md#internal-command-hook).

`init` may create implementation plumbing for the workflow adapter. That
plumbing is not an end-user API. User-owned configuration, prompts, topology,
references, run evidence, and materialized outputs remain under root
Expand Down
61 changes: 61 additions & 0 deletions docs/reference/agent-adapter-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,67 @@ and the gate requires an explicit policy for every adapter actually registered
in `agentFactories`, including a factory imported under an alias or registered
without a matching re-export.

## Command invocation admission

The shared runtime command registry addresses the Codex/OpenRouter incident
path in [#1317](https://github.com/monad-developers/ultrafuzz/issues/1317).
`CompatibleCodexAgent` maps task context onto `PreToolUse` and `PostToolUse`
configuration. The provider owns shell execution; Smithers' output callbacks
observe a command after launch and cannot safely enforce admission themselves.

On Linux, the qualified `@openai/codex@0.146.0` path reserves an invocation
before a Bash or unified-exec command starts. A reservation is scoped to the
run, node, iteration and attempt, plus exact command text and the hook's session
working directory. Exclusive file creation admits only one concurrent caller.
A duplicate receives the existing invocation, provider session and tool-call
identities, a coalescing diagnostic, and `ultrafuzz command-hook --status HANDLE`.
Use the original `write_stdin` handle to retrieve command output. A matching
terminal `PostToolUse`, including completion delivered through `write_stdin`,
permits a subsequent execution. Stream hook messages identify reservations,
coalescing and completion without copying shell commands or output into the
registry. Replaying the same completed provider tool-call identity remains
coalesced; a subsequent execution must have a new native call identity.
Completion finds the original durable reservation by its session/tool-call
identity, not by post-hook command text: existing trusted hooks may rewrite
that text. Ambiguous native identities do not release any reservation.

For intentional concurrent executions, add a unique first-line comment such
as `# ultrafuzz-command-key: second-check`. Different commands and attempts
are independent. The hook does not expose all execution options (notably a
unified-exec workdir override), so the same command text with different hidden
options also needs an explicit key. Shell syntax is not normalized.

Reservations and completion receipts live in the task workspace's
`.ultrafuzz/command-invocations` directory. They survive provider/controller
restarts on that workspace. No elapsed-time lease releases a reservation:
records with a dead controller, corrupt or incomplete records report `completion_unknown` and remain
coalesced, because a child may have outlived its provider. Poll the original
session or confirm that work stopped before deliberately choosing a new key.
`active` means the reserving controller is alive, not independently verified
shell liveness; a missing completion receipt never grants a relaunch.
This is an accidental-duplication guard, not a boundary against an agent that
changes commands or edits its own workspace.

The guard trusts only its exact session-configured Codex hook identities; it
does not use the global hook-trust bypass or change provider-home settings.
Existing user/project hook sources retain their own trust state. Versions not
explicitly qualified, missing task/trusted-CLI context, unsupported platforms,
and operational hook failures report `command guard inactive` and fail open.
Claude/DeepSeek background-completion and settings-composition contracts remain
unqualified, as do direct Kimi/Pi/OpenCode command paths. Their diagnostics make
this limitation explicit. Complete provider-independent enforcement still
requires a managed command runner or equivalent upstream admission surface;
this change does not close #1317.

`command-invocations.test.ts` covers independent-process admission races,
poll/relaunch, distinct keys, completion replay, orphan/PID-reuse and torn-record
handling. Set `ULTRAFUZZ_TEST_CODEX` to the pinned binary to run the deterministic
local Responses-server qualification after building the CLI. It exercises the
real hook CLI through fresh and resumed Codex sessions and verifies that an
unrelated untrusted hook remains disabled. A trusted rewriting hook also proves
that terminal completion permits a later distinct call while completed-call
replays stay coalesced. No paid provider request is made.

| Adapter | Classification | Existing option or missing surface | Upstream dependency |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `claude.tsx` | Thin mapping plus one stated-failure hook in `StatedFailureClaudeCodeAgent`, which `deepseek.tsx` shares. Smithers reports a failed result without an `error` field as `Claude run failed` and drops Claude Code's `result` text, so the `createOutputInterpreter` override keeps that text and the `generate` override, which forgets it when each generation starts, attaches it to that generation's thrown error as `details.agentStatedFailure`. The thrown message is unchanged, because Smithers classifies it. | Uses `model`, `extraArgs`, `addDir`, `permissionMode`, `settingSources`, `apiKey`, `configDir`, and `env`. | [#1084](https://github.com/monad-developers/ultrafuzz/issues/1084): `@smthrs/agents` `src/ClaudeCodeAgent.js` builds a failed result's error as `limitBannerText \|\| resultError \|\| "Claude run failed"` and drops `payload.result`. |
Expand Down
11 changes: 11 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -1033,3 +1033,14 @@ performs no writes.

Eval artifacts are written under `.ultrafuzz/evals/runs/<eval-run-id>/`. See
[Eval Suites](evals.md) for configuration and architecture details.

## Internal command hook

`ultrafuzz command-hook` consumes the provider's hook JSON on stdin. It is an
internal integration command and emits the provider hook protocol rather than
the public CLI result envelope. Within a guarded node attempt,
`ultrafuzz command-hook --status HANDLE` polls the durable invocation identity
reported by a coalesced command. It returns status and native session/tool-call
identities, without executing or cancelling the command. Use the native session
handle to retrieve output. See [command invocation admission](agent-adapter-boundaries.md#command-invocation-admission)
for qualified providers, explicit concurrent invocations and recovery limits.
40 changes: 40 additions & 0 deletions packages/cli/src/commands/command-hook.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
import { Command, Flags } from "@oclif/core";
import {
commandInvocationHook,
commandInvocationRegistryFromEnvironment,
commandInvocationStatus
} from "@ultrafuzz/runtime";

/** Internal provider hook, also usable by the model to inspect a durable handle. */
export default class CommandHook extends Command {
static override hidden = true;
static override summary = "Inspect or update an agent command invocation";
static override flags = { status: Flags.string({ summary: "Poll an invocation handle" }) };

async run(): Promise<void> {
const { flags } = await this.parse(CommandHook);
try {
if (flags.status !== undefined) {
this.log(JSON.stringify(commandInvocationStatus(commandInvocationRegistryFromEnvironment(), flags.status)));
return;
}
let bytes = 0;
const chunks: Buffer[] = [];
for await (const chunk of process.stdin) {
const buffer = Buffer.from(chunk as Uint8Array);
bytes += buffer.length;
if (bytes > 1024 * 1024) throw new Error("hook input limit exceeded");
chunks.push(buffer);
}
this.log(JSON.stringify(commandInvocationHook(JSON.parse(Buffer.concat(chunks).toString("utf8")))));
} catch {
// Provider hooks fail open on operational errors. Surface the missing
// protection rather than misreporting it as a successful reservation.
this.log(
JSON.stringify({
systemMessage: "ultrafuzz.command_invocation unavailable: hook failed; command guard inactive"
})
);
}
}
}
50 changes: 50 additions & 0 deletions packages/cli/test/command-hook.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import path from "node:path";
import test from "node:test";
import { fileURLToPath } from "node:url";
import { temporaryRoot } from "./temporary-root.js";

const cli = fileURLToPath(new URL("../../dist/index.js", import.meta.url));

test("internal hook CLI reserves, coalesces, polls and completes without a public envelope", (t) => {
const root = temporaryRoot("ufz-command-hook-cli-", t);
const env = {
...process.env,
ULTRAFUZZ_COMMAND_INVOCATIONS: path.join(root, "registry"),
ULTRAFUZZ_COMMAND_INVOCATION_OWNER: "fixture:2147483647:0",
ULTRAFUZZ_COMMAND_INVOCATION_ENGINE: "codex"
};
const invoke = (args: string[], input = "") => {
const result = spawnSync(process.execPath, [cli, "command-hook", ...args], {
cwd: root,
env,
input,
encoding: "utf8",
timeout: 10_000
});
assert.equal(result.status, 0, result.stderr);
return JSON.parse(result.stdout) as Record<string, unknown>;
};
const event = {
hook_event_name: "PreToolUse",
tool_name: "Bash",
cwd: root,
session_id: "session",
tool_use_id: "original",
tool_input: { command: "verification --all" }
};
const first = invoke([], JSON.stringify(event));
assert.match(String(first.systemMessage), /reserved/u);
assert.equal(first.schema_version, undefined);
const id = /"invocation_id":"([a-f0-9]{64}-\d+)"/u.exec(String(first.systemMessage))?.[1];
assert.ok(id);
const duplicate = invoke([], JSON.stringify({ ...event, tool_use_id: "duplicate" }));
assert.equal((duplicate.hookSpecificOutput as Record<string, unknown>).permissionDecision, "deny");
assert.equal(invoke(["--status", id]).status, "completion_unknown");
invoke([], JSON.stringify({ ...event, hook_event_name: "PostToolUse", tool_response: "exit code 0" }));
assert.equal(invoke(["--status", id]).status, "completed");
assert.match(String(invoke([], "not json").systemMessage), /guard inactive/u);
assert.match(String(invoke(["--status", "../../outside"]).systemMessage), /guard inactive/u);
assert.match(String(invoke([], " ".repeat(1024 * 1024 + 1)).systemMessage), /guard inactive/u);
});
66 changes: 66 additions & 0 deletions packages/cli/test/e2e/campaign-resume.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { setTimeout as delay } from "node:timers/promises";
import { fileURLToPath } from "node:url";

import { formatEstimatedSpendUsd } from "@ultrafuzz/artifacts";
import { configureCommandInvocationGuard } from "@ultrafuzz/runtime";

// One campaign through the shipped product path: `init`, `run`, `resume`, `status`, `stats`,
// `report`, and `events` each run as their own `ultrafuzz` process, and the generated workflow runs
Expand Down Expand Up @@ -70,6 +71,7 @@ nodes:

interface StubConfig {
logPath: string;
versionLogPath: string;
holdPath: string;
holdNode: string;
promptMarker: string;
Expand Down Expand Up @@ -98,6 +100,13 @@ function stubCodex(config: StubConfig): void {
const path = process.getBuiltinModule("node:path");
const { execFileSync } = process.getBuiltinModule("node:child_process");
const args = process.argv.slice(2);
// Capability probes are not model attempts and must not read a prompt or
// enter the held-node path. Keep separate evidence that the probe ran.
if (args.length === 1 && args[0] === "--version") {
fs.appendFileSync(config.versionLogPath, `${JSON.stringify({ pid: process.pid, args })}\n`);
process.stdout.write("codex-cli 0.146.0\n");
return;
}
const prompt = fs.readFileSync(0, "utf8");
const node = (process.env.SMITHERS_NODE_ID ?? "").replace(/^node:/u, "");
const edited = prompt.includes(config.promptMarker);
Expand Down Expand Up @@ -174,6 +183,7 @@ interface Campaign {
runId: string;
env: NodeJS.ProcessEnv;
logPath: string;
versionLogPath: string;
holdPath: string;
}

Expand Down Expand Up @@ -221,6 +231,7 @@ function prepareCampaign(): Campaign {
project,
runId: `e2e-${process.pid}-${Date.now().toString(36)}`,
logPath: path.join(root, "agent-calls.jsonl"),
versionLogPath: path.join(root, "version-probes.jsonl"),
holdPath: path.join(root, `hold-${INTERRUPTED_NODE}`),
env: {
...Object.fromEntries(
Expand All @@ -242,6 +253,7 @@ function prepareCampaign(): Campaign {
);
const stubConfig: StubConfig = {
logPath: campaign.logPath,
versionLogPath: campaign.versionLogPath,
holdPath: campaign.holdPath,
holdNode: INTERRUPTED_NODE,
promptMarker: PROMPT_EDIT_MARKER
Expand All @@ -262,6 +274,60 @@ function prepareCampaign(): Campaign {
return campaign;
}

test("campaign stub separates configure-only version probes from real model starts", (t) => {
if (process.platform !== "linux") {
t.skip("command guard process identity requires Linux /proc");
return;
}
const campaign = prepareCampaign();
const savedBin = process.env.ULTRAFUZZ_TRUSTED_BIN;
process.env.ULTRAFUZZ_TRUSTED_BIN = path.join(campaign.root, "bin");
try {
const diagnostics: string[] = [];
const binary = path.join(campaign.root, "bin", "codex");
const configured = configureCommandInvocationGuard({
engine: "codex",
command: { command: binary, args: ["exec", "--json", "-"], env: {} },
cwd: campaign.project,
taskContext: { runId: campaign.runId, nodeId: "probe", iteration: 0, attempt: 1 },
diagnostic: (message) => diagnostics.push(message)
});
assert.match(diagnostics.join("\n"), /enabled: codex 0\.146\.0/u);
assert.ok(configured.args.some((arg) => arg.startsWith("hooks.PreToolUse=")));
assert.equal(fs.existsSync(campaign.logPath), false, "configuration must not log a model start or failure");
const probes = fs
.readFileSync(campaign.versionLogPath, "utf8")
.trim()
.split("\n")
.map((line) => JSON.parse(line) as { args: string[] });
assert.deepEqual(
probes.map((probe) => probe.args),
[["--version"]]
);

// Positive control: the real exec path still logs its start and completion.
execFileSync(binary, configured.args, {
cwd: campaign.project,
env: { ...campaign.env, ...configured.env, SMITHERS_NODE_ID: "node:probe" },
input: "- Path: `probe.txt`\n Contract: `ultrafuzz/text@1`\n",
encoding: "utf8"
});
assert.deepEqual(
agentCalls(campaign).map((call) => [call.node, call.event]),
[
["probe", "started"],
["probe", "completed"]
]
);
assert.equal(fs.readFileSync(path.join(campaign.project, "probe.txt"), "utf8"), "stub output for probe\n");
assert.equal(fs.readFileSync(campaign.versionLogPath, "utf8").trim().split("\n").length, 1);
} finally {
if (savedBin === undefined) delete process.env.ULTRAFUZZ_TRUSTED_BIN;
else process.env.ULTRAFUZZ_TRUSTED_BIN = savedBin;
removeTree(campaign.root);
}
});

async function ultrafuzz<T>(campaign: Campaign, args: string[], timeoutMs = 5 * MINUTE): Promise<T> {
const command = `ultrafuzz ${args.join(" ")}`;
const { error, stdout, stderr } = await new Promise<{ error: Error | null; stdout: string; stderr: string }>(
Expand Down
6 changes: 3 additions & 3 deletions packages/prompts/test/agent-preamble.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ describe("agent preamble MDX", () => {
});
expect(sha256(authorization)).toBe("708458aee8dcfc26a35ca5778274b8f81146332ebc3227302aff8a98e60f2535");
expect(sha256(boundary)).toBe("f25d5978ba27833deb062ebae4f7a21e9c3e7cc7ea48a0a49d49aceb3f9214a1");
expect(sha256(runtime)).toBe("05b91daad522fea769116af2bbeb0aa1224129f8192b235ef2d7f8e4973ccfd2");
expect(sha256(runtime)).toBe("8af49367b6aef2ec8544ba1499e2f5ed6e1f9733c031ceaa54e8cff566c15603");

const taskPrompt = "# Node-specific prompt\n\nDo the assigned work.";
const withoutOperator = renderAgentPreambleTemplate("agent-prompt", {
Expand All @@ -30,8 +30,8 @@ describe("agent preamble MDX", () => {
});
const mandatoryPrefix = `${authorization}\n\n${boundary}\n\n${runtime}\n\n`;
expect(withoutOperator).toBe(`${mandatoryPrefix}${taskPrompt}`);
expect(Buffer.byteLength(mandatoryPrefix, "utf8")).toBe(1_478);
expect(sha256(mandatoryPrefix)).toBe("8b796bcd9dd9b811a0a655c928ad58badf1a037e79d9cc9097917a846804bb5f");
expect(Buffer.byteLength(mandatoryPrefix, "utf8")).toBe(1_874);
expect(sha256(mandatoryPrefix)).toBe("cfd9f1ddf6a692c918ef179db0b4934cb9431f90cd822d26440754a711d87179");

const operatorPrompt = "Focus on authorization boundaries.";
expect(
Expand Down
Loading
Loading