[feat] CLI: race one prompt across agents — superset agents wait + superset race

What problem are you trying to solve?

Superset makes it easy to give one task to several agents at once. The next question is which
result to merge, and today I answer it by hand. I watch sidebar spinners until each agent stops,
open every worktree, run the tests in each, and compare diffs by eye. That part doesn't scale past
two agents.

The CLI can already do the fan-out (superset workspaces create --agent … --prompt …). It can't do
the rest:

It can't wait for an agent to finish. superset terminals list reports PTYs, not agent
state. The host service already records the normalized lifecycle (Start / Stop /
PermissionRequest / Failed in terminal_agent_bindings), and the workspace board uses it,
but the CLI and SDK can't read it. feat(sdk): expose terminal agent lifecycle status #7007 exposes it on terminals.list. [feat] Expose provider-neutral resumable agent identity and liveness from the CLI #6980 asks for the
related identity and liveness.
It can't run a check in a worktree and get the result back. terminals create --command
is fire-and-forget: there's no exit code and no output to parse.
It can't compare results. Nothing summarizes N workspaces side by side: tests, new tests,
diff size.

Proposed solution

Two layers. The first is small and useful on its own. The second is built on top of it.

1. superset agents wait: block until an agent settles

superset agents wait --workspace <id> [--terminal <id>] [--timeout 30m] [--settle 15s] [--json]

Polls, or subscribes to, the same binding the board uses. It returns when lastEventType has
been Stop for --settle.
Exit codes: 0 idle · 2 needs input (PermissionRequest) · 3 failed or exited · 124
timeout. --json prints the final agentStatus (the shape from feat(sdk): expose terminal agent lifecycle status #7007).
--terminal defaults to the workspace's only agent terminal. With several, it waits for all of
them.
It also needs a "never started" state. An agent parked on a first-run dialog (folder trust,
hook trust, sign-in) emits no lifecycle event at all, so it looks the same as one that's still
booting. In my first real run, both agents sat on "trust this folder?" for 30 minutes before the
script timed out: the demo repo had never been trusted, and Superset's worktrees inherit trust
from the main checkout. A deadline for the first event, reported as its own exit code, would
make this fail fast.
Useful for CI, automations and any script that chains agents, not just racing.

2. superset race: fan out, wait, verify, score

superset race --project <id> --prompt "<task>" --agents claude,codex,gemini
              [--verify "<cmd>"] [--base-branch main] [--timeout 30m]
              [--baseline] [--cleanup losers] [--json]

create N workspaces ──► agents wait (each) ──► verify in each worktree ──► diff vs fork point ──► scoreboard
  branch race/<run>/<agent>   same state machine    + once on base commit     commits+staged+untracked    race.json
  tag "race" (sidebar folder)  as `agents wait`      (host-side, see below)

Verify command. Order of precedence: --verify, then a new verify key in
.superset/config.json next to setup/teardown/run, then nothing (tests only).
```
{ "setup": ["bun install"], "verify": ["bun run typecheck", "bun test"] }
```
Verify runs on the host, via a small host-service procedure such as
terminal.runToCompletion({ workspaceId, command, timeoutMs }) → { exitCode, output }. That makes
it work the same for local, remote and cloud workspaces, where the CLI can't reach the worktree.
Scoreboard. Tests passed/failed (parsed from common runners), new tests (the delta against
the baseline run, so new parametrize rows count), test files touched, and diff stat. The
default ranking is: changed something → green → fewer failures → more new tests → smaller diff →
faster. It's shown but not enforced. --json gives the raw numbers.
Desktop follow-up (out of scope here). The race tag folder already groups the workspaces.
A compare view could come later.

Suggested order: (a) #7007 lands, (b) agents wait, (c) the host-side run-to-completion
procedure, (d) race. I'm happy to send PRs for (b) through (d) once the shape is agreed.

Open questions for maintainers

Should race be its own command, or workspaces create --agents a,b,c plus superset compare?
Should the ranking be opinionated, or should race only report the numbers?
Is .superset/config.json the right place for verify, given that run there is usually a dev server rather than a check?

Workarounds you've tried

I built the proposal as a standalone script on today's CLI: https://github.com/harshkumawat5/superset-race
(Bun/TypeScript, no dependencies, tested against a mock CLI). Here it is racing Claude and Codex
on a real open bug, python-humanize/humanize#379:

#	AGENT	STATUS	TIME	TESTS	NEW TESTS	TEST FILES	DIFF
1	claude	idle	3m18s	770 passed	+9	1	2 files +43 −2
2	codex	idle	56s	676 passed 90 failed	+4	1	2 files +23 −2
	baseline			761 passed			392aef70

Both agents wrote almost the same fix. Codex finished 3.5× sooner, but its check calls dt.timedelta
at runtime, and humanize imports datetime only under TYPE_CHECKING, so every precisedelta() call
raises NameError. Claude imported it inside the function. That's the call this is meant to make:
the fast diff that looks right breaks the package, and only running the suite in each worktree
shows it.

To wait for idle, the script has to fall back to things a CLI user shouldn't need:

It reads ~/.superset/host/<org>/manifest.json and calls the host's private
terminalAgents.listByWorkspace tRPC procedure directly.
Failing that, it treats "the screen from terminals read hasn't changed for 30s" as idle.
This can't tell a finished agent from one blocked on a permission prompt.
To catch agents that never start, it reads the screen of any agent that hasn't reported
anything for a minute and matches it against known first-run prompts.

Verify and diff run in the local worktree. The script can't support remote or cloud hosts, which is
why the proposal puts verify on the host. agents wait plus a run-to-completion primitive would
shrink the script to about 100 lines of glue, and the built-in race would replace it.

Harsh Kumawat · GitHub · LinkedIn

Link the feature proposal, ParadevAI/Paradev#7939

I read the proposal against the code. This is a feature request, not a bug, so there's nothing to reproduce. Every piece the proposal needs already exists on the host, but the CLI doesn't expose any of them. Nothing I describe below has been run.

Summary

The request is for a CLI way to give one prompt to several agents and then choose a winner. The fan-out already works. Three things are missing: (1) a blocking superset agents wait that returns when an agent's lifecycle settles, with a separate outcome for an agent that never started; (2) a host-side "run this command in the worktree and give me the exit code and output" primitive; (3) a superset race command that runs the check in each workspace and compares the results. The reporter has a prototype script. It reaches into host internals because the CLI has no public surface for this.

Where each piece lives today

Agent lifecycle state already exists on the host:

TerminalAgentBinding holds lastEventType, lastEventAt, endedAt, endReason and subagents: packages/host-service/src/terminal-agents/types.ts:50-65. The doc comment at types.ts:26-31 says a binding is created on the first hook event and disappears from live reads once it ends.
The host router serves it through terminalAgents.listByWorkspace / findActive: packages/host-service/src/trpc/router/terminal-agents/terminal-agents.ts:383-429.
The event-to-status mapping is shared: packages/shared/src/agent-status.ts:22-37 (Start → working, PermissionRequest → permission, Failed → failed, Stop → review).
A wait primitive already exists but is private: waitForBinding (terminal-agents.ts:691-738) subscribes to store.on("change"), which is event-driven, not polling. getOrCreate uses it with a 10s timeout.

The CLI exposes almost none of this:

superset agents only has create, list and read: packages/cli/src/commands/agents/. agents list lists agent configs (settings.agentConfigs.list), not running agents (agents/list/command.ts:35).
terminals list returns terminal.list, which has PTY sessions and no agent status: packages/cli/src/commands/terminals/list/command.ts:31.
The CLI's host client can already reach terminalAgents.*: agents read calls target.client.terminalAgents.transcript (agents/read/command.ts:37). So agents wait can use the same resolveWorkspaceTarget → terminalAgents.listByWorkspace path for local, remote and cloud workspaces. The reporter's manifest-file workaround isn't needed.
agents create returns result.sessionId, which is the terminal ID (agents/create/command.ts:116-131). wait can take that value for --terminal.

Running a command to completion already works on the host, but only for teardown:

runTeardown in packages/host-service/src/runtime/teardown/teardown.ts:54-154 does most of what terminal.runToCompletion needs. It creates an unlisted PTY (listed: false), buffers an output tail, and resolves on pty.onExit with exitCode/signal/timedOut. It kills the PTY on timeout, with a hard-stop grace period. Pulling it out into a general runToCompletion(workspaceId, command, timeoutMs) should be simple.
terminals create --command is fire-and-forget, as the report says: terminals/create/command.ts:37-41 only returns the terminalId.

.superset/config.json keys:

The script keys are a closed list: SCRIPT_KEYS = ["setup", "teardown", "run", "start"] (packages/host-service/src/runtime/setup/config.ts:32), and the same keys appear in SetupConfig, LocalSetupConfig and mergeBaseConfigs (config.ts:11-30, 134-147). A verify key would need changes in all of those places. After that it gets worktree, user-override and config.local.json merging for free, plus the .superset/verify.sh fallback through resolveScript (config.ts:279-308).

Traps an implementation should handle

"Never started" can't be seen as a binding. No hook event means no binding row (types.ts:26). listByWorkspace can't tell "still booting" from "stuck on a trust or sign-in dialog". A first-event deadline has to be measured from terminal creation, for example against terminal.list or the launch time. The Attached event also appears to count as "started but never prompted" (terminal-agents.ts:146-150), so it should probably not count as progress.
An exited agent disappears. When the agent or its terminal ends, the binding leaves live reads (types.ts:28-30). A poller that looks for Stop could spin until the timeout. wait should treat "binding gone, or endedAt set" as the exited outcome (the proposed exit code 3). It can read endReason to tell detached from terminal-exited.
Stop can be forced. clearWorkspaceStatuses sets bindings to Stop without a real hook event (terminal-agents.ts:580-594), for example from the sidebar "Clear Status" or an Esc/Ctrl+C interrupt. The proposed --settle window reduces this risk, but a script can't tell a forced Stop from a real one.
Subagents. A parent can report Stop while subagents is not empty (types.ts:63-64). "Idle" should probably also require no live subagents.

Suggested direction (maintainers decide)

(b) agents wait: add packages/cli/src/commands/agents/wait/command.ts. It resolves the target like agents read, then polls terminalAgents.listByWorkspace, or adds a host subscription or long-poll procedure built on waitForBinding's store.on("change"). It then maps state to exit codes through agentStatusFromEvent. This doesn't strictly need feat(sdk): expose terminal agent lifecycle status #7007, since it reads bindings directly.
(c) host runToCompletion: move the PTY / exit / timeout logic out of runTeardown into a shared helper. Then expose it as a protectedProcedure on the terminal router, returning { exitCode, signal, timedOut, output }, and make teardown call the helper.
(d) race: this is mostly CLI glue over workspaces create, agents wait and runToCompletion. The maintainers' open questions (a separate command or create --agents plus compare, and whether the ranking is opinionated) are product decisions the code can't answer. One data point on the verify question: run resolves per project from the main repo only (config.ts:273-274). verify must resolve per worktree, as setup and teardown do, so that each agent's branch can override it.

Testing sketch for (b)

Build a TerminalAgentStore in a bun:test like the ones in packages/host-service/src/terminal-agents/store.test.ts, and drive a binding through these sequences:

Start → Stop: expect idle after the settle window.
Start → PermissionRequest: expect exit code 2.
Start → markTerminalExited: expect exit code 3, not a timeout.
No events at all: expect the "never started" deadline to fire.

Navigation Menu

Repository navigation

[feat] CLI: race one prompt across agents — `superset agents wait` + `superset race` #7939

What problem are you trying to solve?

Proposed solution

Workarounds you've tried

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[feat] CLI: race one prompt across agents — superset agents wait + superset race #7939

Description

What problem are you trying to solve?

Proposed solution

Workarounds you've tried

Activity

github-actions commented on Sep 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[feat] CLI: race one prompt across agents — `superset agents wait` + `superset race` #7939