Skip to content

[#1288] consult: ship claude-opus-5 and gpt-5.6-sol as default lane models - #1301

Merged
waleedkadous merged 6 commits into
mainfrom
builder/air-1288
Jul 30, 2026
Merged

[#1288] consult: ship claude-opus-5 and gpt-5.6-sol as default lane models#1301
waleedkadous merged 6 commits into
mainfrom
builder/air-1288

Conversation

@waleedkadous

@waleedkadous waleedkadous commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes #1288.

Moves the shipped consult lane defaults forward — the ids every adopter gets when they haven't configured anything.

Lane Before After
claude claude-opus-4-6 claude-opus-5
codex gpt-5.4 @ medium gpt-5.6-sol @ medium
gemini agy's own default unchanged

Review

What changed and why

The ids are now named constants, not inline literals. DEFAULT_CLAUDE_MODEL, DEFAULT_CODEX_MODEL, and DEFAULT_CODEX_REASONING_EFFORT are exported from consult/index.ts. Two reasons: the guard test (issue item 4) needs something CI-safe to assert against, and #1286 turns these exact values into config fallback defaults — having them already extracted makes that a one-line change rather than a re-extraction of the same lines this PR just touched.

Pricing is now per-model, and unknown models cost null. CODEX_PRICING was a bare rate triple hardcoded to gpt-5.4's numbers; it's now a Record<modelId, rates> behind an exported computeCodexCost(model, input, cached, output): number | null.

gpt-5.6-sol's rates were verified against OpenAI's own pricing page (developers.openai.com/api/docs/pricing, fetched 2026-07-30), not an aggregator: $5.00 uncached input / $0.50 cached input / $30.00 output per 1M. So the issue's "record cost_usd as null if rates can't be verified" fallback didn't need to trigger — but the mechanism is in place anyway, which matters once #1286 lets a workspace point the lane at an arbitrary model. Without it, a configured model would silently inherit gpt-5.6-sol's rates. Now it gets null.

Known gap, documented in code and docs: OpenAI publishes a separate higher long-context tier ($10/$1/$45). We model only the standard tier, so a consultation large enough to cross into it under-reports. Modelling the tier boundary is more than this issue asked for; flagging it rather than silently rounding.

A test was lying. codex-sdk.test.ts had its own copy of the CODEX_PRICING constant and a local reimplementation of the cost formula. Its five "cost computation" assertions would have kept passing against gpt-5.4 rates indefinitely while the shipped rates drifted — it was testing a copy of the thing, not the thing. It now imports the real computeCodexCost, and the expected values were recomputed for the new rates.

Tests

New consult/__tests__/default-models.test.ts pins both ids plus the reasoning effort, and covers the null-for-unknown-model path (using gpt-5.4 as one of the unknowns, which doubles as an assertion that the old model's rates really are gone). CI-safe: no network, no model CLIs, no API calls.

The specific thing this guards against is a future drive-by "correction" of the -sol suffix. Both plain gpt-5.6 and gpt-5.6-codex were live-probed on 2026-07-29 and rejected — The '<id>' model is not supported when using Codex with a ChatGPT account. The suffix looks like a typo and isn't one, so the test comment says so out loud.

Existing assertions updated: consult.test.ts (claude lane model), codex-sdk.test.ts (codex lane model, plus a new assertion on reasoning effort that was previously unchecked).

Docs

  • codev/resources/commands/consult.md — Models table gains a Shipped default model id column, plus a -sol warning callout and a "Cost reporting" section explaining the null semantics. Mirrored byte-identically to codev-skeleton/.
  • CLAUDE.md / AGENTS.md — the Multi-Agent Consultation section named "GPT-5.4 Codex (gpt-5.4-codex)", an id the code never actually used. Corrected to gpt-5.6-sol, and the claude lane — which the section had simply omitted — is now listed. Both files remain byte-identical (verified with diff).

Coordination with #1286

No PR open for #1286 as of 2026-07-30, so this lands first. Per the issue, #1286's "default preservation" test vectors must then assert the new ids — default-models.test.ts is the natural place for that to be enforced, since it will fail loudly if the fallback path resolves to anything else.

Verification

  • pnpm build from repo root (core → codev → dashboard → skeleton copy): green.
  • Full unit suite: 3798 passed, 48 skipped, 0 failed (197 files).
  • porch check 1288: build ✓ (4.6s), tests ✓ (27.6s).

No flaky tests encountered; the 48 skips are pre-existing.

Out of scope (unchanged, per the issue)

Lane composition, the gemini/agy lane's model, and the #1286 config mechanism itself.


⚠️ CI note: "Tower Integration Tests" is red, and it is not this PR

Tower Integration Tests failed on both runs of this branch. I diagnosed it rather than re-rolling the dice; it is a pre-existing teardown defect in a file this PR never touches.

Problem. src/agent-farm/__tests__/send-integration.e2e.test.ts fails with Error: Hook timed out in 10000ms. All 5 of its tests pass — the failure is in teardown, after the assertions are green.

Root cause. The suite's afterAll is declared with an explicit 10_000 ms budget (send-integration.e2e.test.ts:273), overriding the config's generous hookTimeout: 300000. That teardown does two fetch(.../deactivate) calls with no timeout of their own (.catch(() => {}) swallows errors but never bounds a hang), then a stopServer that waits up to 2s for SIGTERM before SIGKILL, then four rmSync calls. On a loaded ubuntu runner that comfortably exceeds 10s; on a developer machine it doesn't. The beforeAll in the same suite is budgeted at 120_000 — the asymmetry looks unintentional.

Evidence it's unrelated to this PR:

  • The suite passes locally on this branch: 5/5 in 1.76s (vitest run --config vitest.e2e.config.ts src/agent-farm/__tests__/send-integration.e2e.test.ts).
  • This PR's diff is confined to commands/consult/** and docs. It touches no Tower, websocket, or messaging code.
  • The file has prior form — commit 1546ced5 is literally "Fix flaky send-integration e2e test — retry activation during Tower startup".
  • main is green only because the margin is thin, not because the teardown is safely budgeted.

Fix. I did not apply one, and I did not skip the suite — skipping would drop 5 real integration tests to land a two-line model-id change, and either edit is outside #1288's scope. The minimal fix is to raise that afterAll budget (and ideally bound the two deactivate fetches). Happy to do it in this PR if you'd prefer, or to file it separately — your call at the gate.

…odels

Move the shipped consult lane defaults forward:

- claude lane: claude-opus-4-6 -> claude-opus-5
- codex lane:  gpt-5.4 -> gpt-5.6-sol (reasoning effort unchanged at medium)
- gemini lane: unchanged (agy's own default, no pinned id)

Both ids were live-probed before adoption. The -sol suffix is load-bearing:
plain gpt-5.6 and gpt-5.6-codex are rejected by Codex on a ChatGPT account.

The ids are now exported constants (DEFAULT_CLAUDE_MODEL, DEFAULT_CODEX_MODEL,
DEFAULT_CODEX_REASONING_EFFORT) rather than inline literals, so the new
default-models test can pin them and so #1286 can drop config-driven resolution
in front of them without re-extracting.

CODEX_PRICING becomes a per-model rate table and cost computation moves to an
exported computeCodexCost(). gpt-5.6-sol rates ($5.00 uncached / $0.50 cached /
$30.00 output per 1M) are taken from developers.openai.com/api/docs/pricing.
An unknown model id now yields costUsd: null instead of being billed at some
other model's rates.

codex-sdk.test.ts had its own copy of CODEX_PRICING, so its cost suite would
have kept passing against stale rates forever; it now imports the real function.

Docs: consult.md gains a "Shipped default model id" column and a cost-reporting
note (mirrored to the skeleton); the Multi-Agent Consultation section of
CLAUDE.md/AGENTS.md is corrected and now names both pinned lanes.
@waleedkadous
waleedkadous merged commit 7c21cdc into main Jul 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

consult: change shipped default lane models to claude-opus-5 and gpt-5.6-sol

1 participant