Skip to content

test(provider-switch): run the live probe for quota-independent switching #833

Open

Description

@Juliusolsson05

Split out of #821. The implementation PR for #821 opened on 2026-09-07 without the Stage 7 live probe, by the user's explicit decision. #821's acceptance list still carries two criteria that only the probe can satisfy, so they move here rather than being quietly counted as met.

The two deferred acceptance criteria, verbatim from #821

  • With the source provider's usage row at 100 percent, the bulk modal defaults to that source, disables source-side compaction, and a 10-agent batch completes without any 300 s wait.
  • Typecheck, unit, system, renderer suites green; opt-in live probe run once and its report attached to the PR.

(The suites are green and were reported on the PR; the probe half of that second bullet is what is outstanding. #821's first two acceptance bullets each also end in a live-probe clause — "the resulting Claude transcript resumes and answers a read-only marker prompt in the live probe", and "Codex resumes and answers the marker prompt" — which the same run settles.)

What the probe settles

Four Unknowns from docs/decomposition/quota-independent-provider-switch.md shipped unmeasured:

  1. Whether a shrunk projection's real token count matches the 2.5 characters-per-token estimate closely enough to stay under Codex's 90 percent auto-compact limit; measured by the probe.
    Resolution 2026-09-07: unmeasured. The Stage 7 live probe was not run. The estimate is unchanged, and if it turns out to be off the knob is the reserve fraction, not the ladder.
  2. Whether Codex semantically ignores a custom_tool_call_output whose output is a placeholder, or treats it as an error; probe with a marker prompt that asks for the last three tool results.
    Resolution 2026-09-07: unmeasured. Structural projection tests prove Codex and Claude accept the cleared shape; nothing yet proves a live Codex tolerates it semantically. Same probe dependency as Unknown 2.
  3. Whether a 1M Claude target rejects arrival with "Usage credits required for 1M context" on this account; if so the exhaustion signal must include it.
    Resolution 2026-09-07: unknown. No switch was run against a live 1M target. The switch itself would still succeed; only the arrival compaction would fail, and that failure is non-fatal and reported.
  4. OpenCode import size limits for shrunk envelopes; the 128k planning window is conservative but unverified against opencode import.
    Resolution 2026-09-07: unknown. Unchanged and untested; no OpenCode import of a shrunk envelope was attempted.

Structural projection tests prove the shapes are legal. They cannot prove that a provider includes translated history in its next model request, which is the only thing the probe is for.

The two ladder placeholders the probe would replace with measurements

packages/agent-transcript-parser/src/operations/shrink.ts DEFAULTS:

  • keepRecentTurns: 3 — how many trailing turns rung 4 protects from the drop.
  • maxInputChars: 8_000 — the rung-3 cap on a tool-call input's serialized length.

Both are documented in that file as placeholders. The Stage 0 census measured no per-turn size distribution and no tool-call input percentiles, and rung 3 never fired on realistic data — it is exercised only by its own unit test with an explicit small cap. Real per-turn and tool-input distributions from a probe run are what turn these into chosen numbers.

Shape of the run

One live probe run against real Claude and Codex CLIs on a machine with a genuinely exhausted window: switch a compacted Codex session and an over-budget Claude session, resume each projected transcript, answer a read-only marker prompt, and run a 10-agent batch off the exhausted provider with the modal's defaults. Attach the report to this issue.

Refs #821

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    class:C9-testsTest hygieneneeds-evidenceNeeds a recording/real use to settletype:choreMaintenance, deps, tests, docs

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions