feat(fallback): OCX-11 fallback off per scope, with home_unreachable on Children

devlog/_plan/261021_fallback_off/000_design.md

@@ -0,0 +1,145 @@
+# OCX-11 fallback off — design
+
+> Written 2026-10-21, revised 2026-10-22 after the supervisor decisions (§9) · implemented in PR #7482.
+> Consumer: GOAT hub design, goat-si/goat PR #684 (ADR-0021 §1.6, design doc §1.6 and DH3).
+> The present-tense contract is [structure/fallback-policy.md](../../../structure/fallback-policy.md).
+
+## 1. Goal
+
+Give the operator one clear way to say "do not fall back; fail with a clear error instead", for two families:
+
+- **Hub down.** A Remote Link Child or Hub client whose Home is unreachable never silently serves from
+  its own local providers, and the operator can see and set the per-device local fallback permission.
+- **Fallback chains.** Model and provider fallback (combos, subagent fallback, the main-account
+  hard-lock fallback, routing profiles) can be refused per scope: global, per provider, per model or
+  route, and per device.
+
+Defaults stay exactly as they are today (§7).
+
+## 2. Terms adopted from GOAT
+
+ADR-0021 §1.6 and its design doc define the hub-down policy. This note reuses these terms verbatim:
+
+- "a Child never silently serves its local providers when the Home is unreachable, and model, account and
+  provider fallback chains can be turned off per scope" (design §1.6, describing this package). The
+  account part is settled differently: see §9 C3.
+- "**local fallback** opt-in per device" (DH3: "pause plus decision item; local fallback opt-in per device").
+  GOAT's switch is `hub.devices[alias].localFallback`; the OpenCodex key is
+  `fallback.devices.<deviceId>.localFallback`, so the leaf reads the same on both sides.
+- "goal work on that device **pauses** (health condition `HubUnreachable`), people's own threads see
+  OpenCodex's 503 as today". OpenCodex keeps that 503 and adds `code: "home_unreachable"` (§6).
+- "GOAT reads that state; it never changes OpenCodex fallback settings", shown as "**on, off, unknown**
+  when the Home predates OCX-11". The Home advertises `fallback-policy/1` in `/readyz` `features`;
+  GOAT reads its absence as unknown.
+- "**GOAT fallback**" (ADR-0018 chains) is GOAT's word for moving GOAT work to another provider instance.
+  OpenCodex never uses that phrase; its own hops are "model fallback", "provider fallback" and "local
+  fallback".
+
+## 3. Inventory
+
+A fallback is a hop that serves a request from a **different model** or a **different provider** than
+the one chosen or requested.
+
+| # | Fallback | Kind | Where | Governed |
+|---|---|---|---|---|
+| F1 | Combo failover | provider or model | `src/server/responses/core-combo.ts` `executeComboResponses`; decisions in `src/combos/failover.ts`, picker in `src/combos/resolve.ts` | yes: reactive hop at the dispatch loop, proactive for a cooling first `failover` target |
+| F2 | Subagent model fallback | model or provider | `src/codex/subagent-model-fallback.ts` `applySubagentModelFallback` | yes, proactive |
+| F3 | Main-account hard-lock fallback | provider or model | `src/server/responses/main-hard-lock-routing.ts` `applyMainHardLockFallbackOverride` | yes, proactive |
+| F4 | Routing-profile fallback | provider or model | `src/server/responses/policy-fallback.ts` `handleResponsesWithPolicyFallback` | yes, reactive |
+| F5 | Claude Code classifier fallbacks | — | `src/claude/inbound-model-options.ts` | not a hop: the first configured entry always wins, with no availability check |
+| F6–F9 | Codex pool account switch (including pool → signed-in main), Anthropic OAuth, generic OAuth and API-key pool rotation | — | `src/codex/routing/selection.ts`, `src/oauth/anthropic-routing.ts`, `src/oauth/generic-account-failover.ts`, `src/providers/key-failover.ts` | not a fallback: intra-pool rotation (§9 C3) |
+| F10 | Remote Link Child, Home unreachable | local | `src/client/link-relay.ts` | no local path exists; 503 gains `home_unreachable` |
+| F11 | Hub client, Home unreachable | local | Codex on a Hub client talks to the Hub URL directly; the local listener carries only the dashboard relay (`src/client/hub-relay.ts`, 502) | no local path exists; 502 gains `home_unreachable` |
+
+Neither client kind can serve from local providers: both run the relay-only client runtime
+(`src/client/runtime.ts`), `ocx start` refuses an invalid client state instead of starting standalone
+(`src/cli/index.ts`), and the standalone recycle runs only when the link itself ends, never on an outage.
+`tests/clients/client-home-unreachable.test.ts` pins this with a load-time import-graph check.
+
+Out of scope because the target does not change: same-target retries (`src/lib/upstream-retry.ts`),
+the Antigravity direct lane's bridge fallback (`src/protocols/settings.ts`), transport fallbacks.
+
+### Fallback versus intra-pool rotation
+
+A **fallback** changes *what* answers: another model or another provider, with different capabilities,
+pricing, privacy terms or quality. An operator can reasonably refuse that.
+
+**Intra-pool rotation** changes only *which credential* answers for the same provider and model: the
+next account of an OAuth pool or the next key of an API-key pool when one is rate-limited, or the Codex
+pool's move away from an account at its usage threshold. The request, model and provider are unchanged.
+The maintainer rule recorded in `src/types/config.ts` says rotating away from an account upstream has
+just rate-limited "only ever runs after a refusal, so stranding it while a second logged-in account sits
+idle is a defect rather than a configuration choice". OCX-11 keeps that rule: rotation is resilience
+inside one pool and these controls do not govern it. An operator who wants one account only configures
+one account, or pauses the others, with the existing pool controls.
+
+## 4. Configuration
+
+```jsonc
+"fallback": {
+  "model": true,                                  // absent = true
+  "provider": true,
+  "providers": { "anthropic": { "model": false } },
+  "routes": { "combo/coding": { "provider": false } },
+  "devices": { "lnk_0123456789abcdef": { "localFallback": false, "model": false } }
+}
+```
+
+A hop's kind is decided by what changes: provider differs → `provider`; same provider, different model →
+`model`. A malformed leaf reads as absent and diagnostics name the section. Device keys are a link id
+or a Hub data key id; the resolver maps a request's admission key id to its link through the link store.
+
+CLI: `ocx fallback status|off|on|clear [--kind model|provider|all] [--provider|--route|--device]` and
+`ocx fallback local on|off --device <id|alias>`. Management: `GET|PUT /api/fallback`.
+
+## 5. Resolver
+
+`src/routing/fallback-policy.ts`: `resolveFallbackPolicy`, `fallbackHopKind`, `guardFallbackHop` (records
+and refuses) and a 20-entry event ring. Nothing else reads `config.fallback` on the request path.
+`src/routing/fallback-state.ts` builds the `ocx.fallback/1` view and applies the scoped write.
+
+## 6. Error contract
+
+A refused hop answers with the status of the failure that triggered it (with its `Retry-After`), or 429
+for proactive quota/cooldown/hard-lock refusals and 503 otherwise, and:
+
+```json
+{ "error": { "type": "fallback_refused", "code": "fallback_refused", "message": "… To allow it: ocx fallback on --kind provider --route combo/coding",
+  "fallback": { "kind": "provider", "scope": "route", "scopeKey": "combo/coding", "from": "a/m", "to": "b/n", "trigger": "rate_limited" } } }
+```
+
+plus `x-opencodex-fallback: refused`. A taken hop is recorded in status; no header is added to the
+answering response, because the combo and policy responses are built deep in child dispatch.
+
+Hub down keeps status and retry semantics: `{ "error": "link tunnel unavailable", "code": "home_unreachable", "message": "…" }`
+on a Child (503) and `{ "error": "hub relay unavailable", "code": "home_unreachable", … }` on a Hub
+client's dashboard relay (502).
+
+## 7. Defaults and migration
+
+No default changes and no migration: absent `fallback` keeps every chain as it was, and an absent
+`localFallback` reads false, which is what a Child already does.
+
+## 8. Visibility
+
+`ocx status --json` `fallback` (live, the Home's over the link on a Child, or the local policy), `GET
+/api/fallback` (also over the link relay, aliases omitted), `/readyz` `fallback-policy/1`, and the
+dashboard Fallback panel on the Dashboard overview and on Remote Link. Events carry time, site, route, kind, from, to,
+trigger, outcome, scope and device id; never bodies, keys or account identities.
+
+## 9. Decisions (supervisor, 2026-10-22)
+
+- **C1 precedence: accepted.** Route > provider > global, most specific first; a device set to off always refuses.
+- **C2 `localFallback: true`: accepted as permission only.** OpenCodex keeps answering 503 when the Home is
+  unreachable; GOAT reads the flag and moves the work itself (ADR-0021 §1.6, "pause + decision item").
+- **C3 account rotation: overridden.** Account-pool 429 rotation stays not switchable; the maintainer rule
+  in `src/types/config.ts` stands. Rotation is intra-pool resilience, not fallback, so the `account` kind
+  is out of OCX-11 (§3, "Fallback versus intra-pool rotation").
+
+## 10. Tests
+
+`tests/routing/fallback-policy.test.ts` (81-case precedence matrix, degrade, device identity, contract,
+privacy), `tests/responses/responses-combo-fallback-off.test.ts`, `tests/routing/routing-policy-fallback-off.test.ts`,
+`tests/routing/subagent-model-fallback-off.test.ts`, `tests/codex-integration/main-account-hard-lock-fallback-off.test.ts`,
+`tests/server/fallback-routes.test.ts`, `tests/clients/client-home-unreachable.test.ts`, `tests/cli/cli-fallback.test.ts`,
+`tests/server/readyz-features.test.ts` and `gui/tests/fallback-panel.test.tsx`.

docs-site/astro.config.mjs

@@ -103,6 +103,7 @@ export default defineConfig({
             { label: "Desktop App", translations: { fr: "Application de bureau", ko: "데스크톱 앱", "zh-CN": "桌面应用", "zh-TW": "桌面 App", ru: "Настольное приложение", ja: "デスクトップアプリ", tr: "Masaüstü Uygulaması" }, slug: "guides/desktop-app" },
             { label: "Model Ordering", translations: { fr: "Ordre des modèles", ko: "모델 정렬에 관하여", "zh-CN": "模型排序", "zh-TW": "模型排序", ru: "Сортировка моделей", ja: "モデルの並び順", tr: "Model Sıralaması" }, slug: "guides/model-ordering" },
             { label: "Combos", translations: { fr: "Combinaisons", ko: "콤보", "zh-CN": "组合", "zh-TW": "組合", ru: "Комбо", ja: "コンボ", tr: "Kombolar" }, slug: "guides/combos" },
+            { label: "Fallback off", translations: { fr: "Désactiver le repli", ko: "폴백 끄기", "zh-CN": "关闭回退", "zh-TW": "關閉備援", ru: "Отключение резерва", ja: "フォールバックをオフにする", tr: "Yedeği kapatma" }, slug: "guides/fallback-off" },
             { label: "Protocol Paths", translations: { fr: "Chemins de protocole", ko: "프로토콜 경로", "zh-CN": "协议路径", "zh-TW": "協定路徑", ru: "Пути протоколов", ja: "プロトコル経路", tr: "Protokol Yolları" }, slug: "guides/protocol-paths" },
             { label: "Claude Code", translations: { fr: "Claude Code", ko: "Claude Code", "zh-CN": "Claude Code", "zh-TW": "Claude Code", ru: "Claude Code", ja: "Claude Code", tr: "Claude Code" }, slug: "guides/claude-code" },
             { label: "Grok Build", translations: { fr: "Grok Build", ko: "Grok Build", "zh-CN": "Grok Build", "zh-TW": "Grok Build", ru: "Grok Build", ja: "Grok Build", tr: "Grok Build" }, slug: "guides/grok-build" },

docs-site/src/content/docs/guides/combos.md

@@ -590,6 +590,8 @@ rows show decision count, applied count, and average latency for `typesafe`, `sy
 ## What happens when a target fails
 
 Combo failures are divided into **hop** failures and **terminal** failures.
+To make a combo fail with a clear `fallback_refused` error instead of hopping to another model or
+provider, see [Fallback off](/guides/fallback-off/).
 
 | Result | Behavior |
 | --- | --- |

docs-site/src/content/docs/guides/fallback-off.md

@@ -0,0 +1,127 @@
+---
+title: "Fallback off"
+description: Turn model and provider fallback off per scope, and see what happens on a Remote Link Child when its Home is unreachable.
+---
+
+OpenCodex can serve a request from a different model or provider when the one you chose fails: a
+[combo](/guides/combos/) moves to its next target, a [routing profile](/guides/routing-profile-editor/)
+tries its next candidate, a spawned subagent moves down its model fallback chain, and the main-account
+hard lock can reroute to its configured fallback model. All of these stay on by default.
+
+Turn a kind of fallback **off** when you would rather get a clear error than an answer from somewhere
+you did not pick.
+
+## What counts as a fallback
+
+A fallback is a hop to a **different model** or a **different provider**:
+
+| Kind | Example | Switch |
+|---|---|---|
+| Model fallback | `anthropic/claude-opus` fails, `anthropic/claude-sonnet` would answer | `model` |
+| Provider fallback | `anthropic/claude-opus` fails, `openai/gpt-6-sol` would answer | `provider` |
+
+Rotation among the accounts or API keys of **one provider pool** for the same model is not a fallback.
+It is how a pool absorbs a rate-limited account, and it keeps working whatever you set here. Retrying
+the same target, and transport fallbacks that keep the same target, are not fallbacks either.
+
+## Turn fallback off
+
+```bash
+ocx fallback off                               # model and provider fallback, everywhere
+ocx fallback off --kind provider               # only cross-provider hops
+ocx fallback off --route combo/coding          # one combo, routing profile or model selector
+ocx fallback off --provider anthropic          # hops that leave one provider
+ocx fallback on --route combo/coding           # an exception under a global off
+ocx fallback clear --route combo/coding        # remove an override
+ocx fallback status                            # current settings and the last fallback
+```
+
+The same switches are on the dashboard: the **Fallback** card at the bottom of the **Dashboard** holds the
+global switches, the overrides and the last fallback taken or refused.
+
+### Precedence
+
+1. A **device** set to off always refuses (see [per device](#per-device-settings)). A device setting can
+   only refuse; turning it on never reopens a hop another scope refuses.
+2. Otherwise the most specific setting wins: **route**, then **provider**, then **global**.
+3. Nothing set means allowed, which is today's behavior.
+
+The settings live in `config.json` under `fallback`:
+
+```json
+{
+  "fallback": {
+    "provider": false,
+    "routes": { "combo/coding": { "provider": true } },
+    "providers": { "anthropic": { "model": false } },
+    "devices": { "lnk_0123456789abcdef": { "localFallback": false, "model": false } }
+  }
+}
+```
+
+A malformed leaf is ignored, so policy resolution uses the next applicable scope, and `ocx config`
+diagnostics name it. A whole-config write that contains one is refused instead.
+
+## The error you get
+
+A refused fallback answers with the **status of the failure that triggered it**, so a refused 429 is
+still a 429 with its `Retry-After`, and with this body:
+
+```json
+{
+  "error": {
+    "type": "fallback_refused",
+    "code": "fallback_refused",
+    "message": "combo/coding: anthropic/claude-opus failed (rate_limited). Provider fallback to openai/gpt-6-sol is off for route combo/coding, so OpenCodex did not send the request there. To allow it: ocx fallback on --kind provider --route combo/coding",
+    "fallback": { "kind": "provider", "scope": "route", "scopeKey": "combo/coding",
+                  "from": "anthropic/claude-opus", "to": "openai/gpt-6-sol", "trigger": "rate_limited" }
+  }
+}
+```
+
+The response also carries `x-opencodex-fallback: refused`. When the first target is already known to be
+unavailable, the refusal happens before anything is sent, and the fallback target is never contacted.
+The status depends on why the first target was skipped:
+
+- A combo's first target cooling down, or with its quota spent: 429.
+- A combo's first target disabled or not usable for the request: 503.
+- A blocked main account: 429.
+- An unavailable subagent model: 503.
+
+After a real upstream failure, the refusal keeps that failure's status.
+
+## Remote Link Children and Hub clients
+
+A Remote Link Child and a Hub client never serve requests from their own local providers. When the
+Home is unreachable, a Child answers a retryable `503` (and a Hub client's dashboard relay a `502`) with:
+
+```json
+{ "error": "link tunnel unavailable", "code": "home_unreachable", "message": "The Home is unreachable, and OpenCodex did not serve this request from this device's local providers. …" }
+```
+
+Nothing is sent to a local provider, and the link key and the request body never leave the device.
+
+### Per-device settings
+
+On the Home, each linked device has two switches, on **Remote Link** or with the CLI:
+
+```bash
+ocx fallback local on --device mac-studio      # permission for local fallback while the Home is down
+ocx fallback off --device mac-studio           # refuse model and provider fallback for this device's requests
+```
+
+`--device` takes a link alias or id from `ocx link status`, or a Hub data key id.
+
+**Local fallback** is a permission that GOAT reads: with it on, GOAT may move that device's work to the
+device's own sign-ins while the Home is unreachable. OpenCodex itself still answers `home_unreachable`
+and never serves the device locally. It is off unless you turn it on.
+
+## Status
+
+`ocx status --json` includes a `fallback` object (on a Child it is the Home's, read over the link), and
+`GET /api/fallback` returns the same `ocx.fallback/1` document: the effective global switches, the
+overrides, every device with its local fallback permission, and the last 20 fallbacks taken or refused
+with time, route, from, to, trigger and the scope that decided. The list is kept in memory and resets
+when the proxy restarts. It never contains request bodies, keys or account identities.
+
+A Home that supports this advertises `fallback-policy/1` in `GET /readyz` `features`.

docs-site/src/content/docs/guides/remote-link.md

@@ -97,6 +97,7 @@ When a step fails, the dashboard shows the reason and, when SSH reported one, a
 - **ocx was not found on the remote computer**: OpenCodex looks for `ocx` on the PATH of a non-interactive SSH session first, then in `~/.bun/bin`, `~/.local/bin`, `/opt/homebrew/bin` and `/usr/local/bin`. If it is installed elsewhere, add that directory to PATH in a file the remote shell reads for non-interactive sessions, such as `~/.zshenv` for zsh.
 - **OpenCodex on the remote computer is too old**: run `ocx update` on that computer. Remote Link needs 2.66.0 or later.
 - **The remote computer did not report an OpenCodex version**: `ocx --version` on that computer printed something else, for example the usage text of an unsupported Windows install.
+- **Requests fail with `home_unreachable` while the Home is down**: a Child never serves requests from its own providers. When the tunnel is down before a request is forwarded, the Child answers a retryable 503 and sends nothing to the Home. If the tunnel drops after forwarding started, the answer is also `home_unreachable`, but the Home may already have received the request. A failed link relay authentication answers `link_relay_auth_failed` instead: upgrade the Home and Child or re-link. Whether GOAT may move that device's work to its own sign-ins is the per-device local fallback permission on the Home's **Remote Link** page, or `ocx fallback local on --device <alias>`; see [Fallback off](/guides/fallback-off/).
 
 ## Security