diff --git a/README.md b/README.md index 8197f747220..595ec08b2f9 100644 --- a/README.md +++ b/README.md @@ -335,7 +335,7 @@ OpenCodex tracks process-retained state in the categories below. Each has a docu store, which is pinned and never evicted. - **4 observed buffers** (translator accumulators, image/OAuth/Grok tails) are monitored for in-flight byte pressure without eviction. -- **31 state-store registrations** handle expiry sweeps (60 s interval) and +- **32 state-store registrations** handle expiry sweeps (60 s interval) and config-generation reconciliation so stale provider/account keys are removed. - **Path and fingerprint memos** (workspace metadata, hardened identities, installation salts, mode-hint capabilities) use insertion-order LRU caps (8–128 entries). diff --git a/devlog/_plan/261019_cache_affinity/000_spec.md b/devlog/_plan/261019_cache_affinity/000_spec.md new file mode 100644 index 00000000000..e5778c53d87 --- /dev/null +++ b/devlog/_plan/261019_cache_affinity/000_spec.md @@ -0,0 +1,399 @@ +# Spec — Provider cache affinity for multi-credential routes + +Status: historical design draft. The current implementation contract is in +[`structure/transports/inventory.md`](../../../structure/transports/inventory.md#transport-inventory). +The proposal and contribution notes below are retained for design history and +are not the current implementation contract. + +## Problem Statement + +I use OpenCodex as my local proxy in front of several subscriptions and API +plans at once — an Anthropic subscription, Kimi Code, Ollama Cloud, and +others — with the Codex app as my main client and other apps on top. Several +of these providers have more than one credential available (an API key pool, +or a primary plus a spare key). + +Today, when a conversation makes its second, third, and twentieth request, +OpenCodex is free to send each one through a different credential of the same +provider than the one that answered before. Every time it does, the +provider-side prompt cache that the previous turn paid to create is +abandoned: the new credential has never seen this conversation, so the whole +history is sent and billed as fresh input again. The longer the conversation, +the more this costs, and it costs it on exactly the long agentic sessions +where caching matters most. + +The wire-level pieces are already in place: several providers forward a +stable prompt-cache key, and the Anthropic adapter places cache breakpoints. +What is missing is the routing-level half — remembering which credential +served a conversation, and preferring that same credential for the +conversation's next turn while it is still able to serve. The only affinity +OpenCodex has today is the Codex account pool's thread affinity, which covers +ChatGPT accounts only. A conversation routed to Anthropic, Kimi, or any other +multi-credential provider gets no such protection, and ordinary key rotation +after a rate limit can silently throw away a warmed cache even when another +key was never needed. + +## Solution + +Add **provider cache affinity**: a bounded, per-conversation memory that +records which credential of a provider served a conversation's last +successful turn, together with how much prompt cache that turn read and +wrote. On the conversation's next request, credential selection prefers the +recorded credential — as a preference inside normal eligibility, never as an +override. Quota cooldowns, health holds, per-model barring, request pacing, +and routing policy all still win; affinity only breaks ties among +credentials that could serve anyway. + +Affinity is driven by measured cache evidence, not by assumption. A +conversation stays with its credential across turns only while the cache it +built there is worth keeping and has not gone cold (five minutes, or an hour +where the provider reports long-retention cache writes). Providers that +never report cache usage are treated as silently caching: the prompt that +was sent counts as written. A failed turn releases or rests the binding +according to the same failure classification the key-failover logic already +computes, so a conversation is never pinned to a credential that can no +longer serve it. + +The feature is per-provider configurable (`auto`, `session`, `turn`, `off`), +defaults to `auto`, and is a no-op for single-credential providers. + +## User Stories + +1. As a user with two Anthropic API keys, I want a long conversation to keep + using the key that answered its first turn, so that the prompt cache the + first turn paid to create is read back instead of rebuilt on the other + key. +2. As a Kimi Code user, I want my conversation to stay on the credential + that holds its cache-affinity session key, so that the provider's + documented cache affinity actually accumulates hits across turns. +3. As an Ollama Cloud user, I want affinity to work even though the provider + reports no cache tokens, so that a vendor that caches silently still + keeps my conversation where its prompt was last sent. +4. As a user whose primary key hits a rate limit mid-conversation, I want + the request to fail over to the spare key as it does today, so that + availability is never sacrificed to keep a cache. +5. As that same user, I want the conversation to bind to the spare key once + the spare succeeds, so that subsequent turns rebuild cache once on the + new credential instead of flapping between keys on every request. +6. As a user whose bound credential exhausts its quota, I want affinity to + release the conversation rather than retry the exhausted credential, so + that I never see avoidable quota errors for the sake of a cache. +7. As a user starting a brand-new conversation, I want routing to choose its + first credential exactly as it does today, so that affinity never + distorts initial placement, load balancing, or pacing. +8. As a user running several conversations at once, I want each conversation + bound independently, so that one conversation's affinity never drags + another conversation onto a busy or exhausted credential. +9. As a user who switches models mid-conversation, I want the binding to be + re-evaluated for the new model, so that I am never pinned to a credential + that does not serve the model I just picked. +10. As a user in a long tool-calling turn, I want every tool-result + continuation in that turn to stay on the same credential, so that + within-turn cache reads are never lost between steps of one turn. +11. As a user who returns to a conversation after ten minutes, I want + routing to be free to place it on the best credential again, so that a + cold cache exerts no pull. +12. As a user of Anthropic's one-hour cache retention, I want a conversation + that wrote a one-hour cache to stay bound for that hour, so that the + longer retention I am paying for is actually exploited. +13. As a user with request pacing in round-robin mode, I want affinity to + pin my conversation without breaking per-key rate limits, so that cache + locality and pacing guarantees coexist. +14. As an operator, I want to turn affinity off for a provider with one + setting, so that I can restore today's exact rotation behavior if I ever + suspect affinity of causing a problem. +15. As an operator, I want a `session` mode that keeps a conversation on its + credential for the whole session regardless of cache measurements, so + that I can choose predictability over the automatic heuristic. +16. As an operator, I want a `turn` mode that only holds a conversation + within a single turn, so that cross-turn placement stays fully dynamic + while tool loops keep their cache. +17. As a single-key user, I want affinity to change nothing at all, so that + the feature carries zero risk for the simplest setups. +18. As a combo user, I want affinity to pin the credential inside whichever + combo target served, without changing which target the combo picks, so + that combo failover semantics are untouched. +19. As a Codex subscription user, I want the existing Codex thread affinity + to keep owning my conversations exactly as today, so that this feature + never interferes with the account pool's generation and probe logic. +20. As a user who removes a key from a provider's pool, I want bindings to + that key forgotten, so that no conversation is ever offered a credential + that no longer exists. +21. As a user who disables a provider mid-conversation, I want its bindings + ignored immediately, so that the next request routes elsewhere instead + of failing against a stale binding. +22. As a privacy-conscious user, I want the affinity store and my request + logs to contain no raw API keys, session ids, or account identifiers, + so that keeping this state creates no new credential-leak surface. +23. As a user inspecting my request history, I want each request to show + whether affinity kept, released, or newly bound its credential — and + why — so that an unexpected credential choice is explainable from the + log alone. +24. As a user whose bound credential is in a short cooldown, I want the + conversation served by another credential and not re-offered the cooled + one until it recovers, so that affinity and cooldowns compose instead + of fighting. +25. As a heavy user with hundreds of conversations, I want the affinity + store strictly bounded in count and bytes, so that the feature can + never become an unbounded memory or disk cost. +26. As a user restarting the proxy, I want to be told (in documentation) + that bindings are forgotten on restart, so that a first post-restart + request re-placing a conversation is expected behavior, not a bug. +27. As a maintainer, I want affinity decisions covered by tests at the HTTP + seam with fake upstreams, so that the behavior is proven the way users + experience it rather than by inspecting internal state. +28. As a maintainer, I want affinity to reuse the existing session-lane + identity, credential-selection seam, and attempt ledger, so that the + feature adds one new store and one preference hook instead of new + plumbing through every handler. + +## Implementation Decisions + +- **One new module owns affinity.** A provider cache-affinity store holds, + per conversation, the serving provider, model, and credential identity, + the last turn's cache-read and cache-write token counts, the input size + when no cache telemetry was reported, a timestamp, and a retention class + (short or long). It exposes exactly three operations: *consult* (given a + conversation and candidate provider, return the bound credential if it is + still worth preferring), *record* (a turn completed on a credential, with + its normalized cache telemetry), and *release* (forget or rest a binding, + with a reason). No other module reaches into its internals. + +- **Conversation identity is the existing session lane.** Bindings are keyed + by the session-lane identity the request pipeline already derives for + turn admission and combo recall. Only requests carrying a real, + client-supplied lane identity may create or consult bindings; a + per-request allocated lane is one-shot by definition and never binds. + Lane values are used as internal map keys only and are never written to + logs or disk in raw form. + +- **Credential identity is a selection identity, never key material.** The + store records the same stable per-credential selection identifier the + key-failover machinery already uses to say which pool entry was attempted. + Raw API keys, OAuth account ids, and authorization headers are never + stored, logged, or hashed into the affinity store by this feature. + +- **The consult seam is the shared credential-selection step.** Every + inference path (Responses, Chat Completions, Messages, and Compact) + already funnels through one proactive credential-selection operation + before dispatch. Affinity is consulted there, and only there: if it + returns a credential, that credential is treated as the selection's first + preference; if it returns nothing, selection proceeds byte-for-byte as + today. No request handler gains affinity logic of its own. This is the + feature's single seam. + +- **Eligibility always outranks affinity.** A bound credential is preferred + only if it passes every check normal selection applies: provider enabled, + credential present in the current configuration, model not barred for + that credential, no active quota/billing cooldown, no health hold, and + request-pacing admission. Any failed check releases (or rests) the + binding with a recorded reason and selection falls through to the normal + algorithm in the same request. Affinity can never cause a request to be + sent somewhere normal routing would refuse. + +- **Silent-cache detection uses telemetry provenance.** The finalization + seam already classifies each request's cache telemetry as observed, + synthesized, or unknown. Only "unknown" (the adapter reported no cache + detail at all) triggers the silent-cache rule; an adapter that reports + honest zeros is believed. + +- **Recording happens at attempt finalization.** The request-attempt ledger + already finishes each attempt with the serving credential and normalized + cache usage (cache-read tokens, cache-creation tokens) regardless of + which adapter produced it. The affinity store records from that seam, so + streamed, non-streamed, native, and translated paths all report through + one place. Only successful terminal attempts bind or refresh a + conversation; pre-commit failures never do. + +- **Worthiness is measured, with a silent-cache rule.** In `auto` mode a + binding is kept across turns when the last turn read or wrote at least + 1,024 cached tokens. When an adapter reports no cache telemetry at all, + the turn's input size counts as written once it reaches 1,024 tokens, so + silently caching providers still accumulate affinity. Below the threshold + the binding is kept only for within-turn continuations. + +- **Within-turn means a detected continuation, never an age guess.** A request + is a continuation when it returns tool results for the lane's in-progress + turn (a tool-result final message on the Responses, Chat, or Messages + wire). Timestamps decide only warm/cold, never turn boundaries. `turn` + mode and `auto`'s continuation path both require this signal. + +- **Cold windows follow reported retention.** A binding's cache is + considered warm for five minutes after the turn that built it, matching + the shortest upstream retention; where the turn's cache write carried + long (one-hour) retention — as the Anthropic path reports under + long-retention configuration — the warm window is one hour. Past the warm + window the binding exerts no pull in `auto` mode, though it may be + refreshed by a later turn that lands on the same credential anyway. + +- **Failure handling reuses existing classification.** Affinity introduces + no new error taxonomy. When the failover machinery classifies a failure + as quota, credit, or authentication failure for a credential, a binding + to that credential is released and the credential is not re-preferred + until the cooldown the failover machinery already computes has passed. + Rate-limit rotation proceeds exactly as today; when a rotated credential + ultimately serves the turn, the binding moves to it (recorded at + finalization like any success). + +- **Modes are per-provider configuration.** Providers gain an optional + cache-affinity setting with values `auto` (default), `session`, `turn`, + and `off`, validated at the configuration boundary like other provider + enums, with invalid values rejected rather than silently coerced. + `session` prefers the bound credential whenever it is eligible, ignoring + worthiness and cold windows. `turn` prefers it only for within-turn + continuations. `off` disables both consulting and recording for that + provider. Providers with fewer than two credentials are unaffected in + every mode. + +- **Interaction with request pacing is explicit.** When pacing selects + round-robin or distributed keys, an eligible affinity preference is + honored before the rotation cursor advances, and serving from a binding + does not consume a rotation step. Per-credential pacing limits still + apply in full: if the bound credential is pacing-limited, normal + selection (including its overload refusal) decides the request. + +- **Interaction with combos is deliberately narrow.** Combo target + selection — including existing per-lane combo recall — is unchanged. + Affinity operates only inside the chosen target, pinning that target's + credential. Affinity never changes which combo target a request is + offered, in either direction. + +- **The Codex account pool is excluded.** Requests served by the canonical + OpenAI/Codex forward path keep using the existing Codex thread affinity, + with its credential generations, probe leases, and lineage rules. The + generic store never sees them. This keeps two affinity systems from + owning the same conversation; the generic one owns multi-credential + key-auth provider routes only. + +- **The store is process-local and registered for sweeping.** Following the + precedent of the combo session-recall store, bindings live in memory, + bounded to at most 512 conversations with per-entry and total byte caps, + least-recently-written eviction, a 24-hour hard expiry (the store's bound + on `session` mode), and expiry evaluated on read. The store + registers with the state-store sweeper so expired entries are reaped + periodically and a configuration generation change (provider removed, + credential removed, provider disabled) reconciles bindings away instead + of leaving them to fail eligibility one by one. + +- **Observability uses the existing decision-reporting pattern.** Each + request's log records the affinity outcome — newly bound, kept, released, + or not applicable — with a bounded reason vocabulary (kept for cache, + kept for session, kept within turn, cold, below threshold, credential + ineligible, credential resting, no lane, disabled), mirroring how Codex + affinity decisions are already reported. Reasons never include credential + or conversation identifiers. + +- **No wire behavior changes.** Prompt-cache keys, cache breakpoints, + retention settings, and adapter request shaping are untouched. Affinity + changes only which eligible credential a request is sent with. + +## Testing Decisions + +- **Good tests assert external behavior**: which credential a fake upstream + actually received across a scripted multi-turn conversation, the status + and body the client received, and the affinity outcome recorded on the + request log. Tests must not assert the affinity store's internal map + contents except in the store's own unit tests, and there only through its + three public operations. + +- **Primary seam: end-to-end through the real server** with a two-key + provider whose fake upstream reports cache usage. The suite proves, in + order: first turn binds; second turn arrives on the same credential + (affinity kept); a forced 429 rotates and the third turn rebinds to the + spare; after the first credential's cooldown the binding does not flap + back mid-conversation; a cold conversation (clock advanced past the warm + window) is placed by normal selection again. Prior art: the existing + server-level key-failover end-to-end tests and the Codex session-affinity + tests, which already script multi-account fake upstreams this way. + +- **Store unit tests** cover the decision logic in isolation: worthiness + threshold, silent-cache counting, short and long cold windows, `session` + and `turn` mode semantics, capacity and byte-cap eviction, + expiry-on-read, and generation reconciliation dropping bindings for + removed providers and credentials. Prior art: the combo session-recall + tests, whose bounded-store contract this mirrors. + +- **Credential-selection tests** prove the preference composes with + existing selection rules: barred model on the bound credential, + pacing-limited bound credential, disabled provider, and `off` mode all + fall through to today's selection result unchanged. Prior art: the + existing key-failover unit tests. + +- **Ledger tests** prove recording discipline: a pre-commit failure does + not bind, a successful streamed turn binds with its reported cache + telemetry, and a quota failure releases an existing binding. Prior art: + the request-log attempt tests. + +- **Regression gates**: the existing key-failover, combo-recall, + session-affinity (Codex pool), and chat conversation-affinity suites must + pass unmodified, demonstrating the Codex pool and combo behavior are + untouched. New test files are registered in the test-layout map as the + repository's layout guard requires. Typecheck passes. Because the change + adds a provider configuration option, the provider documentation and the + structure document owning routing/provider behavior are updated in the + same change, per repository rules. + +## Out of Scope + +- Any change to the Codex account pool: its thread affinity, account + selection, probe leases, lineage, or quota logic. +- Persistence of bindings across process restarts (a documented follow-up; + v1 follows the process-local precedent of combo recall and Codex + lineage, and restart behavior is called out in user-facing docs). +- Multi-account generic OAuth providers (for example multiple Anthropic + OAuth logins outside the existing Anthropic pool machinery); v1 covers + key-pool credentials. Extending the store to OAuth account identities is + a follow-up once the key-pool behavior is proven. +- Changing combo target selection, routing profiles, or policy evaluation. +- Any change to prompt-cache wire behavior: cache keys, breakpoints, + retention, or adapter request/response shaping. +- Response caching or result reuse of any kind; this feature only + influences which credential serves a live request. +- GUI or dashboard surfacing of the setting or of affinity decisions; + configuration is file/API-level and decisions appear in request logs. + GUI support is a follow-up. +- Cross-provider affinity (moving a conversation between providers to chase + a cache); bindings only ever prefer a credential of the provider routing + already chose. + +## Further Notes + +- **Viability against the current tree** (checked at `6936d9885`, v2.88.0): + every seam this spec needs already exists. The session-lane identity is + already derived per request and already keys combo recall and turn + admission; the proactive credential-selection operation is already the + single funnel for the Responses, Chat, Messages, and Compact paths; + attempt finalization already normalizes cache-read and cache-creation + tokens from every adapter into one usage shape; and the state-store + sweeper already accepts bounded stores with expiry and + generation-reconciliation hooks, with the combo session-recall store as a + working template for the bounds and eviction contract. No new identity, + ledger, or lifecycle plumbing is required — the change is one store, one + preference consult, one recording hook, and one configuration field. + +- **Design provenance**: the worthiness threshold, cold windows, + silent-cache rule, and mode vocabulary are modeled on Magpie's gateway + affinity design, re-expressed in this repository's terms (session lane, + credential selection, attempt ledger, state-store sweeper). This spec + deliberately diverges from that design in one place — process-local + storage instead of an on-disk file — to match this repository's existing + precedent and keep the change reviewable; persistence is the named + follow-up if maintainers want restart durability. + +- **Contribution hygiene**: this repository's tracker has no + `ready-for-agent` label, and its issue gates close non-templated + submissions, so this spec lives as a local planning record rather than a + freeform upstream issue. Before implementation, open the upstream feature + issue using the repository's feature-proposal template, link this spec, + and target the PR at `dev`. Expect the security-review label set to apply + if the implementation touches credential handling beyond selection + identity; the design above is specifically shaped to avoid that (no key + material is stored or logged). + +- **Expected benefit, stated honestly**: the win scales with credential + multiplicity and conversation length. Single-credential users see no + behavior change. Multi-key users on long sessions should see cache-read + tokens rise and effective input cost fall on second and later turns; the + end-to-end tests assert credential stickiness directly, and a before/after + cache-read comparison on a real multi-key setup is the right acceptance + evidence to attach to the PR. diff --git a/docs-site/astro.config.mjs b/docs-site/astro.config.mjs index e81e8b1fb7e..d0299508789 100644 --- a/docs-site/astro.config.mjs +++ b/docs-site/astro.config.mjs @@ -154,6 +154,7 @@ export default defineConfig({ items: [ { label: "Overview", translations: { fr: "Vue d’ensemble", ko: "개요", "zh-CN": "概览", "zh-TW": "概覽", ru: "Обзор", ja: "概要", tr: "Genel Bakış" }, slug: "reference/configuration" }, { label: "Providers", translations: { fr: "Fournisseurs", ko: "프로바이더", "zh-CN": "提供商", "zh-TW": "供應商", ru: "Провайдеры", ja: "プロバイダー", tr: "Sağlayıcılar" }, slug: "reference/configuration/providers" }, + { label: "Account Auto-pause", slug: "reference/configuration/account-auto-pause" }, { label: "Catalog", translations: { fr: "Catalogue", ko: "카탈로그", "zh-CN": "目录", "zh-TW": "目錄", ru: "Каталог", ja: "カタログ", tr: "Katalog" }, slug: "reference/configuration/catalog" }, { label: "Routing", translations: { fr: "Routage", ko: "라우팅", "zh-CN": "路由", "zh-TW": "路由", ru: "Маршрутизация", ja: "ルーティング", tr: "Yönlendirme" }, slug: "reference/configuration/routing" }, { label: "Agents", translations: { fr: "Agents", ko: "에이전트", "zh-CN": "代理", "zh-TW": "代理", ru: "Агенты", ja: "エージェント", tr: "Ajanlar" }, slug: "reference/configuration/agents" }, diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 6b7439f0b0c..f87e130443c 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -108,6 +108,7 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | --- | --- | --- | | `adapter` | `string` | L'un des `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (ou alias `azure`). | | `baseUrl` | `string` | URL de base de l'API en amont. La plupart des points de terminaison fixes intégrés ignorent une valeur incompatible ; les préréglages de clés protégés contre les collisions préservent une ancienne destination personnalisée portant le même nom. | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | Préférence pour la clé du dernier tour réussi ; nécessite au moins deux entrées de clé et exclut OAuth/forward. Par défaut : `auto`. Les refus locaux conservent la liaison ; un refus amont ne libère que la liaison du fournisseur et de l’entrée qui ont servi. Voir le [contrat complet en anglais](/reference/configuration/providers/#provider-entries-ocxproviderconfig). | | `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. Pour les adaptateurs `runTurn`, dont Cursor, la limite compte les tours actifs plutôt que les envois physiques : RunSSE et BidiAppend peuvent se chevaucher dans un même tour, tandis qu’un autre tour attend. Les envois suivants respectent toujours les intervalles de démarrage. | | `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Sélection des clés pour les requêtes normales : `active` par défaut conserve la clé enregistrée et la rotation sur erreur ; `round-robin` répartit les allocations entre les clés disponibles sans modifier la sélection enregistrée. Cela équilibre les allocations, pas les jetons ni les coûts. | | `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Limites uniformes par référence de clé : RPM, intervalle minimal et concurrence, partagées entre modèles et cumulées avec les plafonds fournisseur et modèle. Les références identiques partagent leur capacité ; plusieurs références du même projet ne garantissent pas des quotas indépendants. TPM et quotas quotidiens ne sont pas gérés. | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 7ecffe3764a..956c0b25894 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -102,6 +102,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai` (または別名 `azure`) のいずれか。 | | `baseUrl` | `string` |アップストリーム API のベース URL。ほとんどの組み込み固定エンドポイントは不一致を無視します。衝突安全キー プリセットは、古い同じ名前のカスタム宛先を保持します。 | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | 直前の成功ターンを処理したキーを優先します。キーが2件以上あるプールが対象で、OAuth/forwardは対象外です。既定値は `auto`。ローカル拒否では関連付けを保持し、上流の拒否では処理したプロバイダーとエントリーが一致する関連付けのみ解除します。[英語の完全な仕様](/reference/configuration/providers/#provider-entries-ocxproviderconfig)を参照してください。 | | `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 Cursor を含む `runTurn` アダプターでは、上限は物理送信数ではなく進行中のターン数を数えます。同じターンの RunSSE と BidiAppend は重なり得ますが、別のターンは待機します。後続送信にも開始間隔は適用されます。 | | `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 通常リクエストのキー選択です。既定の `active` は保存済みキーと既存の障害時切替を維持します。`round-robin` は保存済み選択を変えず、利用可能なプールのキーを順に割り当てます。トークンや料金ではなく割当回数を分散します。 | | `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | キー参照ごとの共通 RPM・開始間隔・同時実行上限です。全モデルで共有し、プロバイダーとモデルの上限も同時に適用します。同一参照は枠を共有しますが、同じ上流プロジェクトの別参照は独立クォータを意味しません。TPM と日次上限は管理しません。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 6102a5442e6..a6249220ace 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -102,6 +102,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` 중 하나이며, `azure`는 별칭입니다. | | `baseUrl` | `string` | 상위 API 기본 URL입니다. 대부분의 내장 고정 엔드포인트는 불일치를 무시합니다. 충돌 안전 키 프리셋은 같은 이름의 이전 사용자 지정 목적지를 보존합니다. | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | 마지막 성공 턴을 처리한 키를 우선합니다. 키 풀 항목이 2개 이상일 때 적용하며 OAuth/forward는 제외합니다. 기본값은 `auto`입니다. 로컬 거부는 바인딩을 유지하고 업스트림 거부는 실제 처리한 공급자와 항목이 일치하는 바인딩만 해제합니다. [전체 영어 계약](/reference/configuration/providers/#provider-entries-ocxproviderconfig)을 참고하세요. | | `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. Cursor를 포함한 `runTurn` 어댑터에서는 물리적 전송 수가 아니라 진행 중인 턴 수를 제한합니다. 같은 턴의 RunSSE와 BidiAppend는 겹칠 수 있지만 다른 턴은 대기합니다. 후속 전송에도 시작 간격은 적용됩니다. | | `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 정상 요청의 키 선택 방식입니다. 기본 `active`는 저장된 키와 기존 오류 전환을 유지하며, `round-robin`은 저장된 활성 키를 변경하지 않고 사용 가능한 풀 키를 요청별 순서대로 배정합니다. 토큰이나 비용이 아니라 배정 횟수를 분산합니다. | | `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 키 참조별 동일한 RPM·최소 시작 간격·동시 요청 제한입니다. 모든 모델에서 공유하며 프로바이더·정확한 모델 상한과 함께 적용합니다. 중복 참조는 한 슬롯을 공유하지만 같은 업스트림 프로젝트의 다른 참조가 독립 할당량을 뜻하지는 않습니다. TPM·일일 한도는 처리하지 않습니다. | diff --git a/docs-site/src/content/docs/reference/configuration/account-auto-pause.md b/docs-site/src/content/docs/reference/configuration/account-auto-pause.md new file mode 100644 index 00000000000..c3f95a3b2fa --- /dev/null +++ b/docs-site/src/content/docs/reference/configuration/account-auto-pause.md @@ -0,0 +1,43 @@ +--- +title: Account Auto-pause +description: Opt-in automatic pausing of accounts after exact permanent refusals. +--- + +## Auto-pause permanently refused accounts + +`providers..accountAutoPause` is opt-in and defaults to off. It applies to that +provider's OAuth accounts, Codex pool accounts (including the main account in pool mode), +and configured API-key pool entries. For example: + +```sh +ocx config set providers.anthropic.accountAutoPause.enabled true +ocx config set providers.anthropic.accountAutoPause.consecutiveFailures 2 +``` + +`consecutiveFailures` is an integer from 1 to 100; omitting it uses 2. Only exact, +account-scoped HTTP 401/403 refusals qualify: an organization forbidding OAuth, an inactive +or expired subscription, an account denied API/Claude Code access or disabled by Google for a Terms of Service violation, or a downgrade to a free +plan with no credits. A model permission error, expired access token, quota window, 429, +5xx, malformed response or network failure never pauses an account. + +The counter is process-local, serialized per account, and resets on a successful HTTP +response, explicit resume or a new login. Token refresh alone does not reset an OAuth +account's streak. Transient errors leave the counter unchanged. Restarting the proxy clears +counters; completed pauses, their fixed reason codes and timestamps persist. Independent +proxy processes maintain independent counters. Changing, disabling or re-enabling the policy, disabling the provider, +or changing its endpoint discards in-flight refusal observations, so a refusal from a request +sent under the earlier policy can never pause an account afterwards. A refusal counts only for the credential that actually +reached the upstream: when a local plugin rewrites the credential or destination of a send, that +send never counts toward a pause. For Codex pool accounts this includes the workspace +(`chatgpt-account-id`) the request was sent for. + +At the threshold, OpenCodex excludes the account using its existing pause mechanism and +attempts an eligible sibling for the current request within the existing retry/send budget. +Fixed Codex account selectors and account-bound uploaded files retain their restrictions. +A pause never cancels already dispatched work or changes credentials. It emits one generic +log line without account identifiers or upstream error text. + +`ocx account list ` shows the pause reason. The dashboard's Paused badge exposes +the reason and time. Use `ocx account resume ` to clear the +reason and allow selection again. On an API-key row, clicking Resume also activates the key. +This policy never resumes an account automatically. diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 16853705892..72936264550 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -330,6 +330,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `apiKeyTransport?` | `"x-api-key" \| "bearer"` | Anthropic key header style. Defaults to native `x-api-key`; valid only for key-auth `anthropic` providers. | | `apiKeyPool?` | `ApiKeyPoolEntry[]` | Multi-key pool. `apiKey` mirrors the active entry; each item has `id`, `key`, optional `label`, and optional numeric `addedAt`. | | `apiKeyPoolStrategy?` | `"round-robin" \| "fill-first" \| "quota"` | How a warm key is chosen **before** the first attempt when the committed key is already cooling. Omitted keeps rotation reactive-only: the pool moves after a 429 or 401 and not before. `round-robin` takes the next key in the pool, `fill-first` keeps the first eligible one, and `quota` prefers the key with the most remaining headroom, falling back to `fill-first` order for a provider whose per-key quota is unknown. A healthy committed key is never overridden, so a manual key selection stands. | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | Prefers the key-pool entry that served a conversation's last successful turn, to reuse its upstream prompt cache. Applies with 2+ key-pool entries, independently of `apiKeyPoolStrategy`; OAuth/forward routes are excluded. Bindings match the exact provider and routed model: a different model does not inherit the binding. Disabled, removed, cooling, or pacing-saturated entries are not preferred, and 429/401 rotation still proceeds. With round-robin request pacing, recovery stays request-local and does not change the saved active key. Logs report `released` only when the matching binding was removed. Proxy-generated refusals preserve a healthy binding; an upstream refusal releases only the binding matching the serving provider and pool entry. `auto` (default) keeps a warm binding with ≥1024 tokens read or written, **or a detected tool-result continuation even below that threshold**. Warmth lasts 5 minutes, or 1 hour after a reported long-retention write. When no cache-write counter is reported, the sent prompt counts as written; an explicit zero is respected. This also pins providers that do not cache, so use `off` where load spreading matters more. `session` ignores warmth and token thresholds while the entry stays eligible; `turn` prefers only warm tool-result continuations. **Every binding expires 24 hours after its last recorded success**, and restart forgets all bindings. Changing the active key manually does not supersede a live binding; use `off` or disable its pool entry to override affinity. Unrelated dashboard provider saves preserve an omitted mode, including `off`. Log rows expose `cacheAffinity`/`cacheAffinityReason`. | | `defaultModel?` | `string` | Model used when this provider is selected without an explicit model. | | `models?` | `string[]` | Seed/fallback model list. With `liveModels: false`, a nonempty `models` list is followed by `retainModels`; an empty or omitted `models` list instead seeds `defaultModel` (if configured), then `retainModels`, removing duplicate ids in first-seen order. | | `liveModels?` | `boolean` | Fetch the live catalog on start/sync (default `true`). Custom providers use `${baseUrl}/models`; built-ins may use a registry URL and filter. | @@ -1927,39 +1928,4 @@ from the tool-call cache for those turns. Preserve their supplied reasoning for ## Auto-pause permanently refused accounts -`providers..accountAutoPause` is opt-in and defaults to off. It applies to that -provider's OAuth accounts, Codex pool accounts (including the main account in pool mode), -and configured API-key pool entries. For example: - -```sh -ocx config set providers.anthropic.accountAutoPause.enabled true -ocx config set providers.anthropic.accountAutoPause.consecutiveFailures 2 -``` - -`consecutiveFailures` is an integer from 1 to 100; omitting it uses 2. Only exact, -account-scoped HTTP 401/403 refusals qualify: an organization forbidding OAuth, an inactive -or expired subscription, an account denied API/Claude Code access or disabled by Google for a Terms of Service violation, or a downgrade to a free -plan with no credits. A model permission error, expired access token, quota window, 429, -5xx, malformed response or network failure never pauses an account. - -The counter is process-local, serialized per account, and resets on a successful HTTP -response, explicit resume or a new login. Token refresh alone does not reset an OAuth -account's streak. Transient errors leave the counter unchanged. Restarting the proxy clears -counters; completed pauses, their fixed reason codes and timestamps persist. Independent -proxy processes maintain independent counters. Changing, disabling or re-enabling the policy, disabling the provider, -or changing its endpoint discards in-flight refusal observations, so a refusal from a request -sent under the earlier policy can never pause an account afterwards. A refusal counts only for the credential that actually -reached the upstream: when a local plugin rewrites the credential or destination of a send, that -send never counts toward a pause. For Codex pool accounts this includes the workspace -(`chatgpt-account-id`) the request was sent for. - -At the threshold, OpenCodex excludes the account using its existing pause mechanism and -attempts an eligible sibling for the current request within the existing retry/send budget. -Fixed Codex account selectors and account-bound uploaded files retain their restrictions. -A pause never cancels already dispatched work or changes credentials. It emits one generic -log line without account identifiers or upstream error text. - -`ocx account list ` shows the pause reason. The dashboard's Paused badge exposes -the reason and time. Use `ocx account resume ` to clear the -reason and allow selection again. On an API-key row, clicking Resume also activates the key. -This policy never resumes an account automatically. +See [Account auto-pause](/reference/configuration/account-auto-pause/) for configuration, qualifying refusals, and recovery. diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 527f8f4761b..11b4cdbc41d 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -115,6 +115,7 @@ cross-route credential fallback не существует. Строки API GPT- | --- | --- | --- | | `adapter` | `string` | Один из `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (или alias `azure`). | | `baseUrl` | `string` | Базовый URL API upstream'а. Большинство built-in fixed-endpoint'ов игнорируют несовпадение; collision-safe key-preset'ы сохраняют старый custom destination с тем же именем. | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | Предпочитает ключ последнего успешного хода при наличии не менее двух ключей; OAuth/forward исключены. По умолчанию `auto`. Локальные отказы сохраняют привязку; отказ upstream снимает только привязку обслужившего поставщика и записи. См. [полный контракт на английском](/reference/configuration/providers/#provider-entries-ocxproviderconfig). | | `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. Для адаптеров `runTurn`, включая Cursor, лимит считает активные ходы, а не физические отправки: RunSSE и BidiAppend могут пересекаться внутри одного хода, пока следующий ход ждёт. Последующие отправки по-прежнему соблюдают интервалы запуска. | | `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Выбор ключа для обычных запросов: `active` по умолчанию сохраняет выбранный ключ и переключение при ошибке; `round-robin` последовательно распределяет запросы между доступными ключами без изменения сохранённого выбора. Балансируются назначения, не токены или стоимость. | | `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Общие для всех моделей ограничения RPM, интервала и параллелизма на ссылку ключа, вместе с лимитами провайдера и модели. Одинаковые ссылки делят ёмкость; разные ссылки одного проекта не означают независимые квоты. TPM и дневные квоты не контролируются. | diff --git a/docs-site/src/content/docs/tr/reference/configuration/providers.md b/docs-site/src/content/docs/tr/reference/configuration/providers.md index 12f2004ed8b..e668c8b0d99 100644 --- a/docs-site/src/content/docs/tr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/tr/reference/configuration/providers.md @@ -117,6 +117,7 @@ alanlı seçilmiş kimlikleri yalın kimliklere yeniden yazar. | Alan | Tip | Anlamı | | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (veya takma ad `azure`) seçeneklerinden biri. | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | En az iki anahtarlı havuzlarda son başarılı turu sunan anahtarı tercih eder; OAuth/forward kapsam dışıdır. Varsayılan `auto`. Yerel retler bağı korur; upstream reddi yalnızca hizmet veren sağlayıcı ve kayıtla eşleşen bağı kaldırır. [İngilizce tam sözleşmeye](/reference/configuration/providers/#provider-entries-ocxproviderconfig) bakın. | | `baseUrl` | `string` | Yukarı akış API temel URL'si. Çoğu yerleşik sabit uç nokta uyumsuzluğu yok sayar; çakışma güvenli anahtar önayarları aynı adlı daha eski özel bir hedefi korur. | | `responsesPath?` | `string` | Anahtar kimlik doğrulamalı `openai-responses` istekleri için göreli kaynak yolu. `/` ile başlamalı ve şema, sorgu veya parça içermemelidir. | | `chatCompletionsPath?` | `string` | `openai-chat` istekleri için göreli kaynak yolu; `responsesPath`'in aynasıdır ve aynı şekil kurallarına tabidir. Bir upstream Chat Completions ve Responses'u farklı öneklerde sunduğunda gereklidir: model başına wire override adaptörü değiştirir ve `baseUrl`'e dokunmaz, bu yüzden bu ayar olmadan etkin bir Chat isteği Responses base'e gönderilir. Gönderilen örnek Z.AI'dir. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index bc201ce0b73..6e3560a7b12 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -101,6 +101,7 @@ selector，而不是分配一个新名称。 | 字段 | 类型 | 含义 | | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或别名 `azure`）之一。 | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | 优先使用处理上一次成功轮次的密钥，适用于至少有两个条目的密钥池，不包括 OAuth/forward。默认值为 `auto`。本地拒绝保留绑定；上游拒绝仅释放与实际提供服务的供应商和条目匹配的绑定。参阅[完整英文约定](/reference/configuration/providers/#provider-entries-ocxproviderconfig)。 | | `baseUrl` | `string` | 上游 API 基础 URL。大多数内置固定端点会忽略不匹配的值；具备冲突安全键的预设会保留一个更早、同名的自定义目标。 | | `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 对包括 Cursor 在内的 `runTurn` 适配器，并发上限计算进行中的回合数，而非物理发送数：同一回合内的 RunSSE 和 BidiAppend 可以重叠，其他回合仍须等待。后续发送仍遵守启动间隔。 | | `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 正常请求的密钥选择：默认 `active` 保留已保存密钥和现有错误切换；`round-robin` 按顺序分配可用池密钥，不修改已保存的选择。均衡的是分配次数，不是令牌或费用。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 5a430f66c07..4fea32ef983 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -81,6 +81,7 @@ ocx models provider openrouter on | 欄位 | 型別 | 意義 | | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或別名 `azure`）之一。 | +| `cacheAffinity?` | `"auto" \| "session" \| "turn" \| "off"` | 優先使用處理上一次成功回合的金鑰，適用於至少有兩個項目的金鑰池，不包括 OAuth/forward。預設值為 `auto`。本機拒絕保留綁定；上游拒絕只釋放與實際提供服務的供應商和項目相符的綁定。請參閱[完整英文契約](/reference/configuration/providers/#provider-entries-ocxproviderconfig)。 | | `baseUrl` | `string` | 上游 API base URL。多數內建固定端點忽略不符；碰撞安全的金鑰預設保留較舊的同名自訂目的地。 | | `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 對包含 Cursor 的 `runTurn` 適配器，並行上限計算進行中的回合數，而非實體傳送數：同一回合內的 RunSSE 與 BidiAppend 可重疊，其他回合仍須等待。後續傳送仍遵守啟動間隔。 | | `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 正常請求的金鑰選擇：預設 `active` 保留已儲存金鑰及既有錯誤切換；`round-robin` 依序分配可用池金鑰，不修改已儲存選擇。均衡的是分配次數，而非權杖或費用。 | diff --git a/readme/README.fr.md b/readme/README.fr.md index 7613afe8b65..cf119d4762e 100644 --- a/readme/README.fr.md +++ b/readme/README.fr.md @@ -330,7 +330,7 @@ OpenCodex suit l'état conservé par le processus dans les catégories ci-dessou rejeu des contrôles natifs, épinglé et jamais évincé. - **4 tampons observés** (accumulateurs de traduction, segments finaux d'images/OAuth/Grok) sont surveillés pour détecter la pression des octets en cours de traitement, sans éviction. -- **31 enregistrements de stockages d'état** gèrent les balayages d'expiration (intervalle de 60 s) et la +- **32 enregistrements de stockages d'état** gèrent les balayages d'expiration (intervalle de 60 s) et la réconciliation des générations de configuration afin de supprimer les clés obsolètes des fournisseurs et des comptes. - **Les mémos de chemins et d'empreintes** (métadonnées de l'espace de travail, identités renforcées, sels d'installation, capacités indiquées par le mode) utilisent des limites LRU selon l'ordre d'insertion (8 à 128 entrées). diff --git a/readme/README.ja.md b/readme/README.ja.md index b396a9d0fa1..2ae16a5d413 100644 --- a/readme/README.ja.md +++ b/readme/README.ja.md @@ -327,7 +327,7 @@ OpenCodex はプロセスが保持する状態を以下のカテゴリで追跡 退避されません。 - **観測バッファ 4 個**（トランスレーターのアキュムレーター、画像・OAuth・Grok の tail）は処理中の バイト圧力を監視するだけで、退避はしません。 -- **state-store の登録 31 個**が期限切れの掃除（60 秒間隔）と config 世代の reconciliation を担い、 +- **state-store の登録 32 個**が期限切れの掃除（60 秒間隔）と config 世代の reconciliation を担い、 古いプロバイダー／アカウントのキーを取り除きます。 - **パスとフィンガープリントのメモ**（ワークスペースのメタデータ、hardened identity、インストール salt、mode-hint の capability）は挿入順の LRU 上限（8〜128 件）を使います。 diff --git a/readme/README.ko.md b/readme/README.ko.md index 5a3530c674e..41d809a2ffa 100644 --- a/readme/README.ko.md +++ b/readme/README.ko.md @@ -316,7 +316,7 @@ OpenCodex는 프로세스가 붙잡고 있는 상태를 아래 항목에서 추 eviction합니다. 단, native control replay 저장소는 고정되어 eviction되지 않습니다. - **관측 버퍼 4개**(translator accumulator, image/OAuth/Grok tail)는 진행 중 바이트 압력을 감시만 하고 eviction하지 않습니다. -- **state-store 등록 31개**는 만료 sweep(60초 간격)과 config-generation reconciliation을 돌려, +- **state-store 등록 32개**는 만료 sweep(60초 간격)과 config-generation reconciliation을 돌려, 낡은 프로바이더/계정 키를 지웁니다. - **경로·fingerprint 메모**(워크스페이스 메타데이터, hardened identity, 설치 salt, mode-hint capability)는 삽입 순서 LRU cap(8–128개)을 씁니다. diff --git a/readme/README.ru.md b/readme/README.ru.md index 0b8428934ce..bda953ce870 100644 --- a/readme/README.ru.md +++ b/readme/README.ru.md @@ -336,7 +336,7 @@ OpenCodex отслеживает состояние, удерживаемое п хранилища native control replay: оно закреплено и не вытесняется. - **4 наблюдаемых буфера** (аккумуляторы транслятора, хвосты image/OAuth/Grok) мониторятся по байтовому давлению in-flight без вытеснения. -- **31 регистрация state-store** выполняет sweeps истечения (интервал 60 с) и сверку +- **32 регистрации state-store** выполняет sweeps истечения (интервал 60 с) и сверку поколений конфигурации, чтобы удалять устаревшие ключи провайдеров и аккаунтов. - **Мемо пути и отпечатков** (метаданные рабочей области, усиленные идентификаторы, соли установки, возможности mode-hint) используют LRU-потолки в порядке вставки diff --git a/readme/README.tr.md b/readme/README.tr.md index dd1eda9f03b..a3ba75112fe 100644 --- a/readme/README.tr.md +++ b/readme/README.tr.md @@ -331,7 +331,7 @@ OpenCodex, süreçte tutulan durumu aşağıdaki kategorilerde izler. Her birini sabitlenmiştir ve hiç tahliye edilmez. - **4 gözlenen arabellek** (çevirici biriktiricileri, görsel/OAuth/Grok kuyrukları) tahliye edilmeden, yalnızca uçuştaki bayt baskısı için izlenir. -- **31 state-store kaydı**, süre dolumu taramalarını (60 sn aralık) ve yapılandırma kuşağı uzlaştırmasını +- **32 state-store kaydı**, süre dolumu taramalarını (60 sn aralık) ve yapılandırma kuşağı uzlaştırmasını yürüterek eski sağlayıcı/hesap anahtarlarını kaldırır. - **Yol ve parmak izi notları** (çalışma alanı meta verileri, sağlamlaştırılmış kimlikler, kurulum tuzları, mod ipucu yetenekleri) ekleme sıralı LRU sınırları kullanır (8–128 girdi). diff --git a/readme/README.zh-CN.md b/readme/README.zh-CN.md index a56eac15ba2..5c28279900b 100644 --- a/readme/README.zh-CN.md +++ b/readme/README.zh-CN.md @@ -317,7 +317,7 @@ OpenCodex 在下列类别中跟踪进程保留状态。每一类都有文档化 不会被逐出。 - **4 个观测缓冲区**（翻译累加器、图片/OAuth/Grok 尾部）会监测飞行中的字节压力， 但不做逐出。 -- **31 个状态存储注册** 负责过期扫描（60 秒间隔）和配置世代对账，从而移除过期的 +- **32 个状态存储注册** 负责过期扫描（60 秒间隔）和配置世代对账，从而移除过期的 提供商/账户键。 - **路径与指纹备忘**（工作区元数据、加固身份、安装盐、模式提示能力）使用按插入顺序的 LRU 上限（8–128 条）。 diff --git a/readme/README.zh-TW.md b/readme/README.zh-TW.md index 78cacab3696..e9904b67d31 100644 --- a/readme/README.zh-TW.md +++ b/readme/README.zh-TW.md @@ -315,7 +315,7 @@ OpenCodex 在下列類別中追蹤行程保留狀態。每一類都有文件化 儲存為固定，不會被淘汰。 - **4 個觀測緩衝區**（翻譯累加器、image/OAuth/Grok 尾端）會 監控進行中的位元組壓力，但不淘汰。 -- **31 個狀態儲存註冊**負責到期清掃（間隔 60 秒）與 +- **32 個狀態儲存註冊**負責到期清掃（間隔 60 秒）與 設定世代調和，以移除過期的供應商／帳號鍵。 - **路徑與指紋 memo**（工作區中繼資料、強化身分、安裝 salt、mode-hint 能力）使用插入順序 LRU 上限（8–128 筆）。 diff --git a/readme/i18n-manifest.json b/readme/i18n-manifest.json index 80ed6b2aa0c..43d72da3dd6 100644 --- a/readme/i18n-manifest.json +++ b/readme/i18n-manifest.json @@ -6,43 +6,43 @@ "file": "readme/README.fr.md", "label": "Français", "docsPath": "fr", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" }, "ko": { "file": "readme/README.ko.md", "label": "한국어", "docsPath": "ko", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" }, "zh-CN": { "file": "readme/README.zh-CN.md", "label": "简体中文", "docsPath": "zh-cn", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" }, "zh-TW": { "file": "readme/README.zh-TW.md", "label": "繁體中文", "docsPath": "zh-tw", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" }, "ru": { "file": "readme/README.ru.md", "label": "Русский", "docsPath": "ru", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" }, "ja": { "file": "readme/README.ja.md", "label": "日本語", "docsPath": "ja", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" }, "tr": { "file": "readme/README.tr.md", "label": "Türkçe", "docsPath": "tr", - "sourceSha256": "f0f17270033cd0af45d92506a6386420b7e509eac18457dac050b8f689789230" + "sourceSha256": "a3297f100495a6a620ebee8b709c6c811abbd23c23b682c1701053b876ba19d1" } } } diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index f4e44633c98..cc15c506bb3 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -251,7 +251,7 @@ "api-key-scope-audio.test.ts": "server", "api-key-scope-images.test.ts": "server", "api-key-scope-live.test.ts": "server", - "api-key-selection-capture.test.ts": "providers", + "api-key-selection-capture.test.ts": "providers", "cache-affinity.test.ts": "providers", "api-keys-routes.test.ts": "server", "api-storage-cleanup.test.ts": "storage", "api-storage-policy-already-running.test.ts": "storage", @@ -1669,7 +1669,7 @@ "server-images-bodyless-content-length.test.ts": "server", "server-images-pool-admission.test.ts": "server", "server-images.test.ts": "server", - "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", + "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", "cache-affinity-e2e.test.ts": "server", "server-jev-combo-e2e.test.ts": "server", "combo-requested-effort.test.ts": "server", "server-kiro-completion-e2e.test.ts": "server", "server-kiro-device-login.test.ts": "server", "server-kiro-oauth-401-replay.test.ts": "server", diff --git a/src/config/schema/leaf-validators.ts b/src/config/schema/leaf-validators.ts index 3eacf076d56..368e9507e28 100644 --- a/src/config/schema/leaf-validators.ts +++ b/src/config/schema/leaf-validators.ts @@ -325,6 +325,9 @@ export const providerConfigSchema = z.object({ // load silently and then be ignored at selection time, which reads as a broken feature // rather than a rejected setting. apiKeyPoolStrategy: z.enum(["round-robin", "fill-first", "quota"]).optional(), + // Same reasoning as apiKeyPoolStrategy: an unrecognized mode must be rejected + // at load, not silently ignored by credential selection. + cacheAffinity: z.enum(["auto", "session", "turn", "off"]).optional(), autoReviewModel: autoReviewModelSchema.optional(), autoReviewModelOverrides: autoReviewModelOverridesSchema.optional(), adapter: z.string().min(1), diff --git a/src/lib/state-store-registrations.ts b/src/lib/state-store-registrations.ts index 22973b92f58..dcb34b35006 100644 --- a/src/lib/state-store-registrations.ts +++ b/src/lib/state-store-registrations.ts @@ -24,6 +24,7 @@ import { import { reconcileComboWarningMemos } from "../combos/request"; import { reconcileComboRotationState } from "../combos/resolve"; import { reconcileComboRecall, sweepExpiredComboRecall } from "../server/responses/combo-session-recall"; +import { reconcileProviderCacheAffinity, sweepExpiredProviderCacheAffinity } from "../providers/cache-affinity"; import { listLiveComboTargetKeys } from "../combos/types"; import { listLiveConfigOwnershipRoots, @@ -107,6 +108,11 @@ export function buildGenerationContext(): GenerationContext { return { generation: 0, providerNames, + disabledProviderNames: new Set(Object.entries(liveServerConfig.providers) + .filter(([, provider]) => provider.disabled === true) + .map(([name]) => name)), + providerKeyEntryIds: new Map(Object.entries(liveServerConfig.providers) + .map(([name, provider]) => [name, new Set((provider.apiKeyPool ?? []).map(entry => entry.id))] as const)), comboIds: new Set(Object.keys(liveServerConfig.combos ?? {})), comboTargets: listLiveComboTargetKeys(liveServerConfig), codexAccountIds: listLiveCodexAccountIds(liveServerConfig), @@ -161,6 +167,11 @@ export const STATE_STORE_REGISTRATIONS = [ sweepExpired: sweepExpiredComboRecall, reconcileGeneration: reconcileComboRecall, }, + { + name: "provider-cache-affinity", + sweepExpired: sweepExpiredProviderCacheAffinity, + reconcileGeneration: reconcileProviderCacheAffinity, + }, { name: "guardian-backoff", reconcileGeneration: reconcileGuardianBackoff }, { name: "codex-reauth", reconcileGeneration: reconcileCodexReauthState }, { name: "oauth-reauth", reconcileGeneration: reconcileOAuthReauthState }, diff --git a/src/lib/state-store-sweeper.ts b/src/lib/state-store-sweeper.ts index 2c99a48fe19..f6f76ff1625 100644 --- a/src/lib/state-store-sweeper.ts +++ b/src/lib/state-store-sweeper.ts @@ -3,6 +3,10 @@ export const STATE_SWEEP_INTERVAL_MS = 60_000; export interface GenerationContext { generation: number; providerNames: ReadonlySet; + /** Providers disabled in the live configuration; stores keyed by provider may drop their rows. */ + disabledProviderNames?: ReadonlySet; + /** Live key-pool entry ids per provider, for stores that bind to one entry. */ + providerKeyEntryIds?: ReadonlyMap>; comboIds: ReadonlySet; comboTargets: ReadonlySet; codexAccountIds: ReadonlySet; diff --git a/src/providers/api-key-selection.ts b/src/providers/api-key-selection.ts index 550f5420867..618e5d4f410 100644 --- a/src/providers/api-key-selection.ts +++ b/src/providers/api-key-selection.ts @@ -13,12 +13,25 @@ import { captureProviderApiKeySelection } from "./api-key-selection-capture"; export { captureProviderApiKeySelection } from "./api-key-selection-capture"; function matchesSelection(provider: OcxProviderConfig, expected: ProviderApiKeySelection): boolean { - if (expected.distributed) return provider.apiKeySelectionRevision === expected.revision + if (expected.distributed && !expected.cacheAffinity) return provider.apiKeySelectionRevision === expected.revision && provider.requestPacing?.enabled === true && provider.requestPacing.keySelection === "round-robin" && !!provider.apiKeyPool?.some(entry => !entry.disabled && entry.id === expected.entryId && entry.key === expected.reference); + if (expected.cacheAffinity === true) { + // Affinity picks are fenced on entry liveness, not the selection revision. + // commitProviderApiKeySelection bumps the revision on EVERY committed key + // change — automatic rotation included — so a revision fence cannot tell + // an operator's re-selection from another conversation's rotation, and + // telling them apart is not affinity's job: the conversation keeps its + // credential while the entry lives, and the provider's cacheAffinity mode + // is the operator's override. (Ordinary captured selections keep the + // revision fence below: a permit for the previously committed key must + // not dispatch after a manual replacement.) + return !!expected.entryId && !!expected.reference + && (provider.apiKeyPool ?? []).some(entry => entry.id === expected.entryId && entry.key === expected.reference); + } const current = captureProviderApiKeySelection(provider); - return current.entryId === expected.entryId && current.reference === expected.reference - && current.revision === expected.revision; + if (current.revision !== expected.revision) return false; + return current.entryId === expected.entryId && current.reference === expected.reference; } function currentKeyProvider(config: OcxConfig, name: string, selection?: ProviderApiKeySelection, allowCooling = false): OcxProviderConfig | null { @@ -27,13 +40,26 @@ function currentKeyProvider(config: OcxConfig, name: string, selection?: Provide let row = configured; if (selection?.distributed) { const entry = configured.apiKeyPool?.find(key => key.id === selection.entryId && key.key === selection.reference); - if (!entry || entry.disabled || (!allowCooling && isKeyInCooldown(name, entry.id)) || configured.apiKeySelectionRevision !== selection.revision + if (!entry || entry.disabled || (!allowCooling && isKeyInCooldown(name, entry.id)) || (!selection.cacheAffinity && configured.apiKeySelectionRevision !== selection.revision) || !configured.requestPacing?.enabled || configured.requestPacing.keySelection !== "round-robin") return null; row = { ...configured, apiKey: entry.key, _apiKeyAttempt: selection }; } + else if (selection?.cacheAffinity && selection.entryId && selection.reference) { + // A request-captured pool selection (cache affinity picks one without + // committing it globally) survives this re-read while its entry is live: + // present with the captured key, enabled, and out of cooldown. It is + // deliberately NOT fenced on the selection revision — that revision bumps + // on every committed change, automatic rotation included, so it cannot + // supersede a pick whose whole purpose is to outlast committed movement. + // Anything else falls back to the committed row rather than refusing. + const entry = configured.apiKeyPool?.find(key => key.id === selection.entryId && key.key === selection.reference); + if (entry && !entry.disabled && (allowCooling || !isKeyInCooldown(name, entry.id))) { + row = { ...configured, apiKey: entry.key, _apiKeyAttempt: selection }; + } + } if (row.apiKeyPool?.some(entry => entry.disabled && entry.key === row.apiKey) || row.apiKeyPool?.length && row.apiKeyPool.every(entry => entry.disabled)) return null; - const current = routedProviderConfig(name, { ...row, _apiKeyAttempt: selection?.distributed ? selection : undefined }); + const current = routedProviderConfig(name, { ...row, _apiKeyAttempt: row === configured ? undefined : selection }); if (current.authMode === "oauth" || current.authMode === "forward") return null; if (current.authMode === "key" && !current.keyOptional && !current.apiKey?.trim()) return null; return current; diff --git a/src/providers/cache-affinity.ts b/src/providers/cache-affinity.ts new file mode 100644 index 00000000000..636a8618242 --- /dev/null +++ b/src/providers/cache-affinity.ts @@ -0,0 +1,237 @@ +/** + * Provider cache affinity: keep a conversation on the key-pool credential that + * served its last turn, so that credential's upstream prompt cache is read again + * instead of rebuilt on another key. + * + * Modeled on Magpie's gateway affinity design, expressed in this codebase's + * terms: conversations are session lanes (server/request-log-conversation), + * credentials are key-pool entries identified by their pool entry id (never key + * material), and outcomes are recorded from attempt finalization, where every + * adapter's cache usage is already normalized into cache-read / cache-creation + * token counts. + * + * Affinity is a preference inside eligibility, never an override: this module + * only answers "which entry served this lane last, and is that worth + * preferring?". Whether that entry may serve right now (disabled, cooling, + * pacing) is decided by the caller in key-failover, which owns those rules. + * + * The store is process-local, following the combo session-recall precedent: + * a restart forgets every binding and the first post-restart turn of a + * conversation is placed by ordinary selection again. + * + * Lane ids are client-supplied, so a client that churns them can fill the + * store and evict other conversations' bindings. The blast radius is lost + * cache locality only — no correctness or credential exposure — and the + * count/byte bounds below are the mitigation. + */ +import type { CacheAffinityMode, OcxProviderConfig } from "../types"; +import type { GenerationContext } from "../lib/state-store-sweeper"; + +/** Tokens read from or written to an upstream cache that make a binding worth keeping across turns. */ +export const CACHE_AFFINITY_WORTH_TOKENS = 1024; +/** How long an ordinary (5-minute-retention) cache stays warm after the turn that built it. */ +export const CACHE_AFFINITY_COLD_MS = 5 * 60_000; +/** How long a long-retention (1-hour) cache write stays warm. */ +export const CACHE_AFFINITY_COLD_LONG_MS = 60 * 60_000; +/** Hard expiry for a binding, however warm: past this the conversation is placed fresh. */ +export const CACHE_AFFINITY_TTL_MS = 24 * 60 * 60_000; +const CAPACITY = 512; +/** Per-entry byte cap over the retained strings, mirroring combo session recall. */ +const ENTRY_BYTES_MAX = 1024; +/** Aggregate byte cap across every retained binding. */ +const TOTAL_BYTES_MAX = 64 * 1024; + +export type CacheAffinityReason = + | "cache" // kept: measured cache worth keeping and still warm (auto) + | "session" // kept: session mode prefers the bound credential while eligible + | "turn" // kept: turn mode, detected continuation of the in-progress turn + | "continuation"; // kept: detected continuation of the in-progress turn (auto) + +export interface CacheAffinityHit { + entryId: string; + reason: CacheAffinityReason; +} + +interface Binding { + providerName: string; + entryId: string; + model: string; + cacheRead: number; + cacheWrite: number; + longRetention: boolean; + at: number; +} + +const bindings = new Map(); +let lastReconciledGeneration = 0; +let bindingsBytes = 0; + +function bindingBytes(lane: string, binding: Binding): number { + return Buffer.byteLength(lane) + Buffer.byteLength(binding.providerName) + + Buffer.byteLength(binding.entryId) + Buffer.byteLength(binding.model); +} + +function deleteBinding(lane: string): void { + const binding = bindings.get(lane); + if (!binding) return; + bindings.delete(lane); + bindingsBytes -= bindingBytes(lane, binding); +} + +export function cacheAffinityMode(provider: Pick): CacheAffinityMode { + return provider.cacheAffinity ?? "auto"; +} + +export function cacheAffinityApplies(provider: OcxProviderConfig): boolean { + if (cacheAffinityMode(provider) === "off") return false; + if (provider.authMode === "oauth" || provider.authMode === "forward") return false; + return (provider.apiKeyPool?.length ?? 0) >= 2; +} + +export interface ConsultProviderCacheAffinityInput { + lane: string | undefined; + providerName: string; + provider: OcxProviderConfig; + model: string; + /** The request continues the lane's in-progress turn (tool results being returned). */ + continuation?: boolean; + now?: number; +} + +/** + * The bound credential for this lane, when affinity says to prefer it. + * Purely a preference: the caller must still validate the entry against the + * live pool (present, enabled, not cooling) before sending with it. + */ +export function consultProviderCacheAffinity(input: ConsultProviderCacheAffinityInput): CacheAffinityHit | null { + const { lane, providerName, provider, model } = input; + if (!lane || !cacheAffinityApplies(provider)) return null; + const binding = bindings.get(lane); + if (!binding) return null; + const now = input.now ?? Date.now(); + if (now - binding.at >= CACHE_AFFINITY_TTL_MS) { + deleteBinding(lane); + return null; + } + if (binding.providerName !== providerName || binding.model !== model) return null; + const mode = cacheAffinityMode(provider); + const age = now - binding.at; + const warmMs = binding.longRetention ? CACHE_AFFINITY_COLD_LONG_MS : CACHE_AFFINITY_COLD_MS; + if (mode === "session") return { entryId: binding.entryId, reason: "session" }; + if (input.continuation === true) { + if (age > warmMs) return null; + return { entryId: binding.entryId, reason: mode === "turn" ? "turn" : "continuation" }; + } + if (mode === "turn") return null; + // auto, across turns: only measured-worth bindings that are still warm. + if (age > warmMs) return null; + const worth = binding.cacheRead + binding.cacheWrite; + return worth >= CACHE_AFFINITY_WORTH_TOKENS ? { entryId: binding.entryId, reason: "cache" } : null; +} + +export interface RecordProviderCacheAffinityInput { + lane: string | undefined; + providerName: string; + entryId: string | undefined; + model: string; + cacheRead?: number; + cacheWrite?: number; + /** Portion of the cache write kept at long (1-hour) retention, when the adapter reports it. */ + cacheWriteLong?: number; + inputTokens?: number; + now?: number; +} + +/** + * Remember who served a completed turn. A turn with no cache telemetry at all + * counts its sent prompt as written (silent-cache vendors such as Ollama + * Cloud): the prompt the vendor cached is the prompt that was sent. + */ +export function recordProviderCacheAffinity(input: RecordProviderCacheAffinityInput): void { + const { lane, providerName, entryId, model } = input; + if (!lane || !entryId) return; + const cacheRead = input.cacheRead ?? 0; + // No write counter reported — reads-only reporters (OpenAI Chat reports + // reads, never writes) and fully silent vendors alike: the prompt that was + // sent is what the vendor cached, so it counts as written. An explicitly + // reported write count, zero included, is believed as-is. + const cacheWrite = input.cacheWrite ?? input.inputTokens ?? 0; + const binding: Binding = { + providerName, + entryId, + model, + cacheRead, + cacheWrite, + longRetention: (input.cacheWriteLong ?? 0) > 0, + at: input.now ?? Date.now(), + }; + if (bindingBytes(lane, binding) > ENTRY_BYTES_MAX) return; + // Re-insert so eviction drops the least recently recorded lane first. + deleteBinding(lane); + bindings.set(lane, binding); + bindingsBytes += bindingBytes(lane, binding); + while (bindings.size > CAPACITY || bindingsBytes > TOTAL_BYTES_MAX) { + const oldest = bindings.keys().next().value; + if (oldest === undefined || oldest === lane) break; + deleteBinding(oldest); + } +} + +export function releaseProviderCacheAffinity( + lane: string | undefined, + providerName?: string, + entryId?: string, +): boolean { + if (!lane) return false; + const binding = bindings.get(lane); + if (!binding) return false; + if ((providerName !== undefined && binding.providerName !== providerName) + || (entryId !== undefined && binding.entryId !== entryId)) return false; + deleteBinding(lane); + return true; +} + +/** The entry a lane is currently bound to, if any (finalization log attribution). */ +export function boundEntryIdForLane(lane: string | undefined): string | undefined { + return lane ? bindings.get(lane)?.entryId : undefined; +} + +/** Periodic expiry for the state-store sweeper; expiry is also evaluated on consult. */ +export function sweepExpiredProviderCacheAffinity(now: number): number { + let removed = 0; + for (const [lane, binding] of bindings) { + if (now - binding.at >= CACHE_AFFINITY_TTL_MS) { + deleteBinding(lane); + removed += 1; + } + } + return removed; +} + +/** + * Drop bindings whose provider left the live configuration, was disabled, or + * lost the bound credential, on a configuration generation change. Consult + * re-validates against the live pool on every request regardless; this keeps + * unconsulted lanes from lingering. + */ +export function reconcileProviderCacheAffinity(context: GenerationContext): number { + if (context.generation <= lastReconciledGeneration) return 0; + lastReconciledGeneration = context.generation; + let removed = 0; + for (const [lane, binding] of bindings) { + const entryIds = context.providerKeyEntryIds?.get(binding.providerName); + if (!context.providerNames.has(binding.providerName) + || context.disabledProviderNames?.has(binding.providerName) + || (entryIds !== undefined && !entryIds.has(binding.entryId))) { + deleteBinding(lane); + removed += 1; + } + } + return removed; +} + +export function clearProviderCacheAffinityForTests(): void { + bindings.clear(); + bindingsBytes = 0; + lastReconciledGeneration = 0; +} diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index c3965c23977..615faf6e7ba 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -9,9 +9,13 @@ * Modelled after src/codex/routing.ts cooldown logic but scoped to plain API-key pools. */ import { isKeyInCooldown, getKeyCooldownUntil, setKeyCooldown } from "./api-key-cooldowns"; -export { clearKeyCooldowns, getKeyCooldownUntil, sweepExpiredApiKeyCooldowns } from "./api-key-cooldowns"; +export { + clearKeyCooldowns, + getKeyCooldownUntil, + sweepExpiredApiKeyCooldowns, +} from "./api-key-cooldowns"; import { commitProviderApiKeySelection, providerApiKeySelectionIsCurrent } from "./api-key-selection"; -import { RequestPacingQueueOverloadError } from "./request-pacing"; +import { providerRequestPacingStatus, REQUEST_PACING_MAX_QUEUE_AGE_MS, RequestPacingQueueOverloadError } from "./request-pacing"; import { isCanonicalMistralChat } from "./mistral-transport"; import type { ProviderApiKeySelection } from "../types/provider"; import { routedProviderConfig } from "./routed-config"; @@ -21,6 +25,7 @@ import type { OcxConfig, OcxProviderConfig, RateLimitRetryPolicy, ResetReplayPol import { OPENCODE_GO_SESSION_HEADER } from "./opencode-go-transport"; import { resolveProviderTransport, type OcxProviderTransport } from "./xai-transport"; import { sweepExpiredOnWrite } from "../lib/state-store-sweeper"; +import { cacheAffinityApplies, consultProviderCacheAffinity, releaseProviderCacheAffinity } from "./cache-affinity"; // quota-key-accounts imports only node:crypto, the key store and the quota types -- NOT // providers/quota.ts -- so the cached reader reaches the dispatch path without dragging the // probe machinery onto it. @@ -459,7 +464,6 @@ function rankKeysByHeadroom( .map(row => row.entry); } - /** * Pick a better key BEFORE the first attempt when the committed one is already cooling. * @@ -537,6 +541,49 @@ export function selectProactiveApiKey( return structuredClone(committed); } +/** Conversation context for the cache-affinity preference inside proactive selection. */ +export interface ProactiveKeyAffinityContext { + /** Explicit session lane of the request; absent lanes never bind or consult. */ + lane?: string; + model?: string; + /** The request continues the lane's in-progress turn (tool results being returned). */ + continuation?: boolean; + /** Decision sink (the request log context); only bounded enum strings are written. */ + log?: { + cacheAffinity?: string; + cacheAffinityReason?: string; + cacheAffinityApplicable?: boolean; + affinityModel?: string; + affinityKeptEntryId?: string; + }; +} + +/** + * Whether the bound entry is pacing-limited right now: its per-key + * concurrency bucket is full, or its next permitted start (minimum interval + * or per-minute cap, as the pacer computes it) is further out than the send + * queue tolerates before expiring a waiter. Below that bound, queueing is + * pacing working as designed — ordinary selection queues too — so affinity + * stands. Only the provider-level keys rule is consulted here; per-model + * caps still apply at the send queue exactly as they do for ordinary + * selection. + */ +function affinityPacingSaturated( + providerName: string, + configured: OcxProviderConfig, + entryId: string, + now: number, +): boolean { + const pacing = configured.requestPacing; + if (!pacing?.enabled) return false; + const key = providerRequestPacingStatus(providerName, configured, now).keys + ?.find(candidate => candidate.id === entryId); + if (!key) return false; + if (key.nextSlotInMs > REQUEST_PACING_MAX_QUEUE_AGE_MS) return true; + const cap = pacing.keys?.maxConcurrentRequests; + return cap !== undefined && cap > 0 && key.inFlight >= cap; +} + /** * Pre-dispatch twin of `rotateProviderTransportOn429`: pick a warm key, then rebuild the * active route from the committed row through the same seam the 429 path uses, so the @@ -552,11 +599,70 @@ export function selectProactiveApiKeyTransport( routedProvider: OcxProviderTransport, promptCacheKey?: string, now = Date.now(), + affinity?: ProactiveKeyAffinityContext, ): OcxProviderTransport | null { const configured = config.providers[providerName]; - if (configured?.requestPacing?.enabled && configured.requestPacing.keySelection === "round-robin" - && hasKeyPoolFailover(configured)) { - return applyRotatedTransport(providerName, routedProvider, allocateDistributedKey(providerName, configured, now), promptCacheKey); + // Cache affinity first: a conversation stays on the credential whose upstream + // prompt cache it built, while that credential is still eligible. The + // preference never commits a global selection — it rebuilds this request's + // transport only — and it yields to every eligibility rule the normal paths + // enforce (entry removed or disabled releases the binding; a cooling or + // pacing-saturated entry falls through to ordinary selection, and whoever + // ultimately serves is who the binding is refreshed to at finalization). + if (affinity?.lane && configured) { + if (affinity.log) { + affinity.log.cacheAffinityApplicable = cacheAffinityApplies(configured); + affinity.log.affinityModel = affinity.model; + } + const hit = consultProviderCacheAffinity({ + lane: affinity.lane, + providerName, + provider: configured, + model: affinity.model ?? "", + continuation: affinity.continuation === true, + now, + }); + if (hit) { + const entry = configured.apiKeyPool?.find((candidate) => candidate.id === hit.entryId); + if (!entry || entry.disabled) { + releaseProviderCacheAffinity(affinity.lane, providerName, hit.entryId); + } else if ( + !isKeyInCooldown(providerName, entry.id, now) && + !affinityPacingSaturated(providerName, configured, entry.id, now) + ) { + if (affinity.log) { + affinity.log.cacheAffinity = "kept"; + affinity.log.cacheAffinityReason = hit.reason; + affinity.log.affinityKeptEntryId = entry.id; + } + const affinityTransport = applyRotatedTransport( + providerName, + routedProvider, + { ...configured, apiKey: entry.key }, + promptCacheKey, + ); + // Mark the pick as request-local affinity so selection re-reads keep it + // while the entry stays live, without committing a global selection. + affinityTransport._apiKeyAttempt = { + ...affinityTransport._apiKeyAttempt!, + cacheAffinity: true, + distributed: configured.requestPacing?.enabled === true && configured.requestPacing.keySelection === "round-robin", + }; + return affinityTransport; + } + } + } + if ( + configured?.requestPacing?.enabled && + configured.requestPacing.keySelection === "round-robin" && + hasKeyPoolFailover(configured) + ) { + return applyRotatedTransport( + providerName, + routedProvider, + allocateDistributedKey(providerName, configured, now), + promptCacheKey, + ); } const committed = selectProactiveApiKey(config, providerName, now); if (!committed) return null; @@ -719,7 +825,7 @@ export function recordDistributedKeyFailure( const provider = config.providers[providerName]; if (!selection?.distributed || !provider || provider.authMode === "oauth" || provider.authMode === "forward") return false; const failed = provider.apiKeyPool?.find(entry => entry.id === selection.entryId && entry.key === selection.reference); - if (!failed || failed.disabled || provider.apiKeySelectionRevision !== selection.revision) return false; + if (!failed || failed.disabled || (!selection.cacheAffinity && provider.apiKeySelectionRevision !== selection.revision)) return false; const delay = status === "billing" ? ANTHROPIC_BILLING_COOLDOWN_MS : status === 401 ? MAX_COOLDOWN_MS : quotaResetAt !== undefined ? Math.max(1, quotaResetAt - now) : parseRetryAfterMs(retryAfter, now) ?? DEFAULT_COOLDOWN_MS; for (const entry of provider.apiKeyPool ?? []) if (entry.key === failed.key) { diff --git a/src/providers/label.ts b/src/providers/label.ts index caf98351669..22f2c68031c 100644 --- a/src/providers/label.ts +++ b/src/providers/label.ts @@ -14,14 +14,18 @@ export function usesApiKeyAccount(provider: Pick, ): void { if (usesApiKeyAccount(provider)) { logCtx.accountLogLabel = apiKeyAccountLogLabel(providerName, provider._apiKeyAttempt); - } else if (KEY_ACCOUNT_LOG_LABEL_RE.test(logCtx.accountLogLabel ?? "")) { - delete logCtx.accountLogLabel; + logCtx.affinityKeyEntryId = provider._apiKeyAttempt?.entryId; + logCtx.affinityProviderName = providerName; + } else { + if (KEY_ACCOUNT_LOG_LABEL_RE.test(logCtx.accountLogLabel ?? "")) delete logCtx.accountLogLabel; + delete logCtx.affinityKeyEntryId; + delete logCtx.affinityProviderName; } } diff --git a/src/providers/model-rename-fields.ts b/src/providers/model-rename-fields.ts index a97202d8bdc..f8c7d1045cc 100644 --- a/src/providers/model-rename-fields.ts +++ b/src/providers/model-rename-fields.ts @@ -130,6 +130,7 @@ export const PROVIDER_MODEL_RENAME_ROLES = { foldDeveloperRoleToSystem: "none", singleLeadingSystemMessage: "none", promptCacheKey: "none", + cacheAffinity: "none", chatServiceTier: "none", responsesItemIdRepair: "none", autoToolChoiceOnlyModels: "list", diff --git a/src/server/auth-cors.ts b/src/server/auth-cors.ts index 11059690ed2..aa975e91bd3 100644 --- a/src/server/auth-cors.ts +++ b/src/server/auth-cors.ts @@ -1197,6 +1197,7 @@ const PROVIDER_CONFIG_FIELD_POLICY = { foldDeveloperRoleToSystem: "editor", singleLeadingSystemMessage: "editor", promptCacheKey: "editor", + cacheAffinity: "editor", chatServiceTier: "editor", responsesItemIdRepair: "editor", autoToolChoiceOnlyModels: "editor", diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index ac3d861eb5e..fffd46c0cee 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -3,6 +3,7 @@ import { createPhysicalSendReporter } from "../lib/request-execution-budget"; import { createInferenceSendBudget } from "./inference/context"; import { observeAdvisorNativeReasoning } from "./inference/advisor-diagnostics"; import { buildOpenAIChatPassthroughRequest, createOpenAIChatAdapter } from "../adapters/openai-chat"; +import { affinityLaneFromRequest } from "./request-log-conversation"; import type { AdapterRequest, ProviderAdapter } from "../adapters/base"; import { isNativeChatRouteEligible } from "./chat-native-eligibility"; import { @@ -336,11 +337,15 @@ export async function runNativeChatAttempt( // Transport variant: the bare picker answers with the persisted row, which for a built-in // provider carries no adapter id or base URL until routedProviderConfig backfills it. try { - const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); + const affinityLane = affinityLaneFromRequest(req.headers, logCtx); + if (affinityLane) logCtx.affinityLane = affinityLane; + const chatMessages = Array.isArray(execution.chatBody.messages) ? execution.chatBody.messages : []; + const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider, undefined, Date.now(), { lane: affinityLane, model: route.modelId, log: logCtx, continuation: (chatMessages.at(-1) as { role?: unknown } | undefined)?.role === "tool" }); if (proactiveKeyProvider) route.provider = proactiveKeyProvider; } catch (error) { cleanupAbort(); upstream.abort(); if (!(error instanceof RequestPacingQueueOverloadError)) throw error; + logCtx.terminalSource = "synthetic"; const refusal = fail(429, error.message, "rate_limit_error"); refusal.headers.set("Retry-After", String(error.retryAfterSeconds)); return refusal; @@ -678,11 +683,13 @@ export async function runNativeChatAttempt( if (req.signal.aborted) return fail(499, "Client cancelled request", "client_cancelled"); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; if (sendError instanceof RequestPacingQueueOverloadError) { + logCtx.terminalSource = "synthetic"; const refusal = fail(429, sendError.message, "rate_limit_error"); refusal.headers.set("Retry-After", String(sendError.retryAfterSeconds)); return refusal; } if (sendError instanceof SendBudgetExhaustedError) { + logCtx.terminalSource = "synthetic"; // A decision this process made, answered as the Responses path answers it: 429, not 502. return fail(429, sendError.message, SEND_BUDGET_EXHAUSTED_CODE, SEND_BUDGET_EXHAUSTED_CODE); } diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index 75a833095ad..c78e3a2905e 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -1257,6 +1257,10 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise !!block && typeof block === "object" && (block as { type?: unknown }).type === "tool_result"); + const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider, undefined, Date.now(), { lane: affinityLane, model: route.modelId, log: logCtx, continuation }); if (proactiveKeyProvider) route.provider = proactiveKeyProvider; } catch (error) { cleanupAbort(); upstream.abort(); if (!(error instanceof RequestPacingQueueOverloadError)) throw error; + logCtx.terminalSource = "synthetic"; const refusal = fail(429, error.message, "rate_limit_error"); refusal.headers.set("Retry-After", String(error.retryAfterSeconds)); return refusal; @@ -834,11 +841,15 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) settleDistributedFailure(); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; if (sendError instanceof RequestPacingQueueOverloadError) { + logCtx.terminalSource = "synthetic"; const refusal = fail(429, sendError.message, "rate_limit_error"); refusal.headers.set("Retry-After", String(sendError.retryAfterSeconds)); return refusal; } - if (sendError instanceof SendBudgetExhaustedError) return fail(429, sendError.message, "send_budget_exhausted", "send_budget_exhausted"); + if (sendError instanceof SendBudgetExhaustedError) { + logCtx.terminalSource = "synthetic"; + return fail(429, sendError.message, "send_budget_exhausted", "send_budget_exhausted"); + } if (sendError instanceof NativeMessagesSpendRefusal) { const refusal = workflowRefusalResponse("workflow-spend-exhausted", logCtx); finishLog(429); diff --git a/src/server/request-log-cache-affinity.ts b/src/server/request-log-cache-affinity.ts new file mode 100644 index 00000000000..8c49c0840b4 --- /dev/null +++ b/src/server/request-log-cache-affinity.ts @@ -0,0 +1,55 @@ +import { + boundEntryIdForLane, + recordProviderCacheAffinity, + releaseProviderCacheAffinity, +} from "../providers/cache-affinity"; +import { isAnthropicBillingFailure } from "../providers/anthropic-billing-refusal"; +import type { OcxUsage } from "../types"; +import type { RequestLogContext } from "./request-log"; + +/** Finalize against the serving credential, never a logical combo's display label. */ +export function finalizeProviderCacheAffinity( + logCtx: RequestLogContext, + status: number, + usage: OcxUsage | undefined, +): void { + const { + affinityLane: lane, + affinityKeyEntryId: entryId, + affinityProviderName: providerName, + } = logCtx; + if (!lane || !entryId || !providerName || logCtx.cacheAffinityApplicable !== true) return; + if (status >= 200 && status < 300 && logCtx.observedTerminalStatus !== "failed") { + const previouslyBound = boundEntryIdForLane(lane); + recordProviderCacheAffinity({ + lane, + providerName, + entryId, + model: logCtx.affinityModel ?? logCtx.model, + cacheRead: usage?.cacheReadInputTokens ?? usage?.cachedInputTokens, + cacheWrite: usage?.cacheCreationInputTokens, + cacheWriteLong: usage?.cacheCreation1hInputTokens, + inputTokens: usage?.inputTokens, + }); + if (logCtx.cacheAffinity === "kept" && logCtx.affinityKeptEntryId !== entryId) { + logCtx.cacheAffinity = "bound"; + logCtx.cacheAffinityReason = "new"; + } else if (!logCtx.cacheAffinity) { + logCtx.cacheAffinity = previouslyBound === entryId ? "kept" : "bound"; + logCtx.cacheAffinityReason = previouslyBound === entryId ? "cache" : "new"; + } + } else if ( + logCtx.terminalSource !== "synthetic" && + (status === 401 || + status === 402 || + status === 429 || + (logCtx.providerAdapter === "anthropic" && + isAnthropicBillingFailure(status, logCtx.upstreamError ?? "", logCtx.upstreamErrorType))) + ) { + // Only this request's refusal is evidence. A shared cooldown may have been + // written by another conversation while this request was in flight. + if (!releaseProviderCacheAffinity(lane, providerName, entryId)) return; + logCtx.cacheAffinity = "released"; + logCtx.cacheAffinityReason = status === 401 ? "auth" : "quota"; + } +} diff --git a/src/server/request-log-conversation.ts b/src/server/request-log-conversation.ts index 6da27a8caf7..4b777a5ffed 100644 --- a/src/server/request-log-conversation.ts +++ b/src/server/request-log-conversation.ts @@ -128,6 +128,26 @@ export function sessionLaneIdFromRequest(headers: Headers): string | undefined { return specific ?? parent; } +/** + * The affinity lane a request binds under, derived once so every entry path agrees. + * + * `sessionLaneIdFromRequest` first (explicit thread/session/parent identity), then the + * explicit `x-opencode-session` header the Claude-Go handoff and + * `getOrAllocateRequestSessionLane` already treat as lane identity, then the log + * conversation id — except the Hermes digest, which is log correlation only and must + * never affect lanes or routing (#330). This expression used to be inlined at four + * entry paths and drifted: three of the four missed the `x-opencode-session` fallback + * until PR #7738 aligned them. + */ +export function affinityLaneFromRequest( + headers: Headers, + slot: LogConversationSlot, +): string | undefined { + return sessionLaneIdFromRequest(headers) + ?? normalizeLogConversationId(headers.get("x-opencode-session")) + ?? (slot.conversationIdIsHermesFallback ? undefined : normalizeLogConversationId(slot.conversationId)); +} + function firstSanitizedConversationId( ...values: Array ): string | undefined { diff --git a/src/server/request-log.ts b/src/server/request-log.ts index 925bd5f2c54..3e263ed8e94 100644 --- a/src/server/request-log.ts +++ b/src/server/request-log.ts @@ -1,6 +1,8 @@ import { existsSync, readFileSync } from "node:fs"; import { randomBytes } from "node:crypto"; import { stampApiKeyAccountLabel, usesApiKeyAccount } from "../providers/label"; +import { finalizeProviderCacheAffinity } from "./request-log-cache-affinity"; + import { KEY_ACCOUNT_LOG_LABEL_RE } from "../codex/account-label"; import { attemptAccountChanged, sealRequestAttemptIdentity } from "./request-log-account-rotation"; export { sealRequestAttemptIdentity }; @@ -21,7 +23,7 @@ import { CODEX_CONFIG_PATH, readRootTomlString } from "../codex/paths"; import type { CodexAffinityMove, CodexAffinityReason } from "../codex/routing"; import type { CodexAuthContext } from "../codex/auth-context"; import { readCodexCatalogPath } from "../codex/catalog"; -import type { AttemptTierOutcome, OcxProviderConfig, OcxUsage } from "../types"; +import type { AttemptTierOutcome, CacheAffinityLogReason, CacheAffinityOutcome, OcxProviderConfig, OcxUsage } from "../types"; import { normalizeRouteDecisionTrace, type RouteDecisionTraceV1 } from "../routing/trace"; import { parseProtocolTraceV1, type ProtocolTraceV1 } from "../protocols/dto"; import { protocolTraceForRequest } from "../protocols/trace"; @@ -36,6 +38,8 @@ import { isKnownAdmissionKind, isKnownAffinityMove, isKnownAffinityReason, + isKnownCacheAffinity, + isKnownCacheAffinityReason, isKnownCacheTelemetryProvenance, isKnownInboundProtocol, isKnownTerminalSource, @@ -263,6 +267,20 @@ export interface RequestLogContext extends UpstreamErrorDiagnostics { * Codex pool account was not the issuer. Never an account identifier. */ conversationStateScrub?: "account-change"; + /** Explicit session lane for provider cache affinity; internal only, never persisted raw. */ + affinityLane?: string; + affinityKeyEntryId?: string; + /** Configured provider owning the serving entry, independent of combo display labels. */ + affinityProviderName?: string; + /** Whether provider cache affinity applies to this request's provider at all. */ + cacheAffinityApplicable?: boolean; + /** Entry the consult kept this request on, when it did; rotation may still move it. */ + affinityKeptEntryId?: string; + /** Model identity affinity consults with; recording reuses it so virtual models match. */ + affinityModel?: string; + /** Provider cache-affinity outcome for this request (bounded enum, persisted on the entry). */ + cacheAffinity?: CacheAffinityOutcome; + cacheAffinityReason?: CacheAffinityLogReason; transportPhase?: "pre_headers" | "mid_stream" | "terminal_sse"; terminalSource?: "upstream" | "synthetic"; /** Closed failure evidence supplied by the transport, never inferred from provider text. */ @@ -373,6 +391,9 @@ export interface RequestLogEntry extends UpstreamErrorDiagnostics { affinity?: CodexAffinityMove; /** Why that decision was made (#4546): a move is the expensive event, so it names its cause. */ affinityReason?: CodexAffinityReason; + /** Provider cache-affinity outcome: kept on, newly bound to, or released from a key-pool credential. */ + cacheAffinity?: CacheAffinityOutcome; + cacheAffinityReason?: CacheAffinityLogReason; /** * Set when this request dropped account-bound continuation after a Codex pool * account change. Never an account identifier. @@ -524,6 +545,7 @@ export function requestLogEntryFromPersistedUsage(entry: PersistedUsageEntry): R ? { cacheProvenance: entry.cacheProvenance } : {}), ...persistedAffinityFields(entry), + ...persistedCacheAffinityFields(entry), ...(isKnownTransportPhase(entry.transportPhase) ? { transportPhase: entry.transportPhase } : {}), ...(isKnownTerminalSource(entry.terminalSource) ? { terminalSource: entry.terminalSource } : {}), ...(routeDecision ? { routeDecision } : {}), @@ -552,6 +574,16 @@ function persistedAffinityFields( }; } +function persistedCacheAffinityFields( + entry: Pick, +): Pick { + if (!isKnownCacheAffinity(entry.cacheAffinity)) return {}; + return { + cacheAffinity: entry.cacheAffinity, + ...(isKnownCacheAffinityReason(entry.cacheAffinityReason) ? { cacheAffinityReason: entry.cacheAffinityReason } : {}), + }; +} + /** * Hydration guard: persisted traces are re-normalized before they enter the * in-memory ring buffer so a hand-edited or corrupt row cannot poison the DTO. @@ -716,6 +748,7 @@ export function addRequestLog(entry: RequestLogEntry) { ? { cacheProvenance: entry.cacheProvenance } : {}), ...persistedAffinityFields(entry), + ...persistedCacheAffinityFields(entry), ...(isKnownTransportPhase(entry.transportPhase) ? { transportPhase: entry.transportPhase } : {}), ...(isKnownTerminalSource(entry.terminalSource) ? { terminalSource: entry.terminalSource } : {}), ...failureDiagnostics, @@ -1506,6 +1539,14 @@ export function addFinalRequestLog( cacheProvenance, ...(logCtx.cacheDiagnosticDraft ? { draft: logCtx.cacheDiagnosticDraft } : {}), }); + // Provider cache affinity: this is the one seam every request passes exactly + // once with its terminal status, the credential that ultimately served (the + // entry id captured beside the account label at each stamp, rotations + // included), and cache usage already normalized from whichever adapter + // answered. A success binds or refreshes the conversation on that + // credential; a quota/auth terminal releases it so the next request is + // placed fresh instead of re-offered a credential that just refused. + finalizeProviderCacheAffinity(logCtx, effectiveStatus, logCtx.usage ?? loggedUsage); // Sanitize at the logging layer, not only at the one call site that populates this today. // The value originates in an upstream-supplied model id, so an unsanitized newline would // let a single field forge a record boundary in any line-oriented log viewer. Doing it here @@ -1571,6 +1612,8 @@ export function addFinalRequestLog( ...(loggedUsage || cacheProvenance !== "unknown" ? { cacheProvenance } : {}), ...(logCtx.affinity ? { affinity: logCtx.affinity } : {}), ...(logCtx.affinityReason ? { affinityReason: logCtx.affinityReason } : {}), + ...(logCtx.cacheAffinity ? { cacheAffinity: logCtx.cacheAffinity } : {}), + ...(logCtx.cacheAffinityReason ? { cacheAffinityReason: logCtx.cacheAffinityReason } : {}), ...(logCtx.conversationStateScrub === "account-change" ? { conversationStateScrub: "account-change" } : {}), diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index 56d8fbbc32a..13270bd6320 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -579,6 +579,7 @@ export async function prepareAdapterExchange( // blaming the provider makes the caller send the whole turn again -- the amplification this // budget exists to stop. The passthrough path has answered 429 here since #4546. if (err instanceof SendBudgetExhaustedError) { + logCtx.terminalSource = "synthetic"; return unboundPoolSpendRefusalResponse(logCtx) ?? formatErrorResponse(429, SEND_BUDGET_EXHAUSTED_CODE, err.message); } @@ -867,6 +868,7 @@ export async function prepareAdapterExchange( // Same rule on the recovery leg: the ladder refused to send again, so the answer names // this proxy rather than the provider it never reached. if (err instanceof SendBudgetExhaustedError) { + logCtx.terminalSource = "synthetic"; return { failed: unboundPoolSpendRefusalResponse(logCtx) ?? formatErrorResponse(429, SEND_BUDGET_EXHAUSTED_CODE, err.message) }; } diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 7e665b0a9cd..9da0d22cdc2 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -227,7 +227,7 @@ import { fetchWithHeaderTimeout, providerFetch, safeHostLabel, safeOriginLabel, import { requestPacingOverloadResponse } from "./pacing-overload"; import { shouldRetryCodexPoolAccountQuota, shouldRetryCodexPoolAccountTransient } from "./core-codex-account"; import { mapCodexAuthContextErrorToResponse, nativeMainRefreshFailureResponse } from "./codex-auth-error"; -import { linkRequestSessionLane, sessionLaneIdFromRequest, reasoningReplayConversationIdFromResponsesRequest, sessionIdHeaderFromRequest } from "../request-log-conversation"; +import { affinityLaneFromRequest, linkRequestSessionLane, reasoningReplayConversationIdFromResponsesRequest, sessionIdHeaderFromRequest, sessionLaneIdFromRequest } from "../request-log-conversation"; import { recallComboForLane } from "./combo-session-recall"; import { redactHostedImageDisplayPaths } from "../responses-hosted-image-display"; @@ -936,10 +936,12 @@ export async function handleResponsesCompact( // Transport variant, for the same reason as the Responses core: the persisted row has no // registry backfills, and compactProvider is read for its transport fields below. let warmKeyProvider; - try { warmKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, compactProvider); } + const affinityLane = affinityLaneFromRequest(req.headers, logCtx); + if (affinityLane) logCtx.affinityLane = affinityLane; + try { warmKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, compactProvider, undefined, Date.now(), { lane: affinityLane, model: route.modelId, log: logCtx }); } catch (error) { const refusal = requestPacingOverloadResponse(error); - if (refusal) return refusal; + if (refusal) { logCtx.terminalSource = "synthetic"; return refusal; } throw error; } if (warmKeyProvider?.apiKey) compactProvider = warmKeyProvider; @@ -1268,6 +1270,7 @@ export async function handleResponsesCompact( now: Date.now(), accountSelector: route.codexAccountNamespace, })); if (response) { + logCtx.terminalSource = "synthetic"; releaseUpstreamHostAdmission(compactHostAdmissionLease); compactHostAdmissionLease = null; releaseCodexAuthContextProbeLease(outcomeCtx); diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 20dae280c0a..e5e8ba878c9 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -1031,7 +1031,8 @@ export async function executeComboResponses( const childLog: RequestLogContext = { model: pick.target.model, provider: pick.target.provider, - ...(logCtx.conversationId ? { conversationId: logCtx.conversationId } : {}), + ...(logCtx.conversationId ? { conversationId: logCtx.conversationId, conversationIdIsHermesFallback: logCtx.conversationIdIsHermesFallback } : {}), + ...(logCtx.affinityLane ? { affinityLane: logCtx.affinityLane } : {}), ...(logCtx.surface ? { surface: logCtx.surface } : {}), }; const targetReasoningEfforts = supportedLadderFor({ diff --git a/src/server/responses/policy-fallback.ts b/src/server/responses/policy-fallback.ts index e4c997f4422..9a4a7a3570c 100644 --- a/src/server/responses/policy-fallback.ts +++ b/src/server/responses/policy-fallback.ts @@ -159,8 +159,17 @@ function finishFailedPolicyAttempt(logCtx: RequestLogContext, status: number): v delete logCtx.usageFromBridge; delete logCtx.upstreamError; delete logCtx.terminalHttpStatus; + delete logCtx.observedTerminalStatus; delete logCtx.terminalErrorCode; delete logCtx.terminalIncompleteReason; + delete logCtx.cacheAffinity; + delete logCtx.cacheAffinityReason; + delete logCtx.affinityKeptEntryId; + delete logCtx.affinityKeyEntryId; + delete logCtx.affinityProviderName; + delete logCtx.affinityModel; + delete logCtx.cacheAffinityApplicable; + delete logCtx.terminalSource; } /** diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index aceb6dfea01..e6f9c3db86f 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -87,7 +87,7 @@ import { recordAnthropicAccountQuotaFromHeadersForInstance, hasPassiveAccountQuo import { checkOutboundBodySize, describeOutboundBodyRefusal } from "./outbound-body-guard"; import { formatErrorResponse } from "../../bridge"; import { bindRouteReasoningReplayScope } from "./core-replay"; -import { assignNativeLogConversationId, sessionIdHeaderFromRequest, normalizeLogConversationId } from "../request-log-conversation"; +import { affinityLaneFromRequest, assignNativeLogConversationId, normalizeLogConversationId, sessionIdHeaderFromRequest } from "../request-log-conversation"; import { redactSecretString } from "../../lib/redact"; import { captureAccountAutoPauseAttempt } from "../../oauth/account-auto-pause"; import { selectProactiveApiKeyTransport } from "../../providers/key-failover"; @@ -1201,11 +1201,15 @@ export async function prepareResponsesTransport( // The Transport variant, not the bare picker: the picker answers with the PERSISTED row, and // a built-in provider stored in its valid minimal form would lose the adapter id, base URL // and static headers registry backfill supplies, throwing `Unknown adapter: undefined`. + const affinityLane = affinityLaneFromRequest(req.headers, logCtx); + if (affinityLane) logCtx.affinityLane = affinityLane; const proactiveKeyProvider = selectProactiveApiKeyTransport( config, route.providerName, route.provider, parsed.options.promptCacheKey, + Date.now(), + { lane: affinityLane, model: route.modelId, log: logCtx, continuation: parsed.context.messages.at(-1)?.role === "toolResult" }, ); if (proactiveKeyProvider) route.provider = proactiveKeyProvider; route.provider = resolveProviderTransport( diff --git a/src/types.ts b/src/types.ts index 8da67a3db8b..343764116c0 100644 --- a/src/types.ts +++ b/src/types.ts @@ -118,6 +118,9 @@ export type { export type { RefreshPolicy, + CacheAffinityMode, + CacheAffinityOutcome, + CacheAffinityLogReason, OpenRouterProviderRouting, VercelGatewayRouting, ResponsesItemIdRepairConfig, diff --git a/src/types/provider.ts b/src/types/provider.ts index c2358cc46d7..0d2b2451b93 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -10,6 +10,21 @@ import type { AnthropicAccountPoolConfig } from "./anthropic-account-pool"; */ export type RefreshPolicy = "proactive" | "lazy-only" | "disabled"; +/** + * Per-provider conversation affinity to the credential that served a conversation's + * last turn, so that credential's upstream prompt cache is read again instead of + * rebuilt on another key. `auto` keeps a binding only while the cache it built is + * measured to be worth keeping and still warm; `session` keeps it while the + * credential stays eligible; `turn` only across immediate continuations; `off` + * disables binding entirely. See src/providers/cache-affinity.ts. + */ +export type CacheAffinityMode = "auto" | "session" | "turn" | "off"; + +/** Provider cache-affinity outcome recorded on a request-log row. */ +export type CacheAffinityOutcome = "kept" | "bound" | "released"; +/** Why that outcome happened; mirrors the store's keep reasons plus lifecycle outcomes. */ +export type CacheAffinityLogReason = "cache" | "session" | "turn" | "continuation" | "new" | "quota" | "auth"; + /** Request-owned identity of the configured key, before env/keychain resolution. */ export interface ProviderApiKeySelection { entryId?: string; @@ -17,6 +32,8 @@ export interface ProviderApiKeySelection { revision?: string; /** Request-local allocation; does not replace the saved active key. */ distributed?: boolean; + /** Request-local cache-affinity pick; survives selection re-reads while its entry stays live. */ + cacheAffinity?: boolean; } export interface OpenRouterProviderRouting { @@ -1027,6 +1044,14 @@ export interface OcxProviderConfig { * fields. Default off; only enable for providers that document this parameter. */ promptCacheKey?: boolean; + /** + * Conversation affinity to the key-pool credential that served a conversation's + * last turn, so its upstream prompt cache is read again on the next turn instead + * of rebuilt on another key. Only affects providers with 2+ key-pool entries; + * eligibility (cooldowns, pacing, disabled keys) always outranks the preference. + * Default `auto`. See src/providers/cache-affinity.ts. + */ + cacheAffinity?: CacheAffinityMode; /** * Opt-in: forward caller `service_tier` values to the upstream `/chat/completions` body. * On a classified route it governs foreign values (for example `flex`), not proxy-owned diff --git a/src/usage/log.ts b/src/usage/log.ts index 51da4e7ee68..ff3d4bf82e4 100644 --- a/src/usage/log.ts +++ b/src/usage/log.ts @@ -18,7 +18,7 @@ import { type RequestCloseReason, type RequestTerminalStatus, } from "./request-outcome"; -import type { AttemptTierOutcome, OcxUsage } from "../types"; +import type { AttemptTierOutcome, CacheAffinityLogReason, CacheAffinityOutcome, OcxUsage } from "../types"; import { normalizeRouteDecisionTrace, type RouteDecisionTraceV1 } from "../routing/trace"; import { parseProtocolTraceV1, type ProtocolTraceV1 } from "../protocols/dto"; import { ACCOUNT_LOG_LABEL_RE, CODEX_ACCOUNT_LOG_LABEL_RE } from "../codex/account-label"; @@ -383,6 +383,9 @@ export interface PersistedUsageEntry { */ affinity?: CodexAffinityMove; affinityReason?: CodexAffinityReason; + /** Provider cache-affinity outcome for key-pool routes; additive, older rows omit it. */ + cacheAffinity?: CacheAffinityOutcome; + cacheAffinityReason?: CacheAffinityLogReason; /** * Set when this request dropped account-bound continuation after a Codex pool * account change. Never an account identifier. @@ -526,6 +529,21 @@ export function isKnownAffinityReason(value: unknown): value is NonNullable); } +const KNOWN_CACHE_AFFINITY = new Set>([ + "kept", "bound", "released", +]); +const KNOWN_CACHE_AFFINITY_REASONS = new Set>([ + "cache", "session", "turn", "continuation", "new", "quota", "auth", +]); + +export function isKnownCacheAffinity(value: unknown): value is NonNullable { + return typeof value === "string" && KNOWN_CACHE_AFFINITY.has(value as NonNullable); +} + +export function isKnownCacheAffinityReason(value: unknown): value is NonNullable { + return typeof value === "string" && KNOWN_CACHE_AFFINITY_REASONS.has(value as NonNullable); +} + export function usageLogPath(configDir?: string): string { return join(configDir ?? getConfigDir(), "usage.jsonl"); } @@ -990,6 +1008,10 @@ function normalizeUsageEntry(entry: PersistedUsageEntry): PersistedUsageEntry { const affinityReason = affinity !== undefined && isKnownAffinityReason(entry.affinityReason) ? entry.affinityReason : undefined; + const cacheAffinity = isKnownCacheAffinity(entry.cacheAffinity) ? entry.cacheAffinity : undefined; + const cacheAffinityReason = cacheAffinity !== undefined && isKnownCacheAffinityReason(entry.cacheAffinityReason) + ? entry.cacheAffinityReason + : undefined; const conversationStateScrub = typeof entry.conversationStateScrub === "string" && KNOWN_CONVERSATION_STATE_SCRUBS.has(entry.conversationStateScrub) ? entry.conversationStateScrub @@ -1085,6 +1107,8 @@ function normalizeUsageEntry(entry: PersistedUsageEntry): PersistedUsageEntry { ...(terminalSource ? { terminalSource } : {}), ...(affinity ? { affinity } : {}), ...(affinityReason ? { affinityReason } : {}), + ...(cacheAffinity ? { cacheAffinity } : {}), + ...(cacheAffinityReason ? { cacheAffinityReason } : {}), ...(conversationStateScrub ? { conversationStateScrub } : {}), ...(entry.errorCode ? { errorCode: entry.errorCode } : {}), // Validated rather than copied on truthiness, like the inbound protocol and transport phase diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 192569907e9..96d32c2a72b 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -275,7 +275,7 @@ per-request first-party callback reads that live object; a failed write leaves i | Startup safety | `GET /api/startup-health` reports whether injected Codex routing is restart-safe, with secret-free service/shim diagnostics. `POST /api/startup-action` provides allowlisted one-click installation for the background service or launcher shim. On Windows a healthy script shim is CLI-only; Codex Desktop requires the background service for full protection. Optional Codex compatibility uses the [confirmed local runtime](clients/codex-desktop.md#optional-compatibility-runtime), [machine-local dashboard consent](clients/codex-desktop.md#dashboard-controls), and [revision-bound next-start preference with shared awaited teardown](clients/codex-desktop.md#proxy-startup-preference); saving that preference performs no runtime action. | | Windows tray | `GET/POST /api/windows-tray` controls an owned, per-user HKCU login tray. The tray delegates fixed actions to the CLI and is never a proxy supervisor or restart-protection signal. | | Updates | `GET /api/update/check`, `POST /api/update/run`, and `GET /api/update/status` own dashboard self-update state. A launched worker PID is persisted in `update-job.json`; dead PIDs recover immediately when a new run starts, and `GET /api/update/status` already reports such a record as `failed` without writing it (`readUpdateJobForStatus`), while legacy active records without a PID recover only after ten minutes. Live PIDs remain exclusive regardless of record age. `GET /api/update/check` and `POST /api/update/run` await one per-channel asynchronous registry lookup and write successful results through to the package cache; run passes that result to the job starter. `GET /api/update/badge` only reads the cache and reports unknown after 40 hours, on missing cache, or on channel mismatch. The badge links to the update surface rather than gating other actions. `GET /api/update/badge?surface=desktop&session=` projects only that process-local Tauri session; missing or expired state is unknown and never falls back to the package cache. `POST /api/update/desktop-snapshot` is a 1 KiB bounded display-state mutation with no install permission. It accepts the existing admin-token principal or a dedicated single-use, ten-second snapshot capability bound to nonce, method, path, PID, port and the SHA-256 digest of the exact body bytes; that capability cannot authorize another route. | -| Providers | Create/update/delete ordinary provider configs and enrich registry metadata. A `POST /api/providers` overwrite of an existing name keeps the ten operator compatibility settings (`PROVIDER_COMPAT_CARRY_FIELDS` in `src/server/management/provider-overwrite-carry.ts`) and the stored key pool only while the destination (adapter, normalized base URL, auth mode when named) is unchanged. It keeps omitted `hideRawReasoning` on every overwrite of the same name, including destination moves, because it is a display policy; a submitted boolean wins. An omitted `inferContextWindow` likewise keeps the latest live same-name state after asynchronous validation, including `false` or a concurrent clear; a submitted boolean wins. It never merges the rest of the old row. `PATCH` is a field mask and keeps every field it does not name; `hideRawReasoning`, `preserveResponsesReasoningContent`, `promoteAdditionalTools` and `inferContextWindow` accept a boolean or `null` to clear them, rejecting other types. `GET /api/providers` exposes configured `inferContextWindow` as an optional boolean without substituting a default; its [custom-only, default-off inference policy](catalog-vendor-context.md) is unchanged. Custom Responses provider forms expose `preserveResponsesReasoningContent` as a default-off plaintext replay switch; unrelated settings saves preserve omission and curated/OpenAI destinations keep their existing defaults. The reserved `openai` card exposes Pool(default)/Direct account mode; `openai-apikey` remains the separate API route. | +| Providers | Create/update/delete ordinary provider configs and enrich registry metadata. A `POST /api/providers` overwrite of an existing name keeps the ten operator compatibility settings (`PROVIDER_COMPAT_CARRY_FIELDS` in `src/server/management/provider-overwrite-carry.ts`) and the stored key pool only while the destination (adapter, normalized base URL, auth mode when named) is unchanged. It keeps omitted `hideRawReasoning` on every overwrite of the same name, including destination moves, because it is a display policy; a submitted boolean wins. An omitted `inferContextWindow` likewise keeps the latest live same-name state after asynchronous validation, including `false` or a concurrent clear; a submitted boolean wins. An omitted `cacheAffinity` keeps the latest live same-name mode, including `off`, so unrelated dashboard saves cannot re-enable affinity; a submitted mode wins. It never merges the rest of the old row. `PATCH` is a field mask and keeps every field it does not name; `hideRawReasoning`, `preserveResponsesReasoningContent`, `promoteAdditionalTools` and `inferContextWindow` accept a boolean or `null` to clear them, rejecting other types. `GET /api/providers` exposes configured `inferContextWindow` as an optional boolean without substituting a default; its [custom-only, default-off inference policy](catalog-vendor-context.md) is unchanged. Custom Responses provider forms expose `preserveResponsesReasoningContent` as a default-off plaintext replay switch; unrelated settings saves preserve omission and curated/OpenAI destinations keep their existing defaults. The reserved `openai` card exposes Pool(default)/Direct account mode; `openai-apikey` remains the separate API route. | | Models | Fetch routed model lists, disabled model visibility, and catalog-facing ids. New non-OAuth registration holds exposure until authoritative discovery; 20 or more distinct switch rows start OFF without disabling the provider. Pending rows cannot accept visibility changes. Ordinary model reads also apply the [new-arrival catalog contract](catalog.md#shared-catalog) before returning models. With a matching persisted configuration, discovery records the baseline, NEW badges, and automatic disables together; HTTP listing does not wait for a separate Codex sync. Inventory already drifted from disk uses a detached policy projection: arrivals remain disabled in management rows without changing live config or disk. | | OAuth | Login/status/logout for OAuth-backed providers, plus multiauth account management: `GET /api/oauth/accounts`, `PUT /api/oauth/accounts/active`, `PUT /api/oauth/accounts/alias`, `PUT /api/oauth/accounts/pause`, and `DELETE /api/oauth/accounts` list masked accounts per provider, switch the active one, edit its display-only alias, pause/resume a generic OAuth account, and remove one. Paused generic OAuth accounts are excluded from request selection, 429 failover, and proactive token refresh; pausing an active account selects the next usable account when available, and resuming an account restores an active selection if the current one remains paused. With no unpaused account, requests return 403 rather than a login error. Kiro account-list rows include the current automatic-selection projection and closed exclusion reason; an active singleton may still send. The login flow itself is `GET /api/oauth/providers`, `POST /api/oauth/login`, `POST /api/oauth/login/code`, `POST /api/oauth/login/cancel`, `POST /api/oauth/logout`, and `GET /api/oauth/status`; pool controls are `GET/PUT/PATCH /api/oauth/accounts/pool` and `POST /api/oauth/accounts/clear-cooldown`. Login accepts `addAccount: true` to force a fresh browser identity. Meta Muse login start and manual-code continuation require the server-resolved `gui-session` principal before credential acquisition or code submission (including reauth); see the [provider contract](providers-and-adapters.md). Device flows return a structured `deviceCode`; the GUI highlights and copies it before the user opens the verification page. Google Antigravity account rows in `gui/src/components/provider-workspace/ProviderAuthPanel.tsx` display the account-summary `plan` as a badge and include it in the account button’s accessible name. The value is the canonical upstream tier ID (for example `free-tier` or `gemini-pro`), preserved literally across locales rather than inferred as a product name. Absent plans render no badge; pause and reauthentication controls keep their existing behavior. | | Key providers | `GET /api/key-providers` exposes API-key provider presets for setup and dashboard flows, and `GET/POST/DELETE /api/keys` owns the proxy's own admission keys. Machine links live in `src/server/management/link-routes.ts`: dashboard sessions reach `GET /api/link/candidates`, `POST /api/link/probe`, `POST /api/link/confirm-host` and `POST /api/link/apply`, meaning a paired session or, on a standalone runtime, the current loopback-issued session on trusted loopback ingress; `POST /api/link/join` instead requires an operator-paired session because it durably redirects local client traffic, and also refuses a runtime that is not standalone (`409 standalone_required`) or not listening on its configured port (`409 join_port_mismatch`) before any SSH, then restarts this runtime as a client; `GET /api/link/status` reports `joinAvailable` and typed `joinDenied` causes only to GUI-session callers, distinguishing pairing, runtime-role and port refusals without changing the admin/CLI DTO, so the dashboard enables the Child role only when the join gates pass; `GET /api/link/status`, `POST /api/link/{id}/reconnect` and `DELETE /api/link/{id}` also accept the admin token on a trusted loopback ingress, and reconnect restarts a Home-initiated link's tunnel now; `POST /api/link/issue` accepts only that admin token. Tailscale-identity sessions are refused on every link route. See [Remote Link](remote-link.md). Multi-key pool per key-auth provider: `GET /api/providers/keys`, `POST /api/providers/keys`, `PUT /api/providers/keys/active`, `PUT /api/providers/keys/alias`, `DELETE /api/providers/keys` masked list, add (upsert + activate), switch, rename, and remove keys. `provider.apiKey` always mirrors the active pool entry so routing stays single-key. | diff --git a/structure/transports/inventory.md b/structure/transports/inventory.md index d0ea8de7b71..f44a6f6a2b8 100644 --- a/structure/transports/inventory.md +++ b/structure/transports/inventory.md @@ -1,5 +1,12 @@ # Transport Inventory +Round-robin API-key cache-affinity picks carry both request-local affinity and distributed +selection ownership. Failure recovery cools their live captured entry and selects a request-local +replacement without writing the saved active key or revision. An unrelated selection revision +does not revoke a live affinity pick; removed, disabled or replaced entries still cannot send +or authorize recovery. Final logging reports `released` only when scoped provider/entry release +actually deletes a binding; a refusal on another entry preserves the binding and prior outcome. + Shared header helpers and bounded local JSON reads follow the [local timeout contract](timeouts.md). Disabling their clocks retains byte caps, cancellation, manual redirects and transport credential fences. @@ -53,7 +60,7 @@ surface is listed here so a maintainer can find the owner without grepping: | Image/video generation loop | `src/images/loop.ts`, `src/images/plan.ts`, `src/images/fulfill.ts`, `src/images/xai-client.ts`, `src/images/xai-video-client.ts`, `src/images/artifacts.ts` | A provider-returned image URL is downloaded into a local artifact once, then served locally; warnings stay URL-free because provider CDN URLs may embed credentials. Artifact downloads go through the pinned-IP transport with a 10 s connect deadline (`DOWNLOAD_CONNECT_TIMEOUT_MS`) that bounds TCP/TLS setup on its own, in addition to the 60 s idle timer, and `pinnedHttpsGet` accepts a per-call `connectTimeoutMs`. | | GitHub Copilot | `src/providers/xai-transport.ts` (`resolveProviderTransport`), `src/providers/github-copilot-transport.ts` | `resolveProviderTransport` selects the Copilot transport when the routed provider name is `github-copilot`; the Copilot module then resolves its headers and base URL, and the registry seeds the provider row and model fallback. Auto session and intent requests stay on the credential-captured origin; adapter construction follows the [Copilot Auto contract](../providers-and-adapters.md#github-copilot-auto-selection). | | Mistral | `src/providers/mistral-transport.ts` | Canonical key-auth Chat is selected by destination, including custom provider names, and shares Retry-After holds by credential and exact model. A real 429 with validated minute pressure and no retry time activates a conservative minute recovery hold and one probe; healthy traffic has no predicted-charge pacing. Local admission uses `src/lib/provider-admission-wait.ts` to pause only the upstream header deadline while retaining client cancellation; the deadline restarts before the send. No fixed account quota, exact server-reset claim or automatic compaction; other providers/destinations retain their transports. See [provider recovery](../providers-and-adapters.md). | -| API-key pools | `src/providers/api-key-selection.ts`, `src/providers/key-failover.ts` | A configured `apiKeyPoolStrategy` plus a cooling committed key rotates before the first send (`selectProactiveApiKeyTransport`); a 429 still rotates after the send and records a cooldown. `provider.apiKey` keeps mirroring the active entry so routing stays single-key. The pick is inert without a strategy or while the committed key is healthy. | +| API-key pools | `src/providers/api-key-selection.ts`, `src/providers/key-failover.ts`, `src/providers/cache-affinity.ts` | A configured `apiKeyPoolStrategy` plus a cooling committed key rotates before the first send; a 429 still rotates after the send and records a cooldown. Before strategy selection, `selectProactiveApiKeyTransport` consults request-local cache affinity without writing the committed `provider.apiKey`. The captured pool entry id and stored reference survive env/keychain resolution. Dispatch re-reads check entry liveness, key identity, and cooldown, not selection revision: another conversation's automatic rotation must not invalidate the pick. Manual key selection does not move an existing binding; `cacheAffinity: "off"` or disabling the entry overrides it. Per-key concurrency saturation or a next slot beyond the pacer's maximum queue age falls through to ordinary selection; shorter pacing waits and model caps remain enforced by the send queue. `auto` prefers the same provider/model while warm (5 minutes, or 1 hour after a long-retention write), with at least 1024 tokens read or written, or for a detected tool-result continuation regardless of token count. Missing write counters count the sent prompt as written; explicit zero is respected. `session` ignores warmth and token thresholds; `turn` prefers only warm continuations. All modes expire 24 hours after the last recorded success. `src/server/request-log-cache-affinity.ts`, called by final logging, records against the serving provider and entry rather than a combo display label, and releases only the matching serving provider/entry binding on this request's upstream auth/quota refusal or exact Anthropic billing evidence. Proxy-generated (`terminalSource: "synthetic"`) refusals and another request's cooldown preserve the binding. Hermes log-only ids are excluded from lanes, including combo children. The process-local store is bounded at 512 lanes, swept on expiry/configuration changes, and exposes only bounded `cacheAffinity`/`cacheAffinityReason` log outcomes. | | OAuth account failover | `src/oauth/generic-account-failover.ts`, `src/oauth/anthropic-routing.ts`, `src/server/responses/antigravity-validation-refusal.ts`, `src/adapters/antigravity-refusal.ts` | Reactive pre-output 429 recovery is presence-driven with 2+ eligible accounts. Anthropic also handles exact revoked-token 401s under its [reauthentication contract](../providers/anthropic-account-pool.md#revoked-oauth-access-token-recovery) and recovers classified pre-output account-entitlement 403s under its [account-pool contract](../providers/anthropic-account-pool.md#account-entitlement-refusal-recovery), and records the refused account's cooldown even when the request has no retry sends left. Primary Antigravity inference through the Google adapter main dispatch may send once on a live sibling after a replayed 401, a terminal credential-refresh failure that marks the account for reauthentication, or a normalized structured `VALIDATION_REQUIRED` 403, within the existing budget. A bounded message-worded verification 403 persists a generation-fenced `verify_account` cause in `src/oauth/store.ts` and `src/oauth/health.ts`, including singleton and last-account refusals. It reads as `needs-reauth(verify)` in `ocx account list`; silent refresh preserves quarantine, while explicit re-login clears it and retires its failover cooldown. Quarantine does not require a retry slot; if persisting it fails, dispatch exits this adapter exchange’s recovery loop and preserves the original bounded 403 without another recovery send in that exchange; enclosing combo and policy fallback rules remain unchanged. A preserved original 403 after sibling build or pre-send admission failure also exits recovery through common bounded error formatting and the combo failure callback; newly constructed failure responses keep their own delivery. Successful sibling replay uses the captured pool activation and the existing send budget. The 401 and structured 403 sibling paths share one request guard; unrelated 403s and transient refresh failures do not rotate. The fetch web-search sidecar accepts only complete bounded structured `VALIDATION_REQUIRED` 403s through the Google normalizer, carries the sibling token/project without a durable reauthentication mark, and uses adapter-owned physical-send accounting; its contract is [Account refusal and rotation boundaries](responses-failover.md#account-refusal-and-rotation-boundaries). Continuations, native passthrough and image sidecars keep their existing behavior. Generic OAuth accounts can be operator-paused; paused accounts are excluded from selection, failover, and proactive Token Guardian refresh. If every account is paused, the request fails with 403 rather than as a login failure. Pool and `oauthAccountFailover` flags govern proactive routing, not the reactive retry: a disabled Anthropic pool recovers through quota ordering rather than its dormant strategy, a per-provider `enabled` beats the global default in either direction, and a non-positive fill-first threshold disables proactive usage-based rotation. Kiro's `kiroAutoSelection` projects the same candidate eligibility into routing and account-list status; unknown evidence stays eligible, while pause, reauth, suspension, cooldown, and confirmed exhaustion have closed reasons. An active singleton or a pool excluded only from automatic Kiro selection may still attempt its active account; an explicitly paused active account is never dispatched, and an all-paused pool fails with 403. A configured Kiro capacity cap can also time out. Kiro's optional process-local account lease is released at response completion or cancellation; other providers retain their admission path. | | OAuth login callback (inbound) | `src/oauth/callback-server.ts` | Every response, including non-callback 404s, closes its connection so a pooled socket cannot deliver a later login to a retired flow on the same callback port. | | Alibaba regions | `src/providers/alibaba-region-backup.ts`, `src/providers/alibaba-region-migration.ts`, `src/providers/alibaba-region-startup.ts` | Region migration backs up before rewriting and is idempotent across restarts. | diff --git a/structure/transports/policy-fallback.md b/structure/transports/policy-fallback.md index 57fdc2e5f32..30f69113fd7 100644 --- a/structure/transports/policy-fallback.md +++ b/structure/transports/policy-fallback.md @@ -23,6 +23,10 @@ their existing budgets. Local skips preserve the last upstream failure and its l route identity, usage, active attempt and spend tracker; preparation-only fields are discarded. Completed physical attempt rows remain intact, and final logging settles the last actual send's usage. The initial decision remains the selection evidence while physical attempts record execution. +Candidate hops clear the previous candidate's cache-affinity outcome, kept entry, serving +credential, model, and applicability before dispatch. A single-key or opted-out destination +cannot inherit a prior candidate's `kept` outcome. Local skips restore the failed owner's +context under the same rollback rule. Cancellation, non-replayable responses and committed output keep the [shared replay boundaries](responses-failover.md). A local policy refusal is identified by its diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 9b05c5b4097..2e2bbbf08d4 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -232,13 +232,15 @@ within their route; neither route falls through to the other. See ### Pre-dispatch API-key pool pick -Key-auth routes with a configured `apiKeyPoolStrategy` and two or more pool entries pick a -warm key before the first send (`selectProactiveApiKeyTransport` in -`src/providers/key-failover.ts`). The pick is inert unless that strategy is set and the -committed key is already cooling or missing from the pool: a healthy committed key, including -a manual selection, is left alone and the common path returns null without a config write. -`forgetApiKeyRotationCursor` drops the process-local round-robin cursor when the operator -edits the pool, so a later pick cannot second-guess that choice. +Key-auth routes with a configured `apiKeyPoolStrategy` and two or more pool entries use +`selectProactiveApiKeyTransport` in `src/providers/key-failover.ts`. Strategy selection +only replaces a cooling or missing committed key; healthy keys need no config write. +`forgetApiKeyRotationCursor` resets the strategy cursor after operator pool edits. +Before strategies, request-local cache affinity may retain the lane's serving entry +without changing the committed selection (see [`inventory.md`](./inventory.md)). +Local native Chat/Messages/compact pacing and send-budget refusals, and adapter-dispatch +send-budget refusals, carry `terminalSource: "synthetic"`; finalization preserves their binding +and releases only a matching credential's upstream refusal. HTTP 2xx structured `failed` results neither create nor refresh affinity; successful results retain ordinary serving-entry recording. On the shared Responses path the assignment lands in `src/server/responses/core.ts` immediately before `resolveProviderTransport`. `route.provider` is copied into diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 23a067d5f98..f2947ddf0ec 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -365,6 +365,7 @@ "api-key-model-scope.test.ts": "server", "api-key-scope-alpha-search.test.ts": "server", "api-key-scope-audio.test.ts": "server", "api-key-scope-images.test.ts": "server", "api-key-scope-live.test.ts": "server", "api-key-selection-capture.test.ts": "providers", + "cache-affinity.test.ts": "providers", "api-keys-routes.test.ts": "server", "api-storage-cleanup.test.ts": "storage", "api-storage-policy-already-running.test.ts": "storage", "api-storage-policy-mutation-busy.test.ts": "storage", "api-storage-policy-put-race.test.ts": "storage", "api-storage-policy-run.test.ts": "storage", @@ -1310,6 +1311,7 @@ "server-antigravity-web-search-validation.test.ts": "server", "server-google-antigravity-oauth-429-budget.test.ts": "server", "server-gui-bundle-freshness.test.ts": "server", "server-images-bodyless-content-length.test.ts": "server", "server-images-pool-admission.test.ts": "server", "server-images.test.ts": "server", "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", + "cache-affinity-e2e.test.ts": "server", "server-jev-combo-e2e.test.ts": "server", "server-kiro-completion-e2e.test.ts": "server", "combo-requested-effort.test.ts": "server", "server-kiro-device-login.test.ts": "server", "server-kiro-oauth-401-replay.test.ts": "server", diff --git a/tests/oauth/state-store-sweeper.test.ts b/tests/oauth/state-store-sweeper.test.ts index 45962eea6ea..b06eceedd48 100644 --- a/tests/oauth/state-store-sweeper.test.ts +++ b/tests/oauth/state-store-sweeper.test.ts @@ -156,6 +156,7 @@ describe("state-store sweeper", () => { "pool-rotation", "combo-rotation", "combo-session-recall", + "provider-cache-affinity", "guardian-backoff", "codex-reauth", "oauth-reauth", diff --git a/tests/providers/cache-affinity.test.ts b/tests/providers/cache-affinity.test.ts new file mode 100644 index 00000000000..0602d505c96 --- /dev/null +++ b/tests/providers/cache-affinity.test.ts @@ -0,0 +1,501 @@ +import { afterEach, describe, expect, test, beforeEach } from "bun:test"; +import { + clearProviderCacheAffinityForTests, + consultProviderCacheAffinity, + recordProviderCacheAffinity, + releaseProviderCacheAffinity, + sweepExpiredProviderCacheAffinity, + reconcileProviderCacheAffinity, + CACHE_AFFINITY_WORTH_TOKENS, +} from "../../src/providers/cache-affinity"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import type { GenerationContext } from "../../src/lib/state-store-sweeper"; +import { rotateKeyOn429, rotateProviderTransportOn429, selectProactiveApiKeyTransport, clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../src/providers/key-failover"; +import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; +import { resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { loadConfig, saveConfig } from "../../src/config"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { setKeyCooldown } from "../../src/providers/api-key-cooldowns"; +import { routedProviderConfig } from "../../src/providers/routed-config"; + +const NOW = 1_700_000_000_000; +function provider(overrides: Partial = {}): OcxProviderConfig { + return { + adapter: "openai-chat", + baseUrl: "https://example.test/v1", + apiKey: "key-a", + apiKeyPool: [ + { id: "a", key: "key-a" }, + { id: "b", key: "key-b" }, + ], + ...overrides, + } as OcxProviderConfig; +} + +function consult(lane: string, p: OcxProviderConfig, model = "m1", now = NOW) { + return consultProviderCacheAffinity({ lane, providerName: "kimi", provider: p, model, now }); +} + +describe("provider cache affinity store", () => { + beforeEach(() => clearProviderCacheAffinityForTests()); + + test("no binding before any recorded turn", () => { + expect(consult("lane-1", provider())).toBeNull(); + }); + + test("a turn with a worthy cache write binds, and auto keeps it while warm", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 0, cacheWrite: CACHE_AFFINITY_WORTH_TOKENS, inputTokens: 5000, now: NOW, + }); + const hit = consult("lane-1", provider(), "m1", NOW + 60_000); + expect(hit?.entryId).toBe("a"); + expect(hit?.reason).toBe("cache"); + }); + + test("below the worth threshold, auto does not keep across turns but session does", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 10, cacheWrite: 20, inputTokens: 100, now: NOW, + }); + expect(consult("lane-1", provider(), "m1", NOW + 120_000)).toBeNull(); + const sessional = provider({ cacheAffinity: "session" }); + expect(consult("lane-1", sessional, "m1", NOW + 120_000)?.entryId).toBe("a"); + }); + + test("silent-cache provider: unreported cache counts the sent prompt as written", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + inputTokens: CACHE_AFFINITY_WORTH_TOKENS + 1, now: NOW, + }); + expect(consult("lane-1", provider(), "m1", NOW + 60_000)?.entryId).toBe("a"); + }); + + test("a cold binding exerts no pull in auto mode", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + expect(consult("lane-1", provider(), "m1", NOW + 6 * 60_000)).toBeNull(); + }); + + test("long-retention writes stay warm for an hour", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 0, cacheWrite: 4000, cacheWriteLong: 4000, inputTokens: 4000, now: NOW, + }); + expect(consult("lane-1", provider(), "m1", NOW + 30 * 60_000)?.entryId).toBe("a"); + expect(consult("lane-1", provider(), "m1", NOW + 61 * 60_000)).toBeNull(); + }); + + test("turn mode keeps only detected continuations, regardless of worth", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 0, cacheWrite: 0, inputTokens: 10, now: NOW, + }); + const turn = provider({ cacheAffinity: "turn" }); + // A new user turn is not a continuation, however fresh the binding is. + expect(consult("lane-1", turn, "m1", NOW + 60_000)).toBeNull(); + const hit = consultProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", provider: turn, model: "m1", continuation: true, now: NOW + 60_000, + }); + expect(hit?.entryId).toBe("a"); + expect(hit?.reason).toBe("turn"); + // Even a continuation cannot use a cache that has gone cold. + expect(consultProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", provider: turn, model: "m1", continuation: true, now: NOW + 6 * 60_000, + })).toBeNull(); + }); + + test("auto keeps a detected continuation before any cache evidence exists", () => { + recordProviderCacheAffinity({ + lane: "lane-1", + providerName: "kimi", + entryId: "a", + model: "m1", + cacheRead: 0, + cacheWrite: 0, + inputTokens: 10, + now: NOW, + }); + const hit = consultProviderCacheAffinity({ + lane: "lane-1", + providerName: "kimi", + provider: provider(), + model: "m1", + continuation: true, + now: NOW + 60_000, + }); + expect(hit?.reason).toBe("continuation"); + expect(consult("lane-1", provider(), "m1", NOW + 60_000)).toBeNull(); + }); + + test("reads-only reporters bind: no write counter means the sent prompt counts as written", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 0, inputTokens: CACHE_AFFINITY_WORTH_TOKENS + 1, now: NOW, + }); + expect(consult("lane-1", provider(), "m1", NOW + 60_000)?.entryId).toBe("a"); + // An explicitly reported zero write count is believed instead. + recordProviderCacheAffinity({ + lane: "lane-2", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 0, cacheWrite: 0, inputTokens: CACHE_AFFINITY_WORTH_TOKENS + 1, now: NOW, + }); + expect(consult("lane-2", provider(), "m1", NOW + 60_000)).toBeNull(); + }); + + test("reconcile drops removed providers, disabled providers, and removed credentials", () => { + const sessional = provider({ cacheAffinity: "session" }); + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + recordProviderCacheAffinity({ + lane: "lane-2", providerName: "gone", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + const context = { + generation: 1, + providerNames: new Set(["kimi"]), + comboIds: new Set(), comboTargets: new Set(), + codexAccountIds: new Set(), oauthAccountKeys: new Set(), + configRoots: new Set(), + providerKeyEntryIds: new Map([["kimi", new Set(["b"])]]), + } as unknown as GenerationContext; + // lane-1's entry "a" left the pool; lane-2's provider left the config. + expect(reconcileProviderCacheAffinity(context)).toBe(2); + expect(consult("lane-1", sessional, "m1", NOW + 1000)).toBeNull(); + recordProviderCacheAffinity({ + lane: "lane-3", providerName: "kimi", entryId: "b", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + expect(reconcileProviderCacheAffinity({ + ...context, generation: 2, disabledProviderNames: new Set(["kimi"]), + })).toBe(1); + expect(consult("lane-3", sessional, "m1", NOW + 1000)).toBeNull(); + }); + + test("capacity evicts the least recently recorded lane; oversize entries are refused", () => { + const sessional = provider({ cacheAffinity: "session" }); + for (let i = 0; i < 513; i++) { + recordProviderCacheAffinity({ + lane: `lane-${i}`, providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW + i, + }); + } + expect(consult("lane-0", sessional, "m1", NOW + 1000)).toBeNull(); + expect(consult("lane-512", sessional, "m1", NOW + 1000)?.entryId).toBe("a"); + recordProviderCacheAffinity({ + lane: "lane-big", providerName: "kimi", entryId: "a", model: `m-${"x".repeat(2000)}`, + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + expect(consultProviderCacheAffinity({ + lane: "lane-big", providerName: "kimi", provider: sessional, model: `m-${"x".repeat(2000)}`, now: NOW + 1000, + })).toBeNull(); + }); + + test("off mode never consults an existing binding", () => { + const off = provider({ cacheAffinity: "off" }); + recordProviderCacheAffinity({ + lane: "lane-1", + providerName: "kimi", + entryId: "a", + model: "m1", + cacheRead: 9000, + cacheWrite: 9000, + inputTokens: 9000, + now: NOW, + }); + expect(consult("lane-1", off, "m1", NOW + 1000)).toBeNull(); + }); + + test("a different model or provider does not inherit the binding", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + expect(consult("lane-1", provider(), "other-model", NOW + 1000)).toBeNull(); + expect(consultProviderCacheAffinity({ + lane: "lane-1", providerName: "other", provider: provider(), model: "m1", now: NOW + 1000, + })).toBeNull(); + }); + + test("release forgets the binding; sweep expires old entries", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + releaseProviderCacheAffinity("lane-1"); + expect(consult("lane-1", provider(), "m1", NOW + 1000)).toBeNull(); + + recordProviderCacheAffinity({ + lane: "lane-2", providerName: "kimi", entryId: "b", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: NOW, + }); + expect(sweepExpiredProviderCacheAffinity(NOW + 25 * 60 * 60_000)).toBe(1); + expect(consult("lane-2", provider(), "m1", NOW + 25 * 60 * 60_000)).toBeNull(); + }); + + test("release is scoped to the failed provider and pool entry", () => { + recordProviderCacheAffinity({ + lane: "lane-1", providerName: "kimi", entryId: "a", model: "m1", + cacheWrite: 9000, now: NOW, + }); + expect(releaseProviderCacheAffinity("lane-1", "other", "a")).toBe(false); + expect(consult("lane-1", provider())?.entryId).toBe("a"); + expect(releaseProviderCacheAffinity("lane-1", "kimi", "b")).toBe(false); + expect(consult("lane-1", provider())?.entryId).toBe("a"); + expect(releaseProviderCacheAffinity("lane-1", "kimi", "a")).toBe(true); + expect(releaseProviderCacheAffinity("lane-1", "kimi", "a")).toBe(false); + expect(releaseProviderCacheAffinity(undefined)).toBe(false); + expect(consult("lane-1", provider())).toBeNull(); + }); + + test("single-credential providers never consult an existing binding", () => { + const single = provider({ apiKeyPool: [{ id: "a", key: "key-a" }] }); + recordProviderCacheAffinity({ + lane: "lane-1", + providerName: "kimi", + entryId: "a", + model: "m1", + cacheRead: 9000, + cacheWrite: 0, + inputTokens: 9000, + now: NOW, + }); + expect(consult("lane-1", single, "m1", NOW + 1000)).toBeNull(); + }); +}); + +describe("cache affinity at the proactive selection seam", () => { + beforeEach(() => { + clearProviderCacheAffinityForTests(); + clearKeyCooldowns(); + }); + + function configWith(committedKey: string): OcxConfig { + return { + providers: { + kimi: provider({ apiKey: committedKey }), + }, + } as unknown as OcxConfig; + } + + function bind(lane: string, entryId: string) { + recordProviderCacheAffinity({ + lane, providerName: "kimi", entryId, model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: Date.now(), + }); + } + + test("selection returns the bound credential's transport, marked request-local", () => { + bind("lane-1", "a"); + const config = configWith("key-b"); + const routed = routedProviderConfig("kimi", config.providers.kimi!); + const picked = selectProactiveApiKeyTransport(config, "kimi", routed, undefined, Date.now(), + { lane: "lane-1", model: "m1", log: {} }); + expect(picked?.apiKey).toBe("key-a"); + expect(picked?._apiKeyAttempt?.cacheAffinity).toBe(true); + // The global committed selection is untouched: affinity is per-request. + expect(config.providers.kimi!.apiKey).toBe("key-b"); + }); + + test("a cooling bound credential falls through to ordinary selection and keeps its binding", () => { + bind("lane-1", "a"); + setKeyCooldown("kimi", "a", Date.now() + 60_000); + const config = configWith("key-b"); + const routed = routedProviderConfig("kimi", config.providers.kimi!); + const picked = selectProactiveApiKeyTransport(config, "kimi", routed, undefined, Date.now(), + { lane: "lane-1", model: "m1", log: {} }); + expect(picked).toBeNull(); + clearKeyCooldowns(); + const after = selectProactiveApiKeyTransport(config, "kimi", routed, undefined, Date.now(), + { lane: "lane-1", model: "m1", log: {} }); + expect(after?.apiKey).toBe("key-a"); + }); + + test("a removed bound credential releases the binding", () => { + bind("lane-1", "a"); + const config = configWith("key-b"); + config.providers.kimi!.apiKeyPool = [{ id: "b", key: "key-b" }, { id: "c", key: "key-c" }]; + const routed = routedProviderConfig("kimi", config.providers.kimi!); + expect(selectProactiveApiKeyTransport(config, "kimi", routed, undefined, Date.now(), + { lane: "lane-1", model: "m1", log: {} })).toBeNull(); + // Binding gone: even with the entry restored, no preference remains. + config.providers.kimi!.apiKeyPool = provider().apiKeyPool; + expect(selectProactiveApiKeyTransport(config, "kimi", routed, undefined, Date.now(), + { lane: "lane-1", model: "m1", log: {} })).toBeNull(); + }); +}); + +describe("cache affinity against committed-key movement (persisted config)", () => { + let home = ""; + let previousHome: string | undefined; + + beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-cache-affinity-fence-")); + process.env.OPENCODEX_HOME = home; + forgetApiKeyRotationCursor(); + clearProviderCacheAffinityForTests(); + clearKeyCooldowns(); + resetProviderRequestPacingForTest(); + }); + + afterEach(() => { + clearKeyCooldowns(); + clearProviderCacheAffinityForTests(); + resetProviderRequestPacingForTest(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (home) removeTreeWithRetry(home); + }); + + function persistedConfig(overrides: Partial = {}): OcxConfig { + const config = { + port: 0, + providers: { + "affinity-pool": provider({ + apiKey: "key-b", + apiKeyPool: [ + { id: "a", key: "key-a" }, + { id: "b", key: "key-b" }, + { id: "c", key: "key-c" }, + ], + ...overrides, + }), + }, + } as unknown as OcxConfig; + saveConfig(config); + return config; + } + + function bindA(lane = "lane-1") { + recordProviderCacheAffinity({ + lane, providerName: "affinity-pool", entryId: "a", model: "m1", + cacheRead: 9000, cacheWrite: 0, inputTokens: 9000, now: Date.now(), + }); + } + + function pick(config: OcxConfig, lane = "lane-1") { + const routed = routedProviderConfig("affinity-pool", config.providers["affinity-pool"]!); + return selectProactiveApiKeyTransport(config, "affinity-pool", routed, undefined, Date.now(), + { lane, model: "m1", log: {} }); + } + + test.each([false, true])("round-robin failover stays request-local with affinity=%s", (affinity) => { + const config = persistedConfig({ + apiKey: "key-a", apiKeySelectionRevision: "original", + requestPacing: { enabled: true, keySelection: "round-robin" }, + }); + if (affinity) bindA(); + const selected = pick(config)!; + expect(selected.apiKey).toBe("key-a"); + const recovered = rotateProviderTransportOn429(config, "affinity-pool", selected, { attemptedKey: "key-a" }); + expect(recovered?.apiKey).toBe("key-b"); + expect(recovered?._apiKeyAttempt?.distributed).toBe(true); + expect(config.providers["affinity-pool"]!.apiKey).toBe("key-a"); + expect(loadConfig().providers["affinity-pool"]!.apiKey).toBe("key-a"); + expect(loadConfig().providers["affinity-pool"]!.apiKeySelectionRevision).toBe("original"); + }); + + test("round-robin affinity survives revision movement through re-read and failover", () => { + const config = persistedConfig({ + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + apiKeySelectionRevision: "before", + }); + bindA(); + const selected = pick(config)!; + expect(selected.apiKey).toBe("key-a"); + expect(selected._apiKeyAttempt?.distributed).toBe(true); + config.providers["affinity-pool"]!.apiKey = "key-c"; + config.providers["affinity-pool"]!.apiKeySelectionRevision = "after"; + saveConfig(config); + expect(providerApiKeySelectionIsCurrent(config, "affinity-pool", selected)).toBe(true); + const rebuilt = resolveCurrentProviderApiKeyTransport(config, "affinity-pool", selected)!; + expect(rebuilt.apiKey).toBe("key-a"); + expect(rebuilt._apiKeyAttempt?.distributed).toBe(true); + const recovered = rotateProviderTransportOn429(config, "affinity-pool", rebuilt, { attemptedKey: "key-a" }); + expect(recovered?.apiKey).toBe("key-b"); + expect(loadConfig().providers["affinity-pool"]!.apiKey).toBe("key-c"); + expect(loadConfig().providers["affinity-pool"]!.apiKeySelectionRevision).toBe("after"); + }); + + test.each(["disabled", "removed", "replaced"])("round-robin affinity fences a %s entry before failover", (change) => { + const config = persistedConfig({ requestPacing: { enabled: true, keySelection: "round-robin" } }); + bindA(); + const selected = pick(config)!; + const entry = config.providers["affinity-pool"]!.apiKeyPool![0]!; + if (change === "disabled") entry.disabled = true; + else if (change === "removed") config.providers["affinity-pool"]!.apiKeyPool!.shift(); + else entry.key = "replacement-key"; + saveConfig(config); + expect(providerApiKeySelectionIsCurrent(config, "affinity-pool", selected)).toBe(false); + expect(rotateProviderTransportOn429(config, "affinity-pool", selected, { attemptedKey: "key-a" })).toBeNull(); + expect(loadConfig().providers["affinity-pool"]!.apiKey).toBe("key-b"); + }); + + test("an affinity pick survives another conversation's rotation (revision bump)", () => { + const config = persistedConfig(); + bindA(); + const affinityTransport = pick(config); + expect(affinityTransport?.apiKey).toBe("key-a"); + // Another conversation's 429 rotates the committed key b -> c, which bumps + // the selection revision. The pin is fenced on entry liveness, not on that + // revision, so dispatch re-reads still honor it. + expect(rotateKeyOn429(config, "affinity-pool", null, Date.now(), "key-b")?.apiKey).toBe("key-c"); + expect(providerApiKeySelectionIsCurrent(config, "affinity-pool", affinityTransport!)).toBe(true); + const rebuilt = resolveCurrentProviderApiKeyTransport(config, "affinity-pool", affinityTransport!); + expect(rebuilt?.apiKey).toBe("key-a"); + expect(providerApiKeySelectionIsCurrent(config, "affinity-pool", rebuilt!)).toBe(true); + expect(resolveCurrentProviderApiKeyTransport(config, "affinity-pool", rebuilt!)?.apiKey).toBe("key-a"); + }); + + test("a 429 on the pinned key still rotates after the committed key moved", () => { + const config = persistedConfig(); + bindA(); + const affinityTransport = pick(config); + expect(affinityTransport?.apiKey).toBe("key-a"); + // Committed key moves b -> c (revision bump) before the pinned send fails. + expect(rotateKeyOn429(config, "affinity-pool", null, Date.now(), "key-b")?.apiKey).toBe("key-c"); + const rotated = rotateProviderTransportOn429(config, "affinity-pool", affinityTransport!, { + attemptedKey: "key-a", + attemptedSelection: affinityTransport!._apiKeyAttempt, + now: Date.now(), + }); + // b is cooling from the earlier rotation; the spare is c, not a 429 surface. + expect(rotated?.apiKey).toBe("key-c"); + }); + + test("a bound key inside a long minimum interval falls through until its slot approaches", async () => { + const config = persistedConfig({ + requestPacing: { enabled: true, keys: { minIntervalMs: 120_000 } }, + }); + bindA(); + const rowA = { ...config.providers["affinity-pool"]!, apiKey: "key-a" }; + // One start on key-a pushes its next permitted start two minutes out: + // past what the send queue tolerates, so affinity must not pin to it. + (await waitForProviderRequestSlot("affinity-pool", rowA, "m1")).release(); + expect(pick(config)).toBeNull(); + resetProviderRequestPacingForTest(); + expect(pick(config)?.apiKey).toBe("key-a"); + }); + + test("a pacing-saturated bound key falls through to ordinary selection", async () => { + const config = persistedConfig({ + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } }, + }); + bindA(); + const rowA = { ...config.providers["affinity-pool"]!, apiKey: "key-a" }; + const held = await waitForProviderRequestSlot("affinity-pool", rowA, "m1"); + try { + expect(pick(config)).toBeNull(); + } finally { + held.release(); + } + expect(pick(config)?.apiKey).toBe("key-a"); + }); +}); diff --git a/tests/routing/routing-policy-fallback.test.ts b/tests/routing/routing-policy-fallback.test.ts index a7a1370da3f..e7bdf9e67de 100644 --- a/tests/routing/routing-policy-fallback.test.ts +++ b/tests/routing/routing-policy-fallback.test.ts @@ -101,7 +101,13 @@ describe("policy candidate fallback", () => { }, ); - for (const hardCode of ["upstream_no_response", "origin_rejected", "origin-rejected", " ORIGIN-REJECTED ", "cyber_policy"]) { + for (const hardCode of [ + "upstream_no_response", + "origin_rejected", + "origin-rejected", + " ORIGIN-REJECTED ", + "cyber_policy", + ]) { test.each(["root", "nested"])(`policy preserves ${hardCode} at %s beside a competing code`, async location => { let calls = 0; const refusal = "The 'gpt-6-astra' model is not supported when using Codex with a ChatGPT account."; @@ -216,6 +222,39 @@ describe("policy candidate fallback", () => { ]); expect(snapshots[1]).toBe(snapshots[0]); }); + test("a policy hop does not persist the previous candidate's cache-affinity outcome", async () => { + const log: RequestLogContext = { model: "", provider: "" }; + let calls = 0; + const response = await handleResponsesWithPolicyFallback( + request(), + fixtureConfig(), + log, + {}, + { + runCore: async (req, _config, context, options) => { + options.onRequestBodyParsed?.(await req.json()); + context.routeDecision = policyTrace(); + if (++calls === 1) { + context.cacheAffinity = "kept"; + context.cacheAffinityReason = "cache"; + context.affinityKeptEntryId = "old-key"; + context.cacheAffinityApplicable = true; + context.affinityKeyEntryId = "old-key"; + return Response.json({ error: { message: "retry next candidate" } }, { status: 503 }); + } + context.provider = "provider-b"; + context.cacheAffinityApplicable = false; + return Response.json({ status: "completed" }); + }, + }, + ); + expect(response.status).toBe(200); + expect(calls).toBe(2); + expect(log.cacheAffinity).toBeUndefined(); + expect(log.cacheAffinityReason).toBeUndefined(); + expect(log.affinityKeptEntryId).toBeUndefined(); + expect(log.affinityKeyEntryId).toBeUndefined(); + }); test("ranks only eligible untried candidates by score and stable original order", () => { const ranked = rankPolicyFallbackCandidates(policyTrace(), new Set(["provider-a\u0000model-a"])); diff --git a/tests/server/cache-affinity-e2e.test.ts b/tests/server/cache-affinity-e2e.test.ts new file mode 100644 index 00000000000..8841618e253 --- /dev/null +++ b/tests/server/cache-affinity-e2e.test.ts @@ -0,0 +1,662 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { handleResponses, handleResponsesCompact } from "../../src/server/responses"; +import { startServer } from "../../src/server"; +import { handleChatCompletions } from "../../src/server/chat-completions"; +import { handleClaudeMessages } from "../../src/server/claude-messages"; +import { loadConfig, saveConfig } from "../../src/config"; +import { addFinalRequestLog, inspectResponseLogJson, nextRequestLogId, type RequestLogContext } from "../../src/server/request-log"; +import { boundEntryIdForLane, clearProviderCacheAffinityForTests, recordProviderCacheAffinity } from "../../src/providers/cache-affinity"; +import { RequestPacingQueueOverloadError } from "../../src/providers/request-pacing"; +import { SendBudgetExhaustedError } from "../../src/lib/upstream-retry"; +import { protocolTraceForRequest } from "../../src/protocols/trace"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import type { OcxConfig } from "../../src/types"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { setProviderKeychainEntryFactoryForTests } from "../../src/providers/key-store"; +import { setKeyCooldown } from "../../src/providers/api-key-cooldowns"; +import { consultProviderCacheAffinity } from "../../src/providers/cache-affinity"; +import { + fillHermesLogConversationFallback, + sessionLaneIdFromRequest, +} from "../../src/server/request-log-conversation"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { ManagementRequest } from "../helpers/management-auth"; +import { SERVER_BUDGET_MS } from "../helpers/test-budget"; + +const REAL_FETCH = globalThis.fetch; +let testDir = ""; +let previousHome: string | undefined; +const seenAuth: string[] = []; +const failingKeys = new Set(); +// Direct handler calls need the spend-ledger ownership startServer acquires. +const trackedServers = new Set>(); + +function makeConfig(): OcxConfig { + const config = { + port: 0, + providers: { + "affinity-e2e": { + adapter: "openai-chat", + authMode: "key", + baseUrl: "https://upstream.invalid/v1", + apiKey: "key-a", + models: ["test"], + liveModels: false, + apiKeyPool: [ + { id: "a", key: "key-a" }, + { id: "b", key: "key-b" }, + ], + }, + "affinity-chat": { + adapter: "openai-chat", + authMode: "key", + baseUrl: "https://upstream.invalid/v1", + apiKey: "key-a", + models: ["test"], + liveModels: false, + apiKeyPool: [ + { id: "a", key: "key-a" }, + { id: "b", key: "key-b" }, + ], + }, + "affinity-compact": { + adapter: "openai-responses", + authMode: "key", + baseUrl: "https://upstream.invalid/v1", + apiKey: "key-a", + models: ["test"], + liveModels: false, + apiKeyPool: [ + { id: "a", key: "key-a" }, + { id: "b", key: "key-b" }, + ], + }, + "affinity-msg": { + adapter: "anthropic", + authMode: "key", + baseUrl: "https://upstream.invalid", + apiKey: "key-a", + models: ["claude-test"], + liveModels: false, + apiKeyPool: [ + { id: "a", key: "key-a" }, + { id: "b", key: "key-b" }, + ], + }, + }, + } as unknown as OcxConfig; + saveConfig(config); + return loadConfig(); +} + +async function turn( + config: OcxConfig, + threadId: string, + model = "affinity-e2e/test", +): Promise { + saveConfig(config); + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleResponses( + new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", "thread-id": threadId }, + body: JSON.stringify({ model, input: "hello", stream: false, store: false }), + }), + config, + logCtx, + {}, + ); + expect(response.status).toBe(200); + await response.text(); + // The ingress layer owns finalization in production; this harness stands in for it. + addFinalRequestLog(nextRequestLogId(), Date.now(), logCtx, response.status); + return logCtx; +} + +describe("provider cache affinity (end-to-end)", () => { + beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + testDir = mkdtempSync(join(tmpdir(), "ocx-cache-affinity-e2e-")); + process.env.OPENCODEX_HOME = testDir; + saveConfig(makeConfig()); + trackedServers.add(startServer(0)); + clearKeyCooldowns(); + clearProviderCacheAffinityForTests(); + seenAuth.length = 0; + failingKeys.clear(); + globalThis.fetch = (async (_url: unknown, init?: RequestInit) => { + const headers = new Headers(init?.headers); + const auth = headers.get("authorization") ?? (headers.get("x-api-key") ? `Bearer ${headers.get("x-api-key")}` : ""); + seenAuth.push(auth); + const key = auth.replace("Bearer ", ""); + if (failingKeys.has(key)) { + return new Response(JSON.stringify({ error: { message: "rate limited" } }), { + status: 429, headers: { "content-type": "application/json" }, + }); + } + if (_url.toString().includes("/responses")) { + return new Response(JSON.stringify({ + output: [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "fixture reply" }] }], + usage: { input_tokens: 5000, output_tokens: 10, input_tokens_details: { cached_tokens: 4000 } }, + }), { status: 200, headers: { "content-type": "application/json" } }); + } + if (_url.toString().includes("/messages")) { + // The native Messages path consumes an Anthropic SSE stream. + const sse = [ + 'event: message_start\ndata: {"type":"message_start","message":{"id":"msg-test","type":"message","role":"assistant","model":"claude-test","content":[],"stop_reason":null,"usage":{"input_tokens":5000,"output_tokens":1,"cache_read_input_tokens":4000,"cache_creation_input_tokens":0}}}', + 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"ok"}}', + 'event: content_block_stop\ndata: {"type":"content_block_stop","index":0}', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":5}}', + 'event: message_stop\ndata: {"type":"message_stop"}', + ].join("\n\n") + "\n\n"; + return new Response(sse, { status: 200, headers: { "content-type": "text/event-stream" } }); + } + return new Response(JSON.stringify({ + id: "chatcmpl-test", + object: "chat.completion", + created: 1, + model: "test", + choices: [{ index: 0, message: { role: "assistant", content: "ok" }, finish_reason: "stop" }], + usage: { + prompt_tokens: 5000, + completion_tokens: 5, + prompt_tokens_details: { cached_tokens: 4000, cache_write_tokens: 4000 }, + }, + }), { status: 200, headers: { "content-type": "application/json" } }); + }) as typeof fetch; + }, SERVER_BUDGET_MS); + + afterEach(async () => { + const stopFailures: unknown[] = []; + for (const server of trackedServers) { + trackedServers.delete(server); + try { await server.stop(true); } catch (error) { stopFailures.push(error); } + } + globalThis.fetch = REAL_FETCH; + setProviderKeychainEntryFactoryForTests(null); + clearKeyCooldowns(); + clearProviderCacheAffinityForTests(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (testDir) removeTreeWithRetry(testDir); + if (stopFailures.length > 0) throw new AggregateError(stopFailures, "tracked server stop failed in afterEach"); + }, SERVER_BUDGET_MS); + + test("a conversation stays on the credential that warmed its cache after the committed key moves", async () => { + const config = makeConfig(); + const first = await turn(config, "conv-1"); + expect(seenAuth).toEqual(["Bearer key-a"]); + expect(first.cacheAffinity).toBe("bound"); + + // Another conversation (or an operator) moved the provider's committed key. + config.providers["affinity-e2e"]!.apiKey = "key-b"; + saveConfig(config); + + const second = await turn(config, "conv-1"); + expect(seenAuth).toEqual(["Bearer key-a", "Bearer key-a"]); + expect(second.cacheAffinity).toBe("kept"); + expect(second.cacheAffinityReason).toBe("cache"); + + // A different conversation is placed by ordinary selection: the moved key. + await turn(config, "conv-2"); + expect(seenAuth.at(-1)).toBe("Bearer key-b"); + }); + + test.each(["env", "keychain"])( + "%s pool references survive affinity dispatch re-reads", + async (kind) => { + const previous = process.env.OCX_AFFINITY_TEST_KEY; + process.env.OCX_AFFINITY_TEST_KEY = "key-a"; + setProviderKeychainEntryFactoryForTests((_service, account) => ({ + getPassword: () => (account.endsWith("/a") ? "key-a" : "key-b"), + setPassword() {}, + deletePassword: () => true, + })); + try { + const config = makeConfig(); + const pool = config.providers["affinity-e2e"]!; + const a = kind === "env" ? "$OCX_AFFINITY_TEST_KEY" : "keychain:affinity-e2e/a"; + pool.apiKey = a; + pool.apiKeyPool = [ + { id: "a", key: a }, + { id: "b", key: "key-b" }, + ]; + await turn(config, "referenced-conversation"); + pool.apiKey = "key-b"; + const second = await turn(config, "referenced-conversation"); + expect(seenAuth).toEqual(["Bearer key-a", "Bearer key-a"]); + expect(second.cacheAffinity).toBe("kept"); + } finally { + if (previous === undefined) delete process.env.OCX_AFFINITY_TEST_KEY; + else process.env.OCX_AFFINITY_TEST_KEY = previous; + } + }, + ); + + test("combo turns retain the serving provider's key binding", async () => { + const config = makeConfig(); + config.combos = { + affinity: { strategy: "failover", targets: [{ provider: "affinity-e2e", model: "test" }] }, + }; + await turn(config, "combo-conversation", "combo/affinity"); + config.providers["affinity-e2e"]!.apiKey = "key-b"; + const second = await turn(config, "combo-conversation", "combo/affinity"); + expect(seenAuth).toEqual(["Bearer key-a", "Bearer key-a"]); + expect(second.provider).toBe("combo"); + expect(second.cacheAffinity).toBe("kept"); + }); + + test.each(["responses", "chat", "messages", "compact", "combo"])( + "a Hermes-only identity is log-only on %s", + async (wire) => { + const config = makeConfig(); + config.combos = { + affinity: { strategy: "failover", targets: [{ provider: "affinity-e2e", model: "test" }] }, + }; + const send = async () => { + saveConfig(config); + const log: RequestLogContext = { model: "", provider: "" }; + const ids = { requestId: nextRequestLogId(), start: Date.now() }; + const headers = { + "content-type": "application/json", + "x-hermes-conversation-id": "hermes-only", + }; + // The HTTP ingress fills this log slot before invoking these handlers. + fillHermesLogConversationFallback(log, new Headers(headers)); + const req = new Request(`http://localhost/v1/${wire}`, { + method: "POST", + headers, + body: JSON.stringify( + wire === "responses" || wire === "compact" || wire === "combo" + ? { + model: + wire === "combo" + ? "combo/affinity" + : `${wire === "compact" ? "affinity-compact" : "affinity-e2e"}/test`, + input: "hello", + stream: false, + } + : { + model: wire === "messages" ? "affinity-msg/claude-test" : "affinity-chat/test", + max_tokens: 16, + messages: [{ role: "user", content: "hello" }], + }, + ), + }); + const response = + wire === "responses" || wire === "combo" + ? await handleResponses(req, config, log, {}) + : wire === "compact" + ? await handleResponsesCompact(req, config, log) + : wire === "chat" + ? await handleChatCompletions(req, config, log, ids) + : await handleClaudeMessages(req, config, log, ids); + expect(response.status).toBe(200); + await response.text(); + if (wire === "responses" || wire === "compact" || wire === "combo") + addFinalRequestLog(ids.requestId, ids.start, log, response.status); + expect(log.conversationId).toBeDefined(); + expect(log.affinityLane).toBeUndefined(); + expect(log.cacheAffinity).toBeUndefined(); + }; + await send(); + for (const provider of Object.values(config.providers)) provider.apiKey = "key-b"; + await send(); + expect(seenAuth).toEqual(["Bearer key-a", "Bearer key-b"]); + }, + ); + + test.each([499, 503])( + "another request's cooldown does not release a binding on status %s", + async (status) => { + const config = makeConfig(); + const log = await turn(config, "unrelated-cooldown"); + setKeyCooldown("affinity-e2e", "a", Date.now() + 60_000); + delete log.cacheAffinity; + delete log.cacheAffinityReason; + addFinalRequestLog(nextRequestLogId(), Date.now(), log, status); + clearKeyCooldowns(); + expect(log.cacheAffinity).not.toBe("released"); + expect( + consultProviderCacheAffinity({ + lane: sessionLaneIdFromRequest(new Headers({ "thread-id": "unrelated-cooldown" })), + providerName: "affinity-e2e", + provider: config.providers["affinity-e2e"]!, + model: "test", + })?.entryId, + ).toBe("a"); + }, + ); + + test.each(["chat", "messages"])("%s local send refusals preserve cache affinity", async (wire) => { + const config = makeConfig(); + config.protocols = { rollout: { managedMessagesNative: true } }; + const providerName = wire === "chat" ? "affinity-chat" : "affinity-msg"; + const model = wire === "chat" ? "test" : "claude-test"; + const lane = sessionLaneIdFromRequest(new Headers({ "thread-id": "send-refusal" })); + recordProviderCacheAffinity({ lane, providerName, entryId: "a", model, cacheWrite: 4000 }); + const fetch = globalThis.fetch; + try { + for (const error of [new RequestPacingQueueOverloadError(providerName, "queue_full", 1), new SendBudgetExhaustedError()]) { + globalThis.fetch = (async () => { throw error; }) as typeof globalThis.fetch; + const log: RequestLogContext = { model: "", provider: "" }; + const request = new Request(`http://localhost/v1/${wire}`, { + method: "POST", headers: { "content-type": "application/json", "thread-id": "send-refusal" }, + body: JSON.stringify({ model: `${providerName}/${model}`, max_tokens: 16, messages: [{ role: "user", content: "hi" }] }), + }); + const ids = { requestId: nextRequestLogId(), start: Date.now() }; + const response = wire === "chat" + ? await handleChatCompletions(request, config, log, ids) + : await handleClaudeMessages(request, config, log, ids); + expect(response.status).toBe(429); + await response.text(); + expect(protocolTraceForRequest(log, log.attempts)?.mode).toBe("native"); + expect(log.terminalSource).toBe("synthetic"); + expect(consultProviderCacheAffinity({ lane, providerName, provider: config.providers[providerName]!, model })?.entryId).toBe("a"); + } + } finally { + globalThis.fetch = fetch; + } + }); + + test.each([401, 402, 429, 400])("a synthetic %s preserves the healthy binding", async (status) => { + const config = makeConfig(); + const log = await turn(config, "local-refusal"); + log.terminalSource = "synthetic"; + log.providerAdapter = "anthropic"; + log.upstreamError = "Your credit balance is too low to access the Anthropic API."; + log.upstreamErrorType = "invalid_request_error"; + addFinalRequestLog(nextRequestLogId(), Date.now(), log, status); + expect(log.cacheAffinity).not.toBe("released"); + config.providers["affinity-e2e"]!.apiKey = "key-b"; + const next = await turn(config, "local-refusal"); + expect(next.cacheAffinity).toBe("kept"); + expect(seenAuth.at(-1)).toBe("Bearer key-a"); + }); + + test.each([false, true])("HTTP 200 structured failure does not bind or replace affinity (bound=%s)", (bound) => { + const lane = "structured-failure"; + if (bound) recordProviderCacheAffinity({ lane, providerName: "affinity-e2e", entryId: "a", model: "test", cacheWrite: 4000 }); + const log: RequestLogContext = { + model: "test", provider: "affinity-e2e", affinityLane: lane, + affinityKeyEntryId: "b", affinityProviderName: "affinity-e2e", cacheAffinityApplicable: true, + }; + inspectResponseLogJson(log, JSON.stringify({ + status: "failed", error: { code: "request_send_budget_exhausted", message: "Local send budget exhausted" }, + usage: { input_tokens: 5000, output_tokens: 0 }, + })); + expect(log.observedTerminalStatus).toBe("failed"); + addFinalRequestLog(nextRequestLogId(), Date.now(), log, 200); + expect(boundEntryIdForLane(lane)).toBe(bound ? "a" : undefined); + expect(log.cacheAffinity).toBeUndefined(); + // A completed structured turn still binds the serving entry. + delete log.observedTerminalStatus; + inspectResponseLogJson(log, JSON.stringify({ status: "completed" })); + addFinalRequestLog(nextRequestLogId(), Date.now(), log, 200); + expect(boundEntryIdForLane(lane)).toBe("b"); + expect(log.cacheAffinity).toBe("bound"); + }); + + test.each(["compact-pacing-pick", "compact-pacing-send", "compact-budget", "adapter-initial", "adapter-recovery"])( + "%s local refusal preserves affinity and carries synthetic attribution", async (kind) => { + const config = makeConfig(); + const compact = kind.startsWith("compact"); + const providerName = compact ? "affinity-compact" : "affinity-e2e"; + const provider = config.providers[providerName]!; + // Explicitly exercise native compact rather than the gateway summarizer fallback. + if (compact) provider.supportsNativeCompactEndpoint = true; + if (kind === "compact-pacing-pick") provider.requestPacing = { enabled: true, keySelection: "round-robin" }; + provider.retryOn429 = { attempts: 1, intervalMs: 1000, maxIntervalMs: 1000 }; + saveConfig(config); + const lane = sessionLaneIdFromRequest(new Headers({ "thread-id": "local-responses-refusal" })); + recordProviderCacheAffinity({ lane, providerName, entryId: "a", model: "test", cacheWrite: 4000 }); + if (kind === "compact-pacing-pick") { + for (const entry of provider.apiKeyPool!) setKeyCooldown(providerName, entry.id, Date.now() + 60_000); + } + let sends = 0; + globalThis.fetch = (async (input) => { + sends++; + if (compact) expect(String(input)).toBe("https://upstream.invalid/v1/responses/compact"); + if (kind === "adapter-recovery" && sends === 1) return Response.json({ error: { message: "rate limited" } }, { + status: 429, headers: { "retry-after": "0" }, + }); + if (kind === "compact-pacing-send") throw new RequestPacingQueueOverloadError(providerName, "queue_full", 1); + throw new SendBudgetExhaustedError(); + }) as typeof globalThis.fetch; + const log: RequestLogContext = { model: "", provider: "" }; + const request = new Request(`http://localhost/v1/responses${compact ? "/compact" : ""}`, { + method: "POST", headers: { "content-type": "application/json", "thread-id": "local-responses-refusal" }, + body: JSON.stringify({ model: `${providerName}/test`, input: "hello", stream: false, store: false }), + }); + const response = compact ? await handleResponsesCompact(request, config, log) : await handleResponses(request, config, log, {}); + expect(response.status).toBe(429); + await response.text(); + addFinalRequestLog(nextRequestLogId(), Date.now(), log, response.status); + // Inspect the actual binding before attribution, so release defects fail visibly too. + expect(boundEntryIdForLane(lane)).toBe("a"); + expect(log.cacheAffinity).not.toBe("released"); + expect(log.terminalSource).toBe("synthetic"); + expect(sends).toBe(kind === "compact-pacing-pick" ? 0 : kind === "adapter-recovery" ? 2 : 1); + }, + ); + + test.each(["provider", "entry"])("a refusal from another %s preserves the healthy binding", async (identity) => { + const config = makeConfig(); + const log = await turn(config, "other-credential-refusal"); + if (identity === "provider") log.affinityProviderName = "affinity-chat"; + else log.affinityKeyEntryId = "b"; + addFinalRequestLog(nextRequestLogId(), Date.now(), log, 429); + expect(log.cacheAffinity).toBe("bound"); + expect(log.cacheAffinityReason).toBe("new"); + config.providers["affinity-e2e"]!.apiKey = "key-b"; + const next = await turn(config, "other-credential-refusal"); + expect(next.cacheAffinity).toBe("kept"); + expect(seenAuth.at(-1)).toBe("Bearer key-a"); + }); + + test.each([401, 402, 429])("a real %s releases the serving credential even with a quota error code", async (status) => { + const config = makeConfig(); + const log = await turn(config, "real-refusal"); + log.errorCode = "usage_limit_reached"; + addFinalRequestLog(nextRequestLogId(), Date.now(), log, status); + expect(log.cacheAffinity).toBe("released"); + expect(consultProviderCacheAffinity({ + lane: log.affinityLane, providerName: "affinity-e2e", + provider: config.providers["affinity-e2e"]!, model: "test", + })).toBeNull(); + }); + + test("an exact Anthropic credit refusal releases its binding without a shared cooldown", async () => { + const config = makeConfig(); + const log = await turn(config, "billing-refusal"); + log.providerAdapter = "anthropic"; + log.upstreamError = "Your credit balance is too low to access the Anthropic API."; + log.upstreamErrorType = "invalid_request_error"; + addFinalRequestLog(nextRequestLogId(), Date.now(), log, 400); + expect(log.cacheAffinity).toBe("released"); + expect(log.cacheAffinityReason).toBe("quota"); + expect( + consultProviderCacheAffinity({ + lane: log.affinityLane, + providerName: "affinity-e2e", + provider: config.providers["affinity-e2e"]!, + model: "test", + }), + ).toBeNull(); + }); + + test("provider POST preserves an omitted cache-affinity mode but accepts an explicit replacement", async () => { + const config = makeConfig(); + config.providers["affinity-chat"]!.cacheAffinity = "off"; + config.providers["affinity-chat"]!.baseUrl = "http://127.0.0.1:11434/v1"; + config.providers["affinity-chat"]!.allowPrivateNetwork = true; + saveConfig(config); + const url = new URL("http://localhost/api/providers"); + for (const mode of [undefined, "session"] as const) { + const response = await handleManagementAPI( + new ManagementRequest(url, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + name: "affinity-chat", + provider: { + adapter: "openai-chat", + authMode: "key", + baseUrl: "http://127.0.0.1:11434/v1", + allowPrivateNetwork: true, + liveModels: false, + models: ["test"], + ...(mode ? { cacheAffinity: mode } : {}), + }, + }), + }), + url, + config, + ); + expect(response?.status).toBe(200); + expect(loadConfig().providers["affinity-chat"]!.cacheAffinity).toBe(mode ?? "off"); + } + }); + test("a 429 on the bound key rotates, and the conversation rebinds to the spare", async () => { + const config = makeConfig(); + await turn(config, "conv-1"); + expect(seenAuth).toEqual(["Bearer key-a"]); + + failingKeys.add("key-a"); + const rotated = await turn(config, "conv-1"); + expect(seenAuth.at(-1)).toBe("Bearer key-b"); + expect(rotated.cacheAffinity).toBe("bound"); + + failingKeys.clear(); + const third = await turn(config, "conv-1"); + expect(seenAuth.at(-1)).toBe("Bearer key-b"); + expect(third.cacheAffinity).toBe("kept"); + }); + + test("when every key fails, the cooled binding survives scoped release and the next turn rebinds", async () => { + const config = makeConfig(); + await turn(config, "conv-1"); + failingKeys.add("key-a"); + failingKeys.add("key-b"); + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", "thread-id": "conv-1" }, + body: JSON.stringify({ model: "affinity-e2e/test", input: "hello", stream: false, store: false }), + }), config, logCtx, {}); + await response.text(); + addFinalRequestLog(nextRequestLogId(), Date.now(), logCtx, response.status); + expect(response.status).toBe(429); + expect(logCtx.cacheAffinity).toBe("kept"); + expect(consultProviderCacheAffinity({ + lane: logCtx.affinityLane, providerName: "affinity-e2e", + provider: config.providers["affinity-e2e"]!, model: "test", + })?.entryId).toBe("a"); + + failingKeys.clear(); + seenAuth.length = 0; + const fresh = await turn(config, "conv-1"); + // The failed turn's rotation committed key-b, so fresh placement starts + // there; the cooled binding does not drag the conversation back to a. + expect(seenAuth[0]).toBe("Bearer key-b"); + expect(fresh.cacheAffinity).toBe("bound"); + }); + + test("cacheAffinity off restores plain committed-key behavior", async () => { + const config = makeConfig(); + config.providers["affinity-e2e"]!.cacheAffinity = "off"; + await turn(config, "conv-1"); + config.providers["affinity-e2e"]!.apiKey = "key-b"; + await turn(config, "conv-1"); + expect(seenAuth).toEqual(["Bearer key-a", "Bearer key-b"]); + // Requests made while off must not leave a latent binding to key-a. + config.providers["affinity-e2e"]!.cacheAffinity = "auto"; + const enabled = await turn(config, "conv-1"); + expect(seenAuth.at(-1)).toBe("Bearer key-b"); + expect(enabled.cacheAffinity).toBe("bound"); + }); + + describe("lane wiring on the chat and messages paths", () => { + test("chat completions: the second turn of a conversation stays on its key", async () => { + const config = makeConfig(); + const turn = async (): Promise => { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleChatCompletions(new Request("http://localhost/v1/chat/completions", { + method: "POST", + headers: { "content-type": "application/json", "thread-id": "chat-conv-1" }, + body: JSON.stringify({ model: "affinity-chat/test", messages: [{ role: "user", content: "hi" }] }), + }), config, logCtx, { requestId: nextRequestLogId(), start: Date.now() }); + expect(response.status).toBe(200); + await response.text(); + return logCtx; + }; + const first = await turn(); + expect(seenAuth).toEqual(["Bearer key-a"]); + expect(first.cacheAffinity).toBe("bound"); + config.providers["affinity-chat"]!.apiKey = "key-b"; + saveConfig(config); + const second = await turn(); + expect(seenAuth.at(-1)).toBe("Bearer key-a"); + expect(second.cacheAffinity).toBe("kept"); + }); + + test("responses compact: the second turn of a conversation stays on its key", async () => { + const config = makeConfig(); + const turn = async (): Promise => { + saveConfig(config); + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleResponsesCompact(new Request("http://localhost/v1/responses/compact", { + method: "POST", + headers: { "content-type": "application/json", "thread-id": "compact-conv-1" }, + body: JSON.stringify({ + model: "affinity-compact/test", + input: [{ role: "user", content: "fixture question" }], + stream: false, + }), + }), config, logCtx); + expect(response.status).toBe(200); + await response.text(); + addFinalRequestLog(nextRequestLogId(), Date.now(), logCtx, response.status); + return logCtx; + }; + const first = await turn(); + expect(seenAuth).toEqual(["Bearer key-a"]); + expect(first.cacheAffinity).toBe("bound"); + config.providers["affinity-compact"]!.apiKey = "key-b"; + saveConfig(config); + const second = await turn(); + expect(seenAuth.at(-1)).toBe("Bearer key-a"); + expect(second.cacheAffinity).toBe("kept"); + }); + + test("claude messages: the second turn of a conversation stays on its key", async () => { + const config = makeConfig(); + const turn = async (): Promise => { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", + headers: { "content-type": "application/json", "session_id": "msg-conv-1" }, + body: JSON.stringify({ + model: "affinity-msg/claude-test", max_tokens: 16, + messages: [{ role: "user", content: "hi" }], + }), + }), config, logCtx, { requestId: nextRequestLogId(), start: Date.now() }); + expect(response.status).toBe(200); + await response.text(); + return logCtx; + }; + const first = await turn(); + expect(seenAuth).toEqual(["Bearer key-a"]); + expect(first.cacheAffinity).toBe("bound"); + config.providers["affinity-msg"]!.apiKey = "key-b"; + saveConfig(config); + const second = await turn(); + expect(seenAuth.at(-1)).toBe("Bearer key-a"); + expect(second.cacheAffinity).toBe("kept"); + }); + }); +}); diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index 937589acc3e..0a226b81686 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -1,4 +1,4 @@ -import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { afterEach, beforeEach, describe, expect, setDefaultTimeout, spyOn, test } from "bun:test"; import { mkdtempSync, readFileSync} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -9,6 +9,7 @@ import { flushConfigDirHardeningForTests } from "../../src/config/paths"; import { flushNativeMainStartupReleases } from "../../src/codex/native-profile-startup"; import { flushWindowsSecretAclReapsBeforeRemoval } from "../../src/lib/windows-secret-acl"; import { clearKeyCooldowns, getKeyCooldownUntil, rotateKeyOn429 } from "../../src/providers/key-failover"; +import { clearProviderCacheAffinityForTests } from "../../src/providers/cache-affinity"; import { deriveXaiConvId } from "../../src/providers/xai-transport"; import { clearReasoningReplayCacheForTests, @@ -31,6 +32,10 @@ import { providerRequestPacingStatus, resetProviderRequestPacingForTest, setProv import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport, commitProviderApiKeySelection } from "../../src/providers/api-key-selection"; import { routedProviderConfig } from "../../src/router"; import type { OcxProviderTransport } from "../../src/providers/xai-transport"; +import { SERVER_BUDGET_MS } from "../helpers/test-budget"; + +// These cases bind real servers and sockets, including their startup and cleanup. +setDefaultTimeout(SERVER_BUDGET_MS); let testDir = ""; let previousHome: string | undefined; @@ -60,6 +65,7 @@ beforeEach(() => { clearKeyCooldowns(); clearReasoningReplayCacheForTests(); clearBridgeSearchReplayCacheForTests(); + clearProviderCacheAffinityForTests(); }); afterEach(async () => { @@ -88,6 +94,7 @@ afterEach(async () => { clearKeyCooldowns(); clearReasoningReplayCacheForTests(); clearBridgeSearchReplayCacheForTests(); + clearProviderCacheAffinityForTests(); if (stopFailures.length > 0) throw new AggregateError(stopFailures, "tracked server stop failed in afterEach"); });