diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 99c8c86963b..031766373e1 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -201,13 +201,29 @@ Combo failures are divided into **hop** failures and **terminal** failures. | Client cancellation (499), `origin_rejected`, cyber-policy refusal, context overflow, or invalid request | Stop and return the error; another target would not make the request valid. | | Any other unclassified error | Stop and return the error. | -A hopped target enters cooldown for 60 seconds by default. If the upstream response includes a -valid `Retry-After` value, opencodex uses it instead. Numeric seconds and HTTP-date values are -accepted, and every cooldown is capped at 10 minutes. +When `cooldownMs` is unset, a hopped target uses an upstream fallback: 5 seconds for request-rate +429s with upstream code `1302` or `1305`, and 60 seconds otherwise. When it is set, `cooldownMs` +applies whenever no usable upstream `Retry-After` or Codex reset signal exists, including those +request-rate 429s. Numeric `Retry-After` seconds and HTTP-date values are accepted, and every +cooldown is capped at 10 minutes. The precedence is, from strongest to weakest, explicit +`Retry-After` → Codex reset headers (`x-codex-primary-reset-at`, `x-codex-secondary-reset-at`, or +`x-codex-tertiary-reset-at`) → the combo's `cooldownMs` (when set) → the 5-second request-rate +fallback for upstream rate-limit codes `1302`/`1305` → the 60-second default. A valid immediate +`Retry-After: 0` remains an immediate upstream directive rather than being replaced by a configured +cooldown. The current request never retries the same attempted target. Later requests skip it until its -cooldown expires. If no eligible target remains, the proxy returns HTTP 503 with -`error.code = "combo_unavailable"`. +cooldown expires. A `Retry-After` HTTP-date that is already in the past is also preserved as an +immediate upstream directive, just like `Retry-After: 0`. Set `waitForCooldownMs` to allow a later +request to wait for the earliest eligible target cooldown, up to that cap on each selection attempt, +and then make one fresh selection. A request may therefore wait up to `hops × waitForCooldownMs` +across multiple failover hops. The default is `0`, which fails closed immediately with HTTP 503 when +every eligible target is cooling; that `combo_unavailable` 503 carries a `Retry-After` header equal +to the earliest remaining cooldown, rounded up to whole seconds with a minimum of 1. Waits are not jittered, so +synchronized wake-ups are possible. An aborted request cancels this wait and returns the normal +`client_cancelled` response; it does not dispatch a backup target after cancellation. A combo target +cooldown is process-local per-combo state and is separate from the account-level Codex quota cooldown +used by native account routing. :::note Failover is intentionally bounded. It helps with target-specific availability, authentication, @@ -321,7 +337,9 @@ combos, and its target picker excludes disabled models and nested combos. Each target also shows a live quota badge: **Available**, **Out of quota**, or **Quota unknown**. Save and Create are disabled only when every enabled target has fresh, complete evidence that its quota is exhausted. Missing, stale, malformed, or incomplete aggregate evidence stays unknown and never locks a control. Polling -continues while the workspace is visible, so recovery automatically restores the action. +continues while the workspace is visible, so recovery automatically restores the action. The dashboard +editor does not yet expose `cooldownMs` or `waitForCooldownMs`; use the configuration file or management +API until the follow-up UI work lands. ### CLI @@ -345,7 +363,12 @@ model alias and a non-empty display name. `create` and `update` are aliases for Headless clients use `GET`, `PUT`, and `DELETE` on `/api/combos`. `GET` lists normalized combo definitions, `PUT` creates or replaces one (and can rename one), and `DELETE` takes the id query parameter. Authentication and request/response details are in the -[Management API reference](/reference/management-api/). +[Management API reference](/reference/management-api/). When a `PUT` body omits `cooldownMs` +or `waitForCooldownMs`, the API preserves the value already stored for that combo; send an explicit +value to change it. An explicit `cooldownMs` (even `60000`) is persisted as-is because it overrides +the request-rate fallback. A stored `cooldownMs` can only be removed by editing the configuration file; +`waitForCooldownMs` resets to its default when a `PUT` explicitly sends `0`, because the sparse +serializer omits that default. Omission preserves both values and the dashboard does not expose them yet. For the complete persisted configuration, see [Configuration](/reference/configuration/). @@ -376,6 +399,8 @@ Combos are stored in the top-level `combos` object, keyed by combo id: | `targets[].weight` | No | `1` | Integer from 1 to 10,000. Used by round-robin and random; ignored by failover, least-used, and reset-window. | | `strategy` | No | `"failover"` | `"failover"`, `"round-robin"`, `"random"`, `"least-used"`, or `"reset-window"`. | | `stickyLimit` | No | `1` | Integer from 1 to 100 successful requests per round-robin selection. Applies only to round-robin. | +| `cooldownMs` | No | unset → upstream fallback (5 s for request-rate 429 codes `1302`/`1305`, otherwise 60 s) | Integer from 1 to 600000. When set, applies as the per-target cooldown whenever no usable upstream `Retry-After` or Codex reset signal exists, including request-rate 429s; when unset, uses the upstream fallback. | +| `waitForCooldownMs` | No | `0` | Integer from 0 to 600000. Maximum time to wait for the earliest eligible cooling target before returning `combo_unavailable`; abort cancels the wait. | | `defaultEffort` | No | `null` | `low`, `medium`, `high`, `xhigh`, `max`, or `ultra`; applied only when the caller omits effort and the target advertises support. | | `reasoningEffortMode` | No | `"strict"` | `"strict"` intersects every known target ladder, so one target advertising no effort control empties the combo's picker. `"adaptive"` excludes those empty ladders from the published intersection. Metadata only; dispatch is unchanged. | | `imageInput` | No | `"auto"` | `"auto"` or `"disabled"`. `"auto"` publishes image support only when every target supports images; `"disabled"` forces text-only (drops image from published modalities and rejects image-bearing requests before dispatch). | @@ -395,8 +420,11 @@ running opencodex instance that receives model requests. Every target is currently ineligible: for example, its provider is disabled, it is cooling down, it has already been attempted for this request, or an encrypted v2 task excludes it. Check target -provider state and recent upstream errors. For cooldowns, wait for the 60-second default or the -upstream `Retry-After` period (never more than 10 minutes), then retry. +provider state and recent upstream errors. For cooldowns, follow an observed `Retry-After` value first; +Codex reset headers also take precedence over `cooldownMs`. +If neither upstream signal is usable, the configured `cooldownMs` applies, or the upstream fallback applies +when it is unset (5 seconds for request-rate codes `1302`/`1305`, otherwise 60 seconds); every cooldown is +capped at 10 minutes. ### Why was my alias rejected? diff --git a/docs-site/src/content/docs/ja/guides/combos.md b/docs-site/src/content/docs/ja/guides/combos.md index 068689bf8cd..655dae22320 100644 --- a/docs-site/src/content/docs/ja/guides/combos.md +++ b/docs-site/src/content/docs/ja/guides/combos.md @@ -125,9 +125,9 @@ ocx combo set balanced \ |クライアントのキャンセル (499)、`origin_rejected`、サイバー ポリシーの拒否、コンテキスト オーバーフロー、または無効なリクエスト |停止してエラーを返します。別のターゲットではリクエストは有効になりません。 | |その他の未分類のエラー |停止してエラーを返します。 | -ホップされたターゲットはデフォルトで 60 秒間のクールダウンに入ります。アップストリーム応答に有効な `Retry-After` 値が含まれている場合、opencodex は代わりにそれを使用します。秒数値と HTTP 日付値が受け入れられ、各クールダウンの上限は 10 分です。 +`cooldownMs` が未設定の場合、ホップされたターゲットはアップストリームのフォールバックを使用します。アップストリームコード `1302` または `1305` を伴うリクエストレート 429 では 5 秒、それ以外では 60 秒です。設定されている場合、使用可能なアップストリームの `Retry-After` または Codex リセットシグナルが存在しないときは、これらのリクエストレート 429 を含め、`cooldownMs` が適用されます。数値の `Retry-After` 秒数と HTTP-date 値が受け入れられ、すべてのクールダウンは 10 分を上限とします。優先順位は強い順に、明示的な `Retry-After` → Codex リセットヘッダー（`x-codex-primary-reset-at`、`x-codex-secondary-reset-at`、または `x-codex-tertiary-reset-at`）→ コンボの `cooldownMs`（設定時）→ アップストリームのレート制限コード `1302`/`1305` に対する 5 秒のリクエストレート フォールバック → 60 秒のデフォルトです。有効な即時指定 `Retry-After: 0` は、設定されたクールダウンで置き換えられず、即時のアップストリーム指示として維持されます。 -現在のリクエストは、同じ試行ターゲットを再試行することはありません。以降のリクエストでは、クールダウンが期限切れになるまでスキップされます。適格なターゲットが残っていない場合、プロキシは `error.code = "combo_unavailable"` を含む HTTP 503 を返します。 +現在のリクエストは、同じ試行ターゲットを再試行しません。後続のリクエストでは、クールダウンが期限切れになるまでそのターゲットをスキップします。すでに過去の時刻である `Retry-After` の HTTP-date も、`Retry-After: 0` と同様に即時のアップストリーム指示として維持されます。`waitForCooldownMs` を設定すると、後続のリクエストは、最も早く利用可能になるターゲットのクールダウンを、その選択試行ごとにこの上限まで待ってから、新たに 1 回選択できます。したがって、複数のフェイルオーバー ホップをまたぐリクエストは合計で `hops × waitForCooldownMs` まで待つ場合があります。デフォルトは `0` です。適格なターゲットがすべて冷却中の場合、HTTP 503 ですぐにフェイルクローズし、その `combo_unavailable` 503 には最も早く終了する残りのクールダウンと等しい `Retry-After` ヘッダーが含まれ、秒単位に切り上げられ、最小値は 1 秒です。待機にはジッターがないため、同期したウェイクアップが発生する可能性があります。中止されたリクエストはこの待機をキャンセルし、通常の `client_cancelled` 応答を返します。キャンセル後にバックアップ ターゲットをディスパッチすることはありません。コンボ ターゲットのクールダウンはプロセス ローカルなコンボごとの状態であり、ネイティブ アカウント ルーティングで使用されるアカウントレベルの Codex クォータ クールダウンとは別です。 :::note フェイルオーバーは意図的に制限されています。これは、ターゲット固有の可用性、認証、クォータ、および過負荷の障害に役立ちます。呼び出し元のエラーやポリシーの拒否は隠蔽されません。 @@ -181,7 +181,7 @@ v1/base/v2 モードと完全な暗号化タスクのワークフローについ 各ターゲットには **利用可能**、**クォータを使い切りました**、**クォータ不明** のライブバッジも表示されます。 保存と作成が無効になるのは、有効な全ターゲットについて、クォータ枯渇を示す新鮮で完全な証拠がある場合だけです。 -欠落、古い、不正、または不完全な集約データは不明のままで、操作をロックしません。クォータが回復すると操作は自動で再び有効になります。 +欠落、古い、不正、または不完全な集約データは不明のままで、操作をロックしません。クォータが回復すると操作は自動で再び有効になります。ダッシュボードのエディターではまだ `cooldownMs` と `waitForCooldownMs` を設定できません。後続の UI 作業が完了するまでは、構成ファイルまたは管理 API を使用してください。 ### CLI @@ -202,7 +202,7 @@ ocx combo remove --yes ### 管理 API -ヘッドレス クライアントは、`/api/combos` 上の `GET`、`PUT`、および `DELETE` を使用します。 `GET` は正規化されたコンボ定義をリストし、`PUT` は 1 つを作成または置換し (名前を変更できます)、`DELETE` は id クエリ パラメーターを受け取ります。認証と要求/応答の詳細は [管理 API リファレンス](/reference/management-api/) にあります。 +ヘッドレス クライアントは、`/api/combos` 上の `GET`、`PUT`、および `DELETE` を使用します。 `GET` は正規化されたコンボ定義をリストし、`PUT` は 1 つを作成または置換し (名前を変更できます)、`DELETE` は id クエリ パラメーターを受け取ります。認証と要求/応答の詳細は [管理 API リファレンス](/reference/management-api/) にあります。`PUT` 本文で `cooldownMs` または `waitForCooldownMs` のいずれかを省略すると、そのコンボに保存済みの値が維持されます。変更するには明示的な値を指定してください。明示的な `cooldownMs`（`60000` でも）はリクエストレート フォールバックを上書きするため、そのまま永続化されます。保存済みの `cooldownMs` を削除できるのは構成ファイルを編集した場合だけです。`waitForCooldownMs` は、`PUT` で `0` を明示的に指定するとデフォルトに戻ります。これはスパース シリアライザーがそのデフォルト値を省略するためです。省略すると両方の値が維持され、ダッシュボードではまだどちらも設定できません。 永続化された設定全体については、「[構成](/reference/configuration/)」を参照してください。 @@ -233,6 +233,8 @@ ocx combo remove --yes | `targets[].weight` |いいえ | `1` | 1 ～ 10,000 の整数。`round-robin` と `random` で使用され、`failover`、`least-used`、`reset-window` では無視されます。 | | `strategy` |いいえ | `"failover"` | `"failover"`、`"round-robin"`、`"random"`、`"least-used"`、`"reset-window"`。 | | `stickyLimit` |いいえ | `1` | `round-robin` の 1 回の選択あたり、成功したリクエスト数を指定する 1 ～ 100 の整数。`round-robin` にのみ適用されます。 | +| `cooldownMs` |いいえ | 未設定 → アップストリーム フォールバック（リクエストレート 429 コード `1302`/`1305` では 5 秒、それ以外では 60 秒） | 1 ～ 600000 の整数。設定時は、使用可能なアップストリーム `Retry-After` または Codex リセットシグナルがない場合に、リクエストレート 429 を含むターゲットごとのクールダウンとして適用されます。未設定時はアップストリーム フォールバックを使用します。 | +| `waitForCooldownMs` |いいえ | `0` | 0 ～ 600000 の整数。最も早く利用可能になる冷却中のターゲットを待ってから `combo_unavailable` を返すまでの最大待機時間。中止すると待機はキャンセルされます。 | | `defaultEffort` |いいえ | `null` | `low`、`medium`、`high`、`xhigh`、`max`、または `ultra`;呼び出し元が努力を省略し、ターゲットがサポートをアドバタイズした場合にのみ適用されます。 | | `alias` |いいえ |なし |オプションのトリミングされたパブリック モデル ID。上記のエイリアス ルールを使用します。空の値はエイリアスなしで保存されます。 | | `nativeAlias` |いいえ | `false` | 現在サポートされている bare native alias に routing/catalog の優先権を明示的に与えます。 | @@ -246,7 +248,7 @@ ocx combo remove --yes ### `combo_unavailable` が発生するのはなぜですか? -現在、すべてのターゲットは不適格です。たとえば、プロバイダーが無効になっている、冷却中である、このリクエストに対してすでに試行されている、暗号化された v2 タスクによってターゲットが除外されているなどです。ターゲットプロバイダーの状態と最近のアップストリームエラーを確認してください。クールダウンの場合は、デフォルトの 60 秒またはアップストリームの `Retry-After` 期間（10 分を超えないでください）待ってから、再試行してください。 +現在、すべてのターゲットは不適格です。たとえば、プロバイダーが無効になっている、冷却中である、このリクエストに対してすでに試行されている、暗号化された v2 タスクによってターゲットが除外されているなどです。ターゲットプロバイダーの状態と最近のアップストリームエラーを確認してください。クールダウンでは、まずレスポンスで確認できる `Retry-After` の値に従ってください。Codex のリセットヘッダーも `cooldownMs` より優先され、どちらのアップストリームシグナルも使用できない場合は、設定した `cooldownMs`、未設定ならアップストリーム フォールバック（リクエストレートコード `1302`/`1305` では 5 秒、それ以外では 60 秒）が適用されますが、いずれのクールダウンも 10 分を超えません。 ### 私のエイリアスが拒否されたのはなぜですか? diff --git a/docs-site/src/content/docs/ko/guides/combos.md b/docs-site/src/content/docs/ko/guides/combos.md index 6028e337cd7..aef5ca1cc19 100644 --- a/docs-site/src/content/docs/ko/guides/combos.md +++ b/docs-site/src/content/docs/ko/guides/combos.md @@ -125,9 +125,9 @@ ocx combo set balanced \ | 클라이언트 취소(499), `origin_rejected`, cyber-policy refusal, context overflow, 또는 invalid request | 멈추고 오류를 반환합니다. 다른 대상을 써도 요청이 유효해지지 않기 때문입니다. | | 그 밖의 분류되지 않은 오류 | 멈추고 오류를 반환합니다. | -홉된 대상은 기본적으로 60초 동안 쿨다운에 들어갑니다. 상위 응답에 유효한 `Retry-After` 값이 있으면 opencodex는 그 값을 대신 사용합니다. 숫자 초와 HTTP-date 값이 모두 허용되며, 모든 쿨다운은 최대 10분으로 제한됩니다. +`cooldownMs`가 설정되지 않으면 홉된 대상은 업스트림 폴백을 사용합니다. 업스트림 코드 `1302` 또는 `1305`인 요청 속도 제한 429는 5초, 그 외에는 60초입니다. 설정하면 사용 가능한 업스트림 `Retry-After` 또는 Codex 재설정 신호가 없을 때, 해당 요청 속도 제한 429를 포함해 `cooldownMs`가 적용됩니다. 숫자로 된 `Retry-After` 초와 HTTP-date 값을 허용하며, 모든 쿨다운은 최대 10분으로 제한됩니다. 우선순위는 강한 순서대로 명시적 `Retry-After` → Codex 재설정 헤더(`x-codex-primary-reset-at`, `x-codex-secondary-reset-at`, 또는 `x-codex-tertiary-reset-at`) → 콤보의 `cooldownMs`(설정된 경우) → 업스트림 속도 제한 코드 `1302`/`1305`의 5초 요청 속도 제한 폴백 → 60초 기본값입니다. 유효한 즉시 지시인 `Retry-After: 0`은 설정된 쿨다운으로 대체하지 않고 업스트림의 즉시 지시로 유지합니다. -현재 요청은 이미 시도한 대상을 다시 시도하지 않습니다. 이후 요청은 그 대상의 쿨다운이 끝날 때까지 건너뜁니다. 적합한 대상이 하나도 남지 않으면 프록시는 HTTP 503과 함께 `error.code = "combo_unavailable"`을 반환합니다. +현재 요청은 이미 시도한 대상을 다시 시도하지 않습니다. 이후 요청은 쿨다운이 끝날 때까지 해당 대상을 건너뜁니다. 이미 지난 시각을 가리키는 `Retry-After` HTTP-date도 `Retry-After: 0`과 마찬가지로 업스트림의 즉시 지시로 유지됩니다. `waitForCooldownMs`를 설정하면 이후 요청은 가장 먼저 적합해지는 대상의 쿨다운을 선택 시도마다 이 한도까지 기다린 뒤 새로 한 번 선택합니다. 따라서 여러 failover 홉을 거치는 요청은 총 `hops × waitForCooldownMs`까지 기다릴 수 있습니다. 기본값은 `0`입니다. 모든 적합한 대상이 쿨다운 중이고 대기 한도가 0이거나 가장 이른 만료 시각이 대기 한도를 넘으면 요청은 즉시 HTTP 503으로 종료됩니다. 이 `combo_unavailable` 503에는 가장 이른 잔여 쿨다운과 같은 `Retry-After` 헤더가 포함되며, 값은 올림해 정수 초로 표시되고 최소 1초입니다. 대기에 지터를 적용하지 않으므로 동시에 깨어날 수 있습니다. 요청이 중단되면 이 대기가 취소되고 정상 `client_cancelled` 응답이 반환됩니다. 취소 후 백업 대상을 디스패치하지 않습니다. 콤보 대상 쿨다운은 프로세스 로컬 콤보별 상태입니다. 네이티브 계정 라우팅에서 사용하는 계정 수준 Codex 쿼터 쿨다운과는 별개입니다. :::note 페일오버는 의도적으로 범위를 제한합니다. 대상별 가용성, 인증, 쿼터, 과부하 실패에는 도움이 되지만, 호출자 오류나 정책 거부를 숨기지는 않습니다. @@ -181,7 +181,7 @@ v1/base/v2 모드와 암호화된 작업의 전체 흐름은 [Sub-agent Surface] 각 대상에는 **사용 가능**, **할당량 소진**, **할당량 알 수 없음** 실시간 배지도 표시됩니다. 저장과 만들기 버튼은 활성화된 모든 대상에 할당량 소진을 입증하는 최신의 완전한 증거가 있을 때만 비활성화됩니다. 누락되거나 오래되거나 -형식이 잘못되었거나 집계가 불완전한 데이터는 알 수 없음으로 남으며 버튼을 잠그지 않습니다. 할당량이 복구되면 버튼도 자동으로 다시 활성화됩니다. +형식이 잘못되었거나 집계가 불완전한 데이터는 알 수 없음으로 남으며 버튼을 잠그지 않습니다. 할당량이 복구되면 버튼도 자동으로 다시 활성화됩니다. 대시보드 편집기에서는 아직 `cooldownMs`나 `waitForCooldownMs`를 설정할 수 없습니다. 후속 UI 작업이 완료될 때까지 구성 파일이나 관리 API를 사용하세요. ### CLI @@ -201,7 +201,7 @@ ocx combo remove --yes ### Management API -헤드리스 클라이언트는 `/api/combos`에 `GET`, `PUT`, `DELETE`를 사용합니다. `GET`은 정규화된 콤보 정의를 나열하고, `PUT`은 새 항목을 만들거나 교체하며(이름 바꾸기도 가능), `DELETE`는 id 쿼리 파라미터를 사용합니다. 인증과 요청/응답 세부 내용은 [Management API reference](/reference/management-api/)에 있습니다. +헤드리스 클라이언트는 `/api/combos`에 `GET`, `PUT`, `DELETE`를 사용합니다. `GET`은 정규화된 콤보 정의를 나열하고, `PUT`은 새 항목을 만들거나 교체하며(이름 바꾸기도 가능), `DELETE`는 id 쿼리 파라미터를 사용합니다. 인증과 요청/응답 세부 내용은 [Management API reference](/reference/management-api/)에 있습니다. `PUT` 본문에서 `cooldownMs` 또는 `waitForCooldownMs`를 생략하면 해당 콤보에 이미 저장된 값이 유지됩니다. 변경하려면 값을 명시적으로 보내세요. 명시적 `cooldownMs`(`60000` 포함)는 요청 속도 제한 폴백을 덮어쓰므로 보낸 값 그대로 저장됩니다. 저장된 `cooldownMs`는 구성 파일을 편집할 때만 삭제할 수 있습니다. `waitForCooldownMs`는 `PUT`에서 `0`을 명시적으로 보내면 기본값으로 돌아갑니다. 희소 직렬화기가 이 기본값을 생략하기 때문입니다. 생략한 값은 유지되고, 대시보드에서는 아직 두 값을 설정할 수 없습니다. 전체 지속 설정은 [Configuration](/reference/configuration/)을 보십시오. @@ -232,6 +232,8 @@ ocx combo remove --yes | `targets[].weight` | 아니요 | `1` | 1에서 10,000 사이의 정수입니다. `round-robin`과 `random`에서 사용되며, `failover`, `least-used`, `reset-window`에서는 무시됩니다. | | `strategy` | 아니요 | `"failover"` | 허용되는 값은 `"failover"`, `"round-robin"`, `"random"`, `"least-used"`, `"reset-window"`입니다. | | `stickyLimit` | 아니요 | `1` | 한 번의 `round-robin` 선택에 유지되는 성공 요청 수로, 1에서 100 사이의 정수입니다. `round-robin`에만 적용됩니다. | +| `cooldownMs` | 아니요 | 미설정 → 업스트림 폴백(요청 속도 제한 429 코드 `1302`/`1305`는 5초, 그 외는 60초) | 1에서 600000 사이의 정수입니다. 설정하면 사용 가능한 업스트림 `Retry-After` 또는 Codex 재설정 신호가 없을 때 요청 속도 제한 429를 포함한 대상별 쿨다운으로 적용됩니다. 설정하지 않으면 업스트림 폴백을 사용합니다. | +| `waitForCooldownMs` | 아니요 | `0` | 0에서 600000 사이의 정수입니다. `combo_unavailable`을 반환하기 전에 가장 먼저 적합해지는 쿨다운 중인 대상을 기다리는 최대 시간입니다. 중단하면 대기가 취소됩니다. | | `defaultEffort` | 아니요 | `null` | `low`, `medium`, `high`, `xhigh`, `max`, 또는 `ultra`입니다. 호출자가 effort를 생략하고 대상이 지원을 광고할 때만 적용됩니다. | | `alias` | 아니요 | 없음 | 선택적으로 앞뒤 공백을 제거한 공개 모델 ID입니다. 위의 alias 규칙을 따릅니다. 빈 값은 alias 없음으로 저장됩니다. | | `nativeAlias` | 아니요 | `false` | 현재 지원되는 bare native alias가 routing/catalog 우선권을 갖도록 명시적으로 허용합니다. | @@ -249,8 +251,7 @@ opencodex 인스턴스에 기록했는지 확인하세요. 모든 대상이 현재 부적격 상태입니다. 예를 들어 프로바이더가 비활성화되었거나, cooldown 중이거나, 이 요청에서 이미 시도되었거나, 암호화된 v2 작업 때문에 제외되었을 수 있습니다. 대상 프로바이더 상태와 -최근 업스트림 오류를 확인하세요. cooldown이라면 기본 60초 또는 업스트림 `Retry-After` 기간(최대 10분)을 -기다린 뒤 다시 시도하세요. +최근 업스트림 오류를 확인하세요. 쿨다운에서는 먼저 응답의 `Retry-After` 값을 따르세요. Codex 재설정 헤더도 `cooldownMs`보다 우선합니다. 두 업스트림 신호를 모두 사용할 수 없을 때 설정된 `cooldownMs`를 적용하고, 미설정이면 업스트림 폴백(요청 속도 제한 코드 `1302`/`1305`는 5초, 그 외는 60초)을 적용하며 모든 쿨다운은 최대 10분으로 제한됩니다. ### alias가 거부된 이유는 무엇인가요? diff --git a/docs-site/src/content/docs/reference/configuration/routing.md b/docs-site/src/content/docs/reference/configuration/routing.md index 4bac3170e9c..25bd32f64e2 100644 --- a/docs-site/src/content/docs/reference/configuration/routing.md +++ b/docs-site/src/content/docs/reference/configuration/routing.md @@ -88,6 +88,8 @@ namespace, and cannot use reserved bare native families such as `gpt-*`, `o1-*`, | `targets` | `{ provider: string; model: string; weight?: number }[]` | required | Ordered concrete routes. `weight` is 1–10000 and defaults to `1`. | | `strategy?` | `"failover" \| "round-robin" \| "random" \| "least-used" \| "reset-window"` | `"failover"` | Selection strategy. Target order is failover priority; weights shape round-robin and random draws; least-used follows recorded successes; reset-window follows the soonest quota reset. | | `stickyLimit?` | `number` | `1` | Successful requests retained in one round-robin batch. Range 1–100. Applies only to round-robin. | +| `cooldownMs?` | `number` | unset → upstream fallback (5 s for request-rate 429 codes `1302`/`1305`, otherwise 60 s) | Range 1–600000. When set, applies whenever no usable upstream `Retry-After` or Codex reset signal exists, including request-rate 429s; when unset, uses the upstream fallback. Upstream signals take precedence and all cooldowns are capped at 10 minutes. | +| `waitForCooldownMs?` | `number` | `0` | Maximum wait for the earliest eligible cooling target on each selection attempt before returning `combo_unavailable`. Range 0–600000; an abort cancels the wait. | | `defaultEffort?` | `"low" \| "medium" \| "high" \| "xhigh" \| "max" \| "ultra" \| null` | unset | Applied only when the caller omits effort and the selected target advertises the requested rung. | | `reasoningEffortMode?` | `"strict" \| "adaptive"` | `"strict"` | `"strict"` intersects every known target effort ladder, so a target advertising no effort control empties the combo's picker. `"adaptive"` excludes those empty ladders from the published intersection. Picker metadata only; target selection and dispatch are unchanged. | | `imageInput?` | `"auto" \| "disabled"` | `"auto"` | `"auto"` publishes image only when every target supports images; `"disabled"` forces text-only (drops image from published modalities and rejects image-bearing requests before dispatch). | diff --git a/docs-site/src/content/docs/ru/guides/combos.md b/docs-site/src/content/docs/ru/guides/combos.md index e025d9cf0e3..3d4820e5210 100644 --- a/docs-site/src/content/docs/ru/guides/combos.md +++ b/docs-site/src/content/docs/ru/guides/combos.md @@ -162,13 +162,9 @@ ocx combo set balanced \ | Отмена клиентом (499), `origin_rejected`, отказ из-за cyber-policy, переполнение контекста или некорректный запрос | Остановиться и вернуть ошибку; другая цель не сделает такой запрос корректным. | | Любая другая неклассифицированная ошибка | Остановиться и вернуть ошибку. | -Цель, по которой произошёл hop, по умолчанию уходит в cooldown на 60 секунд. Если ответ upstream -содержит корректный `Retry-After`, opencodex использует его. Поддерживаются и числовые секунды, и -значения в формате HTTP-date; любой cooldown ограничивается 10 минутами. +Если `cooldownMs` не задан, цель после hop использует upstream fallback: 5 секунд для 429, ограничивающих частоту запросов, с кодом upstream `1302` или `1305`, и 60 секунд в остальных случаях. Если он задан, `cooldownMs` применяется, когда нет пригодного сигнала upstream `Retry-After` или сигнала сброса Codex, включая такие 429, ограничивающие частоту запросов. Принимаются числовые секунды в `Retry-After` и значения HTTP-date; любой cooldown ограничен 10 минутами. Приоритет от сильного к слабому: явный `Retry-After` → заголовки сброса Codex (`x-codex-primary-reset-at`, `x-codex-secondary-reset-at` или `x-codex-tertiary-reset-at`) → `cooldownMs` этой combo (если задан) → 5-секундный fallback для rate-limit-кодов upstream `1302`/`1305` → стандартные 60 секунд. Корректный немедленный `Retry-After: 0` сохраняется как немедленная директива upstream, а не заменяется настроенным cooldown. -Текущий запрос никогда не повторяет уже опробованную цель. Более поздние запросы пропускают её, -пока не истечёт cooldown. Если подходящих целей больше не осталось, прокси возвращает HTTP 503 с -`error.code = "combo_unavailable"`. +Текущий запрос никогда не повторяет уже опробованную цель. Более поздние запросы пропускают её, пока не истечёт cooldown. HTTP-date в `Retry-After`, указывающий на уже прошедшее время, также сохраняется как немедленная директива upstream, как и `Retry-After: 0`. Задайте `waitForCooldownMs`, чтобы следующий запрос мог подождать cooldown цели, которая станет подходящей раньше всех, до этого ограничения при каждой попытке выбора, а затем выполнить один новый выбор. Поэтому запрос с несколькими hop в failover может ждать в общей сложности до `hops × waitForCooldownMs`. По умолчанию это `0`: если все подходящие цели находятся в cooldown, запрос немедленно завершается HTTP 503; этот ответ `combo_unavailable` содержит заголовок `Retry-After`, равный оставшемуся cooldown цели с самым ранним окончанием, округлённый вверх до целых секунд, минимум до 1 секунды. К ожиданиям не добавляется джиттер, поэтому возможны синхронные пробуждения. Отмена запроса отменяет это ожидание и возвращает обычный ответ `client_cancelled`; после отмены резервная цель не запускается. Cooldown цели combo — состояние процесса, отдельное для каждой combo; он не связан с cooldown квоты Codex на уровне аккаунта, который используется нативной маршрутизацией аккаунта. :::note Failover намеренно ограничен. Он помогает при проблемах доступности конкретной цели, @@ -238,7 +234,7 @@ effort вызывающей стороне и цели. У каждой цели также отображается актуальный значок квоты: **Доступно**, **Квота исчерпана** или **Квота неизвестна**. Кнопки сохранения и создания отключаются только тогда, когда для всех включённых целей есть свежие и полные данные об исчерпании квоты. Отсутствующие, устаревшие, некорректные или неполные агрегированные данные остаются -неизвестными и никогда не блокируют управление. Восстановление квоты автоматически снова включает действие. +неизвестными и никогда не блокируют управление. Восстановление квоты автоматически снова включает действие. Редактор дашборда пока не предоставляет `cooldownMs` и `waitForCooldownMs`; до появления соответствующего UI используйте файл конфигурации или Management API. ### CLI @@ -262,7 +258,7 @@ alias и непустой display name. `create` и `update` — alias для `s Headless-клиенты используют `GET`, `PUT` и `DELETE` на `/api/combos`. `GET` возвращает список нормализованных определений combo, `PUT` создаёт или заменяет одну combo (и умеет переименовывать), а `DELETE` принимает id в query-параметре. Аутентификация и детали контрактов запросов/ответов -описаны в [справочнике Management API](/reference/management-api/). +описаны в [справочнике Management API](/reference/management-api/). Если в теле `PUT` не указано `cooldownMs` или `waitForCooldownMs`, API сохраняет уже записанное для этой combo значение; чтобы изменить его, передайте значение явно. Явно переданный `cooldownMs` (в том числе `60000`) сохраняется как есть, поскольку он переопределяет fallback для ограничения частоты запросов. Сохранённый `cooldownMs` можно удалить только редактированием файла конфигурации; `waitForCooldownMs` возвращается к стандартному значению, если `PUT` явно передаёт `0`, поскольку разреженный сериализатор опускает это значение по умолчанию. Пропуск каждого поля сохраняет соответствующее значение, а дашборд пока не позволяет настраивать эти параметры. Полную сохранённую конфигурацию см. в [Конфигурации](/reference/configuration/). @@ -293,6 +289,8 @@ Combo хранятся в объекте верхнего уровня `combos`, | `targets[].weight` | No | `1` | Целое число от 1 до 10 000. Используется стратегиями `round-robin` и `random`; игнорируется стратегиями `failover`, `least-used` и `reset-window`. | | `strategy` | No | `"failover"` | `"failover"`, `"round-robin"`, `"random"`, `"least-used"` или `"reset-window"`. | | `stickyLimit` | No | `1` | Целое число от 1 до 100 успешных запросов на один выбор `round-robin`. Применяется только к `round-robin`. | +| `cooldownMs` | No | не задано → fallback upstream (5 с для rate-limit 429 с кодами `1302`/`1305`, иначе 60 с) | Целое число от 1 до 600000. Если задано, применяется как cooldown каждой цели, когда нет пригодного upstream `Retry-After` или сигнала сброса Codex, включая rate-limit 429; если не задано, используется fallback upstream. | +| `waitForCooldownMs` | No | `0` | Целое число от 0 до 600000. Максимальное время ожидания самой ранней подходящей цели в cooldown перед возвратом `combo_unavailable`; отмена запроса отменяет ожидание. | | `defaultEffort` | No | `null` | `low`, `medium`, `high`, `xhigh`, `max` или `ultra`; применяется только когда вызывающая сторона не указала effort, а цель объявляет поддержку. | | `alias` | No | none | Необязательный обрезанный публичный id модели; используйте правила alias выше. Пустое значение хранится как отсутствие alias. | | `nativeAlias` | No | `false` | Явно разрешает поддерживаемому сейчас bare native alias перехватить приоритет routing/catalog только для неквалифицированного id. Bare `gpt-5.6-*` использует учётные данные Codex Pool/Direct; маршруты с квалификатором аккаунта сохраняют свою идентичность, а provider-qualified `openai-apikey/gpt-5.6-*` использует API-ключ и никогда не переходит на native alias. | @@ -310,8 +308,7 @@ Id combo неизвестен. Ответ — HTTP 404 с типом `invalid_re Сейчас ни одна цель не подходит: например, провайдер отключён, находится в cooldown, уже был испробован для этого запроса или исключён из-за шифрованной задачи v2. Проверьте состояние -провайдеров цели и недавние upstream-ошибки. Для cooldown подождите стандартные 60 секунд или -период из upstream `Retry-After` (но не более 10 минут), затем повторите запрос. +провайдеров цели и недавние upstream-ошибки. Для cooldown сначала следуйте значению `Retry-After` из ответа. Заголовки сброса Codex также имеют приоритет над `cooldownMs`, поэтому если ни один upstream-сигнал не пригоден, применяется заданный `cooldownMs`, а если он не задан — upstream fallback (5 секунд для rate-limit-кодов `1302`/`1305`, иначе 60 секунд); любой cooldown ограничен 10 минутами. ### Почему alias был отклонён? diff --git a/docs-site/src/content/docs/zh-cn/guides/combos.md b/docs-site/src/content/docs/zh-cn/guides/combos.md index d73f9c04f77..fea189deb3e 100644 --- a/docs-site/src/content/docs/zh-cn/guides/combos.md +++ b/docs-site/src/content/docs/zh-cn/guides/combos.md @@ -151,9 +151,9 @@ combo 失败分为 **跳转** 失败和 **终止** 失败。 | 客户端取消（499）、`origin_rejected`、cyber-policy 拒绝、上下文溢出，或无效请求 | 停止并返回错误；换其他目标也无法让请求变得有效。 | | 任何其他未分类错误 | 停止并返回错误。 | -被跳过的目标默认会进入 60 秒冷却。如果上游响应包含有效的 `Retry-After` 值，opencodex 会改用该值。数字秒数和 HTTP-date 值都可以接受，而且每次冷却最多只会封顶到 10 分钟。 +未设置 `cooldownMs` 时，发生跳转的目标使用上游回退值：对于上游代码为 `1302` 或 `1305` 的请求速率限制 429，等待 5 秒；其他情况等待 60 秒。设置后，只要不存在可用的上游 `Retry-After` 或 Codex 重置信号，就会应用 `cooldownMs`，包括这些请求速率限制 429。接受数字形式的 `Retry-After` 秒数和 HTTP-date 值，每次冷却最多封顶 10 分钟。优先级从强到弱依次为：显式 `Retry-After` → Codex 重置标头（`x-codex-primary-reset-at`、`x-codex-secondary-reset-at` 或 `x-codex-tertiary-reset-at`）→ combo 的 `cooldownMs`（已设置时）→ 上游速率限制代码 `1302`/`1305` 的 5 秒请求速率限制回退值 → 60 秒默认值。有效的即时指令 `Retry-After: 0` 会保留为上游即时指令，不会被配置的冷却替换。 -当前请求不会再次重试同一个已经尝试过的目标。后续请求会跳过它，直到冷却结束。如果没有任何合格目标可用，代理会返回 HTTP 503，并带上 `error.code = "combo_unavailable"`。 +当前请求不会再次重试同一个已经尝试过的目标。后续请求会跳过它，直到冷却结束。已过去的 HTTP-date `Retry-After` 同样会像 `Retry-After: 0` 一样保留为上游即时指令。设置 `waitForCooldownMs` 后，后续请求可以等待最早恢复资格的目标的冷却，单次选择尝试最多等待该上限，然后重新选择一次。因此，多次故障切换跳转的请求总共最多等待 `hops × waitForCooldownMs`。默认值为 `0`；当所有合格目标都处于冷却中时，请求会立即失败并返回 HTTP 503；该 `combo_unavailable` 503 会带有 `Retry-After` 标头，其值等于剩余冷却时间最短的目标，向上取整为整秒，最小值为 1 秒。等待不加入抖动，因此可能同时唤醒。请求中止会取消这次等待并返回正常的 `client_cancelled` 响应；取消后不会调度备用目标。combo 目标冷却是进程本地、按 combo 区分的状态，与原生账户路由使用的账户级 Codex 配额冷却彼此独立。 :::note 故障切换是有边界的。它有助于处理特定目标的可用性、认证、配额和过载失败；它不会掩盖调用方错误或策略拒绝。 @@ -211,7 +211,7 @@ combo 失败分为 **跳转** 失败和 **终止** 失败。 每个目标还会显示实时额度徽章：**可用**、**额度已用尽**或**额度未知**。只有当所有已启用目标都有最新、 完整的额度耗尽证据时，保存和创建操作才会被禁用。缺失、过期、格式错误或聚合不完整的证据会保持为未知， -绝不会锁定控件。额度恢复后，操作会自动重新启用。 +绝不会锁定控件。额度恢复后，操作会自动重新启用。dashboard 编辑器目前还不能设置 `cooldownMs` 或 `waitForCooldownMs`；在后续 UI 完成前，请使用配置文件或管理 API。 ### CLI @@ -232,7 +232,7 @@ ocx combo remove --yes ### Management API -无头客户端会对 `/api/combos` 使用 `GET`、`PUT` 和 `DELETE`。`GET` 会列出规范化后的 combo 定义，`PUT` 会创建或替换一个定义（也可以重命名一个），`DELETE` 则使用 id 查询参数。认证以及请求/响应细节请见 [Management API 参考](/reference/management-api/)。 +无头客户端会对 `/api/combos` 使用 `GET`、`PUT` 和 `DELETE`。`GET` 会列出规范化后的 combo 定义，`PUT` 会创建或替换一个定义（也可以重命名一个），`DELETE` 则使用 id 查询参数。认证以及请求/响应细节请见 [Management API 参考](/reference/management-api/)。如果 `PUT` 请求体省略 `cooldownMs` 或 `waitForCooldownMs`，API 会保留该 combo 已存储的值；要更改它，请显式发送一个值。显式设置的 `cooldownMs`（即使是 `60000`）会按原值持久化，因为它会覆盖请求速率限制回退值。已存储的 `cooldownMs` 只能通过编辑配置文件删除；如果 `PUT` 显式发送 `0`，`waitForCooldownMs` 会恢复为默认值，因为稀疏序列化器会省略这个默认值。省略字段会保留对应值，dashboard 目前还不能设置这两个参数。 如需查看完整的持久化配置，请参见 [配置](/reference/configuration/)。 @@ -263,6 +263,8 @@ combo 会存储在顶层的 `combos` 对象中，并以 combo id 作为键： | `targets[].weight` | 否 | `1` | 1 到 10,000 的整数。`round-robin` 和 `random` 会使用它；`failover`、`least-used` 和 `reset-window` 会忽略它。 | | `strategy` | 否 | `"failover"` | `"failover"`、`"round-robin"`、`"random"`、`"least-used"` 或 `"reset-window"`。 | | `stickyLimit` | 否 | `1` | 每次 `round-robin` 选择可连续处理 1 到 100 个成功请求。仅适用于 `round-robin`。 | +| `cooldownMs` | 否 | 未设置 → 上游回退值（请求速率限制代码为 `1302`/`1305` 的 429 为 5 秒，否则为 60 秒） | 1 到 600000 的整数。设置后，只要没有可用的上游 `Retry-After` 或 Codex 重置信号，就会作为每个目标的冷却时间应用，包括请求速率限制 429；未设置时使用上游回退值。 | +| `waitForCooldownMs` | 否 | `0` | 0 到 600000 的整数。在返回 `combo_unavailable` 前等待最早恢复资格的冷却中目标的最长时间；请求中止会取消等待。 | | `defaultEffort` | 否 | `null` | `low`、`medium`、`high`、`xhigh`、`max` 或 `ultra`；仅当调用方省略 effort 且目标声明支持时才会应用。 | | `imageInput` | 否 | `"auto"` | `"auto"` 或 `"disabled"`。`"auto"` 仅在每个目标都支持图片时发布图片能力；`"disabled"` 强制仅文本（从对外能力中去掉图片，并在分发前拒绝带图请求）。 | | `alias` | 否 | 无 | 可选的、已修剪的公开模型 id；使用上面的别名规则。空值会以“无别名”形式存储。 | @@ -277,7 +279,7 @@ combo id 不存在。响应是 HTTP 404，类型为 `invalid_request_error`。 ### 为什么会收到 `combo_unavailable`？ -当前每个目标都不可用：例如，它的 provider 被禁用、它正在冷却、它已经在这次请求中被尝试过，或者加密的 v2 任务把它排除了。检查目标的 provider 状态和最近的上游错误。对于冷却，请等待 60 秒的默认值或上游 `Retry-After` 时长（永远不会超过 10 分钟），然后重试。 +当前每个目标都不可用：例如，它的 provider 被禁用、它正在冷却、它已经在这次请求中被尝试过，或者加密的 v2 任务把它排除了。检查目标的 provider 状态和最近的上游错误。对于冷却，请先遵循响应中的 `Retry-After` 值。Codex 重置标头的优先级也高于 `cooldownMs`；只有在两个上游信号都不可用时，才应用已配置的 `cooldownMs`，未配置时应用上游回退值（请求速率限制代码 `1302`/`1305` 为 5 秒，否则为 60 秒），且所有冷却最多封顶 10 分钟。 ### 为什么我的别名被拒绝了？ diff --git a/src/combos/failover.ts b/src/combos/failover.ts index 3e786ca2f79..3868bc4b0c3 100644 --- a/src/combos/failover.ts +++ b/src/combos/failover.ts @@ -1,3 +1,4 @@ +import { parseResetCooldownMs } from "../codex/routing"; import { classifyError, isCyberPolicyCode } from "../lib/errors"; import type { OcxComboTarget } from "../types"; import { targetKey } from "./types"; @@ -197,6 +198,7 @@ export function coolComboTarget( target: Pick, options?: { retryAfter?: string | null; + resetAt?: unknown | unknown[]; now?: number; cooldownMs?: number; writerGeneration?: number; @@ -209,8 +211,12 @@ export function coolComboTarget( const writerGeneration = options?.writerGeneration ?? captureConfigGeneration(); const ownerKey = `${comboId}::${targetKey(target)}`; if (writerGeneration < lastReconciledGeneration && !liveComboTargets.has(ownerKey)) return; - const cooldownMs = options?.cooldownMs - ?? parseRetryAfterMs(options?.retryAfter, now) + // A server-provided Retry-After is authoritative, including an immediate `0` directive. + // A quota reset is the next-most-specific signal (#3256); configured and default cooldowns + // are only fallbacks when upstream supplied neither usable value. + const cooldownMs = parseRetryAfterMs(options?.retryAfter, now, { preserveImmediate: true }) + ?? parseResetCooldownMs(options?.resetAt, now) + ?? options?.cooldownMs ?? (isTransientRequestRateLimit({ status: options?.status, code: options?.code, @@ -222,6 +228,32 @@ export function coolComboTarget( sweepExpiredOnWrite(now); } +export function earliestComboCooldown( + comboId: string, + targets: Iterable>, + now = Date.now(), +): { expiry: number; target: Pick } | undefined { + let earliest: { expiry: number; target: Pick } | undefined; + for (const target of targets) { + const key = cooldownMapKey(comboId, target); + const entry = targetCooldowns.get(key); + if (!entry || entry.cooldownUntil <= now) continue; + if (earliest === undefined || entry.cooldownUntil < earliest.expiry) { + earliest = { expiry: entry.cooldownUntil, target }; + } + } + return earliest; +} + +/** Public convenience wrapper returning only the earliest cooldown expiry. */ +export function earliestComboCooldownExpiry( + comboId: string, + targets: Iterable>, + now = Date.now(), +): number | undefined { + return earliestComboCooldown(comboId, targets, now)?.expiry; +} + export function reconcileComboTargetCooldowns(context: GenerationContext): number { if (context.generation <= lastReconciledGeneration) return 0; liveComboTargets = new Set(context.comboTargets); diff --git a/src/combos/index.ts b/src/combos/index.ts index 982f87c9e19..92f74f93041 100644 --- a/src/combos/index.ts +++ b/src/combos/index.ts @@ -1,4 +1,5 @@ export { + COMBO_DEFAULT_WAIT_FOR_COOLDOWN_MS, COMBO_NAMESPACE, comboAliasIssues, comboConfigError, @@ -25,6 +26,7 @@ export { noteComboFailure, noteComboSuccess, pickComboTarget, + pickComboTargetWithWait, tryPickComboModel, UnknownComboError, type ComboPick, @@ -34,6 +36,8 @@ export { comboCooldownRetryAfterSeconds, COMBO_REQUEST_RATE_COOLDOWN_MS, coolComboTarget, + earliestComboCooldown, + earliestComboCooldownExpiry, isComboTargetInCooldown, isTransientRequestRateLimit, parseRetryAfterMs, diff --git a/src/combos/resolve.ts b/src/combos/resolve.ts index 5d41d511bf4..58038df0bc5 100644 --- a/src/combos/resolve.ts +++ b/src/combos/resolve.ts @@ -1,7 +1,13 @@ import type { OcxComboTarget, OcxConfig } from "../types"; import { getCachedProviderQuota } from "../providers/quota-routing-cache"; import type { ProviderQuota } from "../providers/quota-types"; -import { coolComboTarget, isComboTargetInCooldown, type ComboFailureCooldownScope } from "./failover"; +import { sleepWithAbort } from "../lib/upstream-retry"; +import { + coolComboTarget, + earliestComboCooldown, + isComboTargetInCooldown, + type ComboFailureCooldownScope, +} from "./failover"; import { quotaResetRemainingMs } from "./reset-window"; import { getCombo, resolveComboId, targetKey } from "./types"; import type { NormalizedComboConfig } from "./types"; @@ -275,7 +281,9 @@ export function advanceComboAfterFailure( pick: ComboPick, options: { retryAfter?: string | null; + resetAt?: unknown | unknown[]; now?: number; + cooldownMs?: number; eligible?: (target: Required) => boolean; cooldownScope?: ComboFailureCooldownScope; status?: number; @@ -294,6 +302,7 @@ export function advanceComboAfterFailure( for (const target of cooldownTargets) { coolComboTarget(pick.comboId, target, { ...options, + cooldownMs: options.cooldownMs ?? combo?.cooldownMs, writerGeneration: pick.writerGeneration, }); } @@ -306,6 +315,61 @@ export function advanceComboAfterFailure( }); } +export async function pickComboTargetWithWait( + config: OcxConfig, + comboId: string, + options: { + exclude?: Iterable; + eligible?: (target: Required) => boolean; + waitForCooldownMs: number; + abortSignal?: AbortSignal; + now?: number; + sleep?: (ms: number, signal?: AbortSignal) => Promise; + }, +): Promise { + const now = options.now ?? Date.now(); + const excluded = new Set(options.exclude ?? []); + const customEligible = options.eligible; + const eligible = (target: Required): boolean => + !isComboTargetInCooldown(comboId, target, now) + && (customEligible?.(target) ?? true); + const pick = pickComboTarget(config, comboId, { exclude: excluded, eligible, now }); + if (pick || options.waitForCooldownMs <= 0 || options.abortSignal?.aborted) return pick; + const combo = getCombo(config, comboId); + if (!combo) throw new UnknownComboError(comboId); + const waitingTargets = combo.targets.filter(target => + targetProviderIsUsable(config, target) + && !cachedProviderQuotaIsExhausted(getCachedProviderQuota(target.provider, now), now) + && !excluded.has(targetKey(target)) + && isComboTargetInCooldown(comboId, target, now) + && (customEligible?.(target) ?? true), + ); + const earliest = earliestComboCooldown(comboId, waitingTargets, now); + if (earliest === undefined) return null; + const delay = earliest.expiry - now; + if (delay > options.waitForCooldownMs) return null; + // The expiry computation above is the single source of truth for the wait budget. + // Its target preserves configured order for ties. + const target = earliest.target; + console.warn( + `[combo] ${comboId}: all targets cooling, waiting ${delay}ms for ${targetKey(target)}`, + ); + try { + await (options.sleep ?? sleepWithAbort)(delay, options.abortSignal); + } catch (error) { + if (options.abortSignal?.aborted) return null; + throw error; + } + if (options.abortSignal?.aborted) return null; + return pickComboTarget(config, comboId, { + exclude: excluded, + now: now + delay, + eligible: targetCandidate => + !isComboTargetInCooldown(comboId, targetCandidate, now + delay) + && (customEligible?.(targetCandidate) ?? true), + }); +} + export function reconcileComboRotationState(context: GenerationContext): number { if (context.generation <= lastReconciledGeneration) return 0; let removed = 0; diff --git a/src/combos/types.ts b/src/combos/types.ts index b3dec16d097..06055300462 100644 --- a/src/combos/types.ts +++ b/src/combos/types.ts @@ -11,6 +11,7 @@ import type { } from "../types"; export const COMBO_NAMESPACE = "combo"; +export const COMBO_DEFAULT_WAIT_FOR_COOLDOWN_MS = 0; export function preservesPhysicalComboProvider( config: Pick, @@ -37,6 +38,8 @@ export interface ComboValidationIssue { export interface NormalizedComboConfig { strategy: OcxComboStrategy; stickyLimit: number; + cooldownMs?: number; + waitForCooldownMs: number; defaultEffort: OcxComboDefaultEffort | null; /** Picker-ladder derivation policy; `strict` preserves the legacy intersection rule. */ reasoningEffortMode: OcxComboReasoningEffortMode; @@ -230,6 +233,18 @@ export function comboConfigIssues( || body.stickyLimit > 100)) { issues.push({ path: ["stickyLimit"], message: "stickyLimit must be an integer from 1 to 100" }); } + if (body.cooldownMs !== undefined + && (typeof body.cooldownMs !== "number" || !Number.isInteger(body.cooldownMs) + || body.cooldownMs < 1 + || body.cooldownMs > 600_000)) { + issues.push({ path: ["cooldownMs"], message: "cooldownMs must be an integer from 1 to 600000" }); + } + if (body.waitForCooldownMs !== undefined + && (typeof body.waitForCooldownMs !== "number" || !Number.isInteger(body.waitForCooldownMs) + || body.waitForCooldownMs < 0 + || body.waitForCooldownMs > 600_000)) { + issues.push({ path: ["waitForCooldownMs"], message: "waitForCooldownMs must be an integer from 0 to 600000" }); + } if (body.defaultEffort !== undefined && body.defaultEffort !== null && (typeof body.defaultEffort !== "string" || !isCodexReasoningEffort(body.defaultEffort))) { @@ -367,6 +382,8 @@ export function normalizeComboConfig(raw: OcxComboConfig): NormalizedComboConfig return { strategy: raw.strategy ?? "failover", stickyLimit: raw.stickyLimit ?? 1, + cooldownMs: raw.cooldownMs, + waitForCooldownMs: raw.waitForCooldownMs ?? COMBO_DEFAULT_WAIT_FOR_COOLDOWN_MS, defaultEffort: raw.defaultEffort ?? null, reasoningEffortMode: raw.reasoningEffortMode === "adaptive" ? "adaptive" : "strict", imageInput: raw.imageInput === "disabled" ? "disabled" : "auto", diff --git a/src/server/management/combo-routes.ts b/src/server/management/combo-routes.ts index 72282d54450..475e72db410 100644 --- a/src/server/management/combo-routes.ts +++ b/src/server/management/combo-routes.ts @@ -52,7 +52,7 @@ import { setDebugSettings, type DebugFlag, } from "../../lib/debug-settings"; -import type { OcxClaudeCodeConfig, OcxConfig, OcxCustomModel, OcxProviderConfig } from "../../types"; +import type { OcxClaudeCodeConfig, OcxComboConfig, OcxConfig, OcxCustomModel, OcxProviderConfig } from "../../types"; import { drainAndShutdown } from "../lifecycle"; import { reconcileLiveStateStores } from "../../lib/state-store-registrations"; import { filterRequestLogs, getRequestLogEntries, type RequestLogEntry } from "../request-log"; @@ -66,6 +66,7 @@ import type { MetricUnavailableReason, TokPerSecondResult, CostEstimateReason, C import type { ManagementContext } from "./context"; import { readManagementJsonBody, rethrowManagementBodyTooLarge } from "./body"; import { shadowCallTargetError } from "./shadow-call-validation"; +import { COMBO_DEFAULT_WAIT_FOR_COOLDOWN_MS } from "../../combos"; /** @@ -75,15 +76,29 @@ import { shadowCallTargetError } from "./shadow-call-validation"; * materialized in every user's config.json. */ function sparseComboConfig(combo: T): Omit & { +}>(combo: T): Omit & { + cooldownMs?: number; + waitForCooldownMs?: number; imageInput?: "disabled"; reasoningEffortMode?: "adaptive"; } { - const { imageInput, reasoningEffortMode, ...rest } = combo; + const { + cooldownMs, + waitForCooldownMs, + imageInput, + reasoningEffortMode, + ...rest + } = combo; return { ...rest, + ...(cooldownMs !== undefined ? { cooldownMs } : {}), + ...(waitForCooldownMs !== undefined && waitForCooldownMs !== COMBO_DEFAULT_WAIT_FOR_COOLDOWN_MS + ? { waitForCooldownMs } + : {}), ...(imageInput === "disabled" ? { imageInput: "disabled" as const } : {}), ...(reasoningEffortMode === "adaptive" ? { reasoningEffortMode: "adaptive" as const } : {}), }; @@ -141,13 +156,28 @@ export async function handleComboRoutes(ctx: ManagementContext): Promise = body.combo; + const effectiveCombo = { + ...requestedCombo, + ...(!Object.hasOwn(requestedCombo, "cooldownMs") && previous?.cooldownMs !== undefined + ? { cooldownMs: previous.cooldownMs } + : {}), + ...(!Object.hasOwn(requestedCombo, "waitForCooldownMs") && previous?.waitForCooldownMs !== undefined + ? { waitForCooldownMs: previous.waitForCooldownMs } + : {}), + }; + const error = comboConfigError(id, effectiveCombo, config.providers, { requireEnabledTarget: true, combos: config.combos, - excludeComboId: renameFrom ?? id, + excludeComboId: sourceId, }); if (error) return jsonResponse({ error }, 400); - const normalized = normalizeComboConfig(body.combo as import("../../types").OcxComboConfig); + const normalized = normalizeComboConfig(effectiveCombo as unknown as OcxComboConfig); // Persist only non-default identity/capability fields so config stays sparse. // Capability defaults (`imageInput`, `reasoningEffortMode`) go through the same // helper the GET/PUT responses use, so the wire shape and the stored shape cannot drift. @@ -157,14 +187,12 @@ export async function handleComboRoutes(ctx: ManagementContext): Promise= 500 && response.status < 600 + && body.displaySafe && !body.truncated + ) { + const quotaMessage = codexQuotaFailureMessage(body.text); + quotaConfirmedByBody = quotaMessage !== undefined + && isRateLimitOrQuotaFailureMessage(quotaMessage); + } if (body.displaySafe) { const normalized = normalizeUpstreamErrorText(body.text, fallback); classificationText = normalized.safeText; @@ -1616,18 +1632,24 @@ export async function consumeComboFailure( ? fallback : `${fallback}: ${classificationText}`; const upstreamRetryAfter = response.headers.get("retry-after"); + // Past HTTP dates are an immediate retry directive, just like the numeric value zero. + // Normalize before the client helper discards them and substitutes a default delay. + const effectiveRetryAfter = parseRetryAfterMs(upstreamRetryAfter, now) === undefined + && parseRetryAfterMs(upstreamRetryAfter, now, { preserveImmediate: true }) !== undefined + ? "0" + : upstreamRetryAfter; // Client response may get the synthetic "2" fallback; cooldown metadata must not — // otherwise coolComboTarget treats it as a 2s cooldown instead of the 60s default. const clientRetryAfter = resolveClientRetryAfter({ status: response.status, message, - upstreamRetryAfter, + upstreamRetryAfter: effectiveRetryAfter, now, }); const cooldownRetryAfter = resolveClientRetryAfter({ status: response.status, message, - upstreamRetryAfter, + upstreamRetryAfter: effectiveRetryAfter, now, includeDefault: false, }); @@ -1644,6 +1666,14 @@ export async function consumeComboFailure( classificationText, ...(normalizedUpstreamCode !== undefined ? { upstreamCode: normalizedUpstreamCode } : {}), ...(!cyberFailure && cooldownRetryAfter !== undefined ? { retryAfter: cooldownRetryAfter } : {}), + // The EFFECTIVE classification decides, not the raw status. An upstream that wraps a quota + // refusal in a 5xx still carries `x-codex-*-reset-at`, and gating on 402/429 alone threw + // those away, so the combo target came back up immediately instead of waiting for the + // window it was told about. `cyberFailure` stays excluded: a policy block is not a quota. + ...(!cyberFailure + && (response.status === 429 || response.status === 402 || quotaConfirmedByBody) + ? { resetAt: codexQuotaOutcomeMeta(response).resetAt } + : {}), ...(usage ? { usage } : {}), }; } @@ -2320,6 +2350,15 @@ export async function handleComboResponses( comboPayloadReadable || !unreadableEncryptedAgentTask || canDecryptUnreadableAgentTask(target); const initialNow = Date.now(); let pick: ReturnType = null; + const pickWithWait = (pickOptions: { + exclude?: Iterable; + eligible?: (target: NonNullable["targets"][number]) => boolean; + now?: number; + }) => pickComboTargetWithWait(config, comboId, { + ...pickOptions, + waitForCooldownMs: combo.waitForCooldownMs, + abortSignal: options.abortSignal, + }); if (unreadableEncryptedAgentTask && !combo.targets.some(canDecryptUnreadableAgentTask)) { const recovery = agentTaskRecoveryConfig(config); @@ -2337,9 +2376,7 @@ export async function handleComboResponses( ); return unreadableEncryptedAgentTaskResponse(); } - pick = pickComboTarget(config, comboId, { - eligible: target => !isComboTargetInCooldown(comboId, target, initialNow), - }); + pick = await pickWithWait({ now: initialNow }); if (!pick) { discardEncryptedAgentTaskRecovery( req, @@ -2347,7 +2384,9 @@ export async function handleComboResponses( config, { parentThreadId: inboundClientThreadId }, ); - return comboUnavailable(comboId); + return options.abortSignal?.aborted + ? clientCancelledResponse() + : comboUnavailable(comboId); } let recovered = false; try { @@ -2377,14 +2416,16 @@ export async function handleComboResponses( comboPayloadReadable = true; comboReplaySnapshot.recoveredPlaintext = true; } else { - pick = pickComboTarget(config, comboId, { - eligible: target => payloadEligible(target) - && !isComboTargetInCooldown(comboId, target, initialNow), + pick = await pickWithWait({ + eligible: payloadEligible, + now: initialNow, }); } if (!pick) { - return comboUnavailable(comboId); + return options.abortSignal?.aborted + ? clientCancelledResponse() + : comboUnavailable(comboId); } // One immutable combo selection trace, before any child dispatch; child // adoption below must never replace it with a concrete child route trace. @@ -2588,9 +2629,12 @@ export async function handleComboResponses( console.warn( `[combo] ${comboId}: ${targetKey(pick.target)} failed with ${failure.response.status} after ${Date.now() - started}ms`, ); + const failureNow = Date.now(); const nextPick = advanceComboAfterFailure(config, pick, { retryAfter: failure.retryAfter, - now: Date.now(), + resetAt: failure.resetAt, + cooldownMs: combo.cooldownMs, + now: failureNow, cooldownScope: comboFailureCooldownScope(failure.response.status, failure.classificationText, { code: failure.upstreamCode, }), @@ -2599,8 +2643,19 @@ export async function handleComboResponses( code: failure.upstreamCode, message: failure.classificationText, }); - if (!nextPick) adoptFailedChildLog(childLog); - pick = nextPick; + if (nextPick) { + pick = nextPick; + } else { + pick = await pickWithWait({ + exclude: pick.attempted, + eligible: payloadEligible, + now: failureNow, + }); + } + if (!pick) { + if (options.abortSignal?.aborted) return clientCancelledResponse(); + adoptFailedChildLog(childLog); + } } if ( lastFailure?.status === 413 diff --git a/src/types/config.ts b/src/types/config.ts index fd2d8c46f20..e7286f97ec5 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -858,6 +858,14 @@ export interface OcxComboConfig { strategy?: OcxComboStrategy; /** Successful requests retained on one RR selection batch. Default 1; range 1..100. */ stickyLimit?: number; + /** + * Optional per-target cooldown used only when the upstream response has no Retry-After or Codex reset signal. + * Unset uses the upstream fallback (5 s for request-rate 429 codes 1302/1305, otherwise 60 s); + * an explicit value overrides that fallback. Range 1..600000. + */ + cooldownMs?: number; + /** Maximum wait for an eligible target cooldown to expire before failing closed. Default 0; range 0..600000, per selection attempt. */ + waitForCooldownMs?: number; /** Used when the client omits reasoning.effort. null/omitted leaves the target default unchanged. */ defaultEffort?: OcxComboDefaultEffort | null; /** diff --git a/tests/codex-integration/combos.test.ts b/tests/codex-integration/combos.test.ts index a1c40616c0b..0261edd9037 100644 --- a/tests/codex-integration/combos.test.ts +++ b/tests/codex-integration/combos.test.ts @@ -1,4 +1,4 @@ -import { afterEach, describe, expect, spyOn, test } from "bun:test"; +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; import { mkdirSync, mkdtempSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -6,6 +6,8 @@ import { advanceComboAfterFailure, clearComboSelectionState, clearComboTargetCooldowns, + earliestComboCooldownExpiry, + pickComboTargetWithWait, comboAliasIssues, comboConfigError, comboConfigIssues, @@ -169,6 +171,12 @@ async function responseJson(response: Response | null): Promise>; } +beforeEach(() => { + clearComboSelectionState(); + clearComboTargetCooldowns(); + clearCachedProviderQuotas(); +}); + afterEach(() => { clearComboSelectionState(); clearComboTargetCooldowns(); @@ -423,6 +431,31 @@ describe("combo target cooldowns", () => { expect(isComboTargetInCooldown("other", target, 1_050)).toBe(false); }); + test("gives immediate Retry-After directives precedence over configured cooldowns", () => { + const now = Date.parse("2026-07-18T00:00:00.000Z"); + const pastDate = new Date(now - 1_000).toUTCString(); + coolComboTarget("retry-zero", target, { now, retryAfter: "0", cooldownMs: 5_000 }); + coolComboTarget("retry-past-date", target, { now, retryAfter: pastDate, cooldownMs: 5_000 }); + expect(isComboTargetInCooldown("retry-zero", target, now)).toBe(true); + expect(isComboTargetInCooldown("retry-zero", target, now + 1)).toBe(false); + expect(isComboTargetInCooldown("retry-past-date", target, now)).toBe(true); + expect(isComboTargetInCooldown("retry-past-date", target, now + 1)).toBe(false); + }); + + test("uses reset-derived cooldown before combo cooldownMs and default cooldown", () => { + const now = 1_000_000; + const resetAt = Math.floor((now + 20_000) / 1_000); + coolComboTarget("reset-wins", target, { now, resetAt, cooldownMs: 5_000 }); + coolComboTarget("configured-wins", target, { now, cooldownMs: 5_000 }); + coolComboTarget("default-wins", target, { now }); + expect(isComboTargetInCooldown("reset-wins", target, now + 5_000)).toBe(true); + expect(isComboTargetInCooldown("reset-wins", target, now + 20_000)).toBe(false); + expect(isComboTargetInCooldown("configured-wins", target, now + 5_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("configured-wins", target, now + 5_000)).toBe(false); + expect(isComboTargetInCooldown("default-wins", target, now + 60_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("default-wins", target, now + 60_000)).toBe(false); + }); + test("uses a short cooldown for request-rate 1302 without Retry-After", () => { coolComboTarget("free", target, { now: 1_000, @@ -431,6 +464,30 @@ describe("combo target cooldowns", () => { }); expect(isComboTargetInCooldown("free", target, 1_000 + COMBO_REQUEST_RATE_COOLDOWN_MS - 1)).toBe(true); expect(isComboTargetInCooldown("free", target, 1_000 + COMBO_REQUEST_RATE_COOLDOWN_MS)).toBe(false); + + const config = baseConfig({ combos: { free: VALID_COMBO } }); + const pick = pickComboTarget(config, "free", { now: 1_000 })!; + advanceComboAfterFailure(config, pick, { + now: 1_000, + status: 429, + code: "1302", + message: "Rate limit reached for requests", + }); + expect(isComboTargetInCooldown("free", pick.target, 1_000 + COMBO_REQUEST_RATE_COOLDOWN_MS - 1)).toBe(true); + expect(isComboTargetInCooldown("free", pick.target, 1_000 + COMBO_REQUEST_RATE_COOLDOWN_MS)).toBe(false); + + const configured = baseConfig({ + combos: { free: { ...VALID_COMBO, cooldownMs: 7_000 } }, + }); + const configuredPick = pickComboTarget(configured, "free", { now: 1_000 })!; + advanceComboAfterFailure(configured, configuredPick, { + now: 1_000, + status: 429, + code: "1302", + message: "Rate limit reached for requests", + }); + expect(isComboTargetInCooldown("free", configuredPick.target, 1_000 + 7_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("free", configuredPick.target, 1_000 + 7_000)).toBe(false); }); test("keeps the default cooldown for usage-window 1308", () => { @@ -471,6 +528,228 @@ describe("combo target cooldowns", () => { expect(response.status).toBe(503); expect(response.headers.get("Retry-After")).toBe("5"); }); + + test("applies Retry-After before reset-derived and combo cooldown values", () => { + const now = 1_000_000; + coolComboTarget("retry", target, { + now, + retryAfter: "10", + cooldownMs: 5_000, + }); + expect(isComboTargetInCooldown("retry", target, now + 10_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("retry", target, now + 10_000)).toBe(false); + + const resetAt = Math.floor((now + 20_000) / 1_000); + coolComboTarget("reset", target, { + now, + resetAt, + cooldownMs: 5_000, + }); + expect(isComboTargetInCooldown("reset", target, now + 20_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("reset", target, now + 20_000)).toBe(false); + + const config = baseConfig({ + combos: { free: { ...VALID_COMBO, cooldownMs: 5_000 } }, + }); + const pick = pickComboTarget(config, "free", { now })!; + advanceComboAfterFailure(config, pick, { + now, + retryAfter: "10", + resetAt, + }); + expect(isComboTargetInCooldown("free", pick.target, now + 10_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("free", pick.target, now + 10_000)).toBe(false); + }); + + test("uses combo cooldownMs instead of the 60-second default", () => { + // The production path passes this normalized combo value to coolComboTarget after a failure. + const config = baseConfig({ combos: { free: { ...VALID_COMBO, cooldownMs: 5_000 } } }); + expect(normalizeComboConfig(config.combos!.free!).cooldownMs).toBe(5_000); + expect(normalizeComboConfig(config.combos!.free!).waitForCooldownMs).toBe(0); + + const now = 1_000_000; + coolComboTarget("short", target, { now, cooldownMs: 5_000 }); + coolComboTarget("default", target, { now }); + expect(isComboTargetInCooldown("short", target, now + 5_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("short", target, now + 5_000)).toBe(false); + expect(isComboTargetInCooldown("default", target, now + 5_000)).toBe(true); + expect(isComboTargetInCooldown("default", target, now + 60_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("default", target, now + 60_000)).toBe(false); + }); + + test("finds the earliest cooldown expiry and waits within the budget", async () => { + const config = baseConfig({ + combos: { + free: { + targets: [ + { provider: "a", model: "m1" }, + { provider: "b", model: "m2" }, + ], + waitForCooldownMs: 10_000, + }, + }, + }); + const now = 1_000_000; + coolComboTarget("free", config.combos!.free!.targets[0]!, { now, cooldownMs: 3_000 }); + coolComboTarget("free", config.combos!.free!.targets[1]!, { now, cooldownMs: 8_000 }); + expect(earliestComboCooldownExpiry("free", config.combos!.free!.targets, now)).toBe(now + 3_000); + + const sleeps: number[] = []; + const pick = await pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 10_000, + sleep: async ms => { sleeps.push(ms); }, + }); + expect(sleeps).toHaveLength(1); + expect(sleeps[0]).toBeCloseTo(3_000, -2); + expect(pick?.target.provider).toBe("a"); + expect(earliestComboCooldownExpiry("free", config.combos!.free!.targets, now + 3_000)).toBe(now + 8_000); + expect(earliestComboCooldownExpiry("free", [{ provider: "c", model: "m3" }], now)).toBeUndefined(); + }); + + test("logs and waits for the target with the earliest cooldown expiry", async () => { + const config = baseConfig({ + combos: { + free: { + targets: [ + { provider: "a", model: "m1" }, + { provider: "b", model: "m2" }, + ], + waitForCooldownMs: 10_000, + }, + }, + }); + const now = 1_000_000; + coolComboTarget("free", config.combos!.free!.targets[0]!, { now, cooldownMs: 8_000 }); + coolComboTarget("free", config.combos!.free!.targets[1]!, { now, cooldownMs: 3_000 }); + const sleeps: number[] = []; + const warning = spyOn(console, "warn").mockImplementation(() => {}); + try { + const pick = await pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 10_000, + sleep: async ms => { sleeps.push(ms); }, + }); + expect(sleeps).toEqual([3_000]); + expect(pick?.target).toMatchObject({ provider: "b", model: "m2" }); + expect(warning).toHaveBeenCalledWith( + "[combo] free: all targets cooling, waiting 3000ms for b/m2", + ); + } finally { + warning.mockRestore(); + } + }); + + test("returns null when real sleepWithAbort rejects after an abort", async () => { + const config = baseConfig({ + combos: { + free: { + targets: [{ provider: "a", model: "m1" }], + waitForCooldownMs: 100, + }, + }, + }); + const now = Date.now(); + coolComboTarget("free", target, { now, cooldownMs: 25 }); + const abort = new AbortController(); + const pending = pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 100, + abortSignal: abort.signal, + }); + setTimeout(() => abort.abort(new Error("test abort")), 5); + expect(await pending).toBeNull(); + }); + + test("fails closed without waiting and honors abort during injected sleep", async () => { + const config = baseConfig({ + combos: { + free: { + targets: [{ provider: "a", model: "m1" }], + waitForCooldownMs: 10_000, + }, + }, + }); + const now = 1_000_000; + coolComboTarget("free", target, { now, cooldownMs: 3_000 }); + const noWait: number[] = []; + expect(await pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 0, + sleep: async ms => { noWait.push(ms); }, + })).toBeNull(); + expect(noWait).toEqual([]); + + const abort = new AbortController(); + let slept = false; + const pending = pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 10_000, + abortSignal: abort.signal, + sleep: async () => { + slept = true; + abort.abort(); + }, + }); + expect(await pending).toBeNull(); + expect(slept).toBe(true); + }); + + test("does not wait for cooling targets with exhausted provider quota", async () => { + const now = 50_000; + const config = baseConfig({ + combos: { + free: { + targets: [{ provider: "a", model: "m1" }], + waitForCooldownMs: 5_000, + }, + }, + }); + setCachedProviderQuotaForTests("a", { + monthlyPercent: 100, + monthlyResetAt: now + 14 * 24 * 60 * 60_000, + updatedAt: now, + }); + coolComboTarget("free", target, { now, cooldownMs: 1_000 }); + const sleeps: number[] = []; + expect(await pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 5_000, + sleep: async ms => { sleeps.push(ms); }, + })).toBeNull(); + expect(sleeps).toEqual([]); + }); + + test("fails closed without sleeping when cooldown expiry exceeds the wait budget", async () => { + const config = baseConfig({ + combos: { + free: { + targets: [{ provider: "a", model: "m1" }], + waitForCooldownMs: 10_000, + }, + }, + }); + const now = 1_000_000; + coolComboTarget("free", target, { now, cooldownMs: 10_000 }); + const sleeps: number[] = []; + expect(await pickComboTargetWithWait(config, "free", { + now, + waitForCooldownMs: 1_000, + sleep: async ms => { sleeps.push(ms); }, + })).toBeNull(); + expect(sleeps).toEqual([]); + }); + + test("applies combo cooldownMs to a failure-derived cooldown", () => { + const config = baseConfig({ + combos: { free: { ...VALID_COMBO, cooldownMs: 5_000 } }, + }); + const combo = getCombo(config, "free")!; + const pick = pickComboTarget(config, "free")!; + advanceComboAfterFailure(config, pick, { now: 1_000_000 }); + expect(isComboTargetInCooldown("free", combo.targets[0]!, 1_000_000 + 5_000 - 1)).toBe(true); + expect(isComboTargetInCooldown("free", combo.targets[0]!, 1_000_000 + 5_000)).toBe(false); + }); }); describe("combo failure policy and advancement", () => { @@ -980,6 +1259,27 @@ describe("combo validation and normalization", () => { } }); + test("validates cooldown knobs and supplies sparse-safe defaults", () => { + const providers = baseConfig().providers; + const cooldownValues: unknown[] = [0, 1.5, 600_001, -1, "5000"]; + for (const value of cooldownValues) { + expect(comboConfigIssues("free", { ...VALID_COMBO, cooldownMs: value }, providers)).toEqual( + expect.arrayContaining([expect.objectContaining({ path: ["cooldownMs"] })]), + ); + } + const waitValues: unknown[] = [1.5, 600_001, -1, "5000"]; + for (const value of waitValues) { + expect(comboConfigIssues("free", { ...VALID_COMBO, waitForCooldownMs: value }, providers)).toEqual( + expect.arrayContaining([expect.objectContaining({ path: ["waitForCooldownMs"] })]), + ); + } + expect(comboConfigIssues("free", { ...VALID_COMBO, waitForCooldownMs: 0 }, providers)) + .not.toEqual(expect.arrayContaining([expect.objectContaining({ path: ["waitForCooldownMs"] })])); + const normalized = normalizeComboConfig(VALID_COMBO); + expect(normalized.cooldownMs).toBeUndefined(); + expect(normalized.waitForCooldownMs).toBe(0); + }); + test("normalizes valid values and returns defensive default efforts", () => { expect(normalizeComboConfig({ defaultEffort: "high", @@ -987,6 +1287,7 @@ describe("combo validation and normalization", () => { })).toEqual({ strategy: "failover", stickyLimit: 1, + waitForCooldownMs: 0, defaultEffort: "high", reasoningEffortMode: "strict", imageInput: "auto", diff --git a/tests/providers/cyber-policy-error-fidelity.test.ts b/tests/providers/cyber-policy-error-fidelity.test.ts index adcf36be018..f949634b3cd 100644 --- a/tests/providers/cyber-policy-error-fidelity.test.ts +++ b/tests/providers/cyber-policy-error-fidelity.test.ts @@ -179,6 +179,21 @@ describe("cyber_policy error fidelity", () => { }); }); + test("drops Codex reset headers as well as Retry-After for a cyber-policy failure", async () => { + const upstream = new Response(JSON.stringify(CYBER_ERROR_BODY), { + status: 429, + headers: { + "retry-after": "120", + "x-codex-primary-reset-at": "2026-09-03T12:00:00Z", + "x-codex-secondary-reset-at": "2026-09-03T13:00:00Z", + "x-codex-tertiary-reset-at": "2026-09-03T14:00:00Z", + }, + }); + const failure = await consumeComboFailure(upstream); + expect(failure.retryAfter).toBeUndefined(); + expect(failure.resetAt).toBeUndefined(); + }); + test("ordinary Responses HTTP failure preserves structured cyber type and exact safe message", async () => { const upstream = Bun.serve({ port: 0, diff --git a/tests/routing/combo-management-api.test.ts b/tests/routing/combo-management-api.test.ts index 2edf73fd546..85f6be0ff6f 100644 --- a/tests/routing/combo-management-api.test.ts +++ b/tests/routing/combo-management-api.test.ts @@ -274,6 +274,136 @@ describe("combo management API", () => { }); }); + test("PUT preserves explicit cooldown knobs and omits sparse defaults", async () => { + await withTempHome(async () => { + const config = baseConfig({ combos: undefined }); + saveConfig(config); + const explicit = await comboApi(config, "PUT", "/api/combos", { + id: "timed", + combo: { + cooldownMs: 5_000, + waitForCooldownMs: 15_000, + targets: [{ provider: "a", model: "m1" }], + }, + }); + expect(explicit?.status).toBe(200); + const explicitBody = await responseJson(explicit); + expect(explicitBody.combo).toMatchObject({ cooldownMs: 5_000, waitForCooldownMs: 15_000 }); + const persistedExplicit = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + expect(persistedExplicit.combos?.timed).toMatchObject({ + cooldownMs: 5_000, + waitForCooldownMs: 15_000, + }); + const listedExplicit = await responseJson(await comboApi(config, "GET", "/api/combos")); + expect((listedExplicit.combos as Array>).find(row => row.id === "timed")) + .toMatchObject({ cooldownMs: 5_000, waitForCooldownMs: 15_000 }); + + const explicitDefault = await comboApi(config, "PUT", "/api/combos", { + id: "default-timed", + combo: { + cooldownMs: 60_000, + targets: [{ provider: "a", model: "m1" }], + }, + }); + expect(explicitDefault?.status).toBe(200); + expect((await responseJson(explicitDefault)).combo).toMatchObject({ cooldownMs: 60_000 }); + expect(config.combos?.["default-timed"]?.cooldownMs).toBe(60_000); + const persistedExplicitDefault = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + expect(persistedExplicitDefault.combos?.["default-timed"]).toMatchObject({ cooldownMs: 60_000 }); + expect(persistedExplicitDefault.combos?.["default-timed"]).not.toHaveProperty("waitForCooldownMs"); + const listedDefault = await responseJson(await comboApi(config, "GET", "/api/combos")); + expect((listedDefault.combos as Array>).find(row => row.id === "default-timed")) + .toMatchObject({ cooldownMs: 60_000 }); + + // Dashboard-shaped updates omit controls it does not render; the server must preserve + // the already persisted values instead of replacing them with sparse defaults. + const dashboardUpdate = await comboApi(config, "PUT", "/api/combos", { + id: "timed", + combo: { targets: [{ provider: "a", model: "m1" }] }, + }); + expect(dashboardUpdate?.status).toBe(200); + const persistedAfterDashboardUpdate = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + expect(persistedAfterDashboardUpdate.combos?.timed).toMatchObject({ + cooldownMs: 5_000, + waitForCooldownMs: 15_000, + }); + const afterDashboardUpdate = await responseJson(await comboApi(config, "GET", "/api/combos")); + expect((afterDashboardUpdate.combos as Array>).find(row => row.id === "timed")) + .toMatchObject({ cooldownMs: 5_000, waitForCooldownMs: 15_000 }); + + const sparse = await comboApi(config, "PUT", "/api/combos", { + id: "plain", + combo: { targets: [{ provider: "a", model: "m1" }] }, + }); + expect(sparse?.status).toBe(200); + const listedSparse = await responseJson(await comboApi(config, "GET", "/api/combos")); + const plain = (listedSparse.combos as Array>).find(row => row.id === "plain"); + expect(plain).toBeDefined(); + expect(plain).not.toHaveProperty("cooldownMs"); + expect(plain).not.toHaveProperty("waitForCooldownMs"); + const persistedSparse = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + const persistedPlain = persistedSparse.combos?.plain; + expect(persistedPlain).toBeDefined(); + expect(persistedPlain).not.toHaveProperty("cooldownMs"); + expect(persistedPlain).not.toHaveProperty("waitForCooldownMs"); + expect(config.combos?.plain).not.toHaveProperty("cooldownMs"); + expect(config.combos?.plain).not.toHaveProperty("waitForCooldownMs"); + }); + }); + + test("PUT resets an explicit zero waitForCooldownMs to the sparse default", async () => { + await withTempHome(async () => { + const config = baseConfig({ combos: undefined }); + saveConfig(config); + + const created = await comboApi(config, "PUT", "/api/combos", { + id: "timed", + combo: { + cooldownMs: 5_000, + waitForCooldownMs: 15_000, + targets: [{ provider: "a", model: "m1" }], + }, + }); + expect(created?.status).toBe(200); + expect(JSON.parse(readFileSync(getConfigPath(), "utf8"))).toMatchObject({ + combos: { timed: { cooldownMs: 5_000, waitForCooldownMs: 15_000 } }, + }); + + const reset = await comboApi(config, "PUT", "/api/combos", { + id: "timed", + combo: { + waitForCooldownMs: 0, + targets: [{ provider: "a", model: "m1" }], + }, + }); + expect(reset?.status).toBe(200); + + const listed = await responseJson(await comboApi(config, "GET", "/api/combos")); + const timed = (listed.combos as Array>).find(row => row.id === "timed"); + expect(timed).toMatchObject({ cooldownMs: 5_000 }); + expect(timed).not.toHaveProperty("waitForCooldownMs"); + + const persisted = JSON.parse(readFileSync(getConfigPath(), "utf8")) as OcxConfig; + expect(persisted.combos?.timed).toMatchObject({ cooldownMs: 5_000 }); + expect(persisted.combos?.timed).not.toHaveProperty("waitForCooldownMs"); + }); + }); + + test("PUT rejects missing and null combo values with the legacy object error", async () => { + await withTempHome(async () => { + const config = baseConfig(); + saveConfig(config); + const before = readFileSync(getConfigPath(), "utf8"); + for (const body of [{ id: "x" }, { id: "x", combo: null }]) { + const response = await comboApi(config, "PUT", "/api/combos", body); + expect(response?.status).toBe(400); + expect(await responseJson(response)).toEqual({ error: "combo must be an object" }); + } + expect(config).toEqual(baseConfig()); + expect(readFileSync(getConfigPath(), "utf8")).toBe(before); + }); + }); + test("PUT persists explicit imageInput disabled", async () => { await withTempHome(async () => { const config = baseConfig({ combos: undefined }); diff --git a/tests/server/server-combo-failover-e2e.test.ts b/tests/server/server-combo-failover-e2e.test.ts index f01878c36aa..14cdc0ceab3 100644 --- a/tests/server/server-combo-failover-e2e.test.ts +++ b/tests/server/server-combo-failover-e2e.test.ts @@ -38,6 +38,7 @@ import { import { clearCursorThreadContinuityForTests } from "../../src/adapters/cursor/thread-continuity"; import { COMPACT_PROMPT, encodeCompactionSummary } from "../../src/responses/compaction"; import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { consumeComboFailure } from "../../src/server/responses/core"; // Full-suite Windows load: startServer + combo rename/delete management flows exceed the // default 5s per-test budget (same flake class as 810fa115 / claude-management-api). @@ -1606,6 +1607,152 @@ describe("server combo failover 030 activation matrix", () => { expect(bHits).toBe(2); }); + test("waits in the hop-level path when later targets are already cooling", async () => { + const t0 = Date.parse("2026-07-18T00:00:00.000Z"); + Date.now = () => t0; + let aHits = 0; + let bHits = 0; + let cHits = 0; + const a = serve(() => { + aHits += 1; + return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + }); + const b = serve(() => { + bHits += 1; + return bHits === 1 + ? Response.json({ error: { message: "rate limited" } }, { status: 429 }) + : chatSuccess("first cooled target recovered", "m2"); + }); + const c = serve(() => { + cHits += 1; + return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + }); + const providers = { + a: provider("openai-chat", baseUrl(a), "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + c: provider("openai-chat", baseUrl(c), "key-c"), + }; + const cooldown = { cooldownMs: 200, waitForCooldownMs: 1_000 }; + + // The prior request only includes B and C, so it cools those targets while A remains + // eligible for the fresh request below. + const prior = await post(comboConfig(providers, [ + { provider: "b", model: "m2" }, + { provider: "c", model: "m3" }, + ], cooldown)); + expect(prior.status).toBe(429); + await prior.text(); + expect([aHits, bHits, cHits]).toEqual([0, 1, 1]); + + const warnings: unknown[][] = []; + const originalWarn = console.warn; + console.warn = (...args: unknown[]) => { warnings.push(args); }; + try { + const fresh = await post(comboConfig(providers, [ + { provider: "a", model: "m1" }, + { provider: "b", model: "m2" }, + { provider: "c", model: "m3" }, + ], cooldown)); + expect(fresh.status).toBe(200); + expect(await fresh.text()).toContain("first cooled target recovered"); + } finally { + console.warn = originalWarn; + } + + expect([aHits, bHits, cHits]).toEqual([1, 2, 1]); + expect(warnings + .filter(args => String(args[0]).includes("all targets cooling, waiting")) + .map(args => args[0])) + .toEqual(["[combo] free: all targets cooling, waiting 200ms for b/m2"]); + }); + + test("returns immediate 503 when all targets cool and the wait budget is unset", async () => { + const t0 = Date.parse("2026-07-18T00:00:00.000Z"); + Date.now = () => t0; + let aHits = 0; + let bHits = 0; + let cHits = 0; + const a = serve(() => { + aHits += 1; + return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + }); + const b = serve(() => { + bHits += 1; + return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + }); + const c = serve(() => { + cHits += 1; + return Response.json({ error: { message: "rate limited" } }, { status: 429 }); + }); + const config = comboConfig({ + a: provider("openai-chat", baseUrl(a), "key-a"), + b: provider("openai-chat", baseUrl(b), "key-b"), + c: provider("openai-chat", baseUrl(c), "key-c"), + }, [ + { provider: "a", model: "m1" }, + { provider: "b", model: "m2" }, + { provider: "c", model: "m3" }, + ], { cooldownMs: 200 }); + + const exhausted = await post(config); + expect(exhausted.status).toBe(429); + await exhausted.text(); + expect([aHits, bHits, cHits]).toEqual([1, 1, 1]); + + const unavailable = await post(config); + expect(unavailable.status).toBe(503); + expect(await unavailable.text()).toContain("No available targets for combo: free"); + expect([aHits, bHits, cHits]).toEqual([1, 1, 1]); + }); + + test("a past Retry-After date remains immediate through response consumption", async () => { + const now = Date.parse("2026-07-18T00:00:00.000Z"); + const failure = await consumeComboFailure(Response.json({ error: { message: "rate limited" } }, { + status: 429, + headers: { "retry-after": new Date(now - 1_000).toUTCString() }, + }), undefined, now); + expect(failure.retryAfter).toBe("0"); + expect(failure.response.headers.get("retry-after")).toBe("0"); + }); + + test("body-confirmed quota inside a 5xx still cools the target for its reset window", async () => { + // consumeComboFailure used to forward x-codex-*-reset-at only when the RAW status was + // 402/429, so an upstream that wrapped a quota refusal in a 5xx lost the window it had + // just advertised: the target fell back to the configured cooldownMs and came back up + // long before the quota did. shouldRetryCodexPoolAccountQuota already treats a + // body-confirmed quota message inside a 5xx as quota, and this is the same normalization + // applied to the cooldown path. One target only, so nothing masks the cooldown by + // failing over. + const t0 = Date.parse("2026-07-18T00:00:00.000Z"); + Date.now = () => t0; + const resetAt = Math.floor((t0 + 3 * 60 * 60_000) / 1000); + let hits = 0; + const a = serve(() => { + hits += 1; + return Response.json({ error: { message: "You exceeded your current quota" } }, { + status: 503, + headers: { "x-codex-primary-reset-at": String(resetAt) }, + }); + }); + const config = comboConfig( + { a: provider("openai-chat", baseUrl(a), "key-a") }, + [{ provider: "a", model: "m1" }], + { cooldownMs: 200 }, + ); + + const response = await post(config); + expect(response.status).toBe(503); + await response.text(); + expect(hits).toBe(1); + + const target = { provider: "a", model: "m1" }; + // The configured 200ms cooldown would have expired here; the advertised window has not. + expect(isComboTargetInCooldown("free", target, t0 + 60_000)).toBe(true); + expect(isComboTargetInCooldown("free", target, t0 + 9 * 60_000)).toBe(true); + // Reset metadata cannot extend the existing ten-minute combo cooldown ceiling. + expect(isComboTargetInCooldown("free", target, t0 + 10 * 60_000)).toBe(false); + }); + test("disabled image input rejects the request before any combo target is called", async () => { let hits = 0; const a = serve(() => {