🕐 feat: Add promptCacheTtl model parameter for 1h/5m cache duration - #13835
Conversation
Adds a user-configurable `promptCacheTtl` parameter (dropdown: 5m | 1h) alongside the existing `promptCache` toggle for Anthropic, Bedrock, and OpenRouter endpoints. Default is undefined so the agents SDK applies its own default (1h), letting users opt down to the legacy 5m TTL. - data-provider: schema, parameterSettings dropdown, types, bedrock picks - data-schemas: convo/preset types + mongoose defaults - api: thread promptCacheTtl into anthropic + openai(OpenRouter) llmConfig - i18n: en translation keys for label/description/default placeholder - tests: anthropic llm.spec coverage for set + unset cases
|
@codex review |
promptCacheTtl model parameter for 1h/5m cache duration
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e15bfa8a65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| 'topP', | ||
| 'stop', | ||
| 'promptCache', | ||
| 'promptCacheTtl', |
There was a problem hiding this comment.
Gate Bedrock 1h TTL to supported Claude models
When a Bedrock user selects 1h, this parser now preserves promptCacheTtl for every Bedrock model, while the cleanup below only toggles promptCache based on claude/nova and never clears or allowlists the TTL. Bedrock's extended 1h cache TTL is only accepted on select Claude 4.5 models; choosing it on other cache-enabled Bedrock models such as Nova or older Claude versions will still send the unsupported TTL and can make those requests fail. Clear promptCacheTtl when caching is off/unsupported and only pass 1h for the Bedrock models that support it.
Useful? React with 👍 / 👎.
| promptCacheTtl: | ||
| options.modelOptions?.promptCacheTtl ?? anthropicSettings.promptCacheTtl.default, |
There was a problem hiding this comment.
Honor custom Anthropic TTL defaults
This resolves promptCacheTtl only from modelOptions before defaultParams/addParams are processed, and the later loops only apply keys from knownAnthropicParams, which does not include this new field. For a custom Anthropic endpoint that exposes promptCacheTtl through customParams.paramDefinitions or forces it with addParams, the configured 5m/1h TTL is silently ignored and the SDK default is used instead. Thread this key through the same default/add handling as the other Anthropic system parameters.
Useful? React with 👍 / 👎.
| if (promptCacheTtl != null) { | ||
| llmConfig.promptCacheTtl = promptCacheTtl; |
There was a problem hiding this comment.
Apply OpenRouter TTL defaults before assigning
This copies promptCacheTtl only from modelOptions; unlike promptCache, the defaultParams and addParams branches above never update a TTL variable, and dropParams: ['promptCacheTtl'] cannot remove a selected value because this assignment happens after the drop checks. For OpenRouter custom endpoints using customParams.paramDefinitions or admin addParams/dropParams to control the new TTL, the setting is ignored or cannot be disabled. Resolve and drop the TTL alongside enablePromptCache before assigning it to llmConfig.
Useful? React with 👍 / 👎.
…TTL params (Codex review) - bedrock.ts: clear promptCacheTtl whenever promptCache is off/unsupported, so an unsupported 1h is never sent on a non-caching Bedrock request - openai/llm.ts: resolve promptCacheTtl through the same defaultParams/ addParams/dropParams machinery as promptCache (via promptCacheTtlValue) so OpenRouter custom endpoints can configure/override/drop it - tests: bedrock TTL-tied-to-promptCache cases; OpenRouter TTL default/add/drop
|
Addressed in P2 — Gate Bedrock 1h TTL to supported models ( P2 — Apply OpenRouter TTL defaults before assigning ( P2 — Honor custom Anthropic TTL defaults ( tsc clean; api (182) + data-provider (268) suites + lint green. @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b64aaf26c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (promptCacheTtlValue != null) { | ||
| llmConfig.promptCacheTtl = promptCacheTtlValue; |
There was a problem hiding this comment.
Honor OpenRouter TTL with default cache switch
This assignment only runs when enablePromptCache is exactly true. In the OpenRouter settings UI, promptCache is displayed as enabled by default, but DynamicSwitch does not persist default values until the switch is toggled; selecting only the new TTL dropdown therefore sends modelOptions.promptCacheTtl without modelOptions.promptCache. That common path silently drops the requested TTL, making the new control ineffective unless the user also toggles prompt caching or an admin defaultParams sets promptCache.
Useful? React with 👍 / 👎.
…t (Codex review) OPENROUTER_DEFAULT_PARAMS injects promptCache:true into defaultParams, so a TTL-only dropdown selection (promptCacheTtl set, promptCache switch untouched) still resolves caching on and forwards the TTL. Add regression tests via the real getOpenAIConfig entry point: TTL-only -> promptCache+TTL both set; explicit promptCache:false -> both dropped.
|
P2 — Honor OpenRouter TTL with default cache switch ( The existing baseline test already encodes this default-on behavior (
Deliberately keeping the TTL assignment inside the @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b80531e8dc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const openRouter: SettingsConfiguration = [ | ||
| ...openAI, | ||
| anthropic.promptCache, | ||
| anthropic.promptCacheTtl, |
There was a problem hiding this comment.
Add the TTL control to auto-detected OpenRouter custom endpoints
This only adds promptCacheTtl to the built-in OpenRouter settings, but the auto-detected custom OpenRouter path is driven by api/server/services/Config/loadCustomConfig.js, where addOpenRouterDefaults still synthesizes customParams.paramDefinitions with only { key: 'promptCache' }. For existing custom/proxy OpenRouter configs whose name or baseURL matches OpenRouter, the UI will therefore expose the cache switch but not the new TTL dropdown even though the schema/backend now support it; include the TTL definition in that injected default set too.
Useful? React with 👍 / 👎.
| if (systemOptions.promptCacheTtl != null) { | ||
| (requestOptions as Record<string, unknown>).promptCacheTtl = systemOptions.promptCacheTtl; |
There was a problem hiding this comment.
Clear Anthropic TTL when prompt caching is dropped
When an Anthropic endpoint config uses dropParams: ['promptCache'] and the conversation/preset still has promptCacheTtl, this assignment runs before the drop-params loop. That loop later deletes only requestOptions.promptCache, leaving promptCacheTtl in llmConfig without caching enabled; in the admin opt-out path, the request can still carry a TTL (or let the SDK infer cache writes) even though prompt caching was dropped. Also clear promptCacheTtl when promptCache is dropped.
Useful? React with 👍 / 👎.
| promptCacheTtl: | ||
| options.modelOptions?.promptCacheTtl ?? anthropicSettings.promptCacheTtl.default, |
There was a problem hiding this comment.
Apply Anthropic custom TTL defaults
With this new setting, admins can configure custom/native Anthropic endpoints with customParams.paramDefinitions: [{ key: 'promptCacheTtl', default: '5m' }]; initializeCustom passes those values as defaultParams, but getLLMConfig snapshots the TTL only from modelOptions here and the later defaultParams/addParams loops do not handle promptCacheTtl. In that supported custom-params path, leaving the dropdown unset still sends no TTL and falls back to the SDK's 1h default instead of the configured 5m default.
Useful? React with 👍 / 👎.
| type: 'enum', | ||
| default: anthropicSettings.promptCacheTtl.default, | ||
| options: ['5m', '1h'], | ||
| component: 'dropdown', |
There was a problem hiding this comment.
Persist custom-endpoint TTL dropdown selections
For custom endpoints, parseCustomParams overwrites these definitions to optionType: 'custom', and DynamicDropdown's custom branch only updates local state before returning instead of calling setOption. So if an admin explicitly exposes promptCacheTtl on a custom Anthropic/OpenRouter endpoint, the new dropdown renders but user selections are never written into the conversation/model parameters, and the request keeps using the default TTL. Either avoid exposing this as a custom dropdown or make the dropdown persist custom values like the other controls.
Useful? React with 👍 / 👎.
| if (typedData.promptCache !== true) { | ||
| typedData.promptCacheTtl = undefined; |
There was a problem hiding this comment.
Gate Bedrock 1h TTL to models that support it
For Bedrock, this only drops the TTL when caching is off, so any cache-capable Bedrock model keeps a selected promptCacheTtl; that includes Claude 4.6/4/3.x and Nova because promptCache is defaulted on above. The AWS Bedrock docs list 1-hour TTL only for Claude Opus/Sonnet/Haiku 4.5, with other listed Claude models limited to 5 minutes, so selecting 1h for those models can send an unsupported cache TTL and fail the request instead of falling back safely.
Useful? React with 👍 / 👎.
…ex review) dropParams: ['promptCache'] deleted requestOptions.promptCache but left promptCacheTtl behind, so the admin opt-out path could still carry a TTL on a request with caching disabled. Clear the TTL alongside promptCache.
|
Round 3 — one fix, the rest are scoped out or false positives. Details: P2 — Clear Anthropic TTL when prompt caching is dropped ( P2 — Gate Bedrock 1h TTL to models that support it ( The remaining three are all about the custom / admin-configured endpoint path, which I'm intentionally keeping out of scope for this PR (the built-in Anthropic/Bedrock/OpenRouter UI works and is tested): P2 — Persist custom-endpoint TTL dropdown selections ( P2 — Add the TTL control to auto-detected OpenRouter custom endpoints ( P2 — Apply Anthropic custom TTL defaults ( |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 35f5e15b4d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| promptCacheTtl: | ||
| options.modelOptions?.promptCacheTtl ?? anthropicSettings.promptCacheTtl.default, |
There was a problem hiding this comment.
Honor Anthropic promptCacheTtl defaults
For native Anthropic custom endpoints, buildAnthropicCustomConfig passes customParams.paramDefinitions as defaultParams, but this value is computed only from modelOptions; the later defaultParams/addParams loops only handle web_search and knownAnthropicParams, so an admin-configured default like { key: 'promptCacheTtl', default: '5m' } is silently ignored and requests continue using the SDK's 1h default unless the user manually selects the dropdown. Thread promptCacheTtl through the same default/add handling used for OpenRouter before applying it to requestOptions.
Useful? React with 👍 / 👎.
| description: 'com_endpoint_anthropic_prompt_cache_ttl', | ||
| descriptionCode: true, | ||
| type: 'enum', | ||
| default: anthropicSettings.promptCacheTtl.default, |
There was a problem hiding this comment.
Preserve unset TTL in custom overrides
When a custom endpoint includes promptCacheTtl in customParams.paramDefinitions just to relabel or expose this setting, loadCustomConfig merges this definition and then validateSettingDefinitions fills any dropdown without a default with options[0], so the intended unset default becomes '5m'. That default is later extracted and sent on OpenRouter/custom requests, unexpectedly opting those endpoints into the legacy 5-minute cache instead of leaving the SDK/provider 1-hour default in effect.
Useful? React with 👍 / 👎.
|
Round 4 — both findings are in the custom/admin-configured endpoint path that I'm intentionally keeping out of scope for this PR (the built-in Anthropic/Bedrock/OpenRouter UI works and is tested). No code changes. P2 — Honor Anthropic promptCacheTtl defaults ( P2 — Preserve unset TTL in custom overrides ( The custom-endpoint cluster (this + round-3 A/C/D) is best handled as a dedicated follow-up that fixes the shared |
…LibreChat-AI#13835) * 🕐 feat: Add promptCacheTtl model parameter for 1h/5m cache duration Adds a user-configurable `promptCacheTtl` parameter (dropdown: 5m | 1h) alongside the existing `promptCache` toggle for Anthropic, Bedrock, and OpenRouter endpoints. Default is undefined so the agents SDK applies its own default (1h), letting users opt down to the legacy 5m TTL. - data-provider: schema, parameterSettings dropdown, types, bedrock picks - data-schemas: convo/preset types + mongoose defaults - api: thread promptCacheTtl into anthropic + openai(OpenRouter) llmConfig - i18n: en translation keys for label/description/default placeholder - tests: anthropic llm.spec coverage for set + unset cases * 🔧 fix: Tie Bedrock promptCacheTtl to promptCache + thread OpenRouter TTL params (Codex review) - bedrock.ts: clear promptCacheTtl whenever promptCache is off/unsupported, so an unsupported 1h is never sent on a non-caching Bedrock request - openai/llm.ts: resolve promptCacheTtl through the same defaultParams/ addParams/dropParams machinery as promptCache (via promptCacheTtlValue) so OpenRouter custom endpoints can configure/override/drop it - tests: bedrock TTL-tied-to-promptCache cases; OpenRouter TTL default/add/drop * 🎨 style: Sort imports in openai/llm.spec.ts (CI sort-imports) * ✅ test: Prove OpenRouter TTL-only selection honors promptCache default (Codex review) OPENROUTER_DEFAULT_PARAMS injects promptCache:true into defaultParams, so a TTL-only dropdown selection (promptCacheTtl set, promptCache switch untouched) still resolves caching on and forwards the TTL. Add regression tests via the real getOpenAIConfig entry point: TTL-only -> promptCache+TTL both set; explicit promptCache:false -> both dropped. * 🔖 chore: Bump librechat-data-provider to 0.8.506 * 🔧 fix: Drop Anthropic promptCacheTtl when promptCache is dropped (Codex review) dropParams: ['promptCache'] deleted requestOptions.promptCache but left promptCacheTtl behind, so the admin opt-out path could still carry a TTL on a request with caching disabled. Clear the TTL alongside promptCache.
…LibreChat-AI#13835) * 🕐 feat: Add promptCacheTtl model parameter for 1h/5m cache duration Adds a user-configurable `promptCacheTtl` parameter (dropdown: 5m | 1h) alongside the existing `promptCache` toggle for Anthropic, Bedrock, and OpenRouter endpoints. Default is undefined so the agents SDK applies its own default (1h), letting users opt down to the legacy 5m TTL. - data-provider: schema, parameterSettings dropdown, types, bedrock picks - data-schemas: convo/preset types + mongoose defaults - api: thread promptCacheTtl into anthropic + openai(OpenRouter) llmConfig - i18n: en translation keys for label/description/default placeholder - tests: anthropic llm.spec coverage for set + unset cases * 🔧 fix: Tie Bedrock promptCacheTtl to promptCache + thread OpenRouter TTL params (Codex review) - bedrock.ts: clear promptCacheTtl whenever promptCache is off/unsupported, so an unsupported 1h is never sent on a non-caching Bedrock request - openai/llm.ts: resolve promptCacheTtl through the same defaultParams/ addParams/dropParams machinery as promptCache (via promptCacheTtlValue) so OpenRouter custom endpoints can configure/override/drop it - tests: bedrock TTL-tied-to-promptCache cases; OpenRouter TTL default/add/drop * 🎨 style: Sort imports in openai/llm.spec.ts (CI sort-imports) * ✅ test: Prove OpenRouter TTL-only selection honors promptCache default (Codex review) OPENROUTER_DEFAULT_PARAMS injects promptCache:true into defaultParams, so a TTL-only dropdown selection (promptCacheTtl set, promptCache switch untouched) still resolves caching on and forwards the TTL. Add regression tests via the real getOpenAIConfig entry point: TTL-only -> promptCache+TTL both set; explicit promptCache:false -> both dropped. * 🔖 chore: Bump librechat-data-provider to 0.8.506 * 🔧 fix: Drop Anthropic promptCacheTtl when promptCache is dropped (Codex review) dropParams: ['promptCache'] deleted requestOptions.promptCache but left promptCacheTtl behind, so the admin opt-out path could still carry a TTL on a request with caching disabled. Clear the TTL alongside promptCache.
…LibreChat-AI#13835) * 🕐 feat: Add promptCacheTtl model parameter for 1h/5m cache duration Adds a user-configurable `promptCacheTtl` parameter (dropdown: 5m | 1h) alongside the existing `promptCache` toggle for Anthropic, Bedrock, and OpenRouter endpoints. Default is undefined so the agents SDK applies its own default (1h), letting users opt down to the legacy 5m TTL. - data-provider: schema, parameterSettings dropdown, types, bedrock picks - data-schemas: convo/preset types + mongoose defaults - api: thread promptCacheTtl into anthropic + openai(OpenRouter) llmConfig - i18n: en translation keys for label/description/default placeholder - tests: anthropic llm.spec coverage for set + unset cases * 🔧 fix: Tie Bedrock promptCacheTtl to promptCache + thread OpenRouter TTL params (Codex review) - bedrock.ts: clear promptCacheTtl whenever promptCache is off/unsupported, so an unsupported 1h is never sent on a non-caching Bedrock request - openai/llm.ts: resolve promptCacheTtl through the same defaultParams/ addParams/dropParams machinery as promptCache (via promptCacheTtlValue) so OpenRouter custom endpoints can configure/override/drop it - tests: bedrock TTL-tied-to-promptCache cases; OpenRouter TTL default/add/drop * 🎨 style: Sort imports in openai/llm.spec.ts (CI sort-imports) * ✅ test: Prove OpenRouter TTL-only selection honors promptCache default (Codex review) OPENROUTER_DEFAULT_PARAMS injects promptCache:true into defaultParams, so a TTL-only dropdown selection (promptCacheTtl set, promptCache switch untouched) still resolves caching on and forwards the TTL. Add regression tests via the real getOpenAIConfig entry point: TTL-only -> promptCache+TTL both set; explicit promptCache:false -> both dropped. * 🔖 chore: Bump librechat-data-provider to 0.8.506 * 🔧 fix: Drop Anthropic promptCacheTtl when promptCache is dropped (Codex review) dropParams: ['promptCache'] deleted requestOptions.promptCache but left promptCacheTtl behind, so the admin opt-out path could still carry a TTL on a request with caching disabled. Clear the TTL alongside promptCache.
Summary
Closes #12965
Closes #13507
Adds a user-configurable Prompt Cache Duration parameter (
promptCacheTtl, dropdown:5m|1h) alongside the existingpromptCachetoggle for Anthropic, Bedrock, and OpenRouter endpoints.The agents SDK now defaults prompt-cache writes to a 1h TTL (danny-avila/agents#249). This parameter defaults to
undefined, so the SDK's default (1h) applies unless a user explicitly opts down to the legacy 5m TTL from the UI.Why
1h cache writes cost 2× base input (vs 1.25× for 5m) but survive much longer, which is a net win for agentic / multi-turn workloads where the cached prefix is reused well beyond the 5-minute window. Making 1h the default while exposing an opt-out keeps cost-sensitive users in control.
Changes
data-provider(parameter definition)schemas.ts:promptCacheTtl: z.enum(['5m','1h']).optional()on the conversation schema; added to anthropic/openRouter picksparameterSettings.ts:enum/dropdownSettingDefinition foranthropicandbedrock, wired into the endpoint parameter arraystypes.ts,bedrock.ts: type union + bedrock picks/field listdata-schemas(persistence)convo.ts/preset.ts:promptCacheTtl?: '5m' | '1h'defaults.ts: mongoose{ type: String }api(backend wiring)anthropic/llm.ts: threadspromptCacheTtlintollmConfiginside thesupportsCacheControlblockopenai/llm.ts: forwards it in the OpenRouter branchtypes/openai.ts: type fieldi18n —
en/translation.json: label, description, and "Default (1 hour)" placeholder keysTesting
data-provider: build + 115 schema tests passdata-schemas: build cleanapi:tsc --noEmit0 errors; anthropicllm.spec99 tests pass (incl. 2 new: set →'1h'lands inllmConfig; unset →undefinedso the SDK default applies)Notes
undefinedmeans no behavior change for existing users beyond the SDK's new 1h default — the control is purely an opt-out path to 5m.dev.