Skip to content

fix(models): resolve Bedrock auto caching for Nova, OpenAI GPT, and xAI (Python) - #6149

Merged
jimbrub merged 2 commits into
strands-agents:mainfrom
strandly-the-agent:port/bedrock-auto-caching-to-python
Nov 16, 2026
Merged

jimbrub merged 2 commits into
strands-agents:mainfrom
strandly-the-agent:port/bedrock-auto-caching-to-python

Conversation

@strandly-the-agent

@strandly-the-agent strandly-the-agent commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Requested by: @jimbrub — port #6082 to Python

Human Overview

Resolving "auto" mode for all Bedrock models in Python.

Description

Python port of #6082. With cache_config set, Python BedrockModel now does the same caching on each Bedrock model as TS:

  • OpenAI GPT and xAI: reported as implicitly cached (prompt_caching="auto"). No cache points are sent and the false "does not support automatic caching" warning is gone.
  • gpt-oss: no longer treated as cached (Converse doesn't cache it), so the warning stays.
  • Nova Pro / Lite / Micro / 2 Lite: get cache points within the limits Converse enforces on Nova. There is no tools cache point, and any TTL is dropped with a one-time warning. A point goes only directly after text, on the last user message that has text, ahead of the per-call trailing blocks. In a tool loop that keeps it on the prompt that started the loop. A caller-placed point only ever moves back, with a warning when no text precedes it.

Six existing tests used Nova as the stand-in for a model without caching. They now use Llama, as the TS PR did.

Related Issues

Refs #6082 (source), #5126, #1571

Documentation PR

None. The source PR shipped no docs either; the site/ prompt-caching update it lists as a follow-up covers both SDKs.

Type of Change

Bug fix

Testing

  • tests/strands/models/: 2049 passed. The 66 ported and changed tests all pass.
  • ruff format --check and ruff check are clean.
  • mypy reports the same 7 errors as main (env stubs), none in bedrock.py.
  • Live Converse us-east-1, agent with a tool, 2 turns, cache_config=CacheConfig(strategy="auto"). Every Nova model is accepted and reads from cache. Nova Pro is accepted even with ttl="1h" and tools_ttl=True. Claude still caches.
Model main (cache read) This PR (cache read)
Nova Lite 0, plus a false warning 21,152
Nova Pro (ttl="1h", tools_ttl=True) 0, plus a false warning 7,012, TTL dropped with a warning
Nova Micro / Nova 2 Lite not run on main 14,125 / 24,840
Claude Haiku 4.5 not run on main 18,928
  • I ran hatch run prepare (not run: the gates above were run individually in a uv venv)

Additional Details

Traceability matrix: source test → Python test

TS line numbers are for strands-ts/src/models/__tests__/ at 42c4e82. Python line numbers are for strands-py/tests/strands/models/test_bedrock.py. All Python rows pass.

TS source test Python test
bedrock-cache.test.ts:4 bedrockCacheModelInfo (15 ids) :4588 test_converse_cache_model_info (same 15)
bedrock-cache.test.ts:26 bedrockLimitsCachePoints (9 ids) :4608 test_converse_limits_cache_points (same 9)
bedrock.test.ts:460 leaves caching unknown for other ids (Nova → Llama) :4468 test_model_info_leaves_caching_unknown_for_other_model_ids
:518 OpenAI GPT or xAI implicitly cached, no point, no warning :4523 test_model_info_reports_openai_gpt_and_xai_models_as_implicitly_cached
:536 warns that gpt-oss has no automatic caching :4540 test_gpt_oss_has_no_automatic_caching_and_warns
:553 places cache points on an implicitly cached OpenAI model under 'anthropic' :4556 test_anthropic_strategy_places_cache_points_on_an_implicitly_cached_openai_model
:3800 Nova caches system + conversation, not tools, no warning :4635 test_nova_caches_system_and_conversation_but_not_tools_without_a_warning
:3815 drops a configured TTL from every point and warns :4652 test_nova_drops_a_configured_ttl_from_every_cache_point_and_warns
:3825 drops a systemPromptTTL and warns :4666 test_nova_drops_a_system_prompt_ttl_and_warns
:3835 drops the TTL of a caller-placed system point :4678 test_nova_drops_the_ttl_of_a_caller_placed_system_cache_point_and_warns
:3846 warns that an explicit toolsTTL has no effect :4691 test_nova_warns_that_an_explicit_tools_ttl_has_no_effect
— (Python only: deprecated cache_tools) :4707 test_nova_drops_the_deprecated_cache_tools_point_and_warns
:3859 applies the Nova rules under 'anthropic' :4718 test_nova_rules_apply_under_the_anthropic_strategy
:3876 keeps the point on the prompt that started a tool loop :4735 test_nova_keeps_the_conversation_cache_point_on_the_prompt_that_started_a_tool_loop
:3900 keeps the point on the prompt when a per-call tail follows a tool result :4754 test_nova_keeps_the_conversation_cache_point_on_the_prompt_when_a_per_call_tail_follows_a_tool_result
:3914 moves a caller point with no text ahead to the prompt, warns :4768 test_nova_moves_a_caller_placed_cache_point_with_no_text_ahead_of_it_to_the_prompt_and_warns
:3949 point directly after text, ahead of image / PDF / image+txt / video :4795 test_nova_places_the_cache_point_directly_after_the_text_ahead_of_media (4 cases)
:3959 moves a caller point after an image back to the text :4807 test_nova_moves_a_caller_placed_cache_point_after_an_image_back_to_the_text
:3977 keeps the point after an image on Claude :4820 test_claude_keeps_the_cache_point_after_an_image
:3990 no conversation point when the only user text is in guardContent :4830 test_nova_places_no_conversation_cache_point_when_the_only_user_text_is_guarded
:4004 moves a caller point off a tool result back, never past the boundary :4848 test_nova_moves_a_caller_placed_cache_point_off_a_tool_result_back_to_the_prompt
:4024 caches a plain-text user message after earlier tool calls :4865 test_nova_caches_a_plain_text_user_message_that_follows_earlier_tool_calls
:4039 keeps the tools point and TTL on Claude :4881 test_claude_keeps_the_tools_cache_point_and_ttl (tools_ttl=True, see decision log)
vercel.test.ts (Vercel → Bedrock) not ported: Python has no Vercel provider

Existing tests switched from Nova to Llama: :4454, :4913, :5400, :6441, :6484, :6873. :4913 is the override test. Nova now defaults to cache_points, so the override was a no-op on Nova. TS bedrock.test.ts:573 still uses Nova.

Runner, lint and type-check output
$ pytest -v tests/strands/models/test_bedrock.py -k "<ported and changed tests>"
====================== 66 passed, 379 deselected in 0.69s ======================

$ pytest -q -n auto tests/strands/models/
2049 passed, 40 warnings in 34.03s

$ ruff format --check src tests
703 files already formatted
$ ruff check src tests
All checks passed!

$ mypy --python-version 3.13 ./src ./tests_typing   # this branch, and identically on main
Found 7 errors in 5 files (checked 378 source files)

The 7 mypy errors are in _exception_notes.py, hooks/ and vended_memory_stores/, and they are identical on main. They come from local stubs and the forced --python-version 3.13: numpy's stubs fail under the configured 3.10 on this 3.13 venv.

Decision log
  • One-time warnings use warnings.warn(..., stacklevel=1) in place of TS warnOnce(logger, …). This is the house Python pattern for once per process (anthropic.py, model.py). The registry key includes the message text, so each model id warns once. The existing "does not support automatic caching" warning stays logger.warning, unchanged.
  • The deprecated cache_tools point is dropped on Nova when cache_config resolves to cache points, with the same tool-level warning (tools_ttl=<value>). TS has no cache_tools; without this, Nova would get a toolConfig point it rejects. Nova with cache_tools and no cache_config is a manual opt-in and is unchanged, like the TS manual opt-in in fix(models): resolve Bedrock auto caching for Nova, OpenAI GPT, and xAI #6082 open question 3.
  • Helpers stay in bedrock.py as private module functions (_converse_cache_model_info, _converse_limits_cache_points, _nova_cache_point_position, _step_back_to_text, _step_back_over_documents). TS has a separate bedrock-cache.ts because two providers share it; Python has one consumer. The Nova id list stays out of shared/model-info.json, as in TS.
  • TS _autoCachePointIdx refactor not mirrored: it has no behaviour change, so the Python auto-placement code is untouched.
  • The Claude tools-TTL test sets tools_ttl=True: Python's tools point needs tools_ttl or cache_tools, which predates this change.
  • No Vercel counterpart: there is no Python Vercel provider.
Dependencies and sensitive surface

No new dependencies or imports in src. The tests add only the stdlib re and warnings. No network, credential, subprocess, env-var or deserialization changes. The diff touches only strands-py/src/strands/models/bedrock.py and its test file.

Open questions
  1. Nova caller point at index 0 ([cachePoint, text]): Nova ends up with no conversation point, while Claude falls back to one at the end of the message. That's because blocks after a caller's boundary count as per-call. TS has the same max(trailingBlocks, content.length - placedIdx) code, so this matches TS. Should both move?
  2. hatch run prepare not run: the gates were run individually (above). A Python 3.10 env with the all extras failed to build in the sandbox, so hatch was not used.
  3. The fix(models): resolve Bedrock auto caching for Nova, OpenAI GPT, and xAI #6082 open questions carry over: multi-turn chat on Nova, and Nova behind an application inference profile ARN.

@strandly-the-agent
strandly-the-agent requested a review from a team as a code owner November 13, 2026 22:08
@github-actions github-actions Bot added bug Something isn't working area-model Related to models or model providers python Pull requests that update python code strands-running complexity/high A touched function exceeds cognitive complexity 25; may be worth splitting size/m labels Nov 13, 2026
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown
Contributor

Assessment: Comment

This port matches TS #6082 closely: caching info by model id, the Nova TTL and tools-point drops, placing the point only after text, and the tool-loop anchor all behave the same. I checked the tool-loop, guardContent, per-call tail and deprecated cache_tools cases by hand at e72cfd8, and the bedrock suite passes locally (445). What's left is mostly how strict the tests are, plus keeping the code comparable with TS. I found no correctness problems.

Review Categories
  • Testing: Several new Nova tests use prefix, length or key-only checks where a single whole-object equality is possible and is what docs/TESTING.md asks for.
  • Complexity / parity: The Nova logic is inlined into two functions that were already over the threshold. TS splits the same logic into separate methods, and doing that here would lower the complexity label and make the two SDKs easier to diff.
  • Follow-up: Adding Python Nova caching was the condition TS gave for keeping the Nova list out of shared/model-info.json. That move is worth tracking as an issue.
  • PR hygiene: The Human Overview section is still empty, and team/AI_USAGE_POLICY.md requires it before merge. hatch run prepare wasn't run either, so CI is the first full gate.

The traceability matrix and decision log made this port easy to check against the TS source.

@mkmeral

mkmeral commented Nov 16, 2026

Copy link
Copy Markdown
Member

@agent-of-mkmeral review pls. is this a good port? any blockers?

@agent-of-mkmeral

Copy link
Copy Markdown
Contributor

TL;DR: Yes, this is a good port — behaviour-faithful to #6082, not a transliteration. No code blockers. The only thing holding merge is the 2 Approvers for Bots check: @jimbrub approved, it needs one more human. Three non-blocking notes below; the bot's four inline threads are polish I agree with but wouldn't gate on.

What I checked
  • Diffed each Python piece against its TS counterpart at main: _converse_limits_cache_points/_converse_cache_model_info ↔ bedrock-cache.ts, _step_back_to_text/_nova_cache_point_position ↔ stepBackToText/novaCachePointPosition, the Nova branch in _honor_placed_cache_point, the tools-point drop ↔ _toolsCacheSection, system TTL drop ↔ _dropUnsupportedSystemTTLs, and the implicit-cache check in format_request ↔ _shouldEnableCaching. Same control flow, same max(trailing, len(content) - placed_idx) arithmetic, same warning text.
  • tests/strands/models/test_bedrock.py: 445 passed locally at e72cfd8. CI green on all 3.10–3.14 × Linux/Windows/macOS matrices plus integ.
  • 17 adversarial probes against format_request, all matching TS: inference-profile ARN id; warning dedup across 2 instances × 2 calls (exactly 2 unique warnings); tool loop keeps point on the prompt; caller point after a toolResult moves back to the prompt with the warning; caller point in an earlier message is stripped before Nova placement; guardrail_latest_message (only guarded text in last turn → point lands on the earlier plain-text turn); image-only first turn → no conversation point; Nova 2 Pro / Premier correctly excluded; cache_tools without cache_config stays a manual opt-in; dynamic_trailing_blocks > len(content) → no crash. Never produced a shape Nova would reject.
  • _IMPLICIT_CACHE / _CACHE_POINTS module singletons: ModelInfo is frozen=True, so sharing them is as safe as TS's Object.freeze.
🟢 Non-blocking notes

1. Warning channel split. The new drops use warnings.warn (→ stderr) while the existing "does not support automatic caching" stays logger.warning, so one feature now emits on two channels. A user with a logging-only pipeline and no logging.captureWarnings(True) won't see the TTL/tools-point drops. TS routes everything through warnOnce(logger, …). The decision log justifies it (free once-per-id dedup, house pattern in anthropic.py), so this is a conscious-choice check, not a request to change. Also: no test pins the dedup — pytest.warns forces an "always" filter so it can't — I verified it by hand.

2. Open question 1 is a TS quirk, faithfully reproduced. trailing_blocks (bumped by the caller's boundary) only feeds the Nova branch; the non-Nova path still reads the raw dynamic_trailing_blocks param, which is why [cachePoint, text] yields no point on Nova but an end-of-message point on Claude. Correct to not fix here. Worth one cross-SDK follow-up issue since it needs to land in both at once.

3. Multi-turn chat on Nova. Confirmed: turn 1 → point on msg0, turn 2 → point moves to msg2, so across chat turns only the system prefix is read; tool loops anchor correctly. Same as TS, carried over from #6082. The site/ prompt-caching follow-up should say this plainly so users don't expect Claude-style lookback on Nova.

Recommend approve + merge once the second approval lands. The inline nits (whole-object asserts, splitting _inject_nova_cache_point out, cache_tools field name in the warning, shared/model-info.json follow-up) can ride along or go in a follow-up — none change behaviour.

@jimbrub
jimbrub enabled auto-merge (squash) November 16, 2026 16:51
@jimbrub
jimbrub merged commit f8ef261 into strands-agents:main Nov 16, 2026
43 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-model Related to models or model providers bug Something isn't working complexity/high A touched function exceeds cognitive complexity 25; may be worth splitting python Pull requests that update python code size/m

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants