Skip to content

Integrations: warn on untested versions when wiring instrument() frameworks by hand (#841) - #856

Merged
LoopGlitch26 merged 1 commit into
Cygnux-Labs:mainfrom
Mr45401:fix/hand-wired-version-warnings
Nov 15, 2026
Merged

LoopGlitch26 merged 1 commit into
Cygnux-Labs:mainfrom
Mr45401:fix/hand-wired-version-warnings

Conversation

@Mr45401

@Mr45401 Mr45401 commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Closes #841

Summary

The seven frameworks tracekit.instrument() wires can also be wired by hand, which is the first setup shown on most of their integration docs pages. Unlike the five hand-wired-only integrations, building those seven by hand did not call version_check.warn_untested(...), leaving untested framework versions silent until a moved hook or internal API failed downstream.

Changes

  • Shared ranges from autowire.FRAMEWORKS (tracekit/integrations/version_check.py): Added version_check.framework(dist) to look up (label, req) directly from autowire.FRAMEWORKS and pick the dist>=... requirement token (extracting langgraph>=1.2,<1.3 from LangGraph's multi-package string and picking up Browser Use's browser-use>=0.13.11,<1.1 range directly without copying).
  • Hand-wired entry points (tracekit/integrations/): Called version_check.warn_untested(*_FRAMEWORK) at the top of all 10 hand-wired entry points across the 7 frameworks:
    • openai_agents.TracekitAgents.__init__
    • langchain.TracekitMiddleware.__init__, tracekit_tool_node, TracekitCheckpointer.__init__
    • claude_agent_sdk.tracekit_hooks, TracekitSessionStore.__init__
    • mcp.TracekitSession.__init__
    • browser_use.trace_tools
    • crewai.hooks
    • pydantic_ai.Tracekit.__init__
  • Single warning under tracekit.instrument() (tracekit/autowire.py, version_check.py): Added version_check.claim(label) / unclaim(label) sharing version_check._WARNED. When instrument() sees an untested version, it claims label before calling wire(run) so neither entry points constructed eagerly during wire(run) (TracekitAgents, TracekitMiddleware, crewai.hooks) nor entry points constructed lazily after instrument() returns (tracekit_hooks, TracekitSession per session, pydantic_ai.Tracekit per agent) warn a second time, and instrument() emits its own single warning. If wire(run) raises, unclaim(label) releases label so falling back to hand-wiring still warns.
  • Tests:
    • Extended VersionCheckCases in tests/factories.py with an overridable entries(integration) hook and added VersionCheck suites across all 7 integration test files covering all 10 entry points (inside tested range -> no warning; outside range -> one warning).
    • Added tests in tests/test_version_check.py and tests/test_instrument.py covering version_check.framework(), single-warning behaviour under instrument() (both eager and lazy per-session wiring), and hand-wired warning after a failed instrument() wire.

Verification

  1. Confirmed new tests fail before the fix:
    python -m pytest -v tests/test_version_check.py tests/test_instrument.py -k "Check or VersionCheck or untested" tests/test_contract_openai_agents.py tests/test_contract_langchain.py tests/test_contract_claude_agent_sdk.py tests/test_contract_mcp.py tests/test_contract_browser_use.py tests/test_crewai.py tests/test_pydantic_ai.py
    # 12 failed, 24 passed, 12 subtests passed
    
  2. Lint & test suites after the fix:
    python -m pyflakes tracekit tracekit_sdk.py tests eval examples contrib integrations
    # exit 0
    
    python -m pytest -v tests/test_version_check.py tests/test_instrument.py -k "Check or VersionCheck or TestInstrument" tests/test_contract_openai_agents.py tests/test_contract_langchain.py tests/test_contract_claude_agent_sdk.py tests/test_contract_mcp.py tests/test_contract_browser_use.py tests/test_crewai.py tests/test_pydantic_ai.py tests/test_llama_index.py tests/test_google_adk.py tests/test_haystack.py tests/test_agent_framework.py tests/test_strands.py
    # 50 passed, 6 skipped, 30 subtests passed
    
    python -m pytest -q tests/test_version_check.py tests/test_crewai.py tests/test_pydantic_ai.py tests/test_integration_langchain.py tests/test_contract_openai_agents.py tests/test_contract_langchain.py tests/test_contract_claude_agent_sdk.py tests/test_contract_mcp.py tests/test_contract_browser_use.py -k "not slow"
    # 324 passed, 125 skipped, 127 subtests passed
    
  3. Ran end-to-end as a user against a live dev signer (tracekit up):
    With distribution metadata reporting pydantic-ai-slim 2.59.0 and langgraph 1.9.0 (both outside the tested ranges):
    • Building two pydantic_ai.Agent(..., capabilities=[Tracekit(r)]) instances and two tracekit_tool_node(...) instances by hand printed one UserWarning per framework pointing at the caller's line (tracekit: Pydantic AI 2.59.0 is wired but untested... and tracekit: LangGraph 1.9.0 is wired but untested; Tracekit is tested with langgraph>=1.2,<1.3).
    • Calling tracekit.instrument() in a fresh process and then building two pydantic_ai.Agent instances and a ToolNode printed one tracekit.instrument(): ... warning per framework at the tracekit.instrument() call and no duplicate warnings when the agents or node were constructed, with tracekit last verifying the signed records.

…eworks by hand (Cygnux-Labs#841)

Signed-off-by: Mr45401 <185895283+Mr45401@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks again for taking on #841, and for the careful write-up. This is ready to merge from my side. Here's what I checked:

  • No file lost code. In every file, the line count changed by exactly the additions minus the deletions. All 21 removed lines are lines the diff replaces, and every file still ends where main's does.
  • All seven frameworks are covered. All ten entry points in the issue's table call warn_untested(*_FRAMEWORK). Each takes its label and range from autowire.FRAMEWORKS through version_check.framework(), and LangGraph passes only its langgraph>=1.2,<1.3 part.
  • The tests pass now and fail on main. I ran them in a sealed container. The 14 related test files pass at your head. With main's code under your tests, 12 fail: the untested-version test for each of the ten entry points, framework(), and the fail-to-wire test. If I remove only your autowire.py change, test_an_untested_framework_warns_once_not_again_in_its_entry_point fails with 4 warnings instead of 2, so that test guards the claim.
  • It behaves right when used. I made pydantic-ai-slim 2.59.0 and langgraph 1.9.0 look installed. Building Tracekit(run) and tracekit_tool_node twice each by hand printed one warning per framework, each pointing at my first call's line. Calling tracekit.instrument() and then building two Agents and a tool node by hand printed one tracekit.instrument(): warning per framework, and nothing more. A tested version prints nothing, and main prints nothing in the hand-wired case.
  • CI is green on every platform, and DCO passes.

One optional nit:

  1. In tests/factories.py:424-426, nothing calls warnings_for without entry any more (both tests pass it). So entry can be a required argument, and the if entry is None: branch can go, along with the import importlib at line 421.

@LoopGlitch26
LoopGlitch26 merged commit 138950f into Cygnux-Labs:main Nov 15, 2026
34 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Merged, thanks @Mr45401. All seven frameworks that instrument() wires now warn once about an untested version, whether you wire them by hand or not, and the warning points at your line. It ships in 1.7.0. If you'd like another, #847 is taken but #843 (Crush: record the model's text from TurnEnd) and #846 (SDKs: say "may not have been recorded" when a write's outcome is unknown) are open and specified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrations: the seven frameworks instrument() wires don't warn about an untested version when wired by hand

2 participants