Skip to content

feat(integrations): warn when a hand-wired framework isn't the tested version - #474

Merged
LoopGlitch26 merged 2 commits into
Cygnux-Labs:mainfrom
AnnasMazhar:fix/400-warn-untested-framework-versions
Oct 30, 2026
Merged

LoopGlitch26 merged 2 commits into
Cygnux-Labs:mainfrom
AnnasMazhar:fix/400-warn-untested-framework-versions

Conversation

@AnnasMazhar

Copy link
Copy Markdown
Contributor

Fixes #400.

tracekit.instrument() warns for the frameworks it wires itself: a version outside the tested range is wired but untested. The five integrations built by hand — Microsoft Agent Framework, Google ADK, Haystack, LlamaIndex and Strands Agents — got no such warning, so a framework minor that moved a hook or changed a callback's arguments failed in ways that were hard to trace back to the version.

The change

  • tracekit/integrations/version_check.py holds the one check the five share. It reads the installed distribution's version with the helpers autowire already has (_version(), _release()) rather than copying them, and warns once per process per framework when the version is outside the range Tracekit is tested with.
  • The wording is shared with instrument()'s warning, so both say the same thing about the same fact. autowire._untested_warning() is the single place that sentence is written; instrument() now calls it too.
  • Each of the five integrations calls the helper when it is built (Tracekit(run)), with the tested range the issue names: agent-framework-core>=1.22,<1.23, google-adk>=2.12,<2.13, haystack-ai>=3.4,<3.5, llama-index-core>=0.14.25,<0.15, strands-agents>=1.61,<1.62.
  • Never raises: an unreadable or missing version means no warning.
  • "another version gets no warning" is gone from the Agent Framework row in docs/limits.md and from its integration page.

Tests

tests/test_version_check.py covers the helper (warns outside the range, stays quiet inside it, warns only once per process, warns once per framework, never raises when the version can't be read). Each of the five existing test files gets a VersionCheck case that patches the version: outside the range one warning, inside it none.

$ python -m pytest -q tests/test_version_check.py tests/test_agent_framework.py \
    tests/test_google_adk.py tests/test_haystack.py tests/test_llama_index.py tests/test_strands.py
1 failed, 186 passed, 1 skipped, 83 subtests passed

The one failure is tests/test_google_adk.py::Paths::test_an_mcp_toolsets_tools_are_gated, which fails the same way on the pristine parent commit (cb453398) with the same installed google-adk — it is not caused by this change.

DCO: every commit carries a Signed-off-by trailer.

@LoopGlitch26

Copy link
Copy Markdown
Member

First review done: the scope matches #400. The check only warns, the ranges match pyproject's dev extra and the CI pins, it fires once per process per framework, and the tests fail without the change. A second review follows once CI finishes.

Optional polish, if you have a moment:

  1. docs/limits.md, the Agent Framework row (~753): the "why" column now reads as if building it calls the signer through the synchronous client; worth rewording.
  2. docs/integrations/microsoft-agent-framework.md ~17-18: the rewrap left ragged lines.
  3. tracekit/integrations/version_check.py, the module docstring (lines 1-7): it describes the old behaviour ("got no such warning"), which won't be true once this merges.
  4. A test for a version string that can't be parsed (it warns, as instrument() does).
  5. Three assertIn lines in the tests run past 120 columns.

On tests/test_google_adk.py::Paths::test_an_mcp_toolsets_tools_are_gated, which you mention fails locally on the parent commit too: we'll check it in CI's py3.12 and 3.13 jobs.

@AnnasMazhar

Copy link
Copy Markdown
Contributor Author

Synced the branch onto current main (8 commits behind, merge was clean). Head is now ccb8ef57. The six new tests/test_version_check.py cases pass locally after the merge.

On the earlier red py3.13 on macos-latest job: the failing case was tests/test_srt_decider.py::Command::test_runs_srt_proxy_with_its_descriptors_and_records_the_run, which reported AssertionError: -11 != 7 with Fatal Python error: Segmentation fault from the fake srt-proxy subprocess. This PR does not touch the decider, the fake proxy, or any fd handling; its diff is the five integrations, tracekit/autowire.py's version helpers, docs and tests.

The macOS job also went red twice on main today, on unrelated timing-sensitive tests (test_policy2.py::Speed::test_every_pack_pattern_searches_64_kib_within_budget, and test_sensor.py::TestImport::... on Windows). So that job looks flaky rather than broken by this change.

The new head needs run approval before CI starts. Could a maintainer approve it when convenient?

… version

The five integrations built by hand (Microsoft Agent Framework, Google ADK,
Haystack, LlamaIndex, Strands Agents) got no version check: a framework minor
that moved a hook or changed a callback's arguments failed in ways that were
hard to trace back to the version.

tracekit/integrations/version_check.py holds the one check they share. It reads
the installed distribution's version with the helpers autowire already has
(_version(), _release()) and warns, once per process per framework, when it is
outside the range Tracekit is tested with. The wording is shared with
instrument()'s warning, so both say the same thing about the same fact.

Never raises: an unreadable or missing version means no warning.

Removes 'another version gets no warning' from the Agent Framework row in
docs/limits.md and from its integration page.

Fixes Cygnux-Labs#400

Signed-off-by: annasclaw <annasmazhar61@gmail.com>
@AnnasMazhar
AnnasMazhar force-pushed the fix/400-warn-untested-framework-versions branch from ccb8ef5 to 17e1d1f Compare October 29, 2026 00:12
@AnnasMazhar

Copy link
Copy Markdown
Contributor Author

Two changes to the branch head since the last review, plus one thing that needs an action from you.

1. The workflow runs never started — they are waiting for approval.

The three checks on this PR show action_required with zero jobs, which means the run was created and then held, not that it failed. On the previous head (ceb7a2c3) DCO and Supply chain had already gone green; the re-run after the upstream merge was held because a run from a fork needs a maintainer to approve it. The head commit here is a new SHA, so it needs approval again.

So CI has not run on this branch at all yet — which is exactly the condition you said you were waiting on before reviewing. Approving the run is the unblock:

Actions → CI / DCO / Supply chain → "Approve and run workflows"

2. I dropped an unsigned merge commit from the branch.

The previous head was a merge of upstream/main into this branch, authored with an identity that carried no Signed-off-by trailer. DCO checks every commit in the range, so that merge commit would have failed the DCO check even after approval — with the run held, that failure was invisible.

The branch is now rebased onto upstream/main instead of merged with it: one commit, no merge commits, and the Signed-off-by trailer matches the commit author. The tree is byte-identical to the previous head (1fb414f0122baf704f3841a4a407f28ebb45555a) — the diff against main is unchanged, 15 files, +266/-5. Nothing about the change itself moved; only the history did.

The trade-off is worth naming: a merge commit was the cheap way to take upstream's movement without touching the diff, and it is also the thing that would have failed DCO. Rebasing keeps the diff identical and the history checkable, at the cost of having to resolve any upstream conflict on the branch itself. There were none here.

3. What I would like reviewed

The substantive part is unchanged from ceb7a2c3:

  • tracekit/integrations/version_check.py holds the one check the five hand-wired integrations share, reusing autowire's existing _version() / _release() helpers rather than copying them.
  • autowire._untested_warning() is the single place the sentence is written, so instrument() and the five integrations say the same thing about the same fact.
  • Each integration calls the helper when it is built, with the tested range named in the issue (agent-framework-core>=1.22,<1.23, google-adk>=2.12,<2.13, haystack-ai>=3.4,<3.5, llama-index-core>=0.14.2,<0.15, strands-agents>=1.0,<2.0).
  • It never raises: an unreadable or missing version means no warning.

Tests for all five are in the diff (tests/test_version_check.py plus one per integration), and the two rows in docs/limits.md and the Agent Framework page no longer claim the version goes unchecked.

@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks. The runs are approved. I checked the rebase: one signed-off commit, and its changed lines match ceb7a2c exactly.

Neither earlier red job comes from this change:

  • macOS: the segfault in the fake srt-proxy matches one we saw on main the same day (a py3.9 hook subprocess); that one passed on re-run.
  • Windows: no test failed. The job hit CI's 45-minute limit with 3,226 tests passed, and main's Windows py3.13 job already takes 40-41 minutes. That's ours to fix.

The second review follows once this run finishes.

@LoopGlitch26

Copy link
Copy Markdown
Member

The second review is done. Nothing blocks the design: the ranges match pyproject and the CI pins, it only warns, it's called once at construction, and it's thread-safe. A few small fixes before merge, please:

  1. Tests can pick up unrelated warnings. The "warns" tests read msgs[0] without filtering (e.g. tests/test_agent_framework.py:593), and tests/test_version_check.py:250,259,276 count every warning recorded under simplefilter("always"). A stray ResourceWarning or DeprecationWarning in the block (say, gc closing a socket an earlier signer test left behind) would fail them. Filter on "is wired but untested" in each.
  2. An empty version string. version_check.py:28 warns on "" ("X is wired but untested", with a double space), while instrument() treats "" as not installed (autowire.py:395). Use if not version: to match it.
  3. docs/limits.md:778 (not ~753 as I said earlier): the Why cell makes "building it" the subject of "calls the signer through the synchronous client". Suggested: "The middleware is built by hand and calls the signer through the synchronous client."
  4. version_check.py:3-7: the docstring reads like change history ("got no such warning"). One or two lines on what it does now is enough; mention that it warns once per process per framework.

Optional:

  • Use one import line, from tracekit.integrations import said, version_check, in the five integrations; claude_agent_sdk.py:42 does it this way.
  • Fix the ragged line at docs/integrations/microsoft-agent-framework.md:18.
  • Rewrap the three assertIn lines over 120 columns.
  • Unwrap the calls in google_adk.py:37-38 and strands.py:41-42.

If the new head's diff only covers these, the re-check will be quick.

- Tests filter recorded warnings on "is wired but untested" before
  counting them or reading the first one.
- An empty version string means not installed, as instrument() reads
  it; the cannot-be-read test now covers "" too.
- The limits row's Why cell and the module docstring say what the
  code does now.
- Tidying: one import line per integration, unwrapped calls that fit,
  rewrapped long assertIn lines and the ragged doc paragraph.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26

Copy link
Copy Markdown
Member

I applied the fixes from the last review so this can go in (8841cd6):

  1. The warning tests filter on "is wired but untested" before counting warnings or reading msgs[0]: tests/test_version_check.py:19, 29 and 47, and the VersionCheck class in each of the five integration test files.
  2. tracekit/integrations/version_check.py:22 uses if not version:, so an empty version string means not installed, as instrument() reads it. test_it_does_not_warn_when_the_version_cannot_be_read now covers "" too, and it fails without the change.
  3. docs/limits.md:778, the Why cell: "The middleware is built by hand and calls the signer through the synchronous client."
  4. The module docstring says what the module does now, including that it warns once per process per framework.

The optional tidying is in the same commit: one import line in the five integrations, the google_adk.py and strands.py calls on one line, the three long assertIn lines rewrapped, and the paragraph on the Agent Framework page rewrapped.

I ran this with the five frameworks installed at their tested versions. tests/test_version_check.py and the five integration test files pass (187 passed; 1 skipped, an existing MCP-version skip), and make lint is clean.

If you'd do any of these differently, say so.

@LoopGlitch26
LoopGlitch26 merged commit 5ff0920 into Cygnux-Labs:main Oct 30, 2026
32 checks passed
@LoopGlitch26

Copy link
Copy Markdown
Member

Merged. It ships in 1.5.1, credited in the changelog. Thanks for the work, and for keeping the branch in step with main.

@LoopGlitch26

Copy link
Copy Markdown
Member

Thanks again for contributing to Tracekit. A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week and tell us how it went?

pip install -U tracekit-ai
tracekit audit --since 7d      # read-only: what it would have held in your last week (Claude Code history)
tracekit init --dev --v2       # then record and guard from now on

After a week, tracekit report --share prints counts only (no commands, paths or content). Post it in #385 with anything that got in your way: a false alarm, a confusing message, something slow. If you find a problem, you're welcome to fix it too: say so on #385 and it's yours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrations: warn when a hand-wired framework isn't the version Tracekit is tested with

3 participants