Skip to content

🔍 fix: exclude deferred tools from instruction token accounting - #105

Merged
danny-avila merged 2 commits into
mainfrom
claude/festive-tu-aa60ad
Apr 17, 2026
Merged

danny-avila merged 2 commits into
mainfrom
claude/festive-tu-aa60ad

Conversation

@danny-avila

@danny-avila danny-avila commented Apr 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Fixes an over-count in AgentContext where tool definitions with defer_loading: true were included in toolSchemaTokens and getTokenBudgetBreakdown().toolCount even though they are never bound to the model until discovered via tool_search.
  • This mismatch with the defer_loading filter in getEventDrivenToolsForBinding inflated the reported instruction token count and produced spurious context-overflow errors (e.g. 292 tools in a LibreChat MCP registry).
  • Both counting paths now go through a new private getActiveToolDefinitions() helper that filters out defer_loading === true entries unless they are already present in discoveredToolNames.

Refs: LibreChat-AI/LibreChat#12702

Upgrade note for consumers

If a consumer caches toolSchemaTokens across runs and passes it back via AgentInputs.toolSchemaTokens (the fast-path added in 8e0ff93), calculateInstructionTokens is skipped entirely in fromConfig. A cached value computed before this fix will continue to reflect the inflated count. Consumers using that cache must invalidate it when upgrading to pick up the corrected accounting.

Out of scope (follow-ups worth filing)

  • getActiveToolDefinitions() does not filter on allowed_callers — a code_execution-only tool passed via toolDefinitions still contributes to toolSchemaTokens even though it is never bound to the model. Pre-existing; narrower fix here keeps the change focused on the issue.
  • toolSchemaTokens is a snapshot taken during calculateInstructionTokens and is not recomputed on markToolsAsDiscovered; toolCount is live. The JSDoc on getTokenBudgetBreakdown now documents this, and a test pins the snapshot semantic.

Test plan

  • npx jest src/agents/__tests__/AgentContext.test.ts — 54/54 pass, including 5 new cases:
    • toolSchemaTokens excludes deferred-undiscovered entries
    • toolSchemaTokens includes deferred entries pre-seeded via discoveredTools
    • getTokenBudgetBreakdown().toolCount excludes deferred-undiscovered entries
    • getTokenBudgetBreakdown().toolCount reflects newly discovered deferred tools
    • toolSchemaTokens snapshot does not auto-update after markToolsAsDiscovered
  • npx tsc --noEmit — clean
  • npx eslint src/agents/AgentContext.ts src/agents/__tests__/AgentContext.test.ts — clean
  • Full suite: only pre-existing live-API failures (missing ANTHROPIC_API_KEY / OPENAI_API_KEY), unrelated to this change

Deferred tool definitions (used for tool_search) were counted in
`toolSchemaTokens` and `getTokenBudgetBreakdown().toolCount` even though
they are never bound to the model until discovered. This mismatch with
`getEventDrivenToolsForBinding` inflated the reported instruction token
count (e.g. 292 tools for a LibreChat MCP registry) and produced
spurious context-overflow errors.

Both paths now go through a new `getActiveToolDefinitions()` helper that
filters out `defer_loading === true` entries unless they are already in
`discoveredToolNames`, matching the filter applied at bind time.

Refs: LibreChat-AI/LibreChat#12702
- Correct JSDoc on getActiveToolDefinitions: it filters for token
  accounting, not bind-time (code_execution-only tools still pass).
- Note the staleness asymmetry on getTokenBudgetBreakdown: toolCount is
  live after markToolsAsDiscovered but toolSchemaTokens is a snapshot.
- Add regression test pinning the snapshot semantic so future changes
  must intentionally re-evaluate whether discovery should recompute.
@danny-avila
danny-avila merged commit b3046a0 into main Apr 17, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant