Skip to content

📭 fix: Deliver Attachment Text Until a File Tool Holds It - #16027

Merged
danny-avila merged 2 commits into
devfrom
lia/ephemeral-file-search-uploads
Sep 16, 2026
Merged

danny-avila merged 2 commits into
devfrom
lia/ephemeral-file-search-uploads

Conversation

@lia-by-librechat

Copy link
Copy Markdown
Contributor

Summary

Turning on the File Search toggle in a plain chat made an upload invisible to the model. Uploading a PDF with the toggle off worked, the extracted text was delivered; uploading the same PDF with the toggle on produced an answer as if no file were attached, with no error and nothing in the log. Enabling a file tool was strictly worse than leaving it off.

Two halves of the pipeline disagreed about what "this conversation has a file tool" means. At upload time the destination is chosen from the agent record, and a plain chat runs an ephemeral agent that has no record, so no tool claims the file and it is never embedded. At turn time the consumers are derived from the live tool set, which does include the toggle, so resolveTurnLLMDeliveryPath suppressed the textFallbackWithoutTools text on the assumption that retrieval would serve the file. The vector store had never received it, so nothing did: the upload path concluded "no file tool, do not file it" and the turn path concluded "there is a file tool, withhold the text".

Withholding the text now requires the record to show that a tool this turn runs actually holds the file, which is the evidence deferred provisioning already reads before queueing one. An attachment no tool has a copy of is delivered as text, on the turn it arrives and on every later turn, and lazy provisioning still embeds it on the first search, so the file is both readable and searchable. A file the sandbox or the vector store does hold is untouched and stays with its tool.

Fixes #15970 (the regression half). The unembedded-upload and .pptx items stay with #15420, which describes them more precisely.

How it works

The delivery decision asks the record, not the tool set, and shares one predicate with the provisioning queue:

-export function hasTurnFileConsumer(mimeType: string, consumers: TurnFileConsumers): boolean {
-  return (
-    (consumers.executeCode && canToolResourceConsume(EToolResources.execute_code, mimeType)) ||
-    (consumers.fileSearch && canToolResourceConsume(EToolResources.file_search, mimeType))
-  );
-}
+export function hasToolResourceProvisioning(file: TurnDeliveryFile, toolResource: string): boolean {
+  if (toolResource === EToolResources.execute_code) {
+    return getCodeEnvRefs(file.metadata).length > 0;
+  }
+  return file.embedded === true || (file.metadata?.embeddedEntities?.length ?? 0) > 0;
+}

The evidence is paired with the tool that can read the type, so vectors do not qualify a code-only turn and a sandbox pointer does not qualify a retrieval-only one. packages/api/src/agents/resources.ts now calls hasToolResourceProvisioning in place of its own copy of that logic, which is what stops the two sides drifting apart again.

Every reader of a turn route goes through the same resolver, so one change covers the first turn, the resend path, and a handoff child:

resolveTurnLLMDeliveryPath          # packages/data-provider
  applyTurnDelivery                 # initialize.ts, first turn
  BaseClient.getAttachmentDeliveryPath
    addPreviousAttachments          # earlier turns
  createRunFileMessageEncoder       # handoff and subagent children

Change Type

  • Bug fix (non-breaking change which fixes an issue)

Testing

Reproduction is the issue's: a custom OpenAI-compatible endpoint with textFallbackWithoutTools: true and application/pdf overridden to none, a new plain chat, File Search toggled on, a text-bearing PDF attached. The model now answers from the extracted text, and the file is embedded on the first search rather than never.

Tests cover both sides of the disagreement, at each layer that resolves a route:

suite result
packages/data-provider resolve-llm-delivery-path.spec.ts 91 passed
packages/api agents/files, resources, initialize, files/upload, files/provision 511 passed
api BaseClient.test.js 148 passed
npx tsc --noEmit in packages/data-provider and packages/api clean
npm run static-checks (staged) ESLint, Prettier, import sorting, circular deps clean

New cases: File Search enabled but the vector store never received the file delivers text; Run Code enabled with no sandbox reference delivers text; a tool that holds the file but cannot read the type delivers text; and the same for the historical resend path and a handoff child. Three existing tests asserted that an enabled tool keeps a file off the prompt using fixtures that carried no provisioning evidence; they now carry the reference or the embedding, which preserves what each was written to check.

Not run: npm run lighthouse (no startup, config, or message-loading path changed) and api/server/routes/files/files.test.js (upload route untouched by this change).

Test Configuration:

ocr:
  strategy: document_parser

fileConfig:
  endpoints:
    "My Custom Endpoint":
      textFallbackWithoutTools: true
      defaultLLMDeliveryPath:
        overrides:
          "application/pdf": "none"

Checklist

  • My code adheres to this project's style guidelines
  • I have performed a self-review of my own code
  • I have commented in any complex areas of my code
  • My changes do not introduce new warnings
  • I have written tests demonstrating that my changes are effective or that my feature works
  • Local unit tests pass with my changes

A `none`-routed upload had its extracted text withheld whenever a file tool was
enabled, on the assumption the tool would serve the file. A plain chat with the
File Search toggle runs an ephemeral agent, so the upload is filed under no tool
resource and never embedded: the text was withheld for a vector store that never
received the file, and the attachment reached nothing.

Withholding now requires the record to show the tool holds the file, which is the
same evidence deferred provisioning reads before queueing it. `resources.ts`
delegates to that shared predicate so the two cannot disagree again.
@lia-by-librechat

Copy link
Copy Markdown
Contributor Author

Ready for review at head 30fc84a76412c197ce0379fa6ecf69d35096601a.

That head carries the whole change: resolveTurnLLMDeliveryPath withholds a none-routed file's extracted text only when a tool this turn runs both reads the type and already holds the file, judged from the record (embedded, metadata.embeddedEntities, codeEnvRef/codeEnvRefs) rather than from the enabled tool set. packages/api/src/agents/resources.ts delegates its own evidence check to the same exported predicate, so the delivery decision and the deferred-provisioning queue cannot disagree again.

The interesting review question is the pairing: evidence for one tool must not qualify the other, and a tool that holds the file but cannot read the type must not count. Both are covered in resolve-llm-delivery-path.spec.ts.

Checks at this head: 91 data-provider, 511 packages/api (agents/files, resources, initialize, files/upload, files/provision), 148 BaseClient.test.js, tsc --noEmit clean in both changed workspaces, and npm run static-checks clean on the staged diff. Three existing tests asserted that an enabled tool keeps a file off the prompt with fixtures carrying no provisioning evidence; they now carry a reference or an embedding, which preserves what each was written to check.

@lia-by-librechat

Copy link
Copy Markdown
Contributor Author

New head 5d71051000224841451f2a7e61a5c741fc8129ab.

Adds one test fixture fix on top of the previous head. client.test.js asserts that a code-running primary agent keeps a tool-routed file out of its prompt while a handoff agent without a reader receives the text; its fixture carried no sandbox reference, so under the new rule the primary agent legitimately received the text too. The fixture now carries the codeEnvRef that its premise assumes, which restores what the test was written to check. No production code changed since the last head.

Locally at this head: api/server/controllers/agents/client.test.js 251 passed, plus the suites reported on the previous head.

On CI, note that TypeScript type checks fails on dev itself, at the same step and with the same message:

npm run -w @librechat/api openapi:check
The committed OpenAPI spec does not match the code. Run: npm run -w @librechat/api openapi:generate

dev head 69f0dd2a121b629648a54dab2242d1b21ffc9eed (job 104911161640) fails there, so this PR inherits it rather than causing it. It came in with #15928, which added the spec and its check; the committed spec needs regenerating on dev. Nothing in this PR touches a route or a schema, and zod-openapi is not installed in my environment, so I have not regenerated it here.

@danny-avila

Copy link
Copy Markdown
Collaborator

@codex review the latest head

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-16T19:43:21.852892Z 5d71051 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 5d71051000

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit 9e83b10 into dev Sep 16, 2026
35 of 36 checks passed
@danny-avila
danny-avila deleted the lia/ephemeral-file-search-uploads branch September 16, 2026 19:46
danny-avila added a commit that referenced this pull request Sep 18, 2026
…reads (#16058)

Since #15694 bounded each turn's attachments across history, a thread whose
history counts more than `fileLimit` model-bound files is refused on every turn,
including turns that attach nothing. Two kinds of file reached that count that
never belonged in the prompt.

Code outputs: priming clears an expired sandbox reference on the turn's copy of
the record so the file is re-provisioned, and a route-less record without a
reference was classified as prompt content, so it counted toward the limit and
could be encoded as media. Code outputs now stay tool-owned regardless of
reference liveness, through one predicate shared by admission, BaseClient
delivery and the child run-file encoder.

Tool-routed spreadsheets: with `textFallbackWithoutTools`, #16027 delivered the
fallback text until a tool held a copy. Run Code receives its copy only on its
first call, which a refused turn never makes, so the text counted on every later
turn. An enabled Run Code is a reader again for the types it can read; File
Search still reads only what its store holds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Toggling File Search in a plain chat makes uploads invisible to the model (ephemeral agent uploads are never filed to a tool resource)

2 participants