fix: preserve system prompt injection through upstream extproc filter - #2745
Open
steffen-karlsson wants to merge 3 commits into
Open
steffen-karlsson wants to merge 3 commits into
steffen-karlsson wants to merge 3 commits into
Conversation
✅ Deploy Preview for theagentrouter ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Hi @steffen-karlsson, from my understanding, this is related to issue #2671 and #2720 |
Signed-off-by: Steffen Karlsson <steffen.karlsson@gmail.com>
…arge bodies Envoy 1.37.1's ext_proc filter has a null-pointer dereference when the UDS send buffer hits the high watermark (data_deferred_ + pending_send_buffer_high_watermark). This occurs when the request body exceeds the HTTP2 per-stream window size (default 64KB). Increase the InitialStreamWindowSize from 64KB to 16MB on the UDS cluster's HTTP2 protocol options. This allows large request bodies to be sent to the extproc without triggering the high watermark condition. Keep BUFFERED mode for the upstream extproc filter (needed for correct auth signing with the mutated body). Also add RequestTrailers handling in the server for future compatibility with FULL_DUPLEX_STREAMED mode. Signed-off-by: Steffen Karlsson <steffen.karlsson@gmail.com>
steffen-karlsson
force-pushed
the
fix/preserve-system-prompt-
branch
from
September 29, 2026 06:41
9207f10 to
034af14
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When multiple ext_proc filters are chained (e.g. a custom filter that modifies the request body, followed by the AI Gateway upstream filter), the upstream filter clobbers any body mutations made by earlier filters.
The upstream filter used
request_body_mode: NONE, so it capturedoriginalRequestBodyRawinProcessRequestHeadersbefore any other HTTP-level filter had a chance to run. It then issuedCONTINUE_AND_REPLACEwith that stale snapshot whenevermodelNameOverride,forceBodyMutation, orhttpBodyMutationwas active — silently replacing the body that earlier filters had modified.Additionally, the
header_mutationfilter (which appendedcontent-lengthfrom dynamic metadata) was redundant:content-lengthis already set directly in theBodyMutationresponse when the body changes.Fix
RequestBodyMode: BUFFERED— the upstream filter now receives the actual request body (after all HTTP-level filters have processed it) inProcessRequestBody, instead of replaying a stale snapshot fromProcessRequestHeaders.Split headers and body processing —
ProcessRequestHeadersnow does header-level mutations only (path rewrite, auth, route header mutations, sensitive header stripping). Body-level mutations (translatorRequestBody,httpBodyMutation) are deferred toProcessRequestBody.Moved backend auth to
ProcessRequestBody— auth handlers like AWS SigV4 need the final body for payload signing. With the body no longer available inProcessRequestHeaders, auth is performed inProcessRequestBodywhere the actual (possibly translated/mutated) body is available.Removed
forceBodyMutationin router processor — thestream_options.include_usagemutation is applied immediately viaBodyMutation+CONTINUE_AND_REPLACEin the router'sProcessRequestBody, so the upstream filter no longer needs to force-replace the body.Removed
header_mutationfilter —content-lengthis set directly on theBodyMutationresponse, making the dynamic-metadata-basedheader_mutationfilter unnecessary.Testing
internal/extproc/...,internal/extensionserver/...)ProcessRequestBodyafterProcessRequestHeadersand assert that body mutations and auth headers are in the body responseRelated Issues/PRs (if applicable)
#2671 and #2720
Special notes for reviewers (if applicable)
N/A