client: carry response metadata on every terminal error - #262
Merged
Merged
Conversation
iainmcgin
force-pushed
the
fix/terminal-error-metadata
branch
from
July 24, 2026 18:33
1540af2 to
c623e55
Compare
Collaborator
Author
|
Filed the conformance gap upstream as connectrpc/conformance#1121: the whole testsuite only ever echoes The upstream issue puts the open question plainly — should a conformant server drop reserved names, forward them verbatim, or reject the RPC — and notes that |
Terminal errors attached response headers and trailers inconsistently across stream shapes and protocols, so whether a caller could read the server's metadata off a failed RPC depended on which path produced it. The metadata is now applied where the terminal record is written rather than at each site that constructs an error. `ServerStream::message()` has the only write, so carried END_STREAM errors, decompression failures and both gRPC classification paths are covered in one place, and the explicit call added earlier for the malformed-JSON case is redundant. Writing it in rather than applying it on read matters because `error()` reads the stored record directly. The gRPC status keys stay out of error metadata. That filter used to live inline in one function; it is extracted so the choke point recomputes the curated view instead of overwriting it, which also makes the two Connect shapes agree. Guarding on an already-populated map would not have worked: trailers carrying only `grpc-status` produce a legitimately empty one. Attaching upstream headers to streaming errors exposed a framing bug on the server. It echoed them onto the wire with a builder that appends, so a gateway propagating an upstream failure over Connect on HTTP/1.1 sent a response hyper could not serialize at all - the connection closed with zero bytes written, and a handler's `permission_denied` reached the caller as `unavailable` with no metadata, every error code collapsing the same way. Measured against connect-go 1.19.1. An echoed `content-encoding` was a second, milder break, and an echoed `date` displaces the one hyper would have set. The echo now lets the response's own headers win and drops names that would describe something other than this response: hop-by-hop, body framing, provenance and status-bearing. Multi-valued metadata still passes through, verified against grpc-go 1.81.1 and connect-go. Trailing metadata is echoed unfiltered still; that needs a policy rather than a predicate, and has its own issue. Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
iainmcgin
force-pushed
the
fix/terminal-error-metadata
branch
from
July 24, 2026 23:23
c623e55 to
0124ac7
Compare
iainmcgin
marked this pull request as ready for review
August 1, 2026 21:15
iainmcgin
enabled auto-merge
August 1, 2026 21:15
asacamano
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #202
Terminal client errors now carry the response headers whenever they were received, and the trailers whenever they were parsed, regardless of stream shape or protocol.
The inventory in #202 listed four sites where an error dropped metadata its sibling path preserved. Rather than adding
set_response_headersat each, the metadata is attached where the terminal record is written, so the rule holds by construction:ServerStream::message()has the only write, and everyStreamEndfunnels through it — carried END_STREAM errors, decompression failures, theclassify_grpc_endvariants and theparse_grpc_error_from_trailersresults. The explicit call added by #201 is now redundant and removed. Writing the metadata in rather than applying it on read matters becauseServerStream::error()reads the stored record directly. The remaining bare sites on the Connect unary and client-streaming paths are covered too, so "consistently" means what it says.Item 5 in the issue is untouched; it needs a
'staticbound on the body error type and is tracked as #237.The choke point initially clobbered curated trailers.
parse_grpc_error_from_trailersdeliberately excludes the gRPC status keys from error metadata, and attaching the raw map on top re-exposed them. Guarding on "already set" would have been unsound — trailers carrying onlygrpc-statusproduce a legitimately empty curated map — so the filter is extracted and the choke point recomputes it, which is idempotent on the gRPC path and a no-op on Connect.The server-side half, and why it is here
Attaching upstream headers to streaming errors would have widened a framing bug that already existed on the unary paths, so it is fixed rather than shipped and filed.
Our server echoed
err.response_headers()onto the wire with a builder that appends. Driven against connect-go v1.19.1 and grpc-go v1.81.1, with a per-header bisect, that is not hygiene — it is a hard break. A gateway propagating an upstream failure over Connect on HTTP/1.1 emits a response hyper cannot serialize at all: the connection closes with zero bytes written, connect-go reportsunexpected EOFwith empty metadata, and a handler'spermission_deniedreaches the caller asunavailable. Every error code collapses the same way, because all the client ever sees is a transport failure.content-lengthis the sole culprit. An echoedcontent-encodingis a second, milder break — connect-go tries to gunzip the plain-JSON error body and falls back to the HTTP status — and an echoeddatedisplaces the one hyper would otherwise set, so the response carries a false generation time.echo_error_headersapplies two rules: a header the response already set wins, and names that would describe something other than this response are dropped — hop-by-hop, body framing, provenance, status-bearing. Legitimate metadata is unaffected:x-upstream-regionand both values of a multi-valuedx-upstream-tagarrive at every client on every shape, which is the case the snapshot-before-append logic could plausibly have broken.Dropping
grpc-encodingwas checked rather than assumed. grpc-go only consults it when a frame's compressed flag is set, and these responses have no data frames, so an echoed one is inert today — confirmed, including underAcceptCompressors("identity"), which makes grpc-go police the advertised encoding. It becomes live the moment a response carries both the echoed encoding and a message frame, and the Connect analogue of the same class already is.Trailing metadata is still echoed unfiltered. The same class exists one frame later, but fixing it needs a policy rather than a predicate — provenance is not derivable where the filtering happens, the obvious signal is remotely drivable, and the source-selection branch has the same bug one level up. An attempt to fix it here was reverted for those reasons and is now #263.
Verification
Conformance: server 3600/0, server Connect+TLS 2396/0, client Connect 2580/0, client gRPC-Web 2838/0. The gRPC client suite fails 1-4 cases in
Timeoutsacross four runs with a different set each time; base flakes identically, including one clean run.duplicate_metadata.yamlis the case worth naming — it echoes multi-valuedx-custom-headeronto error responses across all four RPC kinds. Conformance never asks a server to echo a connection-scoped header, though, and never echoes a name the server also sets, so neither new rule is exercised by the suite; unit tests and the interop harness cover those.One note on CI: two
handler::testselement-budget tests fail onmainright now, independently of this change — fixture rot from buffa 0.9.1, fixed by #239.