server: pass large response payloads through by refcount instead of copying - #219
Merged
Conversation
…opying Envelope framing copied every encoded message into the framing buffer: once into the batching buffer on the streaming paths, and once via Envelope::encode on the gRPC unary path. For a message dominated by one large bytes field this doubled the payload-sized memcpys per response. Payloads of at least 16 KiB (envelope::MIN_CHAIN_SIZE) on the wire are now emitted as their own HTTP body data frame: the 5-byte envelope header goes into the framing buffer and the payload Bytes is passed through by refcount. Envelope framing is independent of HTTP frame boundaries, so the wire bytes are unchanged - the reassembly tests and the full conformance suite (3600/3600) verify byte-identical output. EnvelopeEncoder::encode_chained is the single implementation of the encode policy (the tokio_util Encoder impl delegates with an infinite chain threshold, so the compression and chaining decisions cannot drift), compressed payloads chain on their post-compression size, and the oversized-payload check now runs before any buffer growth. Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
…aining The staging comments still said the chained path was uncompressed-only, which the compressed-chaining test and the changelog contradict; the chaining tests were keyed to STREAM_BATCH_THRESHOLD, which is the flush heuristic, not the chaining threshold - they only passed because the two 16 KiB constants happen to be equal. Reference MIN_CHAIN_SIZE directly. Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
iainmcgin
marked this pull request as ready for review
July 17, 2026 23:58
iainmcgin
enabled auto-merge
July 17, 2026 23:58
asacamano
approved these changes
Jul 18, 2026
EffortlessSteven
pushed a commit
to EffortlessSteven/connect-rust
that referenced
this pull request
Aug 3, 2026
…nectrpc#232) ## Summary Carries buffa 0.9's reference-counted encode segments from the encoder all the way to the HTTP body, so a gRPC or gRPC-Web unary response whose body is an `OwnedView` never copies its large fields. A view's fields are slices into the buffer the view was decoded from. A rope told about that buffer takes a large field by reference instead of copying it into the encode buffer, and the segments then become their own HTTP data frames. Past the framing threshold the encode stops growing with the payload, because only the framing is still being written. Measured on a dedicated bare-metal instance (`benches/rpc/view_rope_encode`, four-string message, contiguous vs segmented): | bytes per field | contiguous | segmented | |---|---|---| | 256 B | 171 ns | 184 ns | | 1 KiB | 210 ns | 204 ns | | 16 KiB | 2.29 µs | 682 ns | | 256 KiB | 48.6 µs | 709 ns | | 1 MiB | 363 µs | 643 ns | The control is the part worth reading: an *unbacked* rope at 1 MiB costs 363 µs, identical to the contiguous encode. The rope machinery is free; the reference-count capture is the entire effect. ## Where this deliberately does nothing Segmenting is skipped wherever it would not pay, each guarded in code rather than left to chance: - **Owned-message bodies.** Their `string` and `bytes` fields are `String` and `Vec<u8>`, which cannot be handed over by reference. Routing them through a rope only adds cost. - **Responses below the framing threshold.** Anything smaller is copied into the framing buffer downstream regardless. - **A response much smaller than the request it borrows from.** Capturing would keep the whole request buffer alive until the response finishes flushing — answering a 64 MiB upload with a 32 KiB summary would hold 64 MiB per in-flight response. Copying is cheaper than that. - **Connect unary, streaming, and any call through an interceptor.** See below. ## Reach, stated plainly This lands on gRPC and gRPC-Web unary responses. Connect unary still flattens because its body type hardcodes `Full<Bytes>`; multiple HTTP/2 DATA frames concatenate to the same Connect body, so that path *could* carry segments and is worth doing — Connect being the primary protocol. Streaming re-envelopes per item and is the next piece of work. A call with an interceptor configured takes the contiguous path, because an interceptor may replace the whole body. ## Breaking change, confined to custom dispatch `EncodedResponse` is now `Response<EncodedBody>` rather than `Response<Bytes>`. This is reachable through the public `Dispatcher` trait, so anyone implementing custom dispatch is affected: construct with `Bytes::into()`, and recover a single buffer with `EncodedBody::into_contiguous()`. Interceptor authors need no change — `UnaryResponse` still carries a contiguous `Payload`. That seam is what keeps the blast radius to dispatch rather than every extension point. ## Builds on Both prerequisites are now merged, and this branch is rebased onto `main`: connectrpc#233 (the buffa 0.9 adoption, which supplies the rope and `EncodeSink`) and connectrpc#219, whose framing emission path this reuses rather than reimplementing. ## Testing Server conformance 3600 passed / 0 failed, 56 test suites, lint, docs, and `task generate:all` (no drift) all green — re-run after rebasing onto `main`, not carried over from the stacked base. `Envelope::encode_body_parts` has a table test sweeping empty, sub-threshold contiguous, sub-threshold segmented, over-threshold contiguous, two-segment and many-segment bodies, asserting in each that the envelope header declares exactly the byte count that follows it and that reassembly equals the contiguous envelope. A high-effort review pass found four defects that produced correct bytes and so would never have failed a test — buffer pinning, a gate that compared whole-message size against a per-field threshold, tiny rope fragments becoming their own frames, and `MaybeBorrowed` silently not forwarding the segmented encode. All four are fixed in the final commit, "response: only segment where it pays, and merge the fragments", whose message carries the reasoning. One known limitation left in place: a message that clears the gate while none of its individual fields do lands everything in a rope tail that grows by doubling rather than one sized allocation. buffa 0.9 exposes no way to pre-size the tail, so it is documented where it bites rather than worked around. --------- Signed-off-by: Iain McGinniss <309153+iainmcgin@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eliminates the second payload-sized copy in the server's response framing. Today every streaming message (Connect, gRPC, and gRPC-Web) and every gRPC unary response is copied twice between the handler and the socket: once by message encoding, and once more when the envelope layer
put_slices the encoded message into the framing buffer. For messages dominated by one largebytesfield, the second copy doubles encode cost — and hands hyper/h2 nothing they can't already handle without it, since h2 chains anyBufpayload above 256 bytes and writes it to the socket with vectored I/O.With this change, payloads of at least 16 KiB on the wire (
envelope::MIN_CHAIN_SIZE) are emitted as their own HTTP body data frame by reference count: the 5-byte envelope header goes into the framing buffer, and the payloadBytespasses through unmoved. Compressed payloads chain on their post-compression size, so enabling compression never makes the copy overhead worse than leaving it off.The wire bytes are unchanged. gRPC/Connect envelope framing has always been independent of HTTP-level frame boundaries; the only observable difference is that the envelope header may arrive in a separate HTTP/2 DATA frame (or HTTP/1.1 chunk) from its payload.
Design notes
EnvelopeEncoder::encode_chainedis the single implementation of the envelope-encode policy; thetokio_util::codec::Encoderimpl delegates to it with an infinite chain threshold, so the compression decision and the chaining decision cannot drift apart.BatchingEnvelopeStreamstages the chained payload aspending_payload, drained ahead of any staged finalizer — order proven bystreaming_error_after_chained_payload_preserves_order.GrpcUnaryBodygains a second data-frame slot fed byEnvelope::encode_parts.> u32::MAX) check now runs before any buffer growth.Measured on a dev machine with a chunk-shaped message (one dominant
bytesfield): encode+frame at 512 KiB drops from 25.3 µs to 10.6 µs (the cost of the encode copy alone); at 2 MiB from 1.24 ms to 125 µs, because the eliminated framing buffer was also a fresh multi-megabyte allocation per message.Testing
Scheduling
Queued for the 0.9.0 release rather than a 0.8.x patch. Client-side request framing has the same second copy; that is a separate follow-up PR.