encode: vectored (rope) output via an EncodeSink abstraction

[iainmcgin]

Summary

Adds vectored ("rope") encode: a message whose dominant content is one large bytes field can now be encoded without copying the payload at all. Draft for design review — targeted at the 0.9.0 breaking bundle.

Today every encode is contiguous: Message::write_to and all the encode helpers take &mut impl BufMut, so a large payload is always memcpy'd into the output buffer, even when the field already holds a refcounted bytes::Bytes and the consumer (hyper/h2, any vectored writer) would happily take segments.

Design

• EncodeSink replaces the BufMut bound on every encode path. A blanket impl over BufMut keeps all existing callers source-compatible (encode_to_vec, encode_to_bytes, msg.encode(&mut vec) unchanged), and const IS_SEGMENTED = false const-folds the zero-copy branch away, so contiguous sinks compile to the exact pre-change code — verified at ±noise on the benchmark. The trait deliberately exposes only the BufMut subset the encoders use.
• Rope is the segmented sink: tags/varints/small fields coalesce into a tail buffer; bytes::Bytes fields at or above min_segment (default 4 KiB) become refcount-shared segments via the new provided ProtoBytes::as_shared hook. Rope::with_backing extends this to views: large borrowed &[u8] fields that lie inside the decode source buffer are captured zero-copy through a containment-checked slice_ref (shared with the existing bytes_from_source helper), making decode→re-encode relays fully zero-copy.
• RopeBuf adapts a finished rope to bytes::Buf with a chunks_vectored override, so the whole rope reaches a vectored writer in one call; consumed segments are released eagerly.
• Codegen emits put_shared_bytes_field for bytes fields in every context (singular, optional, repeated, oneof, and map values through the shared-aware codecs). Vec<u8> fields and view slices keep the copy path — as_shared is a compile-time None there.

Why breaking

Manual Message/ViewEncode implementations must change their write_to bound (impl BufMut → impl EncodeSink), and generated code must be regenerated — the same class of churn as the other queued 0.9.0 items, hence one bundle. cargo semver-checks passes (196/196), but note its known blind spot: trait-method bound changes on implementors are exactly what it does not model; the changelog fragment declares the break explicitly.

Numbers (dev machine, chunk-shaped message: 32 B digest + u64 + one bytes payload)

payload	contiguous encode_to_bytes	through Rope
64 KiB	1.43 µs	150 ns
512 KiB	10.6 µs	150 ns
2 MiB	128 µs	151 ns

Rope encode is payload-size-independent — the ~150 ns is the two-pass size computation plus the tail buffer and one refcount. Contiguous encode_to_bytes is unchanged within run-to-run noise. Worth re-confirming on the quiesced benchmark host and against the full benchmark suite before merge.

Testing

• 2,307 workspace tests pass; 10 new unit tests on Rope/RopeBuf (segment ordering, backing containment, Buf contract, Default parity, reborrow forwarding, eager segment release) and 7 integration tests through real generated code (pointer-identity for singular/repeated/oneof/map fields, byte-identical wire output vs contiguous, zero-copy view re-encode, Vec<u8> fallback).
• check-nostd clean (chunks_vectored is std-gated); clippy clean; MSRV-1.75 compatible.

Notes for review

• The generated-code diff (buffa-descriptor, buffa-types) is the mechanical signature swap plus the put_shared_bytes_field call sites; the reviewable core is buffa/src/encode_sink.rs, the types.rs additions, and the two codegen files.
• examples/logging and examples/bsr-quickstart got the signature swap in their checked-in generated code, but both examples were already broken against current main for unrelated reasons (map hasher drift; MessageView trait drift vs the pinned BSR plugin) — examples aren't built in CI. Worth a separate cleanup.
• Sequencing: this touches the same encode entry points as encode: enforce the protobuf 2 GiB message-size limit #271 (encode-size-guard); whichever lands second rebases.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[iainmcgin]

[claude code] Metal bench confirmation for the numbers flagged in the PR body. Setup: quiesced bare-metal host (turbo disabled, performance governor, RT priority on a pinned core, SMT sibling avoided), layout-normalized builds (lto=true, codegen-units=1, -Cllvm-args=-align-all-nofallthru-blocks=6 -Cllvm-args=-align-loops=64), baseline = main @ 063cda6, 10 s measurement per bench, one box for the whole A/B.

Contiguous-path regression check — full protobuf bench suite, encode as the changed surface and decode as the untouched control:

bench	main	this PR	delta
api_response/encode	2.960 µs	2.982 µs	+0.74%
log_record/encode	9.419 µs	9.449 µs	+0.17%
analytics_event/encode	350.8 µs	354.0 µs	+1.59%
google_message1_proto3/encode	95.61 ns	95.22 ns	−0.31%
media_frame/encode	20.28 µs	20.17 µs	−0.55%
packed_tile/encode	563.3 µs	561.8 µs	−0.31%
mesh/encode	849.3 µs	838.0 µs	−1.27%
packed_signed/encode	114.9 µs	115.5 µs	+0.49%

Encode: mean +0.07%, range [−1.27%, +1.59%]. The decode control — code this PR does not touch — spans [−6.94%, +2.03%] on the same box (packed_tile/decode −6.9%, log_record/decode +2.0%), so every encode delta is inside the residual layout/measurement noise envelope. No measurable contiguous-path regression; the IS_SEGMENTED const-fold claim holds.

Rope absolute numbers (chunk-shaped message, one dominant bytes field; fixed 2.4 GHz so absolute values run ~1.5× the dev-machine turbo numbers in the PR body):

payload	contiguous encode_to_bytes	through Rope	speedup
64 KiB	2.15 µs	230 ns	9×
512 KiB	17.1 µs	228 ns	75×
2 MiB	180.3 µs	223 ns	808×

Rope encode is payload-size-independent (CI widths < 0.1%), confirming the O(header) claim on quiet hardware. This closes the "re-confirm on the quiesced benchmark host" item from the PR body.