decode: limit unknown fields to bound memory amplification (owned, dynamic, and view paths)

CHANGELOG.md

@@ -8,6 +8,55 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ### Added
 
+- **Zero-copy views enforce the unknown-field limit and coalesce adjacent
+  unknown records.** View decoding previously stored one borrowed span (16
+  bytes) per unknown wire record with no bound beyond the input size. Spans
+  for adjacent unknown records now coalesce into a single span — a
+  contiguous run of unknown fields costs one `Vec` slot regardless of field
+  count, and re-encodes byte-identically — and each *new* span (one per
+  unknown run) is counted against the same unknown-field limit that bounds
+  owned-message decoding, configured via
+  `DecodeOptions::with_unknown_field_limit` and honored by
+  `DecodeOptions::decode_view`. As part of this, the view decode path now
+  threads `DecodeContext<'_>`: `MessageView::decode_view_with_limit(buf,
+  depth)` is replaced by `decode_view_with_ctx(buf, ctx)`, and generated
+  views' hidden `_decode_depth` helpers become `_decode_ctx` (**breaking**
+  for code generated by earlier releases, which must be regenerated —
+  consistent with the owned-path change below).
+
+- **View-to-owned conversion is now fallible and honors the decode-time
+  limit.** `MessageView::to_owned_message` and `to_owned_from_source` (and
+  the `OwnedView` wrapper) now return `Result<Owned, DecodeError>`
+  (**breaking**): generated conversions previously swallowed unknown-field
+  re-materialization errors via `unwrap_or_default()`, silently dropping
+  every unknown field. `UnknownFieldsView::to_owned` also now re-materializes
+  under the unknown-field allowance that remained when the view recorded its
+  first unknown field — so a tight `with_unknown_field_limit` configured at
+  `decode_view` time carries through conversion, where each owned
+  `UnknownField` counts individually (unlike the coalesced spans the view
+  stores). Views built manually via `push_raw` fall back to the default
+  limit.
+
+- **Unknown-field decode limit bounds decoder memory amplification.**
+  Unknown wire data can occupy ~20× more memory decoded than encoded:
+  every 2-byte unknown varint field materializes a ~40-byte
+  `UnknownField`, so a 64 MiB payload of minimal unknown fields (flat or
+  nested in a group) could force over 1 GiB of heap — not bounded by
+  `with_max_message_size`, which only caps input length. Decoding now
+  counts every materialized unknown field against a limit shared across
+  the whole decode call and fails with the new
+  `DecodeError::UnknownFieldLimitExceeded` when it is exceeded. The
+  default is 1,000,000 fields per decode (`DEFAULT_UNKNOWN_FIELD_LIMIT`),
+  capping slot overhead at ~40 MB, and applies to all decode entry points
+  including the trait-level convenience methods; tune it with
+  `DecodeOptions::with_unknown_field_limit`. Unknown length-delimited
+  payload bytes are not counted against the limit — the decoder only
+  allocates them once the sender has actually delivered the bytes, so
+  they are bounded by the input size and governed by
+  `with_max_message_size`. The limit covers owned-message and
+  `DynamicMessage` decoding; zero-copy views store unknown fields as
+  borrowed spans and are not affected by the amplification.
+
 - **`chrono` interop for `buffa-types`** (#163). A new off-by-default,
   `no_std`-compatible `chrono` feature adds conversions between the
   well-known `Timestamp` / `Duration` types and `chrono::DateTime` /
@@ -29,6 +78,17 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
   carrier-agnostic `Location { line, column }`. Requires message types
   generated with `json = true`. Contributed by @rsd-darshan.
 
+### Changed
+
+- **Breaking:** the decode-path `Message` trait methods (`merge`,
+  `merge_field`, `merge_to_limit`, `merge_group`, `merge_length_delimited`),
+  `encoding::decode_unknown_field`, and `message_set::merge_item` now take a
+  `DecodeContext<'_>` — carrying the remaining recursion depth and the
+  shared unknown-field allowance — in place of the bare `depth: u32`. Code
+  generated with earlier releases must be regenerated. Callers of the
+  convenience methods (`decode`, `decode_from_slice`, `merge_from_slice`,
+  `DecodeOptions`) are unaffected.
+
 ### Fixed
 
 - **`DecodeOptions::decode_length_delimited_reader` no longer allocates the

DESIGN.md

@@ -372,7 +372,7 @@ pub trait Message: DefaultInstance + Clone + PartialEq + Send + Sync {
     // Required methods (implemented by codegen per message type):
     fn compute_size(&self, cache: &mut SizeCache) -> u32;  // Pass 1
     fn write_to(&self, cache: &mut SizeCache, buf: &mut impl BufMut);  // Pass 2
-    fn merge_field(&mut self, tag: Tag, buf: &mut impl Buf, depth: u32)
+    fn merge_field(&mut self, tag: Tag, buf: &mut impl Buf, ctx: DecodeContext<'_>)
         -> Result<(), DecodeError>;     // Per-field decode dispatch
     fn clear(&mut self);
 
@@ -381,7 +381,7 @@ pub trait Message: DefaultInstance + Clone + PartialEq + Send + Sync {
     fn encode_to_vec(&self) -> Vec<u8>;
     fn encode_to_bytes(&self) -> Bytes;
     fn decode_from_slice(data: &[u8]) -> Result<Self, DecodeError>;
-    fn merge(&mut self, buf: &mut impl Buf, depth: u32) -> Result<(), DecodeError>;
+    fn merge(&mut self, buf: &mut impl Buf, ctx: DecodeContext<'_>) -> Result<(), DecodeError>;
     fn merge_from_slice(&mut self, data: &[u8]) -> Result<(), DecodeError>;
     // ... + length-delimited and io::Read variants
 }

buffa-codegen/src/impl_message.rs

@@ -524,19 +524,15 @@ pub fn generate_message_impl(
                 if tag.field_number() == 1
                     && tag.wire_type() == ::buffa::encoding::WireType::StartGroup
                 {
-                    if depth == 0 {
-                        return ::core::result::Result::Err(
-                            ::buffa::DecodeError::RecursionLimitExceeded,
-                        );
-                    }
-                    let (type_id, bytes) = ::buffa::message_set::merge_item(buf, depth - 1)?;
+                    let (type_id, bytes) =
+                        ::buffa::message_set::merge_item(buf, ctx.descend()?)?;
                     self.__buffa_unknown_fields.push(::buffa::UnknownField {
                         number: type_id,
                         data: ::buffa::UnknownFieldData::LengthDelimited(bytes),
                     });
                 } else {
                     self.__buffa_unknown_fields.push(
-                        ::buffa::encoding::decode_unknown_field(tag, buf, depth)?
+                        ::buffa::encoding::decode_unknown_field(tag, buf, ctx)?
                     );
                 }
             }
@@ -545,13 +541,13 @@ pub fn generate_message_impl(
         quote! {
             _ => {
                 self.__buffa_unknown_fields.push(
-                    ::buffa::encoding::decode_unknown_field(tag, buf, depth)?
+                    ::buffa::encoding::decode_unknown_field(tag, buf, ctx)?
                 );
             }
         }
     } else {
         quote! {
-            _ => { ::buffa::encoding::skip_field_depth(tag, buf, depth)?; }
+            _ => { ::buffa::encoding::skip_field_depth(tag, buf, ctx.depth())?; }
         }
     };
 
@@ -689,7 +685,7 @@ pub fn generate_message_impl(
                 &mut self,
                 tag: ::buffa::encoding::Tag,
                 buf: &mut impl ::buffa::bytes::Buf,
-                depth: u32,
+                ctx: ::buffa::DecodeContext<'_>,
             ) -> ::core::result::Result<(), ::buffa::DecodeError> {
                 #[allow(unused_imports)]
                 use ::buffa::bytes::Buf as _;
@@ -1754,7 +1750,7 @@ fn scalar_merge_arm(
                     ::buffa::Message::merge_length_delimited(
                         self.#ident.get_or_insert_default(),
                         buf,
-                        depth,
+                        ctx,
                     )?;
                 }
             });
@@ -1767,7 +1763,7 @@ fn scalar_merge_arm(
                     ::buffa::Message::merge_group(
                         self.#ident.get_or_insert_default(),
                         buf,
-                        depth,
+                        ctx,
                         #field_number,
                     )?;
                 }
@@ -2071,7 +2067,7 @@ fn repeated_merge_arm(
             #field_number => {
                 #wire_check
                 let mut elem = ::core::default::Default::default();
-                ::buffa::Message::merge_length_delimited(&mut elem, buf, depth)?;
+                ::buffa::Message::merge_length_delimited(&mut elem, buf, ctx)?;
                 self.#ident.push(elem);
             }
         });
@@ -2086,7 +2082,7 @@ fn repeated_merge_arm(
             #field_number => {
                 #wire_check
                 let mut elem = ::core::default::Default::default();
-                ::buffa::Message::merge_group(&mut elem, buf, depth, #field_number)?;
+                ::buffa::Message::merge_group(&mut elem, buf, ctx, #field_number)?;
                 self.#ident.push(elem);
             }
         });
@@ -2478,10 +2474,10 @@ fn oneof_merge_arm(
                 if let ::core::option::Option::Some(
                     #enum_ident::#variant_ident(ref mut existing)
                 ) = self.#field_ident {
-                    ::buffa::Message::merge_length_delimited(#existing_ref, buf, depth)?;
+                    ::buffa::Message::merge_length_delimited(#existing_ref, buf, ctx)?;
                 } else {
                     let mut val = ::core::default::Default::default();
-                    ::buffa::Message::merge_length_delimited(&mut val, buf, depth)?;
+                    ::buffa::Message::merge_length_delimited(&mut val, buf, ctx)?;
                     self.#field_ident = ::core::option::Option::Some(
                         #enum_ident::#variant_ident(#wrapped_val)
                     );
@@ -2494,10 +2490,10 @@ fn oneof_merge_arm(
                 if let ::core::option::Option::Some(
                     #enum_ident::#variant_ident(ref mut existing)
                 ) = self.#field_ident {
-                    ::buffa::Message::merge_group(#existing_ref, buf, depth, #field_number)?;
+                    ::buffa::Message::merge_group(#existing_ref, buf, ctx, #field_number)?;
                 } else {
                     let mut val = ::core::default::Default::default();
-                    ::buffa::Message::merge_group(&mut val, buf, depth, #field_number)?;
+                    ::buffa::Message::merge_group(&mut val, buf, ctx, #field_number)?;
                     self.#field_ident = ::core::option::Option::Some(
                         #enum_ident::#variant_ident(#wrapped_val)
                     );
@@ -2733,7 +2729,7 @@ fn map_element_decode_stmt(
             }
         }
         Type::TYPE_MESSAGE => {
-            quote! { ::buffa::Message::merge_length_delimited(&mut #var, #buf_expr, depth)?; }
+            quote! { ::buffa::Message::merge_length_delimited(&mut #var, #buf_expr, ctx)?; }
         }
         _ => {
             let decode_fn = decode_fn_token(ty);
@@ -2907,7 +2903,7 @@ fn map_merge_arm(
                 match entry_tag.field_number() {
                     1 => { #decode_key }
                     2 => { #decode_val }
-                    _ => { ::buffa::encoding::skip_field_depth(entry_tag, buf, depth)?; }
+                    _ => { ::buffa::encoding::skip_field_depth(entry_tag, buf, ctx.depth())?; }
                 }
             }
             // Correct the buffer position if the entry was not fully consumed.

buffa-codegen/src/owned_view.rs

@@ -273,8 +273,14 @@ pub(crate) fn generate_owned_view_wrapper(
             }
 
             /// Convert to the owned message type.
-            #[must_use]
-            pub fn to_owned_message(&self) -> #owned_path {
+            ///
+            /// # Errors
+            ///
+            /// Returns an error if re-materializing preserved unknown fields
+            /// fails (e.g. the unknown-field limit is exceeded).
+            pub fn to_owned_message(
+                &self,
+            ) -> ::core::result::Result<#owned_path, ::buffa::DecodeError> {
                 self.0.to_owned_message()
             }
 

buffa-codegen/src/tests/view_codegen.rs

@@ -93,10 +93,10 @@ fn test_view_repeated_message_field() {
         content.contains("RepeatedView") && content.contains("ItemView"),
         "ContainerView.items must be RepeatedView<ItemView>: {content}"
     );
-    // _decode_depth must be generated for both view types.
+    // _decode_ctx must be generated for both view types.
     assert!(
-        content.contains("fn _decode_depth"),
-        "missing _decode_depth impl: {content}"
+        content.contains("fn _decode_ctx"),
+        "missing _decode_ctx impl: {content}"
     );
 }
 
@@ -225,9 +225,9 @@ fn test_view_oneof_with_message_variant() {
         content.contains("BodyView") && content.contains("::buffa::alloc::boxed::Box<"),
         "Payload view must have boxed BodyView variant: {content}"
     );
-    // Decode arm for the message variant must check recursion depth.
+    // Decode arm for the message variant must consume one recursion level.
     assert!(
-        content.contains("RecursionLimitExceeded"),
+        content.contains("ctx.descend()?"),
         "message-type oneof variant must check recursion depth: {content}"
     );
 }