Codegen rejects large schemas: element memory limit exceeded (~300 protos)

[iainmcgin]

Codegen rejects moderately large schemas with element memory limit exceeded. Around 300 proto files is enough to break protoc-gen-buffa, and there is currently no option anywhere to raise the limit.

Introduced by #319 (element-memory bound on repeated elements and map entries), shipped in v0.9.0.

Reproduction

Against googleapis, taking the first N files of find google -name '*.proto':

protoc -I. --buffa_out=out @filelist.txt

protos	CodeGeneratorRequest size	result
250	~4.4 MB	OK
300	~5.5 MB	--buffa_out: failed to decode CodeGeneratorRequest: element memory limit exceeded
7071 (all)	~81 MB	same

Why descriptors hit this so easily

The limit charges size_of::<Element>() per repeated length-delimited element, and the descriptor types are unusually large structs:

size_of::<FileDescriptorProto>()  = 704
size_of::<FieldDescriptorProto>() = 416
size_of::<DescriptorProto>()      = 360

Measured against real FileDescriptorSets, the element footprint is consistently about 6.3x the encoded byte size:

descriptor set	bytes	smallest budget that decodes
googleapis, all 7071 protos, with source info	81,431,653	512 MiB
googleapis, source_code_info stripped (#286)	20,721,688	128 MiB
1600 protos	4,582,704	32 MiB
351 protos (aiplatform)	1,164,985	8 MiB

So DEFAULT_ELEMENT_MEMORY_LIMIT (32 MiB) works out to roughly 5 MB of descriptors — and 5 MB * 6.3 = 31.5 MiB lands right on the default. A CodeGeneratorRequest is worse than a bare FileDescriptorSet of the same schema, because protoc includes source_code_info and every transitive import.

Affected paths

All four decode with default options, and none is overridable:

location	symptom
protoc-gen-buffa/src/main.rs:104	failed to decode CodeGeneratorRequest: element memory limit exceeded
protoc-gen-buffa-packaging/src/main.rs:174	same
buffa-build/src/lib.rs:1695	failed to decode FileDescriptorSet: element memory limit exceeded
DescriptorPool::decode (buffa-descriptor/src/pool.rs:280)	see below

DescriptorPool::decode is the worst of the four, because generated descriptor_pool() calls it as:

DescriptorPool::decode(FILE_DESCRIPTOR_SET_BYTES)
    .expect("embedded FileDescriptorSet is well-formed")

A consumer with a large schema and reflection enabled therefore gets a runtime panic asserting the bytes are malformed — when they are perfectly well-formed and it was the limit that rejected them. There is no override on that path at all.

Workarounds today

None for the plugin path. buffa-build::Config exposes no decode-limit method, protoc-gen-buffa parses no such option, and the generated descriptor_pool() is hardcoded. Splitting the schema into smaller protoc invocations is the only avoidance.

Fix

The underlying issue is that a DoS bound designed for untrusted input is being applied to trusted input: protoc's own output, the descriptor set from a protoc that buffa-build just invoked, and bytes that buffa-codegen itself embedded. The generated doc comment already states this ("emitted by buffa-codegen from the same descriptors it generated this code from, so a panic indicates a codegen bug, not consumer input").

Planned:

1. Tooling paths lift or significantly raise the limit — protoc-gen-buffa, protoc-gen-buffa-packaging, buffa-build, and the generated descriptor_pool().
2. DescriptorPool::decode gains an options-taking variant so callers can set the element-memory limit. The existing entry point stays bounded, since it is public and may legitimately be handed an untrusted descriptor set (an uploaded FDS, a reflection service).
3. The protoc plugins get an element_memory_limit option, and a higher default than the library's — the caller controls the protobuf inputs in that context.
4. The panic message in generated descriptor_pool() gets fixed — it is actively misleading when the limit is the cause.

The global DEFAULT_ELEMENT_MEMORY_LIMIT is deliberately not being raised: that would weaken the protection #319 added for genuinely untrusted input. The fix is per-path.

[iainmcgin]

[claude code] For cross-reference: connect-rust has the same four-path exposure, and I've fixed the two we own in connectrpc/connect-rust#239 — protoc-gen-connect-rust's CodeGeneratorRequest decode and connectrpc-build's FileDescriptorSet decode both now raise the budget, since both consume the local toolchain's own output.

Two notes that may be useful for the buffa-side fix:

A finite ceiling rather than usize::MAX. Footprint is not bounded by the 2 GiB input limit — an empty FileDescriptorProto is ~2 bytes encoded and a 704-byte struct — so lifting entirely turns a crafted set into an OOM kill where a ceiling gives an error. We used 16 GiB, which clears anything that fits under the input limit.

connect-rust amplifies the generated descriptor_pool() case. Reflector::from_descriptor_pool requires strategy: all on the buffa plugin, so any one package's pool spans the whole codegen run — which means every generated package embeds the entire schema's closure, not just its own. Users following our documented reflection setup will hit DescriptorPool::decode sooner than the file counts in your table suggest. Point 4 of your plan (fixing the misleading panic message) is worth having for that reason alone.

We're keeping the runtime reflection path bounded to match your stance on DescriptorPool::decode, and holding off on exposing a configurable limit until 0.9.1 lands so we mirror whatever mechanism you pick rather than inventing a second one.

[iainmcgin]

[claude code]

Fixed by #332, merged as 981c4d1. The Fixes #331 line in the PR body did not trigger the auto-close, so closing by hand.

Both overrides are now available: the element_memory_limit=<bytes|unlimited> plugin option for the protoc/buf path, and BUFFA_ELEMENT_MEMORY_LIMIT for buffa-build, which has no parameter string to carry an option. Tooling paths decode under a 1 GiB bound by default rather than the 32 MiB untrusted-input default, and generated descriptor_pool() scales its bound to the length of the bytes compiled into it, so it needs no configuration at any schema size. DEFAULT_ELEMENT_MEMORY_LIMIT is unchanged, so #319's bound on untrusted input is intact.

Verified against the reproduction in this issue: all 7071 googleapis protos now generate (18,134 files), where 300 previously failed.

One gap this did not close, tracked separately as #330: none of these limits apply on the JSON path.