Skip to content

Bound UBJSON optimized arrays of a valueless type - #5504

Merged
nlohmann merged 2 commits into
bjdata-ndarray-size-recursionfrom
ubjson-valueless-count-cap
Sep 11, 2026
Merged

nlohmann merged 2 commits into
bjdata-ndarray-size-recursionfrom
ubjson-valueless-count-cap

Conversation

@nlohmann

@nlohmann nlohmann commented Sep 6, 2026 •

Copy link
Copy Markdown
Owner

Closes #2793. Also the root cause of the OSS-Fuzz out-of-memory and timeout reports for parse_ubjson_fuzzer (issues 42506968 and 42506917, both open since January 2022).

What

An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its type marker alone, so an optimized UBJSON array of one of those has no payload: reading an element consumes no input at all. Its declared count is therefore the only thing that decides how much is allocated, and nothing bounded it.

[$Z#l plus a four-byte count is nine bytes of input describing two billion values. #2793 reports 35 GB and 150 seconds from ten bytes of input.

Everything else is already bounded by the end of the input, because it costs at least one byte per element:

  • 'N' (no-op) is skipped rather than stored
  • objects are safe — every element is preceded by its key, which costs bytes
  • BJData already refuses these markers as an optimized type, so this is a plain UBJSON matter

How

Reject a count above 1,048,576 elements for those three types with out_of_range.408, the code this reader already uses for a declared size it will not honour. The check runs before the SAX start event, so no container is opened and then abandoned.

Rejecting on the read side alone would break the guarantee that anything to_ubjson() writes can be read back, and would trip the round-trip assertion in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized encoding, one byte per element, for arrays of these types above the same limit. Its decision depends only on the array's size, which is identical for a value and for anything parsed back from it, so the round trip is stable:

json j(1048577, nullptr);
json::from_ubjson(json::to_ubjson(j, true, true)) == j;   // still true

Verification

The #2793 payload is now rejected in 0 ms instead of allocating tens of gigabytes. At the limit the optimized form is still used (9 bytes for a million nulls); one past it the writer emits the unoptimized form and it still round-trips. No existing test changes: the largest such count in the test suite is 65,793.

API impact

No breaking changes to the public API. One accepted-input change: a plain UBJSON array of $Z/$T/$F declaring more than 1,048,576 elements is now rejected with out_of_range.408 instead of being materialised. Documented in exceptions.md and ubjson.md.

Checklist

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced.
  • The Code coverage remained at 100%. A test case for every new line of code.
  • If applicable, the documentation is updated.
  • The source code is amalgamated by running make amalgamate.

🤖 Generated with Claude Code

@nlohmann
nlohmann marked this pull request as ready for review September 6, 2026 17:43
@nlohmann nlohmann added the aspect: binary formats BSON, CBOR, MessagePack, UBJSON label Sep 6, 2026
@nlohmann
nlohmann force-pushed the ubjson-valueless-count-cap branch from d89acce to 5bf814b Compare September 7, 2026 02:56
@nlohmann
nlohmann force-pushed the ubjson-valueless-count-cap branch from 5bf814b to ee69490 Compare September 7, 2026 05:59
@nlohmann nlohmann added the review needed It would be great if someone could review the proposed changes. label Sep 7, 2026
@nlohmann
nlohmann force-pushed the ubjson-valueless-count-cap branch from ee69490 to 42420d4 Compare September 8, 2026 11:13
An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576, and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Put (1 << 20) or (0x100000) next to the decimal number so that it's obvious it's a "nice round number" in binary and hex domains? Also applies in other .md file.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea — added (1 << 20) next to the decimal number in both spots (here and the matching text in exceptions.md), matching how max_valueless_container_size is actually defined in binary_reader.hpp. Pushed.

nlohmann added a commit that referenced this pull request Sep 8, 2026
Addresses review feedback from @gregmarr on PR #5504: spell out the
binary/hex form next to the decimal count so it reads as the round
power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp
defines max_valueless_container_size. Applied in both docs/exceptions.md
and ubjson.md, as requested.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann nlohmann added 🚀 ready to merge Ready to merge - just waiting for CI to complete. and removed review needed It would be great if someone could review the proposed changes. labels Sep 8, 2026
@nlohmann nlohmann added this to the Release 3.13.0 milestone Sep 9, 2026
@nlohmann
nlohmann force-pushed the ubjson-valueless-count-cap branch from 5de50cd to 9edfb53 Compare September 9, 2026 08:21
@nlohmann
nlohmann force-pushed the ubjson-valueless-count-cap branch from 9edfb53 to 7c44c0c Compare September 10, 2026 15:15
@nlohmann nlohmann added review needed It would be great if someone could review the proposed changes. and removed 🚀 ready to merge Ready to merge - just waiting for CI to complete. labels Sep 10, 2026
An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its
type marker alone, so an optimized UBJSON array of one of those has no
payload: reading an element consumes no input at all. Its declared count is
therefore the only thing that decides how much is allocated, and nothing
bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing
two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and
OSS-Fuzz has an out-of-memory and a timeout report for the same shape.

Every other type costs at least one byte per element, so the end of the input
bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not
affected either: each element is preceded by its key, which costs bytes. And
BJData already refuses these markers as an optimized type, so this is a plain
UBJSON matter.

Reject a count above 1,048,576 elements for those three types with
out_of_range.408, the code this reader already uses for a declared size it
will not honour. The check runs before the SAX start event, so no container
is opened and then abandoned.

Rejecting on the read side alone would break the guarantee that anything
to_ubjson() writes can be read back, and would trip the round-trip assertion
in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized
encoding, one byte per element, for arrays of these types above the same
limit. Its decision depends only on the array's size, which is identical for
a value and for anything parsed back from it, so the round trip is stable.

No existing test changes: the largest such count in the test suite is 65,793.
The excessive-size test that already used this shape still passes, now
rejected a little earlier than by the max_size() check it used to reach.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Addresses review feedback from @gregmarr on PR #5504: spell out the
binary/hex form next to the decimal count so it reads as the round
power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp
defines max_valueless_container_size. Applied in both docs/exceptions.md
and ubjson.md, as requested.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the ubjson-valueless-count-cap branch from 7c44c0c to f50144c Compare September 11, 2026 06:29
@nlohmann
nlohmann merged commit 44f8ec3 into develop Sep 11, 2026
4 of 155 checks passed
@nlohmann
nlohmann deleted the ubjson-valueless-count-cap branch September 11, 2026 06:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aspect: binary formats BSON, CBOR, MessagePack, UBJSON documentation L review needed It would be great if someone could review the proposed changes. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

An Ubjson Parsing Problem Can Easily Cause DDoS Attack.

2 participants