Skip to content

Flatten deeply nested values without recursing per nesting level - #5792

Merged
nlohmann merged 2 commits into
developfrom
fix-5393-flatten-recursion
Oct 9, 2026
Merged

nlohmann merged 2 commits into
developfrom
fix-5393-flatten-recursion

Conversation

@nlohmann

@nlohmann nlohmann commented Oct 9, 2026

Copy link
Copy Markdown
Owner

Fixes the remaining part of #5393.

#5393 reported three functions that recurse once per nesting level: diff(), merge_patch() and flatten(). #5547 and #5548 fixed the first two, and the issue was closed when #5548 merged. json_pointer::flatten() is still recursive on develop, so

json::parse(std::string(100000, '[') + "0" + std::string(100000, ']')).flatten();

still crashes with SIGSEGV (exit 139, macOS/arm64, -O2).

Change

json_pointer::flatten() now walks the value with an explicit stack of frames. Like diff_iteratively, it keeps the path in one buffer that grows and shrinks with the stack. The signature and the output stay the same, including the entry order of an ordered_json result.

Unlike diff and merge_patch, this version doesn't recurse first and switch to the explicit stack at recursion_depth_limit(). It has one code path. The old version built a new path string for every child, which costs time quadratic in the depth. Sharing one buffer avoids that, so the iterative version is no slower even on shallow values.

Benchmark

Best-of-7 time per call to flatten(), -O2, Apple clang 21, arm64. Old and new produced identical dump() output in every case.

input type old new new/old
wide, ~1.2 MB, 80,000 leaves json 15.20 ms 14.55 ms 0.96
wide ordered_json 2511 ms 2514 ms 1.00
nested, depth 7, 59,296 leaves json 15.31 ms 13.13 ms 0.86
nested ordered_json 3406 ms 3416 ms 1.00
10,000 nested arrays json 4.80 ms 0.183 ms 0.04
10,000 nested objects json 4.76 ms 0.177 ms 0.04

Tests

New TEST_CASE("flatten of structured values") in unit-json_pointer.cpp:

  • 100,000-level arrays and objects flatten to a single entry with the full path
  • interleaved objects and arrays, with empty containers and keys that need escaping (~, /), compared with a result written out by hand
  • entry order of an ordered_json result

Locally: test-json_pointer_cpp11, test-json_pointer_cpp20 and test-json_patch_cpp11 pass, and the build shows no warnings.

The round-trip through unflatten() is only tested at 500 levels. At 100,000 levels unflatten() used about 9.6 GB of memory and was still running after 8 minutes, apparently because it copies pointer prefixes in an amount that grows quadratically with the depth. That's a separate problem and isn't touched here.

Public API

No breaking changes: flatten() keeps its signature and its output, and only the internals of json_pointer::flatten change.

🤖 Generated with Claude Code

}
else
{
stack.push_back({&v, 0, object_const_iterator(), path.size()});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

emplace_back?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e82b29b: the frame got a constructor (C++11 can't emplace_back an aggregate), and both call sites use emplace_back now.

Written by Claude Code.

}
else
{
stack.push_back({&v, 0, v.m_data.m_value.object->begin(), path.size()});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

emplace_back?

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e82b29b: the frame got a constructor (C++11 can't emplace_back an aggregate), and both call sites use emplace_back now.

Written by Claude Code.

@nlohmann nlohmann added this to the Release 3.13.0 milestone Oct 9, 2026
@nlohmann nlohmann added the 🚀 ready to merge Ready to merge - just waiting for CI to complete. label Oct 9, 2026
json_pointer::flatten() called itself once per nesting level, so
flatten() on a value nested deeply enough exhausted the call stack.
#5547 and #5548 fixed merge_patch() and diff() from #5393, but flatten()
was left out.

flatten() now walks the value with an explicit stack and keeps the path
in one buffer that grows and shrinks with it. It has a single code path
and no depth limit: the old version built a new path string per child,
so the iterative one is no slower on shallow values and much faster on
deep ones. The output, including the order of an ordered_json result,
is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Give the frame a constructor so both call sites can use emplace_back, as
suggested in the review; index starts at 0 for every frame.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@nlohmann
nlohmann force-pushed the fix-5393-flatten-recursion branch from e82b29b to f4090fd Compare October 9, 2026 15:06
@nlohmann
nlohmann merged commit 44e8597 into develop Oct 9, 2026
2 of 64 checks passed
@nlohmann
nlohmann deleted the fix-5393-flatten-recursion branch October 9, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L 🚀 ready to merge Ready to merge - just waiting for CI to complete. tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants