Parser callback is still called inside a discarded container, and that container's keys are kept in memory #5643

Description

When a parser callback returns false for an object_start or array_start event, the parser still calls it for the content of that container. It receives every key event, the start events of nested containers, and the value events of elements of nested containers. The documentation says the opposite:

Discarding it at the start event also means the callback is called neither for the content of the value nor for its matching end event.

(parser_callback_t.md)

The same root cause also has a memory effect. Since #5457 it keeps a copy of every key inside a discarded container until the parse ends. Filtering out a large subtree is the main reason to use a callback, and the parser now keeps a large part of that subtree anyway. For a discarded object with 200,000 members, peak heap use during the parse went from 49 KB to 13.7 MB. This applies to both ways of discarding an object: rejecting its object_start event or rejecting its key.

Root cause (json_sax_dom_callback_parser in json_sax.hpp):

• start_object(), start_array() and key() call the callback unconditionally (L586, L707, L625). Only handle_value() checks whether the enclosing container is discarded (L1012). As a result, the value events of the discarded container's direct members are suppressed, but everything below them is not: nested containers get a fresh keep_stack entry of true.
• key() always pushes onto key_keep_stack and key_stack (L626-L629). The matching pop in handle_value() is skipped by its early returns for values that are not stored (L1012-L1015, L1056-L1059). Each key inside a container that is not stored therefore stays on both stacks until the parse ends.
  • Before Do not search a container for the value the callback rejected #5457, this leaked one bit per key (std::vector<bool> key_keep_stack).
  • Do not search a container for the value the callback rejected #5457 (1dc1d09) added std::vector<string_t> key_stack (L1109), so each key is now also kept as a full string copy.

Since when:

• The extra events reproduce on every release from v3.2.0 on, when the callback parser was reimplemented on top of the SAX interface (Add a SAX parser #971).
• v3.1.2 did not call the callback for the content of a discarded container.
• The memory growth started with 1dc1d09 (Do not search a container for the value the callback rejected #5457). The program below reports 49,384 bytes with the headers of its parent commit and 13,663,586 bytes on develop.

Possible fix (tested). Do not call the callback and do not touch the key stacks inside a container that is not stored:

--- a/include/nlohmann/detail/input/json_sax.hpp
+++ b/include/nlohmann/detail/input/json_sax.hpp
@@ -582,8 +582,8 @@
 
     bool start_object(std::size_t len)
     {
-        // check callback for object start
-        const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::object_start, discarded);
+        // check callback for object start; not called inside a discarded container
+        const bool keep = keep_stack.back() && callback(static_cast<int>(ref_stack.size()), parse_event_t::object_start, discarded);
         keep_stack.push_back(keep);
 
         // the key this object will be stored under, read before handle_value()
@@ -619,6 +619,18 @@
 
     bool key(string_t& val)
     {
+        if (!keep_stack.back() || !ref_stack.back())
+        {
+            // the object is not stored: the value of this key is dropped in
+            // handle_value() without touching the key stacks
+            if (keep_stack.back())
+            {
+                BasicJsonType k = BasicJsonType(val);
+                static_cast<void>(callback(static_cast<int>(ref_stack.size()), parse_event_t::key, k));
+            }
+            return true;
+        }
+
         BasicJsonType k = BasicJsonType(val);
 
         // check callback for the key
@@ -704,7 +716,7 @@
 
     bool start_array(std::size_t len)
     {
-        const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::array_start, discarded);
+        const bool keep = keep_stack.back() && callback(static_cast<int>(ref_stack.size()), parse_event_t::array_start, discarded);
         keep_stack.push_back(keep);
 
         // see start_object()

The key callback is still called for a member of an object whose own key was rejected, because the documentation promises that: "The callback is still called for the associated value, but its return value has no further effect". Only the stacks are left alone.

Tests of the sketch:

• The program below prints the documented event sequence, and peak heap drops to 304 bytes (object_start) and 280 bytes (key).
• 400,000 random documents with duplicate keys and random stateless callbacks, for both json and ordered_json, give byte-identical results before and after under ASan/UBSan.
• These unit tests pass: unit-class_parser, unit-regression2, unit-regression3, unit-comparison, unit-diagnostic-positions, unit-locale-cpp.

A regression test would need to check the callback's event sequence (for example, that no key event occurs inside a discarded object) and that key_stack does not grow. The second part can only be checked indirectly, for example with a counting allocator.

Reproduction steps

1. Save the program below as callback.cpp.
2. clang++ -std=c++11 -O2 -I include callback.cpp -o callback && ./callback
3. Part (1): the callback discards the value of "skip" at its object_start event, yet it still receives events from inside that value.
4. Part (2): discarding a 200,000-member object keeps 13.7 MB alive during the parse.

Expected vs. actual results

• Expected (part 1): after the discarded depth 1 object_start, the next event is depth 1 key "keep".
• Actual (part 1): the callback also receives key "k1", key "k2", array_start, value 2, object_start, key "k3" and value 3 from inside the discarded object. The parse result itself is correct.
• Expected (part 2): the memory needed to skip the object does not grow with its size (49 KB before Do not search a container for the value the callback rejected #5457).
• Actual (part 2): 13,663,586 bytes peak for both ways of discarding, because every key is kept as a string until the end of the parse.

Minimal code example

#include <nlohmann/json.hpp>
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <new>
#include <string>

using json = nlohmann::json;

// track the peak number of live heap bytes
static std::size_t live = 0, peak = 0;
void* operator new(std::size_t n)
{
    live += n;
    peak = live > peak ? live : peak;
    auto* p = static_cast<std::size_t*>(std::malloc(n + 16));
    *p = n;
    return reinterpret_cast<char*>(p) + 16;
}
void operator delete(void* p) noexcept
{
    if (p != nullptr)
    {
        auto* q = reinterpret_cast<std::size_t*>(static_cast<char*>(p) - 16);
        live -= *q;
        std::free(q);
    }
}
void operator delete(void* p, std::size_t) noexcept { operator delete(p); }

int main()
{
    const char* names[] = {"object_start", "object_end", "array_start", "array_end", "key", "value"};

    // (1) discard the value of "skip" at its object_start event
    bool first = true;
    json j = json::parse(R"({"skip": {"k1": 1, "k2": [2, {"k3": 3}]}, "keep": 1})",
                         [&](int depth, json::parse_event_t event, json& parsed)
    {
        std::cout << "depth " << depth << "  " << names[static_cast<int>(event)] << "  " << parsed.dump() << '\n';
        if (depth == 1 && event == json::parse_event_t::object_start && first)
        {
            first = false;
            return false;
        }
        return true;
    });
    std::cout << "result: " << j.dump() << "\n\n";

    // (2) discard an object with 200,000 members, once at object_start and once at its key
    std::string s = R"({"skip":{)";
    for (int i = 0; i < 200000; ++i)
    {
        s += (i != 0 ? "," : "");
        s += "\"key_with_some_length_" + std::to_string(i) + "\":" + std::to_string(i);
    }
    s += R"(},"keep":1})";

    std::size_t base = live;
    peak = live;
    json r1 = json::parse(s, [](int depth, json::parse_event_t event, json&)
    {
        return !(depth == 1 && event == json::parse_event_t::object_start);
    });
    std::printf("discarded at object_start: %s, peak heap during parse: %zu bytes\n", r1.dump().c_str(), peak - base);

    base = live;
    peak = live;
    json r2 = json::parse(s, [](int depth, json::parse_event_t event, json& parsed)
    {
        return !(depth == 1 && event == json::parse_event_t::key && parsed == "skip");
    });
    std::printf("discarded at key:          %s, peak heap during parse: %zu bytes\n", r2.dump().c_str(), peak - base);
}

Error messages

depth 0  object_start  <discarded>
depth 1  key  "skip"
depth 1  object_start  <discarded>
depth 2  key  "k1"
depth 2  key  "k2"
depth 2  array_start  <discarded>
depth 3  value  2
depth 3  object_start  <discarded>
depth 4  key  "k3"
depth 4  value  3
depth 1  key  "keep"
depth 1  value  1
depth 0  object_end  {"keep":1}
result: {"keep":1}

discarded at object_start: {"keep":1}, peak heap during parse: 13663586 bytes
discarded at key:          {"keep":1}, peak heap during parse: 13663546 bytes

With the headers of 1dc1d09^ (before #5457), part (2) prints:

discarded at object_start: {"keep":1}, peak heap during parse: 49384 bytes
discarded at key:          {"keep":1}, peak heap during parse: 49312 bytes

Compiler and operating system

Apple clang 21.0.0 (clang-2100.3.34.2), macOS 27.0 (arm64)

Library version

develop @ 633de8e. The extra events also occur on v3.2.0 and v3.12.0, but not on v3.1.2.

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

This issue was written by Claude Code on behalf of @nlohmann.