CheerioCrawler: extracted strings pin the entire page HTML in memory (V8 sliced strings) #3809

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/cheerio (CheerioCrawler)

Issue description

CheerioCrawler parses responses with htmlparser2, which builds DOM text and
attribute values as zero-copy substrings of the raw HTML source. In V8 these are
SlicedStrings: a tiny extracted string internally keeps a reference to the
entire page HTML it was sliced from.

That is harmless while the strings are transient. But the canonical Crawlee usage
pattern — extract a few small fields per page in the requestHandler and collect
them somewhere for the duration of the crawl (an array, a Map, …) — retains those
slices. Every retained string then pins its full source page in the heap:

• page is 3 MB, you keep a 40-byte title → ~3 MB stay live
• one retained string per page over a 10k-page crawl → gigabytes of
  live-but-invisible HTML → JavaScript heap out of memory

The failure is invisible in code review and at runtime until the OOM, and only
diagnosable via heap-snapshot forensics (retainer chain: object field →
sliced string → full <!DOCTYPE html>… body).

Real-world numbers: on a production crawl (~2,500 category pages of 2.3–4 MB
each) the run died with OOM. Heap snapshots showed ~1 GB of live heap consisting of
complete page HTML strings, held by 15,266 sliced strings in two plain string
fields of collected result objects (title: 7,176×, imgUrl: 8,090×); a later
snapshot showed the same counts at ~3× size, i.e. linear retention per page.
After copying extracted fields via Buffer.from(value, 'utf8').toString('utf8')
the same crawl stays flat at ~150–390 MB instead of climbing past 2.4 GB.

Steps to reproduce with the code sample below:

1. run the crawler over many URLs with large pages
2. observe heap usage growing by roughly one full page size per retained string
3. replace the push with the detach(...) variant → heap stays flat

Note this is not limited to Cheerio APIs: regex capture groups over a kept
HTML/XML string (e.g. extracting URLs from a downloaded sitemap before enqueueing)
slice the same way, so a request queue can pin multi-MB sitemap documents for the
whole run, too.

Why report this against Crawlee rather than cheerio/htmlparser2: this was
reported to cheerio in 2013 (cheeriojs/cheerio#263) and closed by the htmlparser2
maintainer as a V8 issue ("if you consume multiple documents, add the fix
yourself"; upstream: https://issues.chromium.org/issues/41480525, unfixed since). Defensible for a parsing
library — most cheerio users consume one document. Crawlee's core use case is
consuming thousands of large documents and retaining small extracts from each,
so the sharp edge passes silently through two abstraction layers and surfaces as
an undebuggable OOM in user code.

Suggested resolution: detach by default. The exposed $ should copy
text/attribute values on extraction (e.g. Buffer.from(v, 'utf8').toString('utf8')).
The cost is a small allocation per extracted string — nanoseconds, and completely
dwarfed by the network I/O and parsing Crawlee already does per page. In exchange,
the dominant usage pattern (retain small extracts from many pages) stops OOMing
silently.

For users who genuinely need zero-copy extraction (huge extracts, hot loops), an
opt-out (detachExtractedStrings: false) would preserve today's behavior.

If changing the default is off the table, fallbacks in order of usefulness:

1. opt-in option (detachExtractedStrings: true),
2. an exported, documented detachString() helper in @crawlee/utils, so the fix
   has a canonical, discoverable name,
3. at minimum a prominent docs warning on CheerioCrawler and in the scraping
   tutorials.

The current silent zero-copy default is a micro-optimization that costs Crawlee's
main audience gigabytes of heap and an undebuggable crash.

Code sample

import { CheerioCrawler } from 'crawlee';

const titles: string[] = [];

const crawler = new CheerioCrawler({
    requestHandler({ $ }) {
        // Looks like it stores ~40 bytes. Actually pins the whole page HTML,
        // because htmlparser2 node data are V8 sliced strings of the source.
        titles.push($('title').text());

        // Workaround — copying detaches the string from the source buffer:
        // const detach = (s: string) => Buffer.from(s, 'utf8').toString('utf8');
        // titles.push(detach($('title').text()));
    },
});

await crawler.run([/* many URLs pointing at large pages */]);
// Heap grows linearly with page count (~pageSize per retained string).
// Verify: take a heap snapshot and inspect retainers of the string fields.

Package version

crawlee@3.17.0

Node.js version

v26.4.0

Operating system

Linux x64 (Arch)

Other context

Related: cheeriojs/cheerio#263 (same root cause, closed as V8-specific in 2013),
V8 upstream ticket https://issues.chromium.org/issues/41480525 (
open since 2013). We have added our production heap-forensics numbers as a comment on that
V8 issue, but engine-side options (GC-time flattening, a runtime --no-string-slices flag,
an official flatten API) have been stalled there for over a decade — which is why a
mitigation at the Crawlee layer seems like the pragmatic place.

The SlicedString optimization applies to
substrings ≥ 13 characters, so virtually every extracted text/attribute is affected.