Skip to content

NDJSON stream decoder leaks source on parse failure and ignores reader backpressure #4649

Closed

Description

@nsxdavid

Failure

In Eve 0.73.0 (Node 26.8.2, Linux, Local World), parseNdjsonStream in dist/src/execution/ndjson-stream.js releases its source reader when JSON parsing or the event normalizer throws, but never cancels the source. Cancelling the errored output afterwards rejects without invoking the underlying cancel hook. A Local World source therefore retains its chunk:<stream> and close:<stream> listeners and 100 ms directory poller.

An isolated reproduction using the bundled createWorld and parseNdjsonStream left 12 listeners after 12 failed decodes. Twelve successful reads followed by cancellation left zero. This is an error-path cleanup defect; it does not require a model or HTTP server.

The same parser also reads eagerly in start() for the lifetime of the stream. A downstream reader that stops consuming does not apply backpressure to decoding: the parser keeps parsing and enqueuing events. On a long session that can retain the entire decoded replay in addition to the underlying stream buffers.

Minimal reproduction

Run this from a project with eve@0.73.0 installed:

import { createRequire } from 'node:module';
import { dirname, join } from 'node:path';
import { pathToFileURL } from 'node:url';
const require = createRequire(import.meta.url);
const root = dirname(require.resolve('eve/package.json'));
const { parseNdjsonStream } = await import(pathToFileURL(join(root, 'dist/src/execution/ndjson-stream.js')));
let cancelled = 0;
for (let i = 0; i < 12; i++) {
  const source = new ReadableStream({
    start(c) { c.enqueue(new TextEncoder().encode('invalid json\n')); },
    cancel() { cancelled++; }
  });
  const reader = parseNdjsonStream(() => source).getReader();
  await reader.read().catch(() => {});
  await reader.cancel().catch(() => {});
  reader.releaseLock();
}
console.log({ cancelled }); // actual 0; expected 12

The event normalizer throwing with syntactically valid JSON produces the same result.

Expected

Cancel and release the source on decode/normalization failure, preserving the original error. Decode from pull() so the output consumer controls read-ahead. Keep ordinary cancellation and final-line/UTF-8 handling intact.

Related: #199 (closed; its cancellation hook is present in this release), #2880 (similar listener warnings, different demonstrated trigger). The existing fixes do not cover this reproduction. This report does not claim that decoding failure explains every observed listener warning or all memory growth.

Activity

  1. roboeve commented on Oct 11, 2026

    @roboeve
    Contributor

    This no longer reproduces on origin/main (b6b353f1bd0512d0adde137b31406656a123e981). Both behaviors changed in #4547 (13059ba5b), first released in eve@0.75.1. I ran the checks on Node v26.8.2.

    • On a decode or normalizer failure, parseNdjsonStream now errors the output and cancels the source with the original error. The 12-iteration parse-failure check produced 12 source cancellations on main and 0 on eve@0.73.0; the throwing-normalizer variant behaved the same.
    • With a pull-based 100-line source, decoding made 3 source pulls after one read on main versus 100 on eve@0.73.0.

    Both behaviors are covered by tests in packages/eve/src/execution/ndjson-stream.test.ts. If you still see listener growth on the latest release (eve@0.76.3), please share steps that reproduce it there.

  2. benpankow commented on Oct 14, 2026

    @benpankow
    Collaborator

    Thanks @nsxdavid for reporting. Believe this is fixed by #4547 in eve 0.75.1. If you're still running into this, please feel free to reopen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions