Skip to content

Encode string chunks as UTF-8 when pumping TypeScript streams - #8053

Merged
jasnell merged 1 commit into
mainfrom
jasnell/ts-streams-string-body-chunks
Oct 28, 2026
Merged

jasnell merged 1 commit into
mainfrom
jasnell/ts-streams-string-body-chunks

Conversation

@jasnell

@jasnell jasnell commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Under typescript_implemented_streams, the C++ bridge's pump for a queued (JS-sourced) stream rejected string chunks with "This ReadableStream did not return bytes." The C++ implementation's pump has always written them as their UTF-8 encoding (valueToBytes() in queue.c++), and code such as server-sent-event handlers that enqueue strings without a TextEncoder relies on it.

Every sink that sends a body on goes through this pump: a fetch() request body, iterable request bodies included, a Response the fetch handler returns, cache.put(), KV put(), HTMLRewriter's input and streamed replacement content, EventSource.from() and JS RPC. As the request body pump runs apart from the response, its failure did not reject fetch(): the request went out with its body cut short at the batch holding the string, and only the source's cancel() saw the error. Elsewhere the client got an empty reply, or an RPC receiver a premature disconnect.

The pump now writes a string chunk with toUSVString(), exactly as valueToBytes() does: lone surrogates become U+FFFD and embedded NULs are kept. Other non-byte chunks still reject. Reading a body in JavaScript (text() and the like) still rejects strings in both implementations, as the spec has it.

Tests pin the parity in each sink's suite: the readable suite (request, iterable and returned response bodies), htmlrewriter, cache, the cross-implementation RPC test, and EventSource, which gains a TypeScript-streams config. The readable suite's main cells pin fetch_iterable_type_support for the iterable body. The bridge unit test that pinned the rejection with a string now uses a number.

typescript_implemented_streams is still experimental, with no enable date, so the change needs no compatibility flag.

Fixes: #8041

@jasnell
jasnell requested review from a team and guybedford October 28, 2026 00:13
@jasnell
jasnell requested review from a team as code owners October 28, 2026 00:13
@clanky-the-clanker

clanky-the-clanker Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

💬 Approved with comments — The fix is correct and matches valueToBytes(), and the approver's buffer-adoption nit has been applied. The only remaining comment is an optional clarification to the developer docs.

Findings: 1 suggestion

The decision is advisory: this comment never approves, blocks or requests changes on the pull request.


📝 Documentation: 💬 1 suggestion

One optional clarification about the JS-to-kj bridge's byte-only wording

Recommendation: 💬 Optional Nits

  • [SUGGESTION] The JS-to-kj bridge docs say only bytes are accepted. Consider noting that both pumps write string chunks as UTF-8, while other non-byte chunks still reject (also src/workerd/api/streams/README.md:259-261). (docs/streams.md:395)
View full review

docs/streams.md:394-396 and src/workerd/api/streams/README.md:259-261 still say that data crossing the JS-to-kj bridge is restricted to bytes. Both pumps actually write string chunks as UTF-8, while other non-byte chunks still reject. It would help to mention that exception.


✅ All other checks passed

🔒 Security: ✅ LGTM

Lone surrogates are replaced with U+FFFD, so the pump always writes valid UTF-8. The byte array keeps its backing string alive.


📖 AGENTS.md: ✅ LGTM

The AGENTS.md additions in the test suites correctly describe the new string-chunk tests.


📋 Code Quality: ✅ LGTM

The JsString branch now takes ownership of the toUSVString() buffer through webstreams::stringToBytes() and leaves out the NUL terminator. Its behaviour matches the C++ implementation's valueToBytes().


⚡ Performance: ✅ LGTM

The string chunk path no longer makes an extra copy. The encoded buffer is taken over as it is.


Reviewed commit: 40beacd4

@ask-bonk

ask-bonk Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Since last review: 0 resolved, 0 still open, 0 new.
LGTM!


Reviewed commit: 40beacd4 · github run

@jasnell
jasnell force-pushed the jasnell/ts-streams-string-body-chunks branch from 02f8c22 to e38d9d2 Compare October 28, 2026 12:34
@jasnell
jasnell enabled auto-merge October 28, 2026 20:40

@ry ry left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Matches valueToBytes() exactly.

Follow-up question: other non-byte chunks (e.g. a number) still fail the request-body pump without rejecting fetch(), leaving the body cut short. Does C++ behave the same? If not, worth an issue.

Comment thread src/workerd/api/js-readable-stream.c++ Outdated
Under typescript_implemented_streams, the C++ bridge's pump for a
queued (JS-sourced) stream rejected string chunks with "This
ReadableStream did not return bytes." The C++ implementation's pump
has always written them as their UTF-8 encoding (valueToBytes() in
queue.c++), and code such as server-sent-event handlers that
enqueue strings without a TextEncoder relies on it.

Every sink that sends a body on goes through this pump: a fetch()
request body, iterable request bodies included, a Response the
fetch handler returns, cache.put(), KV put(), HTMLRewriter's input
and streamed replacement content, EventSource.from() and JS RPC. As
the request body pump runs apart from the response, its failure did
not reject fetch(): the request went out with its body cut short at
the batch holding the string, and only the source's cancel() saw
the error. Elsewhere the client got an empty reply, or an RPC
receiver a premature disconnect.

The pump now writes a string chunk with toUSVString(), exactly as
valueToBytes() does: lone surrogates become U+FFFD and embedded NULs
are kept. Other non-byte chunks still reject. Reading a body in
JavaScript (text() and the like) still rejects strings in both
implementations, as the spec has it.

Tests pin the parity in each sink's suite: the readable suite
(request, iterable and returned response bodies), htmlrewriter,
cache, the cross-implementation RPC test, and EventSource, which
gains a TypeScript-streams config. The readable suite's main cells
pin fetch_iterable_type_support for the iterable body. The bridge
unit test that pinned the rejection with a string now uses a number.

typescript_implemented_streams is still experimental, with no enable
date, so the change needs no compatibility flag.

Fixes: #8041
@jasnell
jasnell force-pushed the jasnell/ts-streams-string-body-chunks branch from e38d9d2 to 40beacd Compare October 28, 2026 21:59
@jasnell
jasnell merged commit 1c1b9f2 into main Oct 28, 2026
29 checks passed
@jasnell
jasnell deleted the jasnell/ts-streams-string-body-chunks branch October 28, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Bug Report — Runtime APIs: a string chunk in a fetch() request body silently sends an empty body with typescript_implemented_streams

2 participants