Repository navigation
Hangs on await response.text() using AbortController on unidci 6.21.1 #4089
Description
Bug Description
I am upgrading my project from undici 5.x to 6.21.1. After the upgrade, I found that under high concurrency, when using undici.fetch to download files(The file sizes range from 10KB to 10MB, with concurrency typically between 2 and 10 requests. The total number of files is over 200, involving 4 different hosts.), there is a chance that it will block on await response.text(). I can’t consistently reproduce the issue, but I have added a lot of logs and gathered some information bellow. I would like to ask if, logically, there are cases where [await reponse.text()] does not return in some race condition?
Reproducible By
Our project is a Node.js web service that downloads many files during startup. This issue occurs when downloading the files at startup. The project code is quite complex, so I wrote a minimal demo for the downloading part. The only parts related to Undici and the APIs used are as follows. (We’re not sure if any of the third-party packages we depend on use Undici, as Node.js has Undici built-in, but the issue happens within the logic below.)
By the way, it cannot be reliably reproduced, and every time the issue occurs on different file, not the same file.
const downloadSingleFile = async (urlToDownload)=>{
const options = {
url: urlToDownload,
method: 'GET',
headers: {}
}
const controller = new AbortController();
const timeout = 500
const reqTimer = setTimeout(() => {
logger.info(`fetch abort by timeout ${timeout}`);
controller.abort();
}, timeout);
options.signal = controller.signal;
let result;
try{
const response = await undici.fetch(options.url, options)
const { status, statusText, headers } = response
if (status >= 200 && status < 300) {
const bodyTxt = await response.text();
result = { statusCode: status, headers, body: bodyTxt };
} else {
result = { statusCode: status, headers, body: statusText };
}
}catch(e){
result = {statusCode: -100, body: e.message}
}
clearTimeout(reqTimer);
return result;
}
const demoEntry = async()=>{
const res = [
'https://xxxxxx/xxx.json'
//...
//about 200-300 urls
]
for (let index = 0; index < res.length; index++) {
const element = res[index];
//...
//concurrency is 1-10, not all urls
//file size is between 10KB-10MB
downloadSingleFile(element,index)
}
}
demoEntry()Expected Behavior
All promise return normally.
Logs & Screenshots
As the code shown above, I've add a lot of logs on response.text(), as it's hanged here.
The log i added is readAllBytes in /node_modules/undici/lib/web/fetch/util.js
//in my code
const controller = new AbortController();
const reqTimer = setTimeout(() => {
logger.info(`fetch abort by timeout ${timeout} tempIndex=${tempIndex}`);
controller.abort();
}, timeout);
//in /node_modules/undici/lib/web/fetch/util.js
const sleep = (ms)=>{
return new Promise((resolve) => setTimeout(() => resolve({value: false}), ms));
}
async function readAllBytes (reader) {
const bytes = []
let byteLength = 0
const logger = global.__asyncLogger;
const tempIndexIdx = reader["__tempIndexIdx"] //this is injected from the download index
logger.info(`fetch enter readAllBytes with idx ${tempIndexIdx} step0 ${!!reader}`)
let times = 0
let timesInfo = []
while (true) {
times++;
// logger.info(`fetch enter readAllBytes with dix ${tempIndexIdx} step0 times=${times}`)
const { done, value: chunk } = await Promise.race([reader.read(), sleep(15000)])
timesInfo.push({
time: new Date().toISOString(),
times,
length: byteLength
})
if(chunk === false){
try{
logger.info(`fetch enter controller readAllBytes with idx ${tempIndexIdx} step3 timeout ${!!reader} times=${times} length=${byteLength} ${JSON.stringify(timesInfo)}`)
logger.info(`fetch enter readAllBytes with idx ${tempIndexIdx} step3 info 1`, {reader})
const infoOfReader = {}
const symbolKey = Object.getOwnPropertySymbols(reader)
const kStateSymbol = symbolKey.find(sym => sym.toString().includes('kType'))
const kStreamSymbol = symbolKey.find(sym => sym.toString().includes('kState'))
const _innerState = reader[kStateSymbol]
const _innerReader = reader[kStreamSymbol]
infoOfReader.state = _innerState
infoOfReader.state = _innerState
const waitingLength = _innerReader.readRequests?.length || -1
const waitingReq = _innerReader.readRequests[0]
const waitingReqPromimse = waitingReq?.promise
let waitingReqStatus = 'none';
if(waitingReqPromimse){
waitingReqStatus = await Promise.race([
waitingReqPromimse.then(() => "fulfilled").catch(() => "rejected"),
new Promise(resolve => setTimeout(() => resolve("pending"), 0))
]);
}
infoOfReader.waitingLength = waitingLength
const _stream = _innerReader.stream
const streamKeys = Object.getOwnPropertySymbols(_stream)
const kStreamStatus = streamKeys.find(sym => sym.toString().includes('kState'))
const stateOfStream = _stream[kStreamStatus]
infoOfReader.stateOfStream = stateOfStream
const controller = stateOfStream.controller
const stateOfController = controller[kStreamStatus]
infoOfReader.controllerState = stateOfController
infoOfReader.waitingReqStatus = waitingReqStatus
logger.info(`fetch enter readAllBytes with idx ${tempIndexIdx} status=${waitingReqStatus} step3 info ${JSON.stringify(infoOfReader)}`)
}catch(e){
logger.info(`fetch enter readAllBytes with idx ${tempIndexIdx} step3 timeout error ${e.message}`, e)
}
//when hangs happened, it will await here, so I can catch the hang.
await sleep(600000)
}
if (done) {
logger.info(`fetch enter readAllBytes with idx ${tempIndexIdx} step2 done`)
// 1. Call successSteps with bytes.
return Buffer.concat(bytes, byteLength)
}
// 1. If chunk is not a Uint8Array object, call failureSteps
// with a TypeError and abort these steps.
if (!isUint8Array(chunk)) {
throw new TypeError('Received non-Uint8Array chunk')
}
// 2. Append the bytes represented by chunk to bytes.
bytes.push(chunk)
byteLength += chunk.length
// 3. Read-loop given reader, bytes, successSteps, and failureSteps.
}
}Here is useful log I've got when hangs.
2025-03-12 12:51:15,604 downloadIndex=340 startFetch
2025-03-12 12:51:15,613 downloadIndex=340 get fetch response, start await reponse.text()
2025-03-12 12:51:15,613 fetch enter readAllBytes with idx 340 step0 true
2025-03-12 12:51:16,127 fetch abort by timeout 500 tempIndex=340
2025-03-12 12:51:30,913 fetch enter controller readAllBytes with idx 340 step3 timeout true times=452 length=2899968 [...(ignores),{"time":"2025-03-12T12:51:15.912Z","times":451,"length":2890356},{"time":"2025-03-12T12:51:30.913Z","times":452,"length":2899968}]
2025-03-12 12:51:30,915 fetch enter readAllBytes with idx 340 step3 info 1 data: {
reader: ReadableStreamDefaultReader {
stream: ReadableStream { locked: true, state: 'readable', supportsBYOB: true },
readRequests: 1,
close: Promise {
<pending>,
[Symbol(async_id_symbol)]: 1145919,
[Symbol(trigger_async_id_symbol)]: 1144674,
[Symbol(kResourceStore)]: [Object]
}
}
}
}
2025-03-12 12:51:30,916 INFO 128 [agent][node@v20.11.1] [ssr:1.202.0] fetch enter readAllBytes with idx 340 status=pending step3 info {"state":"ReadableStreamDefaultReader","waitingLength":1,"stateOfStream":{"disturbed":true,"reader":{"__tempIndexIdx":340},"state":"readable","transfer":{},"controller":{}},"controllerState":{"byobRequest":null,"closeRequested":false,"pullAgain":false,"pulling":false,"started":true,"stream":{},"queue":[],"queueTotalSize":0,"highWaterMark":0,"pendingPullIntos":[]},"waitingReqStatus":"pending"}
As we can see the promise in the reader is still pennding, but the chunk is fully pulled and the abortController.sigal has emitted.
Environment
Debian GNU/Linux 12
tested on Node v20.11.1 and v22.12.0
Activity
ferrisxie commentedon Mar 12, 2025
From my code it hangs on
await reponse.text()Inside the undici code, it hangs on
await reader.read()in '/lib/web/fetch/util.js'From the log and code above, we can get the timeline:
2025-03-12 12:51:15,604: start fetch
2025-03-12 12:51:15,613: get fetch reponse, start call await response.text()
2025-03-12 12:51:15.912: await reader.read() returned 451 times, and current chunkSize is 2899968 (total size is 3032884)
2025-03-12 12:51:16,127: abort signal called after 500ms, as expected
hangs
2025-03-12 12:51:30.913: returned by Promise.race in 15 seconds
2025-03-12 12:51:30,916: print info todo with the reader, the reader promise status is still pendding.
fwx5618177 commentedon Mar 13, 2025
It looks like:
fwx5618177 commentedon Mar 24, 2025
I reproduce this issue by using express.
Here are code:
server.js
test.js
asyncLogger.js
we need to update lib/web/fetch/util.js
The root cause of the problem is:
fwx5618177 commentedon Mar 24, 2025
mcollina commentedon Mar 27, 2025
Can you put all the above in a repository and make it easy for use to reproduce?
fwx5618177 commentedon May 27, 2025
I have pushed these files on branch fix-4089, we could find files on reproduce-script.
Directly run:
mcollina commentedon Jun 1, 2025
What repository?
rene84 commentedon Jun 18, 2025
We also ran into an issue that very accurately resembles the issue reported here. We saw hangs on
await response.text(), but only when the server was responding with 'transfer-encoding': 'chunked'.What may be relevant for this issue is: bumping to undici 7.10.0 solved it for us
fwx5618177 commentedon Jun 18, 2025
Yes, please read this repo, and I reproduced it. But I still have no idea to fix it.
@mcollina here: https://github.com/fwx5618177/undici/tree/fix-4089/reproduce-script
ferrisxie commentedon Jun 19, 2025
hi, @rene84 ,as you said :
I have noticed that there is the same response header
chunkedin my case.WIP: Use Undici's fetch implementation
pacocartones commentedon Aug 9, 2026
I was able to reproduce this deterministically and pin down the exact fix.
Root cause (undici 6.x). The hang is the same bug that #3754 fixed on the 7.x line (a regression introduced by 49e33a8 / #2823). In 6.21.1, when the abort lands after the transport has completed the body but before the body stream is closed:
onCompleteremoves theterminatedlistener (lib/web/fetch/index.js, theoff('terminated', fetchParams.controller.onAborted)added by remove all fetchParam event handlers #2823).terminated, butonAbortedis gone, sofetchParams.controller.controller.error(...)never runs — the body stream is never errored.isAborted(fetchParams)andbreaks without closing or erroring the stream.ReadableStreamstays'readable'forever, soreadAllBytesparks atawait reader.read()inlib/web/fetch/util.jsandresponse.text()never settles — exactly what @ferrisxie traced.Verified by instrumenting 6.21.1:
onAbortedis registered,onCompleteremoves it, the loop breaks onisAborted, andonAbortedis never called. The read promise stays pending indefinitely.Repro. The deterministic repro is essentially the regression test added in #3754 (
test/fetch/issue-1711.js, "aborting a Stream throws"): server sends a small full body immediately, client startsresponse.arrayBuffer()and aborts right away. On 6.21.1 this hangs 50/50 runs (Windows, node 24.14.1; also 20/20 on Linux node 18.20.8). On 7.10.0 and on current main, 50/50 reject withAbortErrorin ~1 ms.Note: the reproduce-script in https://github.com/fwx5618177/undici/tree/fix-4089/reproduce-script stalls the server mid-body, so the transport never completes and the abort path works (text() rejects with AbortError) — at least on node 24, I could not get it to hang across several 100-request runs. The hang window requires the abort to land after
onComplete(body fully received at the socket level, still being drained), which matches the original timeline (451 reads returned, abort at 500 ms, then hang).Version bisect.
7.0.0-alpha.3hangs,7.0.0-alpha.4is fixed — the only fetch/abort change in that range is afeb626 (#3754). First stable release with the fix: 7.0.0. This also explains @rene84's report that 7.10.0 fixed it.Still affected: the whole 6.x line — I verified 6.21.1, 6.21.3 and 6.28.0 (latest 6.x) all hang. The fix was never backported.
So: fixed in 7.x by afeb626 (#3754). Two options: backport #3754 to the 6.x branch for users pinned to 6.x, or close this as fixed in 7.x with a note that 6.x won't get the fix.