You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Memory issues / many instances of MaxListenersExceededWarning when using Undici via Elasticsearch #3131
This is a follow-up to #1711. @KhafraDev asked that I file a separate issue because Elasticsearch does not appear to be using fetch, which is what was fixed in that issue (via #2823).
I've been running into what I think is the same or similar issue to what was reported in #1711 - lots of MaxListenersExceededWarnings being logged, and my service eventually runs out of memory and dies. When I run with --trace-warnings, the stacktrace points to Undici:
(node:14480) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 101 abort listeners added to [EventEmitter]. Use emitter.setMaxListeners() to increase limit
at _addListener (node:events:588:17)
at EventEmitter.addListener (node:events:606:10)
at addAbortListener (/Users/daveyarwood/code/spark/node_modules/undici/lib/core/util.js:443:10)
at addSignal (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/abort-signal.js:36:3)
at new RequestHandler (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:66:5)
at Pool.request (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:171:25)
at /Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:164:15
at new Promise (<anonymous>)
at Pool.request (/Users/daveyarwood/code/spark/node_modules/undici/lib/api/api-request.js:163:12)
This is with Undici 6.13.0, which is a transitive dependency of @elastic/transport 8.5.0.
Reproducible By
I would be happy to provide a minimal repro case if I had any idea how to do that! :)
Please let me know if you have any ideas of things I can test in my setup.
Expected Behavior
No MaxListenersExceededWarning messages should be logged.
In a deployed environment, the memory graphs should not show available memory declining significantly, and the machine shouldn't run out of memory and die.
Environment
Local
2021 Macbook Pro M1 Max
macOS Sonoma 14.11
Node 20.9.0
I'm seeing the MaxListenersExceededWarnings when I run my service locally.
Deployed
My service is also running on a Linux server, in a Docker container with Node 20. I'm also seeing the MaxListenersExceededWarnings logged there, and the memory graph shows a significant decline in memory usage from the time the instance spins up to when it eventually runs out of memory and dies.
Additional context
I don't fully understand what @KhafraDev meant by Elasticsearch not using fetch. I looked through the @elastic/transport code base (on the v8.5.0 tag, which is the version I'm using) and I found UndiciConnection.ts, which appears to be where it's creating an Undici pool and calling request on it, but beyond that, I'm not sure what to look for.
It's quite possible that there is some other reason that my service is running out of memory and dying. But at any rate, it would be good for us to get to the bottom of the MaxListenersExceededWarnings, which (as far as I can tell, from my limited understanding) appear to suggest a memory leak somewhere in Undici.
Thanks for pointing this out @ronag. Seems I didn't address your comment there when I initially thought the problem was resolved in elasticsearch 8.8.1.
Not in the Elasticsearch client, yet. I have to run down what @ronag suggested and establish the best way to fix that without introducing any breaking changes.
Activity
daveyarwood commentedon Apr 16, 2024
It's quite possible that there is some other reason that my service is running out of memory and dying. But at any rate, it would be good for us to get to the bottom of the MaxListenersExceededWarnings, which (as far as I can tell, from my limited understanding) appear to suggest a memory leak somewhere in Undici.
ronag commentedon Apr 16, 2024
I believe I open end an issue over at elasticsearch a long time ago. They don't properly cleanup signals.
ronag commentedon Apr 16, 2024
elastic/elasticsearch-js#1716
Uzlopak commentedon Apr 17, 2024
Just one remark: when running the wpt suite for fetch, i see the MaxListenersExceededWarning emitted too.
JoshMock commentedon Apr 17, 2024
Thanks for pointing this out @ronag. Seems I didn't address your comment there when I initially thought the problem was resolved in elasticsearch 8.8.1.
sibelius commentedon Apr 23, 2024
is this fixed?
JoshMock commentedon Apr 23, 2024
Not in the Elasticsearch client, yet. I have to run down what @ronag suggested and establish the best way to fix that without introducing any breaking changes.
Updates will be shared on elastic/elastic-transport-js#63.
daveyarwood commentedon May 28, 2024
@elastic/elastic-transport8.5.2 has just been released - updating to that version fixed my issue.