This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Repository navigation
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
media repo workers use an awful lot of memory #9365
Description
A lot of it gets cleaned up when we eventually do a gen 2 GC, but 44G seems a bit insane for a process which is essentially a glorified web server:
It would be good to find out what is using that memory, with a view to either not using it in the first place, or making it easier to free so that it gets removed in a gen 0 or gen 1 GC.
Activity
richvdh commentedon Feb 10, 2021
(this tends to cause the box to go into swap, which is obviously bad)
benbz commentedon Feb 10, 2021
FYI the media repo's memory profile seem to change after a deploy on 18th Jan:
erikjohnston commentedon Feb 11, 2021
Previous issue: #8653, though the situation seems to have changed somewhat
richvdh commentedon Feb 11, 2021
I don't think it's minor.
michaelkaye commentedon Feb 15, 2021
So we have a tactical cronjob in place on matrix.org for this ; are we going to be able to look into this soon or should we make the automatic restarting more robust?
clokep commentedon Feb 16, 2021
This might be a regression from #9108 or one of the earlier PRs updating that code.
clokep commentedon Feb 16, 2021
From @richvdh doing a bit of research:
Looking a bit at this I think setting
unbuffered=Trueon the call totreq.request"fixes" this. I'm not sure of the other ramifications of this though, so looking into that.clokep commentedon Feb 19, 2021
Hopefully the changes in #9421 will help out with this, but I guess it'll be a few days before we know.
clokep commentedon Feb 22, 2021
After this change it seems that the huge spikes in memory allocation are gone:
It also seems that nothing is being allocated into gen2 anymore:
Due to the above I'm going to call this complete. But @richvdh or @benbz please let me know if I'm missing anything here!