You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Prometheus high memory utilization because of krr queries #83
I first noticed a problem when I clicked Rescan on the Efficiency panel of Robusta.dev that connects to Prometheus and gathers metrics from my cluster where I have installed the Robusta Helm chart.
My prometheus Pod that normally uses less than 1.5 GiB of memory suddenly needs a lot more than 2.5 GiB. Because it’s running on a node with only 4 GiB ram (and is limited to 3GiB in the Pod spec), the pod is OOM-killed. You can see that behavior in the graph below. Robusta UI also shows the OOM-killed pod.
I believe that feature uses krr in the background, so I also tried running it directly on the CLI pointing at the same Prometheus endpoint, which produced the same result.
Anything I could do to improve that (besides the obvious increase in memory limits)?
To Reproduce
Steps to reproduce the behavior:
Run krr pointing to a Prometheus pod with a 3 GiB memory limit
Prometheus should have 250k+ time series and 10k+ label pairs
Do not limit the namespaces scanned by krr
Expected behavior
Not cause Prometheus to crash
Screenshots
Additional context
When scanning a single namespace Prometheus doesn't crash but you can still notice a big hike in memory utilization
So I did some testing with different number of workers. For our setup, it seems 3 workers is the ideal number as you can see in the graph below (and considering that W=3 runs 50% faster than W=1 but only 15% slower than W=6)
Prometheus didn't crash anymore but that's because we doubled the node instance size from 4GiB to 8GiB.
In general, I would say, without looking at the source code, if you need to send such heavy queries to Prometheus, you should probably try to use precomputed time series as much as possible.
Thanks @macmiranda for your testing
We were doing an investigation on how to improve the performance additionally. Can you also try the latest version from main branch, as it should now use even less memory.
I am closing the issue currently, but feel free to open it again if you think that there is something else to improve
Thanks for the work you put into this. I have tested krr from the main branch and the results are promising.
(it ran for all namespaces with -w 1 in about 3 minutes)
As you can see there is still a peak memory utilization when krr runs but it seems manageable now.
I'm eager to test it running as a Job in the cluster. Do you know when the next Robusta release will happen? (I'm asking because the next release will contain a fix that limits the memory consumption of the krr Job as well).
Activity
LeaveMyYard commentedon Jun 23, 2023
It seems like it is related to the amount of parallel queries being really high
We are doing the fix currently, I will update here on the progress
Avi-Robusta commentedon Jun 27, 2023
Just wanted to update there is a pr open that might fix this issue
#82
macmiranda commentedon Jun 27, 2023
Cool, @Avi-Robusta, I saw it was just merged. I'll try it out and report results here.
macmiranda commentedon Jun 27, 2023
So I did some testing with different number of workers. For our setup, it seems 3 workers is the ideal number as you can see in the graph below (and considering that W=3 runs 50% faster than W=1 but only 15% slower than W=6)
Prometheus didn't crash anymore but that's because we doubled the node instance size from 4GiB to 8GiB.
In general, I would say, without looking at the source code, if you need to send such heavy queries to Prometheus, you should probably try to use precomputed time series as much as possible.
LeaveMyYard commentedon Jul 5, 2023
Thanks @macmiranda for your testing
We were doing an investigation on how to improve the performance additionally. Can you also try the latest version from main branch, as it should now use even less memory.
I am closing the issue currently, but feel free to open it again if you think that there is something else to improve
macmiranda commentedon Jul 6, 2023
Hi @LeaveMyYard,
Thanks for the work you put into this. I have tested
krrfrom themainbranch and the results are promising.(it ran for all namespaces with
-w 1in about 3 minutes)As you can see there is still a peak memory utilization when
krrruns but it seems manageable now.I'm eager to test it running as a Job in the cluster. Do you know when the next Robusta release will happen? (I'm asking because the next release will contain a fix that limits the memory consumption of the
krrJob as well).ROB-1574 Bump idna to 3.15 and drop bundled pip wheel to fix CVEs (#555)