Skip to content

Prometheus high memory utilization because of krr queries #83

Closed

Description

@macmiranda

Describe the bug

I first noticed a problem when I clicked Rescan on the Efficiency panel of Robusta.dev that connects to Prometheus and gathers metrics from my cluster where I have installed the Robusta Helm chart.

My prometheus Pod that normally uses less than 1.5 GiB of memory suddenly needs a lot more than 2.5 GiB. Because it’s running on a node with only 4 GiB ram (and is limited to 3GiB in the Pod spec), the pod is OOM-killed. You can see that behavior in the graph below. Robusta UI also shows the OOM-killed pod.

I believe that feature uses krr in the background, so I also tried running it directly on the CLI pointing at the same Prometheus endpoint, which produced the same result.

Anything I could do to improve that (besides the obvious increase in memory limits)?

To Reproduce
Steps to reproduce the behavior:

  1. Run krr pointing to a Prometheus pod with a 3 GiB memory limit
  2. Prometheus should have 250k+ time series and 10k+ label pairs
  3. Do not limit the namespaces scanned by krr

Expected behavior
Not cause Prometheus to crash

Screenshots
image (6)

Additional context

When scanning a single namespace Prometheus doesn't crash but you can still notice a big hike in memory utilization
image (7)

Activity

  1. self-assigned this
    on Jun 23, 2023
  2. LeaveMyYard commented on Jun 23, 2023

    @LeaveMyYard
    Contributor

    It seems like it is related to the amount of parallel queries being really high
    We are doing the fix currently, I will update here on the progress

  3. Avi-Robusta commented on Jun 27, 2023

    @Avi-Robusta
    Contributor

    Just wanted to update there is a pr open that might fix this issue
    #82

  4. macmiranda commented on Jun 27, 2023

    @macmiranda
    Author

    Cool, @Avi-Robusta, I saw it was just merged. I'll try it out and report results here.

  5. macmiranda commented on Jun 27, 2023

    @macmiranda
    Author

    So I did some testing with different number of workers. For our setup, it seems 3 workers is the ideal number as you can see in the graph below (and considering that W=3 runs 50% faster than W=1 but only 15% slower than W=6)

    krr_workers_prometheus

    Prometheus didn't crash anymore but that's because we doubled the node instance size from 4GiB to 8GiB.

    In general, I would say, without looking at the source code, if you need to send such heavy queries to Prometheus, you should probably try to use precomputed time series as much as possible.

  6. LeaveMyYard commented on Jul 5, 2023

    @LeaveMyYard
    Contributor

    Thanks @macmiranda for your testing
    We were doing an investigation on how to improve the performance additionally. Can you also try the latest version from main branch, as it should now use even less memory.

    I am closing the issue currently, but feel free to open it again if you think that there is something else to improve

  7. macmiranda commented on Jul 6, 2023

    @macmiranda
    Author

    Hi @LeaveMyYard,

    Thanks for the work you put into this. I have tested krr from the main branch and the results are promising.

    image
    (it ran for all namespaces with -w 1 in about 3 minutes)

    As you can see there is still a peak memory utilization when krr runs but it seems manageable now.

    I'm eager to test it running as a Job in the cluster. Do you know when the next Robusta release will happen? (I'm asking because the next release will contain a fix that limits the memory consumption of the krr Job as well).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions