Skip to content

High L7 lost samples on cortex distributor node pools #369

Open

Description

@vishnukumarkvs

Issue

  • The coroot node agents running in Cortex Distributor node pools are loosing lot of l7 samples. Likely due to ring-buffer getting full
  • For every lost sample, the coroot node agent sends a error log which are getting spammed heavily
  • I also observed memory leak on these particular node agents

Is there a way to fix lost samples on cortex nodepools. Also, these logs are too much and throttling Loki in our setup. I have tried fine tuning LOG_PER_SECOND and LOG_BURST params but seems no effect on these error logs.

Also, Instead of error log for lost samples, can we have a prometheus counter?

Activity

  1. vishnukumarkvs commented on Sep 20, 2026

    @vishnukumarkvs
    ContributorAuthor

    On digging a bit more, I feel L7 tracing cant be scaled on cortex or mimir node pools. The amount of ingestion traffic will cause buffer overflow to lost samples. This is also causing node cpu saturation. The load average is peaking on the node

  2. def commented on Sep 21, 2026

    @def
    Member

    Could you please gather a CPU/Memory profile of the agent on this node?

  3. vishnukumarkvs commented on Sep 23, 2026

    @vishnukumarkvs
    ContributorAuthor

    Could you please gather a CPU/Memory profile of the agent on this node?

    I might not be able to get this now, as we scaled down in this nodepool. Let me check if I can recreate it in lower envs

  4. vishnukumarkvs commented on Sep 30, 2026

    @vishnukumarkvs
    ContributorAuthor

    coroot-node-agent-profiles.zip

    Hi @def , sorry for the delay but sharing cpu and heap profiles I could gather on mimir node pool, where I had lot of L7 lost samples ( 11,931 lost samples during CPU window: 30s)

  5. vishnukumarkvs commented on Sep 30, 2026

    @vishnukumarkvs
    ContributorAuthor

    Also, just curious, in L7 tracing, is DNS the only protocol used to detect outside connections to domains in servicemap? Like connection to an RDS instance. I mean does node agent uses only DNS protocol to construct this?

    If so, would it be a good idea to create seperate option like enable DNS tracing? So that even when L7 is disabled, we can still see substancial amount of outside connections in servicemap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions