You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
High L7 lost samples on cortex distributor node pools #369
The coroot node agents running in Cortex Distributor node pools are loosing lot of l7 samples. Likely due to ring-buffer getting full
For every lost sample, the coroot node agent sends a error log which are getting spammed heavily
I also observed memory leak on these particular node agents
Is there a way to fix lost samples on cortex nodepools. Also, these logs are too much and throttling Loki in our setup. I have tried fine tuning LOG_PER_SECOND and LOG_BURST params but seems no effect on these error logs.
Also, Instead of error log for lost samples, can we have a prometheus counter?
On digging a bit more, I feel L7 tracing cant be scaled on cortex or mimir node pools. The amount of ingestion traffic will cause buffer overflow to lost samples. This is also causing node cpu saturation. The load average is peaking on the node
Hi @def , sorry for the delay but sharing cpu and heap profiles I could gather on mimir node pool, where I had lot of L7 lost samples ( 11,931 lost samples during CPU window: 30s)
Also, just curious, in L7 tracing, is DNS the only protocol used to detect outside connections to domains in servicemap? Like connection to an RDS instance. I mean does node agent uses only DNS protocol to construct this?
If so, would it be a good idea to create seperate option like enable DNS tracing? So that even when L7 is disabled, we can still see substancial amount of outside connections in servicemap
Activity
vishnukumarkvs commentedon Sep 20, 2026
On digging a bit more, I feel L7 tracing cant be scaled on cortex or mimir node pools. The amount of ingestion traffic will cause buffer overflow to lost samples. This is also causing node cpu saturation. The load average is peaking on the node
def commentedon Sep 21, 2026
Could you please gather a CPU/Memory profile of the agent on this node?
vishnukumarkvs commentedon Sep 23, 2026
I might not be able to get this now, as we scaled down in this nodepool. Let me check if I can recreate it in lower envs
vishnukumarkvs commentedon Sep 30, 2026
coroot-node-agent-profiles.zip
Hi @def , sorry for the delay but sharing cpu and heap profiles I could gather on mimir node pool, where I had lot of L7 lost samples ( 11,931 lost samples during CPU window: 30s)
vishnukumarkvs commentedon Sep 30, 2026
Also, just curious, in L7 tracing, is DNS the only protocol used to detect outside connections to domains in servicemap? Like connection to an RDS instance. I mean does node agent uses only DNS protocol to construct this?
If so, would it be a good idea to create seperate option like enable DNS tracing? So that even when L7 is disabled, we can still see substancial amount of outside connections in servicemap