You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue happened to coroot node agent pod in node where our own internal prometheus was running. This one agent sends 39.51 Mb decoded metrics payload , exceeding 32Mb hard limit.
97% of payload comes from Coroot's HTTP/TCP observations of prometheus workload scraping many endpoints
One Scrape -> 85K series -> one 39Mb request -> rejected
85K split
Metric Description
Count / Value
HTTP request-duration histogram buckets
42,000
HTTP duration sum + count
7,150
HTTP request counters
3,469
TCP connect time, successful connect and byte counters
23,000
I feel this cardinality is normal if we are scraping prometheus. If not pls correct me
And in this case, would splitting payload to batches help?
Activity