Skip to content

Node agent metrics exceeded Prometheus 32Mb remote-write limit #404

Open

Description

@vishnukumarkvs

This issue happened to coroot node agent pod in node where our own internal prometheus was running. This one agent sends 39.51 Mb decoded metrics payload , exceeding 32Mb hard limit.

97% of payload comes from Coroot's HTTP/TCP observations of prometheus workload scraping many endpoints

One Scrape -> 85K series -> one 39Mb request -> rejected

85K split

Metric Description Count / Value
HTTP request-duration histogram buckets 42,000
HTTP duration sum + count 7,150
HTTP request counters 3,469
TCP connect time, successful connect and byte counters 23,000

I feel this cardinality is normal if we are scraping prometheus. If not pls correct me

And in this case, would splitting payload to batches help?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions