Repository navigation
docs(integrations): list Utilization & Health and the trace heatmap for Splunk Enterprise #936
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
reckless-sherixx
wants to merge
1
commit into
main
Choose a base branch
from
docs/splunk-enterprise-heatmap-utilization-webhook
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -180,9 +180,25 @@ Metrics need **Splunk Metric Index**, and it must be a `datatype=metric` index. | |||||
|
|
||||||
| What uses Splunk metrics: | ||||||
|
|
||||||
| - **Utilization & Health.** When Splunk Enterprise is the account's metrics provider, which **Default Metrics Provider** sets, the cluster and workload utilization views read the metric index: | ||||||
| - On the cluster's **Summary** tab, the **Utilization & Health** card shows CPU and memory **Total**, **Usage**, **Request**, **Limit**, **P50 Usage**, **P90 Usage**, and **Max Usage**. **Trend** charts usage, total, request, and limit over the selected time range. | ||||||
| - On a workload's **Utilization Trends** tab, the CPU and memory charts show its usage, requested, and limit. | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Use "request" instead of "requested" to maintain parallel noun structure with "usage" and "limit", and to remain consistent with the terminology used in the previous bullet point.
Suggested change
References
|
||||||
| - **nubi.** For accounts whose metrics provider is Splunk Enterprise, nubi uses a dedicated Splunk metrics agent. The agent writes `mstats` SPL, not PromQL. nubi can list the metrics a workload emits and read their values. With no metric index configured, nubi answers that Splunk metrics are not configured, not that there is no data. | ||||||
| - **Event evidence.** On workload alerts, NudgeBee finds a CPU metric in the index, such as `k8s.pod.cpu.usage` or `container.cpu.usage`, and charts it for the affected pods. | ||||||
|
|
||||||
| **Utilization metric names.** NudgeBee reads the metrics index's catalog and, for each value, uses the first of these names it finds. A value whose names are all missing shows `-`. | ||||||
|
|
||||||
| | Value | Splunk metric names checked, in order | | ||||||
| |-------|---------------------------------------| | ||||||
| | Cluster CPU total | `k8s.node.allocatable_cpu`, `k8s.node.allocatable.cpu`, `kube.node.allocatable_cpu` | | ||||||
| | Cluster CPU usage | `k8s.node.cpu.usage`, `k8s.node.cpu.utilization`, `k8s.node.cpu_usage` | | ||||||
| | Cluster memory total | `k8s.node.allocatable_memory`, `k8s.node.allocatable.memory`, `kube.node.allocatable_memory` | | ||||||
| | Cluster memory usage | `k8s.node.memory.usage`, `k8s.node.memory.working_set`, `k8s.node.memory_usage` | | ||||||
| | Workload CPU usage | `k8s.pod.cpu.usage`, `k8s.pod.cpu.utilization`, `container.cpu.usage`, `k8s.container.cpu.usage` | | ||||||
| | Workload memory usage | `k8s.pod.memory.usage`, `k8s.pod.memory.working_set`, `container.memory.usage`, `k8s.container.memory.usage` | | ||||||
| | CPU request / limit | `k8s.container.cpu_request` / `k8s.container.cpu_limit`, or the dotted forms `k8s.container.cpu.request` / `k8s.container.cpu.limit` | | ||||||
| | Memory request / limit | `k8s.container.memory_request` / `k8s.container.memory_limit`, or the dotted forms `k8s.container.memory.request` / `k8s.container.memory.limit` | | ||||||
|
|
||||||
| A hand-written metrics query must follow three rules: | ||||||
|
|
||||||
| - It must start with `| mstats` or `| mcatalog`. | ||||||
|
|
@@ -207,7 +223,7 @@ Traces need **Splunk Trace Index**, which must hold spans written by the OpenTel | |||||
|
|
||||||
| What the Traces page supports with Splunk Enterprise: | ||||||
|
|
||||||
| - The trace list, trace grouping, and the span waterfall are supported. | ||||||
| - The trace list, trace grouping, and the trace heatmap are supported. The heatmap is a trace's **Service & Operation** tab: a **Trace Chart** of its spans by service and operation, drawn as a waterfall. | ||||||
| - **The service map is not available.** A Splunk span names the service it called, but not that service's namespace. A map drawn from names alone would merge same-named services in different namespaces. | ||||||
| - For the same reason, you cannot filter by the destination workload's namespace. | ||||||
|
|
||||||
|
|
@@ -219,6 +235,7 @@ What the Traces page supports with Splunk Enterprise: | |||||
| 2. Open a **Kubernetes workload** in NudgeBee and go to the **Logs** tab. Confirm that its logs from Splunk appear. | ||||||
| 3. If you set a trace index, open the **Traces** tab and confirm that spans appear. | ||||||
| 4. If you set a metric index, ask nubi, for example: *"Which metrics does the checkout deployment in the prod namespace emit?"* | ||||||
| 5. If Splunk Enterprise is the **Default Metrics Provider**, open the cluster's **Summary** tab and confirm that **Utilization & Health** shows CPU and memory figures. | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
|
|
@@ -244,7 +261,7 @@ What the Traces page supports with Splunk Enterprise: | |||||
| - **No cluster filter.** NudgeBee does not add a cluster filter to the searches it generates. If one Splunk holds data from several clusters, workloads with the same namespace and name in different clusters are mixed together. | ||||||
| - **One provider, three signals.** Logs, metrics, and traces each have their own index setting. You can connect only logs, and add metrics and traces later. | ||||||
| - **Read-only.** NudgeBee never writes to Splunk. It blocks write and send commands in any query it runs, but the Splunk role is the real control. | ||||||
| - **Telemetry only.** This integration reads logs, metrics, and traces. It does not turn Splunk alerts into NudgeBee events. | ||||||
| - **Telemetry only.** This integration reads logs, metrics, and traces. It does not turn Splunk alerts into NudgeBee events. The separate [Splunk Webhook](../Webhooks/splunk_webhook.md) integration does that: Splunk posts each alert to NudgeBee, and it becomes an event. Its tile in **Admin** > **Integrations** > **Webhooks** is grayed out, so you cannot add a Splunk Webhook connection. | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
|
|
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The phrase "which Default Metrics Provider sets" is grammatically awkward. It is clearer to refer to it as a setting.
References