Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions doc-server/docs/integrations/Observability/splunk_enterprise.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,25 @@ Metrics need **Splunk Metric Index**, and it must be a `datatype=metric` index.

What uses Splunk metrics:

- **Utilization & Health.** When Splunk Enterprise is the account's metrics provider, which **Default Metrics Provider** sets, the cluster and workload utilization views read the metric index:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The phrase "which Default Metrics Provider sets" is grammatically awkward. It is clearer to refer to it as a setting.

Suggested change
- **Utilization & Health.** When Splunk Enterprise is the account's metrics provider, which **Default Metrics Provider** sets, the cluster and workload utilization views read the metric index:
- **Utilization & Health.** When Splunk Enterprise is the account's metrics provider (configured via the **Default Metrics Provider** setting), the cluster and workload utilization views read the metric index:
References
  1. Use US English spelling consistently throughout the documentation (e.g., 'utilization' instead of 'utilisation').

- On the cluster's **Summary** tab, the **Utilization & Health** card shows CPU and memory **Total**, **Usage**, **Request**, **Limit**, **P50 Usage**, **P90 Usage**, and **Max Usage**. **Trend** charts usage, total, request, and limit over the selected time range.
- On a workload's **Utilization Trends** tab, the CPU and memory charts show its usage, requested, and limit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Use "request" instead of "requested" to maintain parallel noun structure with "usage" and "limit", and to remain consistent with the terminology used in the previous bullet point.

Suggested change
- On a workload's **Utilization Trends** tab, the CPU and memory charts show its usage, requested, and limit.
- On a workload's **Utilization Trends** tab, the CPU and memory charts show its usage, request, and limit.
References
  1. Use US English spelling consistently throughout the documentation (e.g., 'utilization' instead of 'utilisation').

- **nubi.** For accounts whose metrics provider is Splunk Enterprise, nubi uses a dedicated Splunk metrics agent. The agent writes `mstats` SPL, not PromQL. nubi can list the metrics a workload emits and read their values. With no metric index configured, nubi answers that Splunk metrics are not configured, not that there is no data.
- **Event evidence.** On workload alerts, NudgeBee finds a CPU metric in the index, such as `k8s.pod.cpu.usage` or `container.cpu.usage`, and charts it for the affected pods.

**Utilization metric names.** NudgeBee reads the metrics index's catalog and, for each value, uses the first of these names it finds. A value whose names are all missing shows `-`.

| Value | Splunk metric names checked, in order |
|-------|---------------------------------------|
| Cluster CPU total | `k8s.node.allocatable_cpu`, `k8s.node.allocatable.cpu`, `kube.node.allocatable_cpu` |
| Cluster CPU usage | `k8s.node.cpu.usage`, `k8s.node.cpu.utilization`, `k8s.node.cpu_usage` |
| Cluster memory total | `k8s.node.allocatable_memory`, `k8s.node.allocatable.memory`, `kube.node.allocatable_memory` |
| Cluster memory usage | `k8s.node.memory.usage`, `k8s.node.memory.working_set`, `k8s.node.memory_usage` |
| Workload CPU usage | `k8s.pod.cpu.usage`, `k8s.pod.cpu.utilization`, `container.cpu.usage`, `k8s.container.cpu.usage` |
| Workload memory usage | `k8s.pod.memory.usage`, `k8s.pod.memory.working_set`, `container.memory.usage`, `k8s.container.memory.usage` |
| CPU request / limit | `k8s.container.cpu_request` / `k8s.container.cpu_limit`, or the dotted forms `k8s.container.cpu.request` / `k8s.container.cpu.limit` |
| Memory request / limit | `k8s.container.memory_request` / `k8s.container.memory_limit`, or the dotted forms `k8s.container.memory.request` / `k8s.container.memory.limit` |

A hand-written metrics query must follow three rules:

- It must start with `| mstats` or `| mcatalog`.
Expand All @@ -207,7 +223,7 @@ Traces need **Splunk Trace Index**, which must hold spans written by the OpenTel

What the Traces page supports with Splunk Enterprise:

- The trace list, trace grouping, and the span waterfall are supported.
- The trace list, trace grouping, and the trace heatmap are supported. The heatmap is a trace's **Service & Operation** tab: a **Trace Chart** of its spans by service and operation, drawn as a waterfall.
- **The service map is not available.** A Splunk span names the service it called, but not that service's namespace. A map drawn from names alone would merge same-named services in different namespaces.
- For the same reason, you cannot filter by the destination workload's namespace.

Expand All @@ -219,6 +235,7 @@ What the Traces page supports with Splunk Enterprise:
2. Open a **Kubernetes workload** in NudgeBee and go to the **Logs** tab. Confirm that its logs from Splunk appear.
3. If you set a trace index, open the **Traces** tab and confirm that spans appear.
4. If you set a metric index, ask nubi, for example: *"Which metrics does the checkout deployment in the prod namespace emit?"*
5. If Splunk Enterprise is the **Default Metrics Provider**, open the cluster's **Summary** tab and confirm that **Utilization & Health** shows CPU and memory figures.

---

Expand All @@ -244,7 +261,7 @@ What the Traces page supports with Splunk Enterprise:
- **No cluster filter.** NudgeBee does not add a cluster filter to the searches it generates. If one Splunk holds data from several clusters, workloads with the same namespace and name in different clusters are mixed together.
- **One provider, three signals.** Logs, metrics, and traces each have their own index setting. You can connect only logs, and add metrics and traces later.
- **Read-only.** NudgeBee never writes to Splunk. It blocks write and send commands in any query it runs, but the Splunk role is the real control.
- **Telemetry only.** This integration reads logs, metrics, and traces. It does not turn Splunk alerts into NudgeBee events.
- **Telemetry only.** This integration reads logs, metrics, and traces. It does not turn Splunk alerts into NudgeBee events. The separate [Splunk Webhook](../Webhooks/splunk_webhook.md) integration does that: Splunk posts each alert to NudgeBee, and it becomes an event. Its tile in **Admin** > **Integrations** > **Webhooks** is grayed out, so you cannot add a Splunk Webhook connection.

---

Expand Down
Loading