Skip to content

feat: host filesystem, load and swap metrics in standalone mode; fix meminfo KiB - #381

Merged
mayankpande88 merged 4 commits into
mainfrom
feat/vm-host-metrics
Oct 9, 2026
Merged

mayankpande88 merged 4 commits into
mainfrom
feat/vm-host-metrics

Conversation

@mayankpande88

@mayankpande88 mayankpande88 commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Host metrics for hosts without node_exporter, plus a fix to the existing memory metrics.

  • Standalone mode (no Kubernetes): the node collector exports the host's filesystem space on every mount, load average and swap, under node_exporter's names:

    • node_filesystem_{size,free,avail}_bytes, node_filesystem_files, node_filesystem_files_free and node_filesystem_readonly (labels device, fstype, mountpoint);
    • node_load1, node_load5, node_load15;
    • node_memory_SwapTotal_bytes, node_memory_SwapFree_bytes.

    Before this, the agent had no filesystem space except for volumes a workload writes to, and no load or swap.

  • Kubernetes: these stay off. node_exporter usually runs there, and sums by instance would count the same host twice.

  • /proc/meminfo units: the "kB" there are KiB, but values were multiplied by 1000, so node_resources_memory_* read about 2.3% low (upstream has the same bug). They now report correct values, about 2.4% higher than before, matching node_exporter's node_memory_*_bytes.

Engineering detail

Mounts:

  • Read from /proc/1/mounts, with node_exporter's default exclusions:
    • pseudo filesystems (sysfs, proc, cgroup, overlay, squashfs, tracefs, ...);
    • /dev, /proc, /sys, /run/credentials/..., /var/lib/docker/... and /var/lib/containers/storage/....
  • Octal escapes in paths (for example \040 for a space) are decoded, and read-only mounts are reported.

statfs:

  • Runs through /proc/1/root/<mountpoint>, with a 2s timeout.
  • A mount whose statfs hangs (a dead network mount) is skipped for 5 minutes, so it can't block a scrape.
  • No second statfs starts for a mount while one is still in flight (from review). Each hung call holds an OS thread, so retries would leak threads; TestStatfsHungMountSingleCall covers it.

Where it's switched on: standalone mode is when the IP resolver is the VM resolver (no Kubernetes API).

Tests: load average parsing, mount filtering, escapes and read-only flags, and swap plus the KiB multiplier in the memory test.

CI: gofmt, goimports, vet, golangci-lint, go test (excluding /containers) and the build all pass in a Linux container with Go 1.26.5.

Local e2e: I built agent binaries from this branch and from main and ran both as systemd services on a local Debian 12 VM (kernel 6.1), with a 256 MiB swapfile.

  • Root filesystem: size and avail matched df -B1 exactly (12468432896 / 8699375616).
  • Mounts: /, /boot/efi, /run, /run/lock, /run/user/501 and the VM's two host shares were reported; /dev/shm was left out, as in node_exporter.
  • Load and swap: node_load* matched /proc/loadavg, and swap matched /proc/meminfo.
  • Memory: node_resources_memory_total_bytes was 2081710080 (= MemTotal × 1024) on this branch and 2032920000 on main.
  • Main exported no node_filesystem_* or node_load*.

memoryInfo multiplied /proc/meminfo values by 1000, but the kernel's kB
there are 1024 bytes, so node_resources_memory_* read about 2.3% low
(upstream has the same bug). They now match node_exporter's
node_memory_*_bytes.
Without Kubernetes there is usually no node_exporter, and the agent had
no filesystem space (only volumes a workload writes to), load or swap
metrics. In standalone mode the node collector now exports them under
node_exporter's names: node_filesystem_{size,free,avail}_bytes,
node_filesystem_files{,_free} and node_filesystem_readonly per mount
(pid 1's mounts, without node_exporter's default pseudo and container
filesystems), node_load{1,5,15} and node_memory_Swap{Total,Free}_bytes.
statfs runs with a timeout, and a mount that hangs is skipped for 5
minutes, so a dead network mount cannot block a scrape. On Kubernetes
these stay off: node_exporter usually runs there and sums would count
the same host twice.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces host metrics collection (load average, swap memory, and filesystem statistics) under node_exporter names for standalone/VM environments where node_exporter is not running. It also corrects the memory unit multiplier from 1000 to 1024 (KiB) in /proc/meminfo parsing. The review feedback focuses on improving the robustness of the statfs timeout mechanism by refactoring it into a method on hostCollector and tracking active statfs calls to prevent spawning duplicate goroutines and leaking them when network mounts are hung.

Comment thread node/host.go Resolved
Comment thread node/host.go Resolved
Comment thread node/host.go Resolved
Comment thread node/host.go Resolved
Comment thread node/host.go Resolved
A mount whose statfs hung was retried every 5 minutes, and each retry
started another goroutine while the previous one was still blocked in
the kernel, holding an OS thread: a dead network mount leaked one
thread every 5 minutes. A second statfs is no longer started while one
is in flight for that mount point, and a successful call clears the
mount's stuck mark.
@blue4209211

Copy link
Copy Markdown
Contributor

LGTM. The meminfo KiB fix is correct, and the statfs guard (timeout, one call in flight per mount, 5-minute skip) is right. Two notes:

  1. Duplicate mounts produce duplicate series (node/host.go, hostMounts). /proc/1/mounts can list the same mount point more than once, for example a path mounted over itself or a repeated bind mount, and nothing is deduplicated. When device/fstype/mountpoint repeat, Prometheus rejects the duplicate series. With ContinueOnError the rest of the scrape still goes out, but a gather error is logged on every scrape. node_exporter deduplicates these; keeping the last entry per mount point (the visible one) would match it.
  2. Release notes: the KiB fix applies in Kubernetes mode too, so node_resources_memory_* steps up about 2.4% on upgrade. Worth a line, so dashboards and alerts on absolute values aren't surprised.

@mayankpande88
mayankpande88 merged commit 20efed0 into main Oct 9, 2026
@mayankpande88
mayankpande88 deleted the feat/vm-host-metrics branch October 9, 2026 08:50
blue4209211 added a commit that referenced this pull request Oct 9, 2026
* fix(node): report a mount point listed more than once only once

/proc/1/mounts lists a mount point again when something is mounted over it
or bind-mounted again. Identical label sets made every scrape log a gather
error. Keep the last entry: it is the visible mount, the one statfs reads
(#381 review).

* fix(node): filter ignored file systems after picking a mount point's visible mount

An ignored file system (squashfs, proc, ...) mounted over a real one was
dropped before deduplication, so the hidden mount below it was reported
with the top mount's statfs (#389 review).

---------

Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants