Repository navigation
feat: host filesystem, load and swap metrics in standalone mode; fix meminfo KiB - #381
Merged
Merged
Conversation
memoryInfo multiplied /proc/meminfo values by 1000, but the kernel's kB there are 1024 bytes, so node_resources_memory_* read about 2.3% low (upstream has the same bug). They now match node_exporter's node_memory_*_bytes.
Without Kubernetes there is usually no node_exporter, and the agent had
no filesystem space (only volumes a workload writes to), load or swap
metrics. In standalone mode the node collector now exports them under
node_exporter's names: node_filesystem_{size,free,avail}_bytes,
node_filesystem_files{,_free} and node_filesystem_readonly per mount
(pid 1's mounts, without node_exporter's default pseudo and container
filesystems), node_load{1,5,15} and node_memory_Swap{Total,Free}_bytes.
statfs runs with a timeout, and a mount that hangs is skipped for 5
minutes, so a dead network mount cannot block a scrape. On Kubernetes
these stay off: node_exporter usually runs there and sums would count
the same host twice.
There was a problem hiding this comment.
Code Review
This pull request introduces host metrics collection (load average, swap memory, and filesystem statistics) under node_exporter names for standalone/VM environments where node_exporter is not running. It also corrects the memory unit multiplier from 1000 to 1024 (KiB) in /proc/meminfo parsing. The review feedback focuses on improving the robustness of the statfs timeout mechanism by refactoring it into a method on hostCollector and tracking active statfs calls to prevent spawning duplicate goroutines and leaking them when network mounts are hung.
A mount whose statfs hung was retried every 5 minutes, and each retry started another goroutine while the previous one was still blocked in the kernel, holding an OS thread: a dead network mount leaked one thread every 5 minutes. A second statfs is no longer started while one is in flight for that mount point, and a successful call clears the mount's stuck mark.
Contributor
|
LGTM. The meminfo KiB fix is correct, and the statfs guard (timeout, one call in flight per mount, 5-minute skip) is right. Two notes:
|
blue4209211
approved these changes
Oct 9, 2026
This was referenced Oct 9, 2026
blue4209211
added a commit
that referenced
this pull request
Oct 9, 2026
* fix(node): report a mount point listed more than once only once /proc/1/mounts lists a mount point again when something is mounted over it or bind-mounted again. Identical label sets made every scrape log a gather error. Keep the last entry: it is the visible mount, the one statfs reads (#381 review). * fix(node): filter ignored file systems after picking a mount point's visible mount An ignored file system (squashfs, proc, ...) mounted over a real one was dropped before deduplication, so the hidden mount below it was reported with the top mount's statfs (#389 review). --------- Co-authored-by: Shiv <3078106+blue4209211@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Host metrics for hosts without node_exporter, plus a fix to the existing memory metrics.
Standalone mode (no Kubernetes): the node collector exports the host's filesystem space on every mount, load average and swap, under node_exporter's names:
node_filesystem_{size,free,avail}_bytes,node_filesystem_files,node_filesystem_files_freeandnode_filesystem_readonly(labelsdevice,fstype,mountpoint);node_load1,node_load5,node_load15;node_memory_SwapTotal_bytes,node_memory_SwapFree_bytes.Before this, the agent had no filesystem space except for volumes a workload writes to, and no load or swap.
Kubernetes: these stay off. node_exporter usually runs there, and sums by instance would count the same host twice.
/proc/meminfounits: the "kB" there are KiB, but values were multiplied by 1000, sonode_resources_memory_*read about 2.3% low (upstream has the same bug). They now report correct values, about 2.4% higher than before, matching node_exporter'snode_memory_*_bytes.Engineering detail
Mounts:
/proc/1/mounts, with node_exporter's default exclusions:/dev,/proc,/sys,/run/credentials/...,/var/lib/docker/...and/var/lib/containers/storage/....\040for a space) are decoded, and read-only mounts are reported.statfs:
/proc/1/root/<mountpoint>, with a 2s timeout.TestStatfsHungMountSingleCallcovers it.Where it's switched on: standalone mode is when the IP resolver is the VM resolver (no Kubernetes API).
Tests: load average parsing, mount filtering, escapes and read-only flags, and swap plus the KiB multiplier in the memory test.
CI: gofmt, goimports, vet, golangci-lint,
go test(excluding/containers) and the build all pass in a Linux container with Go 1.26.5.Local e2e: I built agent binaries from this branch and from main and ran both as systemd services on a local Debian 12 VM (kernel 6.1), with a 256 MiB swapfile.
df -B1exactly (12468432896 / 8699375616)./,/boot/efi,/run,/run/lock,/run/user/501and the VM's two host shares were reported;/dev/shmwas left out, as in node_exporter.node_load*matched/proc/loadavg, and swap matched/proc/meminfo.node_resources_memory_total_byteswas 2081710080 (= MemTotal × 1024) on this branch and 2032920000 on main.node_filesystem_*ornode_load*.