Skip to content

collector/filesystem: fix readonly false positives under ro rootfs bind mount - #3758

Open
locker95 wants to merge 1 commit into
prometheus:masterfrom
locker95:fix/filesystem-readonly-regression-3755
Open

collector/filesystem: fix readonly false positives under ro rootfs bind mount#3758
locker95 wants to merge 1 commit into
prometheus:masterfrom
locker95:fix/filesystem-readonly-regression-3755

Conversation

@locker95

Copy link
Copy Markdown

Fixes #3755

The regression

After upgrading from 1.11.x to 1.12.0/1.12.1, node_filesystem_readonly reports 1 for host filesystems that are clearly writable, when node_exporter runs in the recommended containerized deployment (-v /:/host:ro,rslave + --path.rootfs=/host).

Root cause

isFilesystemReadOnly() (introduced for #3157/#3484) considers a filesystem read-only when either the per-mount options (mountinfo field 6) or the super options (last mountinfo field) contain ro:

In the recommended containerized deployment the host root is bind-mounted read-only into the container. On current Docker (25.0+/API 1.44+, kernel >= 5.12) ro bind mounts are recursively read-only, so every host mount propagated via rslave carries a per-mount ro flag in the exporter's mount namespace — even though the filesystems are writable on the host. Unless the container also runs with --pid=host (in which case /proc/1/mountinfo provides the host's view), the exporter reads its own namespace's mountinfo and now reports readonly 1 for every host filesystem. That is exactly the reporter's setup in #3755, and the same false positive was previously observed with the statfs-based detection in #2506.

The per-mount ro seen through --path.rootfs describes the exporter's own view of the host mounts (an artifact of the deployment), not the state of the host filesystem.

The fix

For mount points below the configured --path.rootfs prefix, only consider the super options for the read-only decision. The superblock is shared across mount namespaces, so ro/emergency_ro there reliably reflects the state of the host filesystem regardless of how the exporter observes it:

  • Host filesystems observed through a read-only bind mount are no longer falsely reported (superOptions=rw, per-mount ro ignored) — fixes the regression.
  • Truly read-only filesystems are still reported: superblock ro (e.g. mount -o ro, or an ext4 emergency remount triggered by errors=remount-ro) and emergency_ro (Linux 6.15+, filesystem: support ext4 super block emergency_ro flag #3717) are honored for host mounts.
  • Deployments without a rootfs prefix (bare metal, or containers with --pid=host where mountinfo is read from the host's PID 1) are unchanged: the per-mount ro semantics from node_filesystem_readonly does not report readonly bind mounts correctly #3484 (read-only bind mounts of a writable device) are preserved.

Tests

  • New fixture collector/fixtures_bindmount_ro/ mirrors the reporter's environment as seen from the container's mount namespace: /host and /host/var/data with per-mount ro but superblock rw (must be 0), plus a superblock-ro mount and an emergency_ro mount under /host (must stay 1), and a ro bind mount of the container itself outside the rootfs prefix (must stay 1, node_filesystem_readonly does not report readonly bind mounts correctly #3484 semantics).
  • TestPathRootfsReadOnly runs mountPointDetails + isFilesystemReadOnly against that fixture. It fails on current master (/ and /var/data are reported read-only) and passes with this change.
  • New hostMount cases in Test_isFilesystemReadOnly cover the unit-level behavior.

Verified locally with GOOS=linux go build, GOOS=linux go vet and GOOS=linux go test -c (development on a non-Linux host — the Linux test suite runs in CI).

…nd mount

Since 65902fa (prometheus#3659) fixed the missing comma separator in
mountOptionsString, isFilesystemReadOnly actually matches the per-mount
"ro" flag. In the recommended containerized deployment the host root is
bind-mounted read-only into the container (`-v /:/host:ro,rslave` with
--path.rootfs=/host), so every host mount carries a per-mount "ro" flag
in the exporter's mount namespace, and node_filesystem_readonly now
reports 1 for host filesystems that are writable on the host.

Only consider the superblock options, which are shared across mount
namespaces, for mount points below --path.rootfs. Filesystems that
really are read-only (superblock "ro", ext4 "emergency_ro") are still
reported, and mounts outside the rootfs prefix keep the per-mount
semantics from prometheus#3484.

Fixes prometheus#3755

Signed-off-by: Dean Chen <862469039@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

node_filesystem_readonly reports writable filesystem as read-only after upgrade to 1.12.0/1.12.1

1 participant