Repository navigation
feat(btrfs): notify on degraded filesystems, device errors and scrub errors - #1127
Merged
Merged
Conversation
…errors The Btrfs collector already reports a degraded filesystem, per-device error counters and scrub results, but no notification path consumed them. A missing device or a scrub that found checksum errors was only visible on the Btrfs page. Each metrics upload now compares the uploaded state with the row stored by the previous upload and notifies when the filesystem became DEGRADED, a device error counter grew, or a scrub finished with errors. The previous row is read before the register call because that call deletes and recreates the device rows that hold the counters. One notification carries every reason found; its failure_type is the most severe one (BtrfsDegraded, BtrfsDeviceErrors, BtrfsScrubErrors). A muted or archived filesystem never notifies. With repeat notifications enabled a problem still present notifies on every upload. Dispatch goes through the shared notification gate, so quiet hours, the rate limit and the outbox apply. sendNotificationWithGate now takes the loaded settings instead of loading them itself, so the Btrfs path can read the repeat setting once and share the helper with the MDADM path. Closes #1110 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XyPthdH33ZeTJLDutdTt24
…readable GetBtrfsFilesystemDetails returns the row together with an error when only the devices query fails. Treating that as "no previous state" would count every existing device error counter as new and send one spurious notification. Only a not-found error now means a first upload; any other read error skips the notification check for that upload and the next one compares against a complete row again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XyPthdH33ZeTJLDutdTt24
This was referenced Nov 5, 2026
Starosdev
added a commit
that referenced
this pull request
Nov 5, 2026
…1131) ## Summary The ZFS collector reports pool status, pool-level error totals and scrub results, but nothing turned them into a notification. Each metrics upload now compares the uploaded pool with the row stored by the previous upload and notifies when the status changed away from `ONLINE`, an error total grew, or a scrub finished with errors. Sibling of #1127 for Btrfs. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Closes #1130 ## Changes Made - `notify/zfs.go`: `ZFSFailureTypes` decides which problems are new (or still present, with repeat notifications on) and `NewZFSNotify` builds the payload. Failure types by severity: `ZFSPoolDegraded` (any status other than `ONLINE`, including one unhealthy status worsening to another), `ZFSDeviceErrors` (pool-level `total_read_errors`, `total_write_errors`, `total_checksum_errors` grew), `ZFSScrubErrors` (scrub finished with `scrub_errors_count` above 0, told apart from the previous scrub by `scrub_end_time`). `device_type` is `ZFS`, `device_serial` is the pool GUID. - `handler/zfs_upload_metrics.go`: reads the stored row before `RegisterZFSPool`, because that call replaces the vdev rows. Muted or archived pools never notify. A read error other than not-found skips the check for that upload. A healthy upload never loads settings. - `handler/state_notify.go`: new `notifyStateChange` holds the mute, settings, repeat and gate handling once; `btrfs_upload_metrics.go` now calls it instead of its own copy. No behaviour change for Btrfs; its existing tests pass unchanged. - Docs: new "Notifications" section in `docs/ZFS_POOL_MONITORING.md`; README feature line mentions ZFS alerts. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [x] All existing tests pass `go test ./webapp/backend/pkg/notify/ -run 'ZFS|Btrfs' -count=1` and `go test ./webapp/backend/pkg/web/handler/ -count=1` - `notify`: `TestZFSFailureTypes` (18 cases) and `TestNewZFSNotify` pass; Btrfs tests unchanged and passing. The full notify package still has the macOS-only `TestScriptStreamsOutputAndPreservesEnvironment` failure noted in #1127, unrelated. - `handler`: whole package ok (35s), including four new ZFS upload tests: healthy upload loads no settings, muted pool skips, unreadable previous state skips, first upload of a degraded pool delivers one webhook with `failure_type=ZFSPoolDegraded` after reading the stored row before registering. - `go build ./webapp/backend/...` and `go vet` on both packages clean; `gofmt -l` reports nothing in the touched files. Review: not required (lands on `develop`; reaches users through the promotion PR and release). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Screenshots (if applicable) N/A ## Additional Notes Vdev-level error matching was left out on purpose: the pool-level totals already move whenever any vdev counter moves, and the notification message points at the pool page for the per-vdev detail. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
11 of 15 tasks
Starosdev
added a commit
that referenced
this pull request
Nov 5, 2026
…reshold (#1133) ## Summary Adds one setting, `metrics.filesystem_usage_threshold_percent` (0 disables, 1 to 100), and a usage alert for Btrfs filesystems and ZFS pools on top of the per-upload notification path from #1127 and #1131. This is the optional item from #1110. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Closes #1132 ## Changes Made - `models/settings.go`: `FilesystemUsageThresholdPercent` under `metrics`; `MaxFilesystemUsageThresholdPercent = 100`. Zero is the disabled sentinel, so no default is needed and existing installs are unchanged. - `handler/save_settings.go`: rejects values outside 0..100 with a 400. - `notify/btrfs.go`, `notify/zfs.go`: new least-severe failure types `BtrfsUsageHigh` and `ZFSUsageHigh`. `BtrfsUsagePercent` mirrors the dashboard formula (`statfs_used / (statfs_used + statfs_available)`, falling back to `used / device_size`); ZFS uses `capacity_percent`. Shared `usageCrossed` fires at or above the threshold when newly crossed, first seen, or repeating. Both messages now carry a `Usage` line. - `handler/state_notify.go`: settings load on every unsilenced upload (the threshold lives there) instead of after a repeat-only pre-check. Muted and archived rows still return before the load. `failureTypes` takes the threshold. - Frontend: `filesystem_usage_threshold_percent` in `app.config.ts`; number input with validation in the dashboard settings dialog under the temperature settings; Save disabled while invalid via the new `notificationSettingsInvalid` getter. - Docs: Btrfs and ZFS monitoring notification sections, `openapi.yaml` settings schema. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [x] All existing tests pass `go test ./webapp/backend/pkg/web/handler/ ./webapp/backend/pkg/models/ -count=1` ok; `go test ./webapp/backend/pkg/notify/ -run 'Btrfs|ZFS'` ok. - `notify`: 7 new Btrfs and 5 new ZFS usage cases in the failure-type tables (cross, at threshold, stays above with and without repeat, first upload, below, disabled), plus `TestBtrfsUsagePercent`. Full notify package still has the macOS-only `TestScriptStreamsOutputAndPreservesEnvironment` failure noted in #1127; unrelated. - `handler`: `TestSaveSettingsFilesystemUsageThresholdValidation` (0, 1, 100 accepted; -1, 101 rejected). Healthy-upload tests now return settings and assert `GetNotifyUrls` is never called. - Frontend: one new spec case for the threshold validation getter. Not run locally (no `node_modules` here); relies on the `Test Frontend` CI job. - `go build ./webapp/backend/...`, `go vet` and `gofmt -l` clean on the touched packages. Review: not required (lands on `develop`; reaches users through the promotion PR and release). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Screenshots (if applicable) N/A ## Additional Notes Cost: one extra `LoadSettings` (a `setting_entries` table read) per Btrfs or ZFS upload, so once per collector interval per filesystem or pool. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Starosdev
pushed a commit
that referenced
this pull request
Nov 5, 2026
## [1.80.0](v1.79.3...v1.80.0) (2026-11-05) ### Features * **btrfs:** notify on degraded filesystems, device errors and scrub errors ([#1127](#1127)) ([36e1e23](36e1e23)), closes [#1110](#1110) * manual release trigger ([44f3e44](44f3e44)) * **notify:** alert when Btrfs or ZFS usage reaches a configurable threshold ([#1133](#1133)) ([6e79118](6e79118)), closes [#1127](#1127) [#1132](#1132) [#1127](#1127) * **zfs:** notify on unhealthy pools, error totals and scrub errors ([#1131](#1131)) ([9c6544c](9c6544c)), closes [#1127](#1127) [#1130](#1130) [#1127](#1127) ### Bug Fixes * **database:** inline validated ids in Btrfs and ZFS history queries for InfluxDB OSS ([#1111](#1111)) ([#1124](#1124)) ([b0a1c69](b0a1c69)), closes [#157](#157) [#155](#155) [#157](#157) [#157](#157) * **mdadm:** honor repeat notifications and retry failed degraded alerts ([#1128](#1128)) ([62f9cef](62f9cef)), closes [#1109](#1109) * **notify:** notify on manufacturer SMART failures at every notify level ([#1126](#1126)) ([5cb3622](5cb3622))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Btrfs collector reports degraded filesystems, per-device error counters and scrub results, but nothing turned them into a notification. Each metrics upload now compares the uploaded state with the row stored by the previous upload and notifies when the filesystem became
DEGRADED, a device error counter grew, or a scrub finished with errors.Type of Change
Related Issues
Closes #1110
Changes Made
notify/btrfs.go:BtrfsFailureTypesdecides which problems are new (or still present, with repeat notifications on) andNewBtrfsNotifybuilds the payload. Failure types by severity:BtrfsDegraded,BtrfsDeviceErrors,BtrfsScrubErrors. One notification carries every reason found in an upload;failure_typeis the most severe one,device_typeisBtrfs,device_serialis the filesystem UUID.handler/btrfs_upload_metrics.go: reads the stored row beforeRegisterBtrfsFilesystem, because that call deletes and recreates the device rows holding the error counters. Muted or archived filesystems never notify. A healthy upload never loads settings. Dispatch goes through the shared notification gate, so quiet hours, the rate limit and the outbox apply.handler/mdadm_upload_metrics.go:sendNotificationWithGatetakes the loaded settings instead of loading them itself, so both paths share it. No behaviour change for MDADM.docs/BTRFS_FILESYSTEM_MONITORING.md; README feature line mentions Btrfs alerts.Skipped from the issue: the optional usage threshold. No threshold setting exists for it today; it can follow as its own change if wanted.
Testing
go test ./webapp/backend/pkg/notify/ ./webapp/backend/pkg/web/handler/ -count=1handler: ok (35s), including the three new upload tests: healthy upload loads no settings, muted filesystem skips, ONLINE to DEGRADED delivers one webhook withfailure_type=BtrfsDegradedand reads the stored row before registering.notify: newTestBtrfsFailureTypes(16 cases) andTestNewBtrfsNotifypass.TestScriptStreamsOutputAndPreservesEnvironmentfails on this macOS host both on this branch and on a cleanorigin/developsnapshot; unrelated to this change.go build ./webapp/backend/...andgo veton both packages clean;gofmt -lreports nothing in the touched files.Review: not required (lands on
develop; reaches users through the promotion PR and release).Checklist
Screenshots (if applicable)
N/A
Additional Notes
ZFS pools have the same gap (
zfs_upload_metrics.gonever notifies). Out of scope here; worth a sibling issue.