Repository navigation
feat(zfs): notify on unhealthy pools, error totals and scrub errors - #1131
Merged
Merged
Conversation
The ZFS collector reports pool status, pool-level read, write and checksum error totals and scrub results, but no notification path consumed them. Each metrics upload now compares the uploaded pool with the row stored by the previous upload and notifies when the status changed away from ONLINE (including one unhealthy status worsening to another), an error total grew, or a scrub finished with errors. The previous row is read before the register call because that call replaces the vdev rows. One notification carries every reason found; its failure_type is the most severe one (ZFSPoolDegraded, ZFSDeviceErrors, ZFSScrubErrors). A muted or archived pool never notifies. With repeat notifications enabled a problem still present notifies on every upload. Dispatch goes through the shared notification gate. A read error other than not-found skips the check for that upload rather than counting every existing total as new. The Btrfs path from #1127 and this one share notifyStateChange in state_notify.go, so the mute, settings, repeat and gate handling is written once. Closes #1130 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XyPthdH33ZeTJLDutdTt24
… one ZFSFailureTypes notifies on any status change while the pool is not ONLINE; the docs line claimed only a worsening does. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XyPthdH33ZeTJLDutdTt24
11 of 15 tasks
Starosdev
added a commit
that referenced
this pull request
Nov 5, 2026
…reshold (#1133) ## Summary Adds one setting, `metrics.filesystem_usage_threshold_percent` (0 disables, 1 to 100), and a usage alert for Btrfs filesystems and ZFS pools on top of the per-upload notification path from #1127 and #1131. This is the optional item from #1110. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Closes #1132 ## Changes Made - `models/settings.go`: `FilesystemUsageThresholdPercent` under `metrics`; `MaxFilesystemUsageThresholdPercent = 100`. Zero is the disabled sentinel, so no default is needed and existing installs are unchanged. - `handler/save_settings.go`: rejects values outside 0..100 with a 400. - `notify/btrfs.go`, `notify/zfs.go`: new least-severe failure types `BtrfsUsageHigh` and `ZFSUsageHigh`. `BtrfsUsagePercent` mirrors the dashboard formula (`statfs_used / (statfs_used + statfs_available)`, falling back to `used / device_size`); ZFS uses `capacity_percent`. Shared `usageCrossed` fires at or above the threshold when newly crossed, first seen, or repeating. Both messages now carry a `Usage` line. - `handler/state_notify.go`: settings load on every unsilenced upload (the threshold lives there) instead of after a repeat-only pre-check. Muted and archived rows still return before the load. `failureTypes` takes the threshold. - Frontend: `filesystem_usage_threshold_percent` in `app.config.ts`; number input with validation in the dashboard settings dialog under the temperature settings; Save disabled while invalid via the new `notificationSettingsInvalid` getter. - Docs: Btrfs and ZFS monitoring notification sections, `openapi.yaml` settings schema. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [x] All existing tests pass `go test ./webapp/backend/pkg/web/handler/ ./webapp/backend/pkg/models/ -count=1` ok; `go test ./webapp/backend/pkg/notify/ -run 'Btrfs|ZFS'` ok. - `notify`: 7 new Btrfs and 5 new ZFS usage cases in the failure-type tables (cross, at threshold, stays above with and without repeat, first upload, below, disabled), plus `TestBtrfsUsagePercent`. Full notify package still has the macOS-only `TestScriptStreamsOutputAndPreservesEnvironment` failure noted in #1127; unrelated. - `handler`: `TestSaveSettingsFilesystemUsageThresholdValidation` (0, 1, 100 accepted; -1, 101 rejected). Healthy-upload tests now return settings and assert `GetNotifyUrls` is never called. - Frontend: one new spec case for the threshold validation getter. Not run locally (no `node_modules` here); relies on the `Test Frontend` CI job. - `go build ./webapp/backend/...`, `go vet` and `gofmt -l` clean on the touched packages. Review: not required (lands on `develop`; reaches users through the promotion PR and release). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Screenshots (if applicable) N/A ## Additional Notes Cost: one extra `LoadSettings` (a `setting_entries` table read) per Btrfs or ZFS upload, so once per collector interval per filesystem or pool. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Starosdev
pushed a commit
that referenced
this pull request
Nov 5, 2026
## [1.80.0](v1.79.3...v1.80.0) (2026-11-05) ### Features * **btrfs:** notify on degraded filesystems, device errors and scrub errors ([#1127](#1127)) ([36e1e23](36e1e23)), closes [#1110](#1110) * manual release trigger ([44f3e44](44f3e44)) * **notify:** alert when Btrfs or ZFS usage reaches a configurable threshold ([#1133](#1133)) ([6e79118](6e79118)), closes [#1127](#1127) [#1132](#1132) [#1127](#1127) * **zfs:** notify on unhealthy pools, error totals and scrub errors ([#1131](#1131)) ([9c6544c](9c6544c)), closes [#1127](#1127) [#1130](#1130) [#1127](#1127) ### Bug Fixes * **database:** inline validated ids in Btrfs and ZFS history queries for InfluxDB OSS ([#1111](#1111)) ([#1124](#1124)) ([b0a1c69](b0a1c69)), closes [#157](#157) [#155](#155) [#157](#157) [#157](#157) * **mdadm:** honor repeat notifications and retry failed degraded alerts ([#1128](#1128)) ([62f9cef](62f9cef)), closes [#1109](#1109) * **notify:** notify on manufacturer SMART failures at every notify level ([#1126](#1126)) ([5cb3622](5cb3622))
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The ZFS collector reports pool status, pool-level error totals and scrub results, but nothing turned them into a notification. Each metrics upload now compares the uploaded pool with the row stored by the previous upload and notifies when the status changed away from
ONLINE, an error total grew, or a scrub finished with errors. Sibling of #1127 for Btrfs.Type of Change
Related Issues
Closes #1130
Changes Made
notify/zfs.go:ZFSFailureTypesdecides which problems are new (or still present, with repeat notifications on) andNewZFSNotifybuilds the payload. Failure types by severity:ZFSPoolDegraded(any status other thanONLINE, including one unhealthy status worsening to another),ZFSDeviceErrors(pool-leveltotal_read_errors,total_write_errors,total_checksum_errorsgrew),ZFSScrubErrors(scrub finished withscrub_errors_countabove 0, told apart from the previous scrub byscrub_end_time).device_typeisZFS,device_serialis the pool GUID.handler/zfs_upload_metrics.go: reads the stored row beforeRegisterZFSPool, because that call replaces the vdev rows. Muted or archived pools never notify. A read error other than not-found skips the check for that upload. A healthy upload never loads settings.handler/state_notify.go: newnotifyStateChangeholds the mute, settings, repeat and gate handling once;btrfs_upload_metrics.gonow calls it instead of its own copy. No behaviour change for Btrfs; its existing tests pass unchanged.docs/ZFS_POOL_MONITORING.md; README feature line mentions ZFS alerts.Testing
go test ./webapp/backend/pkg/notify/ -run 'ZFS|Btrfs' -count=1andgo test ./webapp/backend/pkg/web/handler/ -count=1notify:TestZFSFailureTypes(18 cases) andTestNewZFSNotifypass; Btrfs tests unchanged and passing. The full notify package still has the macOS-onlyTestScriptStreamsOutputAndPreservesEnvironmentfailure noted in feat(btrfs): notify on degraded filesystems, device errors and scrub errors #1127, unrelated.handler: whole package ok (35s), including four new ZFS upload tests: healthy upload loads no settings, muted pool skips, unreadable previous state skips, first upload of a degraded pool delivers one webhook withfailure_type=ZFSPoolDegradedafter reading the stored row before registering.go build ./webapp/backend/...andgo veton both packages clean;gofmt -lreports nothing in the touched files.Review: not required (lands on
develop; reaches users through the promotion PR and release).Checklist
Screenshots (if applicable)
N/A
Additional Notes
Vdev-level error matching was left out on purpose: the pool-level totals already move whenever any vdev counter moves, and the notification message points at the pool page for the per-vdev detail.