Skip to content

feat(zfs): notify on unhealthy pools, error totals and scrub errors - #1131

Merged
Starosdev merged 2 commits into
developfrom
feat/1130-zfs-notifications
Nov 5, 2026
Merged

Starosdev merged 2 commits into
developfrom
feat/1130-zfs-notifications

Conversation

@Starosdev

Copy link
Copy Markdown
Owner

Summary

The ZFS collector reports pool status, pool-level error totals and scrub results, but nothing turned them into a notification. Each metrics upload now compares the uploaded pool with the row stored by the previous upload and notifies when the status changed away from ONLINE, an error total grew, or a scrub finished with errors. Sibling of #1127 for Btrfs.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or infrastructure change

Related Issues

Closes #1130

Changes Made

  • notify/zfs.go: ZFSFailureTypes decides which problems are new (or still present, with repeat notifications on) and NewZFSNotify builds the payload. Failure types by severity: ZFSPoolDegraded (any status other than ONLINE, including one unhealthy status worsening to another), ZFSDeviceErrors (pool-level total_read_errors, total_write_errors, total_checksum_errors grew), ZFSScrubErrors (scrub finished with scrub_errors_count above 0, told apart from the previous scrub by scrub_end_time). device_type is ZFS, device_serial is the pool GUID.
  • handler/zfs_upload_metrics.go: reads the stored row before RegisterZFSPool, because that call replaces the vdev rows. Muted or archived pools never notify. A read error other than not-found skips the check for that upload. A healthy upload never loads settings.
  • handler/state_notify.go: new notifyStateChange holds the mute, settings, repeat and gate handling once; btrfs_upload_metrics.go now calls it instead of its own copy. No behaviour change for Btrfs; its existing tests pass unchanged.
  • Docs: new "Notifications" section in docs/ZFS_POOL_MONITORING.md; README feature line mentions ZFS alerts.

Testing

  • I have tested this locally
  • I have added/updated tests that prove my fix/feature works
  • All existing tests pass

go test ./webapp/backend/pkg/notify/ -run 'ZFS|Btrfs' -count=1 and go test ./webapp/backend/pkg/web/handler/ -count=1

  • notify: TestZFSFailureTypes (18 cases) and TestNewZFSNotify pass; Btrfs tests unchanged and passing. The full notify package still has the macOS-only TestScriptStreamsOutputAndPreservesEnvironment failure noted in feat(btrfs): notify on degraded filesystems, device errors and scrub errors #1127, unrelated.
  • handler: whole package ok (35s), including four new ZFS upload tests: healthy upload loads no settings, muted pool skips, unreadable previous state skips, first upload of a degraded pool delivers one webhook with failure_type=ZFSPoolDegraded after reading the stored row before registering.
  • go build ./webapp/backend/... and go vet on both packages clean; gofmt -l reports nothing in the touched files.

Review: not required (lands on develop; reaches users through the promotion PR and release).

Checklist

  • My code follows the project's code style
  • I have performed a self-review of my code
  • I have commented my code where necessary (particularly complex areas)
  • I have updated the documentation if needed
  • My changes generate no new warnings
  • No console.log, debug statements, or commented-out code

Screenshots (if applicable)

N/A

Additional Notes

Vdev-level error matching was left out on purpose: the pool-level totals already move whenever any vdev counter moves, and the notification message points at the pool page for the per-vdev detail.

The ZFS collector reports pool status, pool-level read, write and checksum
error totals and scrub results, but no notification path consumed them.
Each metrics upload now compares the uploaded pool with the row stored by
the previous upload and notifies when the status changed away from ONLINE
(including one unhealthy status worsening to another), an error total grew,
or a scrub finished with errors. The previous row is read before the
register call because that call replaces the vdev rows.

One notification carries every reason found; its failure_type is the most
severe one (ZFSPoolDegraded, ZFSDeviceErrors, ZFSScrubErrors). A muted or
archived pool never notifies. With repeat notifications enabled a problem
still present notifies on every upload. Dispatch goes through the shared
notification gate. A read error other than not-found skips the check for
that upload rather than counting every existing total as new.

The Btrfs path from #1127 and this one share notifyStateChange in
state_notify.go, so the mute, settings, repeat and gate handling is written
once.

Closes #1130

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XyPthdH33ZeTJLDutdTt24
… one

ZFSFailureTypes notifies on any status change while the pool is not ONLINE;
the docs line claimed only a worsening does.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XyPthdH33ZeTJLDutdTt24
@Starosdev
Starosdev merged commit 9c6544c into develop Nov 5, 2026
19 checks passed
Starosdev added a commit that referenced this pull request Nov 5, 2026
…reshold (#1133)

## Summary

Adds one setting, `metrics.filesystem_usage_threshold_percent` (0
disables, 1 to 100), and a usage alert for Btrfs filesystems and ZFS
pools on top of the per-upload notification path from #1127 and #1131.
This is the optional item from #1110.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Refactoring (no functional changes)
- [ ] CI/CD or infrastructure change

## Related Issues

Closes #1132

## Changes Made

- `models/settings.go`: `FilesystemUsageThresholdPercent` under
`metrics`; `MaxFilesystemUsageThresholdPercent = 100`. Zero is the
disabled sentinel, so no default is needed and existing installs are
unchanged.
- `handler/save_settings.go`: rejects values outside 0..100 with a 400.
- `notify/btrfs.go`, `notify/zfs.go`: new least-severe failure types
`BtrfsUsageHigh` and `ZFSUsageHigh`. `BtrfsUsagePercent` mirrors the
dashboard formula (`statfs_used / (statfs_used + statfs_available)`,
falling back to `used / device_size`); ZFS uses `capacity_percent`.
Shared `usageCrossed` fires at or above the threshold when newly
crossed, first seen, or repeating. Both messages now carry a `Usage`
line.
- `handler/state_notify.go`: settings load on every unsilenced upload
(the threshold lives there) instead of after a repeat-only pre-check.
Muted and archived rows still return before the load. `failureTypes`
takes the threshold.
- Frontend: `filesystem_usage_threshold_percent` in `app.config.ts`;
number input with validation in the dashboard settings dialog under the
temperature settings; Save disabled while invalid via the new
`notificationSettingsInvalid` getter.
- Docs: Btrfs and ZFS monitoring notification sections, `openapi.yaml`
settings schema.

## Testing

- [x] I have tested this locally
- [x] I have added/updated tests that prove my fix/feature works
- [x] All existing tests pass

`go test ./webapp/backend/pkg/web/handler/ ./webapp/backend/pkg/models/
-count=1` ok; `go test ./webapp/backend/pkg/notify/ -run 'Btrfs|ZFS'`
ok.

- `notify`: 7 new Btrfs and 5 new ZFS usage cases in the failure-type
tables (cross, at threshold, stays above with and without repeat, first
upload, below, disabled), plus `TestBtrfsUsagePercent`. Full notify
package still has the macOS-only
`TestScriptStreamsOutputAndPreservesEnvironment` failure noted in #1127;
unrelated.
- `handler`: `TestSaveSettingsFilesystemUsageThresholdValidation` (0, 1,
100 accepted; -1, 101 rejected). Healthy-upload tests now return
settings and assert `GetNotifyUrls` is never called.
- Frontend: one new spec case for the threshold validation getter. Not
run locally (no `node_modules` here); relies on the `Test Frontend` CI
job.
- `go build ./webapp/backend/...`, `go vet` and `gofmt -l` clean on the
touched packages.

Review: not required (lands on `develop`; reaches users through the
promotion PR and release).

## Checklist

- [x] My code follows the project's code style
- [x] I have performed a self-review of my code
- [x] I have commented my code where necessary (particularly complex
areas)
- [x] I have updated the documentation if needed
- [x] My changes generate no new warnings
- [x] No console.log, debug statements, or commented-out code

## Screenshots (if applicable)

N/A

## Additional Notes

Cost: one extra `LoadSettings` (a `setting_entries` table read) per
Btrfs or ZFS upload, so once per collector interval per filesystem or
pool.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Starosdev pushed a commit that referenced this pull request Nov 5, 2026
## [1.80.0](v1.79.3...v1.80.0) (2026-11-05)

### Features

* **btrfs:** notify on degraded filesystems, device errors and scrub errors ([#1127](#1127)) ([36e1e23](36e1e23)), closes [#1110](#1110)
* manual release trigger ([44f3e44](44f3e44))
* **notify:** alert when Btrfs or ZFS usage reaches a configurable threshold ([#1133](#1133)) ([6e79118](6e79118)), closes [#1127](#1127) [#1132](#1132) [#1127](#1127)
* **zfs:** notify on unhealthy pools, error totals and scrub errors ([#1131](#1131)) ([9c6544c](9c6544c)), closes [#1127](#1127) [#1130](#1130) [#1127](#1127)

### Bug Fixes

* **database:** inline validated ids in Btrfs and ZFS history queries for InfluxDB OSS ([#1111](#1111)) ([#1124](#1124)) ([b0a1c69](b0a1c69)), closes [#157](#157) [#155](#155) [#157](#157) [#157](#157)
* **mdadm:** honor repeat notifications and retry failed degraded alerts ([#1128](#1128)) ([62f9cef](62f9cef)), closes [#1109](#1109)
* **notify:** notify on manufacturer SMART failures at every notify level ([#1126](#1126)) ([5cb3622](5cb3622))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant