Repository navigation
Conversation
…errors (#1127) ## Summary The Btrfs collector reports degraded filesystems, per-device error counters and scrub results, but nothing turned them into a notification. Each metrics upload now compares the uploaded state with the row stored by the previous upload and notifies when the filesystem became `DEGRADED`, a device error counter grew, or a scrub finished with errors. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Closes #1110 ## Changes Made - `notify/btrfs.go`: `BtrfsFailureTypes` decides which problems are new (or still present, with repeat notifications on) and `NewBtrfsNotify` builds the payload. Failure types by severity: `BtrfsDegraded`, `BtrfsDeviceErrors`, `BtrfsScrubErrors`. One notification carries every reason found in an upload; `failure_type` is the most severe one, `device_type` is `Btrfs`, `device_serial` is the filesystem UUID. - `handler/btrfs_upload_metrics.go`: reads the stored row before `RegisterBtrfsFilesystem`, because that call deletes and recreates the device rows holding the error counters. Muted or archived filesystems never notify. A healthy upload never loads settings. Dispatch goes through the shared notification gate, so quiet hours, the rate limit and the outbox apply. - `handler/mdadm_upload_metrics.go`: `sendNotificationWithGate` takes the loaded settings instead of loading them itself, so both paths share it. No behaviour change for MDADM. - Docs: new "Notifications" section in `docs/BTRFS_FILESYSTEM_MONITORING.md`; README feature line mentions Btrfs alerts. Skipped from the issue: the optional usage threshold. No threshold setting exists for it today; it can follow as its own change if wanted. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [x] All existing tests pass `go test ./webapp/backend/pkg/notify/ ./webapp/backend/pkg/web/handler/ -count=1` - `handler`: ok (35s), including the three new upload tests: healthy upload loads no settings, muted filesystem skips, ONLINE to DEGRADED delivers one webhook with `failure_type=BtrfsDegraded` and reads the stored row before registering. - `notify`: new `TestBtrfsFailureTypes` (16 cases) and `TestNewBtrfsNotify` pass. `TestScriptStreamsOutputAndPreservesEnvironment` fails on this macOS host both on this branch and on a clean `origin/develop` snapshot; unrelated to this change. - `go build ./webapp/backend/...` and `go vet` on both packages clean; `gofmt -l` reports nothing in the touched files. Review: not required (lands on `develop`; reaches users through the promotion PR and release). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Screenshots (if applicable) N/A ## Additional Notes ZFS pools have the same gap (`zfs_upload_metrics.go` never notifies). Out of scope here; worth a sibling issue. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…1131) ## Summary The ZFS collector reports pool status, pool-level error totals and scrub results, but nothing turned them into a notification. Each metrics upload now compares the uploaded pool with the row stored by the previous upload and notifies when the status changed away from `ONLINE`, an error total grew, or a scrub finished with errors. Sibling of #1127 for Btrfs. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [x] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Closes #1130 ## Changes Made - `notify/zfs.go`: `ZFSFailureTypes` decides which problems are new (or still present, with repeat notifications on) and `NewZFSNotify` builds the payload. Failure types by severity: `ZFSPoolDegraded` (any status other than `ONLINE`, including one unhealthy status worsening to another), `ZFSDeviceErrors` (pool-level `total_read_errors`, `total_write_errors`, `total_checksum_errors` grew), `ZFSScrubErrors` (scrub finished with `scrub_errors_count` above 0, told apart from the previous scrub by `scrub_end_time`). `device_type` is `ZFS`, `device_serial` is the pool GUID. - `handler/zfs_upload_metrics.go`: reads the stored row before `RegisterZFSPool`, because that call replaces the vdev rows. Muted or archived pools never notify. A read error other than not-found skips the check for that upload. A healthy upload never loads settings. - `handler/state_notify.go`: new `notifyStateChange` holds the mute, settings, repeat and gate handling once; `btrfs_upload_metrics.go` now calls it instead of its own copy. No behaviour change for Btrfs; its existing tests pass unchanged. - Docs: new "Notifications" section in `docs/ZFS_POOL_MONITORING.md`; README feature line mentions ZFS alerts. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [x] All existing tests pass `go test ./webapp/backend/pkg/notify/ -run 'ZFS|Btrfs' -count=1` and `go test ./webapp/backend/pkg/web/handler/ -count=1` - `notify`: `TestZFSFailureTypes` (18 cases) and `TestNewZFSNotify` pass; Btrfs tests unchanged and passing. The full notify package still has the macOS-only `TestScriptStreamsOutputAndPreservesEnvironment` failure noted in #1127, unrelated. - `handler`: whole package ok (35s), including four new ZFS upload tests: healthy upload loads no settings, muted pool skips, unreadable previous state skips, first upload of a degraded pool delivers one webhook with `failure_type=ZFSPoolDegraded` after reading the stored row before registering. - `go build ./webapp/backend/...` and `go vet` on both packages clean; `gofmt -l` reports nothing in the touched files. Review: not required (lands on `develop`; reaches users through the promotion PR and release). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Screenshots (if applicable) N/A ## Additional Notes Vdev-level error matching was left out on purpose: the pool-level totals already move whenever any vdev counter moves, and the notification message points at the pool page for the per-vdev detail. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…for InfluxDB OSS (#1111) (#1124) ## Summary The Btrfs filesystem and ZFS pool detail pages never show their usage history charts on InfluxDB OSS. `GetBtrfsMetricsHistory` and `GetZFSPoolMetricsHistory` used Flux query parameters (`params.uuid`, `params.guid`), which only InfluxDB Cloud supports. OSS rejects every call with `undefined identifier params`, the handlers log a warning, and the API returns `metrics_history: null`. `GetZFSPoolMetricsHistory` was fixed in #157 and regressed one day later: merge `98a1b439` resolved a conflict back to the parameterized form. The Btrfs query was added in May with the same construct. Nothing in CI caught either. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Refactoring (no functional changes) - [x] CI/CD or infrastructure change ## Related Issues Fixes #1111 Related to #155, #157 ## Changes Made - `scrutiny_repository_btrfs.go`, `scrutiny_repository_zfs.go`: validate the identifier inside the repository function (`ValidateUUID` / `ValidateGUID`) and inline it as a `strconv.Quote` Flux string literal via `Query`. This is the pattern `GetMdadmMetricsHistory` and the temperature queries already use. Handler-level validation stays as a second layer. - `.golangci.yml`: enable `forbidigo` with a rule that fails lint on `QueryWithParams` / `QueryRawWithParams`, so a future merge-conflict resolution cannot restore the pattern silently. The message points at #157 and #1111. - `scrutiny_repository_metrics_history_test.go`: a recording `QueryAPI` stub asserts the generated query inlines the quoted id and never contains `params.`, and that an invalid id is rejected before any query runs. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [ ] All existing tests pass Local: `go build ./webapp/backend/pkg/database/` and `go vet ./webapp/backend/pkg/database/` pass; `gofmt -l` clean on changed files; pre-commit gofmt and vet hooks pass. The Go test suite and `golangci-lint` run in CI on this PR; I did not run the full suite locally. No Btrfs or ZFS host was available to verify the chart end to end; the issue reporter's reproduction (`metrics_history: null` plus the `undefined identifier params` warning) is the failing path these changes remove. Review: not required (change ships through the develop-to-master promotion PR and a manual release, no direct production effect on merge) ## Follow-up Even with the query working, both charts are capped at the main bucket's 15-day retention: `btrfs_filesystem` and `zfs_pool` are not part of the downsampling tasks, and `lookupBucketName` routes month and year ranges to `_weekly` / `_monthly` buckets that hold no data for them. Not addressed here; an enhancement issue for downsampling the capacity fields and adding a range selector on both detail pages is still to be filed. `beta` carries the same two `params.*` queries (`scrutiny_repository_btrfs.go:215`, `scrutiny_repository_zfs.go:382`) and gets this fix through the next master-into-beta sync. The `forbidigo` rule will flag them there until the sync lands. ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [ ] I have updated the documentation if needed - [x] My changes generate no new warnings
…reshold (#1133) ## Summary Adds one setting, `metrics.filesystem_usage_threshold_percent` (0 disables, 1 to 100), and a usage alert for Btrfs filesystems and ZFS pools on top of the per-upload notification path from #1127 and #1131. This is the optional item from #1110. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Closes #1132 ## Changes Made - `models/settings.go`: `FilesystemUsageThresholdPercent` under `metrics`; `MaxFilesystemUsageThresholdPercent = 100`. Zero is the disabled sentinel, so no default is needed and existing installs are unchanged. - `handler/save_settings.go`: rejects values outside 0..100 with a 400. - `notify/btrfs.go`, `notify/zfs.go`: new least-severe failure types `BtrfsUsageHigh` and `ZFSUsageHigh`. `BtrfsUsagePercent` mirrors the dashboard formula (`statfs_used / (statfs_used + statfs_available)`, falling back to `used / device_size`); ZFS uses `capacity_percent`. Shared `usageCrossed` fires at or above the threshold when newly crossed, first seen, or repeating. Both messages now carry a `Usage` line. - `handler/state_notify.go`: settings load on every unsilenced upload (the threshold lives there) instead of after a repeat-only pre-check. Muted and archived rows still return before the load. `failureTypes` takes the threshold. - Frontend: `filesystem_usage_threshold_percent` in `app.config.ts`; number input with validation in the dashboard settings dialog under the temperature settings; Save disabled while invalid via the new `notificationSettingsInvalid` getter. - Docs: Btrfs and ZFS monitoring notification sections, `openapi.yaml` settings schema. ## Testing - [x] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [x] All existing tests pass `go test ./webapp/backend/pkg/web/handler/ ./webapp/backend/pkg/models/ -count=1` ok; `go test ./webapp/backend/pkg/notify/ -run 'Btrfs|ZFS'` ok. - `notify`: 7 new Btrfs and 5 new ZFS usage cases in the failure-type tables (cross, at threshold, stays above with and without repeat, first upload, below, disabled), plus `TestBtrfsUsagePercent`. Full notify package still has the macOS-only `TestScriptStreamsOutputAndPreservesEnvironment` failure noted in #1127; unrelated. - `handler`: `TestSaveSettingsFilesystemUsageThresholdValidation` (0, 1, 100 accepted; -1, 101 rejected). Healthy-upload tests now return settings and assert `GetNotifyUrls` is never called. - Frontend: one new spec case for the threshold validation getter. Not run locally (no `node_modules` here); relies on the `Test Frontend` CI job. - `go build ./webapp/backend/...`, `go vet` and `gofmt -l` clean on the touched packages. Review: not required (lands on `develop`; reaches users through the promotion PR and release). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Screenshots (if applicable) N/A ## Additional Notes Cost: one extra `LoadSettings` (a `setting_entries` table read) per Btrfs or ZFS upload, so once per collector interval per filesystem or pool. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…1125) ## Summary #1116 reports that the dashboard's **Run collectors** action ignores `COLLECTOR_*_ENABLED`. It does, and that is the correct behaviour. Only the standalone collector omnibus manager reads those variables (`collector/pkg/omnibus/config.go`, `applyEnvironmentOverrides`). That manager ships only in the non-Docker omnibus archive, which has no web server. `TriggerCollectors` runs only in the all-in-one image. There, cron (`rootfs/etc/cont-init.d/50-cron-config`) and the startup services (`rootfs/etc/services.d/collector-*-once/run`) enable optional collectors by schedule and startup flag alone. The manual gate already matches them. If the manual gate honoured `COLLECTOR_ZFS_ENABLED=false`, then in the all-in-one image with a ZFS schedule set, cron would still run ZFS and only the manual run would skip it. That would create the disagreement the issue describes. This PR documents the scope of the variables and pins the current behaviour with tests. It does not change runtime behaviour. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Refactoring (no functional changes) - [ ] CI/CD or infrastructure change ## Related Issues Related to #1116. The issue can be closed as not planned once this lands. No code change is needed. ## Changes Made - `docs/TROUBLESHOOTING_DEVICE_COLLECTOR.md`: states that `COLLECTOR_*_ENABLED` applies only to the standalone omnibus manager, and explains how to disable an optional collector in the all-in-one image. - `docs/INSTALL_COLLECTOR_OMNIBUS.md`: states that the Docker images ignore the `*_ENABLED` variables. - `collector_run.go`: a comment at `manualCollectorEnabled` explains why `spec.EnabledEnv` is not read. - `collector_run_test.go`: two cases. An explicit `false` with a schedule set still runs ZFS, MDADM and SMART. An explicit `true` with no schedule still skips the optional collectors. ## Testing - [ ] I have tested this locally - [x] I have added/updated tests that prove my fix/feature works - [ ] All existing tests pass The pre-commit hook passed `gofmt` and `go vet`. The test suite was not run locally. CI runs it. The new cases pin existing behaviour, so they pass on the current code and fail only if a later change starts honouring `*_ENABLED` in the manual gate. Evidence checked on `master`, `beta` and `develop`: no collector `*_ENABLED` variable is read in `rootfs`, `docker/entrypoint-*`, the Unraid templates or the compose examples. The collector binaries' Viper `COLLECTOR_` prefix maps no `*.enabled` key to these names. `Dockerfile.collector-omnibus` uses cron and does not ship the manager binary. Review: not required (documentation and tests only; no runtime, security, contract or data change). ## Checklist - [x] My code follows the project's code style - [x] I have performed a self-review of my code - [x] I have commented my code where necessary (particularly complex areas) - [x] I have updated the documentation if needed - [x] My changes generate no new warnings - [x] No console.log, debug statements, or commented-out code ## Additional Notes Making the all-in-one image honour `COLLECTOR_*_ENABLED` everywhere (cron, startup services and the manual gate) is possible as a separate feature. It is out of scope here because it changes behaviour for existing deployments. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ts (#1128) Records the degradation last announced per array in an in-memory MDADMTracker, only after TrySend reports delivery or queueing, so failed deliveries retry on the next collection. Repeat Notifications on notifies every collection; off notifies on new or worsening degradation. Removes the InfluxDB history lookup from the notification path. sendNotificationWithGate now returns the delivery result. Documents the behavior and permanently degraded Synology md0/md1 arrays. Fixes #1109
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Product changes
Summary
Btrfs filesystems now send a notification when they become degraded, when a device error counter grows, or when a scrub finishes with errors.
ZFS pools now send a notification when their status changes away from
ONLINE, when a pool error total grows, or when a scrub finishes with errors.New optional usage alert for Btrfs filesystems and ZFS pools. Set
metrics.filesystem_usage_threshold_percent(1 to 100) to be notified when usage reaches that percent. The default is 0, which keeps the alert off.Manufacturer SMART failures now send a notification at every notify level, including Warn. smartctl's
when_failedvalues are now matched correctly in 1.x.Degraded MDADM arrays now follow the Repeat Notifications setting, and a degraded alert that fails to deliver is retried until it is sent.
Btrfs and ZFS usage history charts now load on InfluxDB OSS. Before this fix the history queries failed with
undefined identifier params.Related Issues
Fixes #1108
Fixes #1109
Fixes #1110
Fixes #1111
Fixes #1130
Fixes #1132
Reason
This promotion carries seven
developcommits tomaster:feat(btrfs)): Btrfs problem notifications. Closes [FEAT] Notifications for Btrfs filesystem problems (degraded, device errors, scrub errors) #1110.feat(zfs)): ZFS pool problem notifications. Closes [FEAT] Notifications for ZFS pool problems (status, error totals, scrub errors) #1130.feat(notify)): configurable Btrfs and ZFS usage threshold alert, built on the per-upload notification path from feat(btrfs): notify on degraded filesystems, device errors and scrub errors #1127 and feat(zfs): notify on unhealthy pools, error totals and scrub errors #1131. Closes [FEAT] Notify when Btrfs filesystem or ZFS pool usage passes a configurable threshold #1132.fix(notify)): manufacturer SMART failures notify at every notify level. Fixes [BUG] Manufacturer SMART failures don't notify at notify level "Warn": when_failed never matches in 1.x, and Warn ignores the device-level SMART failure #1108.fix(mdadm)): MDADM notifications honor Repeat Notifications and retry failed deliveries. Fixes [BUG] MDADM degraded notification is sent only once: ignores "Repeat Notifications" and is lost if delivery fails #1109.fix(database)): Btrfs and ZFS history queries inline validated ids instead of Flux parameters, which only InfluxDB Cloud supports. Fixes [BUG] Btrfs usage history is never shown: parameterized Flux query fails on InfluxDB OSS ("undefined identifier params"), same as #155 #1111.docs(collectors)): documents thatCOLLECTOR_*_ENABLEDapplies only to the standalone omnibus manager. No behaviour change. Manual collector runs ignore COLLECTOR_*_ENABLED #1116 was closed as not planned.Validation
develop.master(thechore(release): 1.79.3commit) was merged intodevelopasd30d8250to satisfy strict branch protection. That merge changed onlyCHANGELOG.mdandversion.go(to 1.79.3).Review: not required (each change was reviewed and merged into
develop; merging this PR publishes nothing until the release workflow runs).Follow-up
masteronce this merges. Three new features (feat(btrfs): notify on degraded filesystems, device errors and scrub errors #1127, feat(zfs): notify on unhealthy pools, error totals and scrub errors #1131, feat(notify): alert when Btrfs or ZFS usage reaches a configurable threshold #1133) make this a minor release.