Skip to content

Promote develop to master: collector device-type fixes and menu scroll fix - #680

Merged
Starosdev merged 114 commits into
masterfrom
develop
Jul 28, 2026
Merged

Starosdev merged 114 commits into
masterfrom
develop

Conversation

@Starosdev

Copy link
Copy Markdown
Owner

Promotes the current develop tree to master. Same tree is already on beta (afa51031).

Fixes #663.
Fixes #664.
Fixes #672.

What is going to master

  1. Preserve jmb39x-q device type and accept usable SMART JSON (fix(collector): preserve jmb39x-q device type and accept usable SMART JSON #663, PR fix(collector): preserve jmb39x-q device type and accept usable SMART JSON #677)

    • The collector dropped an explicitly configured jmb39x-q,N device type and discarded SMART JSON that smartctl returned alongside a non-zero exit status.
    • Adds webapp/backend/pkg/smartctl/exit_status.go plus detect/config coverage.
  2. Pass explicitly configured smartctl device types verbatim (fix(collector): support explicit device types for Hitachi and Toshiba drives #664, PR fix(collector): pass explicitly configured smartctl device types verbatim #678)

    • Explicit device types for Hitachi and Toshiba drives were rewritten by autodetection instead of being passed through as configured.
    • Covered by collector/pkg/detect/detect_test.go and new smartctl fixtures.
  3. Keep scroll position when a menu closes ([BUG] Overlay scrolling issue #672, PR fix(frontend): keep scroll position when a menu closes (#672) #679)

    • MatMenuTrigger restored focus on close with a bare focus(), so the browser scrolled the trigger back into view and the page jumped to where it was when the menu opened.
    • MenuTriggerRestoreFocusDirective takes focus restoration over and restores it with { preventScroll: true }, keeping keyboard focus return intact.
    • Also registers the mobile layout content as a CDK scroll container, and adds a CI guard so a component that renders a menu trigger without importing the directive fails the frontend job.

Supporting changes: docs/TROUBLESHOOTING_DEVICE_COLLECTOR.md, example.collector.yaml device-type guidance, and the Guard menu triggers step in ci.yaml.

Verification

  • CI green on each source PR (frontend, backend, lint, 8 build targets, Trivy)
  • Frontend: 160 karma specs passing, including a control spec that reproduces the scroll jump without the directive
  • Deployed to the Zeus testing instance from develop-omnibus; the fix was confirmed in the served bundle

Release is manual after this merge: Actions > Release > Run workflow.

dependabot Bot and others added 30 commits May 18, 2026 03:08
Bumps [@angular-eslint/schematics](https://github.com/angular-eslint/angular-eslint/tree/HEAD/packages/schematics) from 21.3.1 to 21.4.0.
- [Release notes](https://github.com/angular-eslint/angular-eslint/releases)
- [Changelog](https://github.com/angular-eslint/angular-eslint/blob/main/packages/schematics/CHANGELOG.md)
- [Commits](https://github.com/angular-eslint/angular-eslint/commits/v21.4.0/packages/schematics)

---
updated-dependencies:
- dependency-name: "@angular-eslint/schematics"
  dependency-version: 21.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [ng-apexcharts](https://github.com/apexcharts/ng-apexcharts) from 2.3.0 to 2.4.0.
- [Release notes](https://github.com/apexcharts/ng-apexcharts/releases)
- [Changelog](https://github.com/apexcharts/ng-apexcharts/blob/master/CHANGELOG.md)
- [Commits](https://github.com/apexcharts/ng-apexcharts/commits/v2.4.0)

---
updated-dependencies:
- dependency-name: ng-apexcharts
  dependency-version: 2.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [eslint-plugin-prettier](https://github.com/prettier/eslint-plugin-prettier) from 4.2.5 to 5.5.5.
- [Release notes](https://github.com/prettier/eslint-plugin-prettier/releases)
- [Changelog](https://github.com/prettier/eslint-plugin-prettier/blob/main/CHANGELOG.md)
- [Commits](prettier/eslint-plugin-prettier@v4.2.5...v5.5.5)

---
updated-dependencies:
- dependency-name: eslint-plugin-prettier
  dependency-version: 5.5.5
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/go-gormigrate/gormigrate/v2](https://github.com/go-gormigrate/gormigrate) from 2.1.5 to 2.1.6.
- [Release notes](https://github.com/go-gormigrate/gormigrate/releases)
- [Changelog](https://github.com/go-gormigrate/gormigrate/blob/master/CHANGELOG.md)
- [Commits](go-gormigrate/gormigrate@v2.1.5...v2.1.6)

---
updated-dependencies:
- dependency-name: github.com/go-gormigrate/gormigrate/v2
  dependency-version: 2.1.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [github.com/nicholas-fedor/shoutrrr](https://github.com/nicholas-fedor/shoutrrr) from 0.14.3 to 0.16.0.
- [Release notes](https://github.com/nicholas-fedor/shoutrrr/releases)
- [Changelog](https://github.com/nicholas-fedor/shoutrrr/blob/main/CHANGELOG.md)
- [Commits](nicholas-fedor/shoutrrr@v0.14.3...v0.16.0)

---
updated-dependencies:
- dependency-name: github.com/nicholas-fedor/shoutrrr
  dependency-version: 0.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
…3 updates

Bumps the dev-dependencies group with 3 updates in the /webapp/frontend directory: [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin), [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser) and [apexcharts](https://github.com/apexcharts/apexcharts.js).


Updates `@typescript-eslint/eslint-plugin` from 8.59.2 to 8.60.1
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.60.1/packages/eslint-plugin)

Updates `@typescript-eslint/parser` from 8.59.2 to 8.60.1
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.60.1/packages/parser)

Updates `apexcharts` from 5.11.0 to 5.14.0
- [Release notes](https://github.com/apexcharts/apexcharts.js/releases)
- [Commits](apexcharts/apexcharts.js@v5.11.0...v5.14.0)

---
updated-dependencies:
- dependency-name: "@typescript-eslint/eslint-plugin"
  dependency-version: 8.60.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: "@typescript-eslint/parser"
  dependency-version: 8.60.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
- dependency-name: apexcharts
  dependency-version: 5.14.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: dev-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Add heartbeatEnabled to NotifyUrlEntry interface, update NotifyUrlService
to map heartbeat_enabled from the API and send it on create/PATCH, and
surface a mat-slide-toggle in the notify-URLs table and add-URL form.
Add HelpLinkIconComponent — a standalone icon-button anchor — and wire it into
the Notification Channels expansion panel title. Also add an inline "Learn more"
link to the Consumer Drive Profiles field hint pointing to CONSUMER_DRIVE_PROFILES.md.
…orkload (#588)

Adds four boolean settings (navigation.show_zfs_pools, show_mdadm, show_btrfs,
show_workload) to hide unused nav links. Configurable via settings UI, scrutiny.yaml,
or env vars (SCRUTINY_WEB_NAVIGATION_SHOW_*). Defaults to true (all visible).
MDADM arrays and Workload devices now render in sections grouped by
host_id, matching the existing ZFS Pools and SMART dashboard pattern.

- Add host_id field to MDADM array across the full stack: collector
  model, backend DB model (migration m20260610000000), registration
  handler, summary API response, and frontend TS model
- Populate host_id from collector config host.id during array detection
- MDADM component builds hostGroups map and renders per-host card grids
  with h3 host headers; header hidden when host_id is empty
- Workload component builds per-host MatTableDataSource instances,
  wires @ViewChildren(MatSort) per group; both desktop table and mobile
  card views render host sections
…github.com/go-gormigrate/gormigrate/v2-2.1.6

chore(deps): bump github.com/go-gormigrate/gormigrate/v2 from 2.1.5 to 2.1.6
…github.com/nicholas-fedor/shoutrrr-0.16.0

chore(deps): bump github.com/nicholas-fedor/shoutrrr from 0.14.3 to 0.16.0
…/frontend/develop/angular-eslint/schematics-21.4.0

chore(deps): bump @angular-eslint/schematics from 21.3.1 to 21.4.0 in /webapp/frontend
…/frontend/develop/ng-apexcharts-2.4.0

chore(deps): bump ng-apexcharts from 2.3.0 to 2.4.0 in /webapp/frontend
…/frontend/develop/eslint-plugin-prettier-5.5.5

chore(deps): bump eslint-plugin-prettier from 4.2.5 to 5.5.5 in /webapp/frontend
…/frontend/develop/dev-dependencies-72597602f0

chore(deps): bump the dev-dependencies group across 1 directory with 3 updates
github-actions Bot and others added 29 commits July 4, 2026 23:41
…0260715

chore(sync): reconcile develop and beta
… JSON (#677)

Fixes #663.

## Summary

QNAP TR-004 enclosures in Individual mode address each bay through the
smartmontools `jmb39x-q,N` backend. Two independent collector defects
made those disks unmonitorable. Both are fixed here.

## Bug A: comma-containing device types were split at config parse time

The argument construction was never at fault. All three `--device` sites
already append the type as a single element, and the shell layer uses
`exec.CommandContext` with no re-tokenization. The damage happened
earlier: viper installs its own `stringToWeakSliceHookFunc(",")` decode
hook, which splits a scalar string when the destination is a slice.
`type: 'jmb39x-q,0'` therefore decoded to the two bogus types `jmb39x-q`
and `0`.

A comma belongs to the smartctl device-type grammar (`megaraid,14`,
`sat,auto`, `jmb39x-q,0`), it is not a separator between types.
`GetDeviceOverrides` now overrides `DecodeHook` for the `devices` key so
`WeaklyTypedInput` lifts a scalar into a one-element slice verbatim. The
list form is unchanged.

Two edge cases that removing the hook opens up are handled by a new
`normalizeDeviceOverrides`: configured types are trimmed, and an
explicitly empty type stays an empty but non-nil slice so
`buildOverrideDeviceGroup` keeps dropping the device rather than falling
back to the scanned type.

## Bug B: any non-zero smartctl exit aborted device identification

`SmartCtlInfo` masked the exit code with `0xBF`, but the tolerant branch
logged "Continuing." and then fell straight into the pre-existing `if
err != nil { return err }`. It was dead code: no non-zero smartctl exit
had ever survived detection, including the bit-6 case the mask was added
for.

These enclosures report `Read Device Statistics page 0x00 failed` and
exit 4 on every run while printing a complete JSON document. The disk
was therefore registered with no model, serial or WWN. The empty WWN is
the actual cause of the reported "No Data": every InfluxDB query filters
on the `device_wwn` tag. This means users on the documented list-form
workaround were still broken, because their `Collect` calls were
publishing fine under the existing `0x03` rule while detection threw the
identity away.

Detection now applies the same rule the collection path and the server's
`validateSmartExitStatus` already used: only bits `0x01` and `0x02` mean
the payload cannot be trusted, since every other bit describes a
condition of the disk rather than of the output. The three copies of
that rule are replaced by a shared `webapp/backend/pkg/smartctl`
package. Detection additionally rejects empty output, so genuinely
unusable results are still refused.

## Behavior changes reviewers should weigh

**Scalar comma semantics.** `type: 'sat,auto'` previously yielded the
two types `sat` and `auto`. It now yields one. Both old outputs were
nonsense as smartctl device types and neither has ever worked, so no
realistic configuration can depend on the old behavior, and no
compatibility path is included. The list form is unaffected;
`TestConfiguration_GetScanOverrides_Raid` is the unchanged proof.

**Affected devices re-register under a new id.** `deviceid` is untouched
and `DeviceType` is not an identity input, so the parsing change cannot
re-id anything by itself. But a drive that is identity-less today
resolves through the `device_name`+`host_id` fallback; once its model,
serial and WWN can be read it hashes differently and appears as a new
device, orphaning the old "No Data" row. This is unavoidable, and is the
desired outcome. The troubleshooting doc tells affected users to delete
the stale entry, whose history is empty-WWN and cannot be merged
usefully.

**This supersedes a documented asymmetry.** The knowledge base recorded
the two exit-code masks as deliberately different and instructed against
unifying them. `git log -S 0xBF` shows the mask arrived as a narrow
bit-6 spot-fix whose lines were inserted above the existing `return
err`, so the documented tolerance never executed. The invariant
described an intent the code never implemented. ADR 0008 records the
decision.

## Tests

13 new tests, all passing.

- Config: comma preservation in scalar and list form, the four-bay
`jmb39x-q` layout, whitespace trimming, and the empty-vs-nil device type
distinction.
- Detect: valid JSON accepted on a non-fatal exit 4 with identity and a
lowercase WWN populated; output rejected on fatal exit 2 with no fields
copied; empty output rejected; `jmb39x-q,0` asserted as one `--device`
argument. The existing `SmartCtlInfo` test used an empty device type, so
that branch had no coverage at all.
- Metrics collector: `--device` argv asserted on both the `--xall` and
FARM calls (the two sites besides detection that must stay identical),
publish on non-fatal exit, and error report without publish on fatal
exit. `Collect` previously had no coverage.

Exit statuses come from a real `*exec.ExitError`, produced by
re-executing the test binary as a helper process rather than shelling
out to `sh -c`, so the tests stay portable.

Verified: `go test ./collector/...`, backend handler exit-status tests,
`go vet`, `gofmt`, `go mod vendor` clean, and `grep 0xBF` returns
nothing.

## Not covered here

No TR-004 hardware was available, so the end-to-end acceptance criterion
is unverified on real hardware. The detect fixture is modeled on a
standard SAT `--info --json` response rather than captured from a real
`jmb39x-q` device; replacing it with sanitized real output would make
the regression test stronger.
…atim (#678)

## Summary

Some Hitachi and Toshiba drives answer an auto-detected `smartctl` query
with no identity, `SMART support is: Unavailable` and no attributes.
Scrutiny renders that as a failed disk with zero metrics. The documented
remedy is to pin the device type in `collector.yaml`; for most of these
drives that type is `sat`, which already reached smartctl. A configured
type of `scsi` or `ata` did not.

All three smartctl call sites (detection `--info`, collection `--xall`,
FARM log) carried a hand-copied guard:

```go
if len(deviceType) > 0 && deviceType != "scsi" && deviceType != "ata" {
    args = append(args, "--device", deviceType)
}
```

The guard is correct for a type that came from `smartctl --scan`:
container scans routinely mislabel ATA disks as SCSI, and forcing `-d
scsi` on one loses its ATA attributes. It is wrong for a type the user
wrote down, but `models.Device` carries no provenance, so the two were
indistinguishable. The comparison was also exact-case, so `type: 'ATA'`
was passed through while `type: 'ata'` was dropped.

## Changes

- New `detect.AppendDeviceTypeArgs` is now the only place a device type
becomes smartctl arguments. All three call sites route through it, so
they cannot drift apart.
- A type configured for that device path in `collector.yaml` is passed
verbatim, including `scsi` and `ata`. A type that is not configured
keeps the existing suppression, now trimmed and case-insensitive.
- Provenance is resolved per call from `Config.GetDeviceOverrides()`
(already on the interface, memoized). It cannot be a field on
`models.Device`: the device type round-trips through
`/api/devices/register` before `Collect` reads it back, so a
collector-only field would not survive.
- `nvme` is unchanged and still passed through.

## Behavior change on upgrade

A `type: 'scsi'` or `type: 'ata'` entry that was a silent no-op in
earlier releases now takes effect. For a drive that was working through
auto-detection, that can lose attributes. The troubleshooting doc calls
this out and tells users to delete the entry if they want
auto-detection. Worth repeating in the release notes.

## Tests

`go test ./collector/...` passes. New coverage:

- `detect_test.go`: configured `sat` reaches the `--info` argv and
populates model, serial, WWN and `smart_support.available`; the
auto-detected response for the same drive yields the empty identity that
produces the false failure; configured `scsi` is now passed; a
configured device that inherits a scanned `scsi` still suppresses it; an
uppercase scanned `SCSI` is suppressed.
- `TestDetect_AppendDeviceTypeArgs`: table over the resolution rules,
including a type configured for a different device path.
- `metrics_collect_test.go`: configured `sat` reaches both the `--xall`
and FARM calls; configured `scsi` is passed; a scanned `scsi` with
nothing configured produces neither `--device`.
- Two new fixtures under `collector/pkg/detect/testdata/`.

## Docs

- `docs/TROUBLESHOOTING_DEVICE_COLLECTOR.md`: new section on drives
reported as failed with zero SMART values, with the `smartctl -a` vs
`smartctl -a -d sat` diagnosis and the upgrade note.
- `example.collector.yaml`: states that a configured type is passed
verbatim.

Fixes #664
Take focus restoration over from MatMenuTrigger and restore it with { preventScroll: true }, so closing a menu no longer scrolls the trigger back into view. Adds a CI guard for components that render menu triggers, and registers the mobile layout content as a CDK scroll container.

Fixes #672.
@Starosdev
Starosdev merged commit 323c295 into master Jul 28, 2026
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants