Add disconnected drive visibility and stale-device lifecycle management #533

Problem

As drives are swapped or retired, Scrutiny keeps their device records indefinitely. That preserves history, but it creates two practical problems:

1. There is no clear dashboard state showing that a drive is no longer connected or has not been seen recently.
2. Cleaning up stale devices is manual and tedious because the operator has to inspect drives individually and archive or delete them one at a time.

The backend already has manual archive, unarchive, and delete APIs, and it already computes device last-seen timestamps for missed-ping notifications. What is missing is a first-class stale/disconnected device lifecycle in the main summary API and dashboard.

Current State

• Manual device lifecycle actions already exist:
  • POST /api/device/:id/archive
  • POST /api/device/:id/unarchive
  • DELETE /api/device/:id
• The backend already queries the most recent SMART submission time per device through GetDevicesLastSeenTimes().
• The missed-ping monitor already uses that last-seen data to determine when a device has stopped reporting.
• The dashboard summary does not expose last_seen, stale, or disconnected state.
• There is no bulk lifecycle action and no policy-based auto-archive or auto-delete setting.

Proposed Scope

Implement this in phases, with visibility first and destructive automation last.

Phase 1: Expose disconnected/stale state in the summary API and dashboard

Extend the main device summary response so each device can surface:

• last_seen_at
• stale or disconnected
• stale_for_minutes or stale_for_days
• optionally stale_threshold_minutes for UI explanation

Use the existing last-seen query path instead of introducing a new persistence model.

Compute stale state using the same timeout logic already used by missed-ping monitoring:

• global metrics.missed_ping_timeout_minutes
• optional per-device missed_ping_timeout_override

Dashboard updates:

• show a clear badge/state such as Disconnected
• show a human-readable last seen value such as Last seen 18 days ago
• add filters for Active, Disconnected, and Archived
• keep archived devices visually distinct from merely disconnected devices

Phase 2: Add manual bulk stale-device actions

Add operator-facing actions to reduce one-by-one cleanup:

• archive all stale devices older than X days
• optionally delete all stale devices older than X days
• support dry-run or preview behavior before the action executes

This can be exposed either as dedicated endpoints or as a small lifecycle endpoint set.

Recommended first pass:

• implement bulk archive first
• treat bulk delete as optional or follow-up work

Phase 3: Add policy-based auto-archive for stale devices

Add new settings for automatic stale-device cleanup, for example:

• metrics.auto_archive_stale_devices
• metrics.auto_archive_stale_after_days

Run the archive check from the existing monitor loop or a closely related lifecycle job.

Recommended constraints:

• archive only in the first implementation
• never auto-delete in the same first pass
• skip already archived devices
• preserve history by default

Phase 4: Consider auto-delete only after archive behavior is proven

If automatic deletion is added later, it should be a separate retention phase, for example:

• archive after 30 days stale
• delete after 180 days archived

This should be treated as higher risk because deletion removes both SQLite metadata and historical InfluxDB data.

Recommended Implementation Order

1. Add last_seen and disconnected/stale fields to the summary API.
2. Update the dashboard to make disconnected drives obvious and filterable.
3. Add manual bulk archive stale behavior.
4. Add optional auto-archive settings and lifecycle job logic.
5. Evaluate auto-delete only as a separate follow-up after the archive workflow is validated.

Acceptance Criteria

API

• The device summary API returns enough data for the frontend to distinguish:
  • active devices
  • disconnected/stale devices
  • archived devices
• Stale/disconnected state uses the same timeout basis as missed-ping monitoring.
• Per-device missed-ping timeout overrides are honored in stale-state calculations.

Dashboard

• A disconnected drive is visibly distinct from a healthy active drive.
• A disconnected drive is visibly distinct from an archived drive.
• Operators can see when a device was last seen.
• Operators can filter the dashboard to focus on disconnected devices.

Bulk Lifecycle

• Operators can archive stale devices without processing each one manually.
• Bulk archive behavior targets only devices meeting the stale threshold.
• Any destructive bulk delete path, if included, is clearly separated and explicitly confirmed.

Settings / Automation

• Auto-archive is optional and disabled by default.
• Auto-archive preserves device history and does not delete metrics.
• Auto-delete, if later added, is a separate opt-in phase.

Tests

• Backend tests cover stale-state calculation with:
  • global timeout
  • per-device timeout override
  • archived devices
  • devices with no last-seen data
• Frontend tests cover disconnected badge/filter behavior.
• Any bulk lifecycle endpoint includes tests for threshold targeting and non-target exclusion.

Notes

A key design requirement here is to avoid conflating three different states:

• active and reporting
• disconnected/stale but still historically relevant
• intentionally archived

That separation should be preserved in both backend semantics and frontend presentation.