Clean up completed background jobs and one-shot reminders that are no longer needed

[Aaronontheweb]

Summary

Completed background jobs and one-shot reminders are never cleaned up, so their definitions accumulate indefinitely in the daemon's on-disk stores. We need a bounded retention policy that removes what is no longer useful while preserving failure visibility.

Background jobs — never cleaned up (since inception)

BackgroundJobDefinitionStore.Delete(BackgroundJobId) exists (src/Netclaw.Actors/Jobs/BackgroundJobDefinitionStore.cs:135) but is never called — it is dead code. BackgroundJobManagerActor transitions terminal states (Completed/Failed/Cancelled/Reaped/Lost) in HandleCompleted and delivers the full result to the owning session via DeliverResultToSession (line 408), but never removes the definition file or its output log. Every background job submitted (shell_execute + _background:true, e.g. the 234fee75dbff release build) leaves a {jobId}.json + output.log behind forever.

Proposed: delete the definition and output log once the job reaches a terminal state and the result has been delivered to the session. Add a reconcile sweep for terminal jobs that were never delivered (e.g. session passivated) — mirroring the reminders reconcile pattern. A short grace period after delivery is reasonable so a slow polling session can still check_background_job.

One-shot reminders — soft-disabled only since #1812

PR #1812 / issue #1803 changed one-shot settlement so definitions are retained (soft-disabled with Enabled=false, TerminalOutcome) instead of hard-deleted, to preserve failed executions for diagnosis. The reconcile path no longer deletes stale one-shots either.

Important — do NOT regress #1803. The retain-on-failure behavior is intentional and must stay.

Proposed:

• Success → delete the definition (and its history) at settlement, restoring the pre-fix(reminders): retain failed one-shot executions #1812 behavior for successful one-shots. Nobody needs to re-inspect a reminder that fired successfully — the channel delivery already has it.
• Failure → retain (disabled, TerminalOutcome=Failed) for a bounded retention window so the user can still see why it failed (netclaw reminders list, get_reminder_history), then sweep it in reconcile. This preserves the Retain and retry a one-shot reminder after execution failure #1803 diagnostic value while capping growth.
• ExpiresAt is currently rejected for one-shots (ReminderManagerActor.cs:270), so the retention window needs its own field (e.g. RetainUntilMs) rather than piggybacking on expiry.

Acceptance criteria

1. A background job that completes, fails, is cancelled, is reaped, or is lost gets its definition + output log deleted after result delivery (with a documented grace period).
2. A reconcile sweep removes terminal background jobs that were never delivered.
3. A one-shot reminder that fires successfully is hard-deleted at settlement.
4. A one-shot reminder that fails remains visible (disabled, with failure state) for the retention window, then is swept.
5. Failed-reminder retention behavior from Retain and retry a one-shot reminder after execution failure #1803 is preserved (regression tests for the failure path).
6. No unbounded growth in ReminderDefinitionStore, ReminderHistoryStore, BackgroundJobDefinitionStore, or the Akka.Reminders shard state.

Related

• Closes nothing directly; follow-up to Retain and retry a one-shot reminder after execution failure #1803 and PR fix(reminders): retain failed one-shot executions #1812 (retain-failed-one-shots) and the check_background_job approval work in check_background_job status queries require approval and wedge non-interactive turns #1818 / PR fix(approvals): bypass approval for background job control #1817.

[Aaronontheweb]

Closed via #1821