CP: Fix host activity queues blocked by stuck app installs (#51197) - #51378
Merged
Conversation
**Related issue:** Resolves #50681 An App Store or in-house app install that a device acknowledges but never verifies holds the head of the host's activity queue for good. Fleet records these installs as successful only on verification, so nothing behind one runs, scripts and package installs included, and `UnblockHostsUpcomingActivityQueue` cannot rescue the host because something is activated. One reported host held 1 activated install and 1,328 waiting behind it, unchanged for 72 days. - Added `ReapStuckActivatedMDMInstalls`, which fails App Store and in-house app installs that have been activated past a timeout and can no longer make progress, then releases the queue. - Ran it from a new `reap_stuck_activated_mdm_installs` job on the existing `upcoming_activities_maintenance` schedule. - Added `server.vpp_install_reap_timeout`, default 24h. - Released the verification command when a result arrives with nothing left to verify. It was previously held until the daily cleanup, suppressing the next install's verification on that host. ## Notes for reviewers - An install is reaped only once it cannot make progress on its own. An answered install is judged on the age of the answer alone. Only an unanswered one is judged on delivery, having either lost its queue row or gone past the seven-day push window. Age alone would fail every install to a device that is merely switched off, and judging an answered install on delivery would fail one a returning device had just started running. A `NotNow` reply is not an answer, since nanomdm keeps the command queued and re-serves it. - The delivery window is measured from `activated_at`, not from `nano_enrollment_queue.created_at`. Both enqueue paths copy the queue row's `created_at` from the activity's to preserve ordering, so a command that activates after a long wait is born outside the window. That is the state of every install behind a head this fix has just freed. - An acknowledged install ages from `ncr.updated_at`, the same column the verify handler measures its own budget from, so a device that returns after a long absence keeps its verification window. - Reaping is per host. Activation batches up to five installs and stops at the first row still activated, so failing one of a batch would advance nothing. - No migration. The reaper clears existing stuck installs on its first pass. - Scan cost on MySQL 8.0.44: 23 ms with 24,300 queued App Store installs, 253 ms with 250,300, and 217 ms with nothing stuck. It is an index range scan on `(activity_type, host_id)`, and the subqueries run once per reap candidate rather than once per row. # Checklist for submitter - [x] Changes file added for user-visible changes in `changes/`, `orbit/changes/` or `ee/fleetd-chrome/changes`. See [Changes files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files) for more information. - [x] Input data is properly validated, `SELECT *` is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters. - [x] Timeouts are implemented and retries are limited to avoid infinite loops ## Testing - [x] Added/updated automated tests - [x] Where appropriate, [automated tests simulate multiple hosts and test for host isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing) (updates to one hosts's records do not affect another) - [x] QA'd all new/changed functionality manually The datastore test seeds fifteen hosts covering each way an install does or does not qualify, and asserts that reaping one host leaves the others untouched. Every assertion that claims to catch the defect was checked by reverting the fix and confirming it fails on the expected assertion. Manual QA ran the final predicate against a copy of a dev database at the default timeout, with six synthetic iPadOS hosts covering one rule each, all activated well past the floor so only the answer and delivery rules decided. The two that could no longer make progress were reaped, releasing their queues and deactivating their nano rows; on the stuck host the verification command was released and the script queued behind the install activated. The four still in flight were left alone: unanswered but recently queued, answered `NotNow`, carrying a backdated queue row, and returning after a long absence to acknowledge. Both config paths were exercised at startup, one below the verify timeout and one at zero. In-house installs, the sub-second timeout, and the duplicate-UUID lock delete rest on the automated tests above. ## New Fleet configuration settings - [x] Setting(s) is/are explicitly excluded from GitOps <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Automatically cleans up App Store and in-house app installations stuck for more than 24 hours. * Adds a configurable cleanup timeout, which can be disabled when set to zero. * Preserves commands that have not yet been delivered to devices. * **Bug Fixes** * Clears obsolete verification commands and releases blocked activity queues so subsequent installations can proceed. * Handles duplicated host identifiers correctly when removing stale commands. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Member
Author
|
I believe the test failures are unrelated and due to upstream vulnerability data. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## rc-minor-fleet-v4.91.0 #51378 +/- ##
=========================================================
Coverage ? 68.56%
=========================================================
Files ? 3982
Lines ? 256588
Branches ? 13805
=========================================================
Hits ? 175935
Misses ? 65008
Partials ? 15645
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
jkatz01
marked this pull request as ready for review
August 17, 2026 19:39
georgekarrv
approved these changes
Aug 17, 2026
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #51197 into
rc-minor-fleet-v4.91.0. Resolves #50681.Testing