Skip to content

Windows MDM improved host profile status performance - #44225

Merged
ksykulev merged 5 commits into
mainfrom
44189-host-profile-perf
Apr 28, 2026
Merged

Windows MDM improved host profile status performance#44225
ksykulev merged 5 commits into
mainfrom
44189-host-profile-perf

Conversation

@ksykulev

@ksykulev ksykulev commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Related issue: Resolves #44189

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.
  • Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements), JS inline code is prevented especially for url redirects, and untrusted data interpolated into shell scripts/commands is validated against shell metacharacters.

Testing

Summary by CodeRabbit

  • Performance

    • Optimized Windows MDM profile removal to skip redundant database writes for terminal removals.
  • Bug Fixes

    • Ensure terminal remove responses (both verified and failed) delete the corresponding profile records without affecting concurrent installs.
  • Tests

    • Added coverage for mixed install/remove responses and re-install after a verified removal.

Copilot AI review requested due to automatic review settings April 27, 2026 17:45
@ksykulev
ksykulev requested a review from a team as a code owner April 27, 2026 17:45

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves Windows MDM host profile ACK handling performance by avoiding a redundant UPSERT+DELETE cycle when processing verified remove acknowledgements, aligning the flow more closely with the Apple MDM approach.

Changes:

  • Partition Windows profile ACK updates into “upsert” vs “delete” buckets to skip unnecessary writes for terminal remove acknowledgements.
  • Add MySQL datastore tests covering verified remove deletion, failed remove persistence/detail, and mixed install+remove batches.
  • Add a user-facing changelog entry describing the performance improvement.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File Description
server/datastore/mysql/microsoft_mdm.go Partitions profile status updates and introduces a batched DELETE path for terminal remove ACKs.
server/datastore/mysql/microsoft_mdm_test.go Adds regression tests validating delete vs upsert behavior for Windows profile remove/install responses.
changes/44189-host-profile-perf Adds changelog note for Windows MDM profile removal performance.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread server/datastore/mysql/microsoft_mdm.go Outdated
@coderabbitai

coderabbitai Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e8356ac2-abae-4030-8332-f6bf7b73158c

📥 Commits

Reviewing files that changed from the base of the PR and between 9be9484 and 1a09e9e.

📒 Files selected for processing (2)
  • server/datastore/mysql/microsoft_mdm.go
  • server/datastore/mysql/microsoft_mdm_test.go

Walkthrough

A changelog entry was added. The Windows MDM host-profile ACK handler now partitions matched host_mdm_windows_profiles rows into an UPSERT bucket (installs and remove operations still in-flight/Verifying) and a DELETE bucket (remove operations resolving to terminal statuses Verified or Failed). UPSERTs are executed only when there are values; terminal remove command_uuids are removed via a single batched DELETE. Tests were added to cover terminal vs non‑terminal remove semantics and mixed install/remove batches.

Possibly related PRs

  • fleetdm/fleet PR 42206: Also modifies updateMDMWindowsHostProfileStatusFromResponseDB to delete terminal remove command UUIDs instead of UPSERTing them.
  • fleetdm/fleet PR 42566: Makes related changes in server/datastore/mysql/microsoft_mdm.go around batched host/profile upserts and command_uuid state transitions.
  • fleetdm/fleet PR 42495: Alters Windows MDM response handling and the batching/semantics for deleting host_mdm_windows_profiles rows.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: Windows MDM host profile removal performance improvement through eliminating redundant UPSERT operations.
Description check ✅ Passed The description adequately covers key requirements: related issue is linked, changes file was added, input validation/SQL injection prevention was addressed, and automated tests were updated with host isolation coverage.
Linked Issues check ✅ Passed All coding objectives from issue #44189 are met: removes at terminal status (Verified/Failed) are deleted instead of upserted, payload partitioning isolates upsert/delete buckets, at most two batched statements per call are issued, and failed-remove tombstones are cleaned up.
Out of Scope Changes check ✅ Passed All changes directly support the stated objectives: changelog entry documents the performance improvement, core logic refactors Windows MDM ACK handling, and tests validate the new remove semantics and host isolation.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 44189-host-profile-perf

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
server/datastore/mysql/microsoft_mdm_test.go (1)

3739-3749: Optional: collapse the two SELECTs into one round-trip.

Minor readability/perf nit — status and detail can be fetched in a single query into a small struct, which also makes the assertions read as a single observation of the row rather than two independent ones.

♻️ Suggested refactor
-		// Row should persist with failed status and "Failed to remove: ..." detail.
-		var status, detail string
-		ExecAdhocSQL(t, ds, func(q sqlx.ExtContext) error {
-			return sqlx.GetContext(t.Context(), q, &status, `
-SELECT status FROM host_mdm_windows_profiles
-WHERE host_uuid = ? AND command_uuid = ?`, enrolledDevice1.HostUUID, deleteCommandUUID)
-		})
-		ExecAdhocSQL(t, ds, func(q sqlx.ExtContext) error {
-			return sqlx.GetContext(t.Context(), q, &detail, `
-SELECT detail FROM host_mdm_windows_profiles
-WHERE host_uuid = ? AND command_uuid = ?`, enrolledDevice1.HostUUID, deleteCommandUUID)
-		})
-		assert.Equal(t, "failed", status, "failed remove row should persist")
-		assert.Contains(t, detail, "Failed to remove:", "detail should have remove prefix")
+		// Row should persist with failed status and "Failed to remove: ..." detail.
+		var row struct {
+			Status string `db:"status"`
+			Detail string `db:"detail"`
+		}
+		ExecAdhocSQL(t, ds, func(q sqlx.ExtContext) error {
+			return sqlx.GetContext(t.Context(), q, &row, `
+SELECT status, detail FROM host_mdm_windows_profiles
+WHERE host_uuid = ? AND command_uuid = ?`, enrolledDevice1.HostUUID, deleteCommandUUID)
+		})
+		assert.Equal(t, "failed", row.Status, "failed remove row should persist")
+		assert.Contains(t, row.Detail, "Failed to remove:", "detail should have remove prefix")
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@server/datastore/mysql/microsoft_mdm_test.go` around lines 3739 - 3749, The
test currently issues two separate queries to fetch status and detail (using
ExecAdhocSQL and sqlx.GetContext into the variables status and detail) which
should be collapsed into a single SELECT that returns both columns from
host_mdm_windows_profiles for the given enrolledDevice1.HostUUID and
deleteCommandUUID; create a small local struct (e.g. with Status and Detail
fields), run one sqlx.GetContext into that struct, and update the subsequent
assertions to read from the struct instead of the separate status/detail
variables.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@server/datastore/mysql/microsoft_mdm_test.go`:
- Around line 3754-3834: Add a new subtest that verifies
"resurrect-after-delete": insert a remove row into host_mdm_windows_profiles for
enrolledDevice1 using mdmWindowsInsertCommandForHostsDB + ExecAdhocSQL with a
delete command UUID and a specific profile_uuid, call MDMWindowsSaveResponse
with an enriched SyncML entry for Delete (StatusCode 200) via
createResponseAsEnrichedSyncML and assert the remove row is deleted, then insert
a fresh install row for the same profile_uuid but a different command UUID, call
MDMWindowsSaveResponse again with a Replace (StatusCode 200) entry and assert
the install row is upserted and marked "verified"; use the same helpers and
table (host_mdm_windows_profiles, mdmWindowsInsertCommandForHostsDB,
ExecAdhocSQL, createResponseAsEnrichedSyncML, MDMWindowsSaveResponse,
enrolledDevice1) to implement the flow.

---

Nitpick comments:
In `@server/datastore/mysql/microsoft_mdm_test.go`:
- Around line 3739-3749: The test currently issues two separate queries to fetch
status and detail (using ExecAdhocSQL and sqlx.GetContext into the variables
status and detail) which should be collapsed into a single SELECT that returns
both columns from host_mdm_windows_profiles for the given
enrolledDevice1.HostUUID and deleteCommandUUID; create a small local struct
(e.g. with Status and Detail fields), run one sqlx.GetContext into that struct,
and update the subsequent assertions to read from the struct instead of the
separate status/detail variables.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 39d17b67-9b5c-4d87-af36-3edb226fc931

📥 Commits

Reviewing files that changed from the base of the PR and between 65b4da9 and b649664.

📒 Files selected for processing (3)
  • changes/44189-host-profile-perf
  • server/datastore/mysql/microsoft_mdm.go
  • server/datastore/mysql/microsoft_mdm_test.go

Comment thread server/datastore/mysql/microsoft_mdm_test.go
@codecov

codecov Bot commented Apr 27, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.79%. Comparing base (911343e) to head (88e7ea4).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
server/datastore/mysql/microsoft_mdm.go 66.66% 3 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #44225      +/-   ##
==========================================
+ Coverage   66.72%   66.79%   +0.06%     
==========================================
  Files        2626     2627       +1     
  Lines      211177   211165      -12     
  Branches     9392     9408      +16     
==========================================
+ Hits       140914   141043     +129     
+ Misses      57480    57303     -177     
- Partials    12783    12819      +36     
Flag Coverage Δ
backend 68.56% <66.66%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@server/datastore/mysql/microsoft_mdm_test.go`:
- Around line 3700-3752: The test "remove failed persists with detail prefix"
currently asserts that a failed remove persists with status "failed" and a
"Failed to remove:" detail; update it to reflect the new terminal-delete
behavior by asserting the host_mdm_windows_profiles row for the given host_uuid
and deleteCommandUUID is removed (e.g., query and expect no row /
sqlx.GetContext to return sql.ErrNoRows) after calling MDMWindowsSaveResponse,
rather than checking for status/detail; keep references to
MDMWindowsSaveResponse, enrolledDevice1.HostUUID and deleteCommandUUID so the
check targets the same record and remove the assert.Contains on detail and
assert.Equal on status.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ac0b45a1-e9a0-4ff0-b62c-ea06c54f9f2c

📥 Commits

Reviewing files that changed from the base of the PR and between b649664 and 9be9484.

📒 Files selected for processing (2)
  • server/datastore/mysql/microsoft_mdm.go
  • server/datastore/mysql/microsoft_mdm_test.go

Comment thread server/datastore/mysql/microsoft_mdm_test.go Outdated

@getvictor getvictor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made a couple of comments. Might be able to merge as is.

Comment thread server/datastore/mysql/microsoft_mdm.go Outdated
Comment thread server/datastore/mysql/microsoft_mdm_test.go Outdated
@ksykulev

Copy link
Copy Markdown
Contributor Author

Local load test results

Setup: 1000 Windows hosts enrolled via osquery-perf, 20 Windows profiles assigned per host (20,000 total profile rows). Hosts transferred between teams to trigger a remove storm of 20,000 profile removals.

Results:

  • All 20,000 removes processed in ~25 seconds
  • InnoDB Row Writes chart confirms the expected pattern:
    • Rows deleted spike (removes going straight to DELETE, no preceding UPSERT)
    • Rows inserted/updated spike is from the new team's profile installs, not from the remove path
    • Old code would have doubled the insert spike at the same moment (redundant UPSERT before DELETE for each remove). The old code would have shown yellow and red spiking together at 18:31 (UPSERT then DELETE for every remove). Instead, removes are DELETE-only.

InnoDB Row Writes at transfer time (zoomed):

Screenshot 2026-04-27 at 6 38 27 PM Peak at 18:31:25: 560 rows deleted, 4.11K rows inserted (new installs), 3.68K rows updated (new install status updates). Remove storm fully drained by 18:32:00.

@ksykulev
ksykulev merged commit 9ec20e6 into main Apr 28, 2026
48 checks passed
@ksykulev
ksykulev deleted the 44189-host-profile-perf branch April 28, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skip UPSERT for verified-remove ACKs in Windows MDM (Apple parity)

3 participants