Skip to content

Defer Windows MDM profile removals via pending-delete retention - #47156

Merged
getvictor merged 31 commits into
mainfrom
46993-pending-delete
Jun 11, 2026
Merged

Defer Windows MDM profile removals via pending-delete retention#47156
getvictor merged 31 commits into
mainfrom
46993-pending-delete

Conversation

@getvictor

@getvictor getvictor commented Jun 9, 2026

Copy link
Copy Markdown
Member

Related issue: Resolves #46993

Requires #47071 to merge first

Loadtest shows reduction of batch delete of 40 profiles for 30K hosts down to ~3.9 seconds.

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.

Testing

Database migrations

  • Ensured the correct collation is explicitly set for character columns (COLLATE utf8mb4_unicode_ci).

Summary by CodeRabbit

Release Notes

  • Bug Fixes

    • Resolved timeout issues when removing large numbers of Windows configuration profiles from teams with many hosts.
  • New Features

    • Windows profile deletions now process asynchronously in the background, enabling faster API responses and consistent behavior with profile delivery operations.

getvictor added 15 commits June 7, 2026 09:30
Pure refactor, no behavior change. First step for #45635 (Windows
batched in-memory reconciler).

- New server/mdm/reconcile package holds the include/exclude label
  handlers and the team+label applicability dispatcher. The Apple
  platform gate stays in the Apple wrapper since platform eligibility
  is platform-specific.
- New platform-neutral fleet types (MDMProfileLabelRef,
  MDMProfileIncludeMode, MDMLabeledEntity); the Apple names are now
  type aliases so existing code and tests are unchanged.
- BulkGetHostLabelMemberships moves from apple_mdm_batched.go to a
  neutral file; it was already platform-agnostic.
- The existing Apple label-scenario tests keep covering the shared
  logic through the delegating wrappers; the shared package also gets
  its own handler/dispatcher tests.
# Conflicts:
#	server/mdm/reconcile/reconcile_test.go
Batch/single/team Windows config-profile deletes were O(profiles x hosts) and synchronous:
they marked host_mdm_windows_profiles rows and generated <Delete> commands inside the request,
so large removals (e.g. 40 profiles x 30K hosts) timed out and rolled back, never succeeding.

Deletes now copy the profile definition into a new mdm_windows_configuration_profiles_pending_delete
table before the live row is removed, keep only the cheap host-row cleanup (phases 0/1), and let the
profile-manager cron generate the removals asynchronously in its bounded 2,000-host batches, the same
path used for team transfers. The delete endpoints become O(profiles). GetMDMWindowsProfilesContents
falls back to the retention table so the cron can build <Delete> commands after the definition is gone.
Retention is GC'd by age (default 7-day grace) in the existing Windows MDM cleanup cron.

The reconciler's <Delete> generation now protects LocURIs per host using the host's still-desired
(applicable) profiles, so a removed profile's shared LocURI is not reverted where another applicable
profile still enforces it. Protection is label-aware (a label-scoped profile only protects the hosts
it applies to), matching the old synchronous two-pass behavior; hosts that share the same protected
subset are grouped into one command.
@getvictor
getvictor requested a review from Copilot June 9, 2026 10:46
@getvictor

Copy link
Copy Markdown
Member Author

@coderabbitai full review

@getvictor

Copy link
Copy Markdown
Member Author

/agentic_review

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Jun 9, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0)

Grey Divider


Remediation recommended

1. Quadratic remove payload build ✓ Resolved 🐞 Bug ➹ Performance
Description
executeWindowsProfileReconcileBatch rebuilds remove payloads by scanning the full per-profile remove
list for every protection group, which can degrade to O(groups×hosts) and approach O(hosts²) when
many groups form (e.g., label-scoped protection differences). This can significantly slow large
reconcile windows and increase the chance of missing the reconciler’s time/budget constraints.
Code

server/service/microsoft_mdm.go[R3996-4023]

+			groupHosts := make(map[string]struct{}, len(g.hostUUIDs))
+			for _, h := range g.hostUUIDs {
+				groupHosts[h] = struct{}{}
}
-			hp := &fleet.MDMWindowsBulkUpsertHostProfilePayload{
-				ProfileUUID:   rp.ProfileUUID,
-				HostUUID:      rp.HostUUID,
-				ProfileName:   rp.ProfileName,
-				CommandUUID:   target.cmdUUID,
-				OperationType: fleet.MDMOperationTypeRemove,
-				Status:        &fleet.MDMDeliveryPending,
-				Checksum:      checksum,
+			removePayloadsForCommand := []*fleet.MDMWindowsBulkUpsertHostProfilePayload{}
+			for _, rp := range removePayloadData[profUUID] {
+				if _, ok := groupHosts[rp.HostUUID]; !ok {
+					continue
+				}
+				// Remove operations don't need a checksum; use a zero value if none exists (defensive coding).
+				checksum := rp.Checksum
+				if len(checksum) == 0 {
+					checksum = make([]byte, 16)
+				}
+				removePayloadsForCommand = append(removePayloadsForCommand, &fleet.MDMWindowsBulkUpsertHostProfilePayload{
+					ProfileUUID:   rp.ProfileUUID,
+					HostUUID:      rp.HostUUID,
+					ProfileName:   rp.ProfileName,
+					CommandUUID:   cmdUUID,
+					OperationType: fleet.MDMOperationTypeRemove,
+					Status:        &fleet.MDMDeliveryPending,
+					Checksum:      checksum,
+				})
+				logger.DebugContext(ctx, "removing profile", "profile.uuid", rp.ProfileUUID, "host.uuid", rp.HostUUID, "profile.name", rp.ProfileName)
+			}
+			if err := ds.MDMWindowsInsertCommandAndUpsertHostProfilesForHosts(ctx, g.hostUUIDs, command, removePayloadsForCommand); err != nil {
+				return ctxerr.Wrap(ctx, err, "inserting remove commands for hosts")
}
-			removePayloadsForCommand = append(removePayloadsForCommand, hp)
-			logger.DebugContext(ctx, "removing profile", "profile.uuid", rp.ProfileUUID, "host.uuid", rp.HostUUID, "profile.name", rp.ProfileName)
-		}
-		if err := ds.MDMWindowsInsertCommandAndUpsertHostProfilesForHosts(ctx, target.hostUUIDs, command, removePayloadsForCommand); err != nil {
-			return ctxerr.Wrap(ctx, err, "inserting remove commands for hosts")
Evidence
The updated remove path introduces an inner loop that scans all remove payloads for a profile for
each computed group, producing potentially quadratic work when group count grows with host count.

server/service/microsoft_mdm.go[3922-4024]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`executeWindowsProfileReconcileBatch` builds per-host protection groups for profile removals, but then for each group it iterates over the entire `removePayloadData[profUUID]` slice to filter payloads for that group. In worst cases (many distinct protection subsets), this becomes quadratic-ish work per profile and can slow down large Windows reconcile ticks.
## Issue Context
The code currently:
- builds `groups` per profile,
- for each group, builds a `groupHosts` set,
- scans *all* `removePayloadData[profUUID]` to pick payloads for that group.
## Fix Focus Areas
- server/service/microsoft_mdm.go[3922-4024]
Suggested direction:
- Pre-index `removePayloadData[profUUID]` by `HostUUID` once (e.g., `map[string]*fleet.MDMWindowsProfilePayload` or `map[string][]...` if duplicates are possible).
- For each group, build `removePayloadsForCommand` by iterating `g.hostUUIDs` and doing O(1) lookups.
- Alternatively, while building `groups`, also accumulate the payload pointers per group key to avoid a second pass entirely.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Stale pending-delete created_at ✓ Resolved 🐞 Bug ☼ Reliability
Description
copyWindowsConfigProfilesToPendingDeleteDB uses INSERT IGNORE, so if a profile UUID already exists
in the pending-delete table its created_at is not refreshed on subsequent delete attempts. Because
DeleteTeam writes pending-delete retention in a separate pre-DeleteTeam transaction, a
failed/aborted team delete followed by a later retry can leave a stale created_at that allows the GC
cron to remove retained SyncML earlier than the intended grace window from the eventual deletion,
leading the reconciler to skip <Delete> generation due to missing content.
Code

server/datastore/mysql/microsoft_mdm.go[R1722-1731]

+	stmt, args, err := sqlx.In(`
+		INSERT IGNORE INTO mdm_windows_configuration_profiles_pending_delete (profile_uuid, team_id, name, syncml, created_at)
+		SELECT profile_uuid, team_id, name, syncml, NOW(6)
+		FROM mdm_windows_configuration_profiles
+		WHERE profile_uuid IN (?)`, profileUUIDs)
+	if err != nil {
+		return ctxerr.Wrap(ctx, err, "building IN for pending-delete copy")
}
-
-	// Update host-profile rows only for profiles that had delete commands enqueued.
-	// This covers both install rows (being flipped to remove) and remove+NULL rows
-	// (being given a command_uuid and set to pending).
-	//
-	// Flatten (host_uuid, profile_uuid, cmd_uuid) triples across all profiles and
-	// batch them into a single UPDATE per batch. Each batch can span multiple
-	// profiles, with a CASE mapping each row's profile_uuid to its command_uuid.
-	// The WHERE clause uses a tuple IN on (host_uuid, profile_uuid), which matches
-	// the PK and lets the optimizer perform direct PK point lookups. This avoids
-	// the previous per-profile loop, which under-utilized batches when profiles
-	// affected fewer than batchSize hosts.
-	//
-	// Profile UUIDs are iterated in sorted order so concurrent callers
-	// acquire InnoDB row locks on host_mdm_windows_profiles in the same
-	// order, reducing the deadlock surface on this path. The SQL text
-	// itself is placeholder-only and already deterministic for a given
-	// batch size, so iteration order does not affect plan-cache / query
-	// digest stability.
-	type pendingRemoveRow struct {
-		hostUUID    string
-		profileUUID string
-		cmdUUID     string
-	}
-	sortedProfUUIDs := slices.Sorted(maps.Keys(enqueuedTargets))
-	totalRows := 0
-	for _, profUUID := range sortedProfUUIDs {
-		totalRows += len(enqueuedTargets[profUUID].hostUUIDs)
-	}
-	rows := make([]pendingRemoveRow, 0, totalRows)
-	for _, profUUID := range sortedProfUUIDs {
-		target := enqueuedTargets[profUUID]
-		for _, hostUUID := range target.hostUUIDs {
-			rows = append(rows, pendingRemoveRow{
-				hostUUID:    hostUUID,
-				profileUUID: profUUID,
-				cmdUUID:     target.cmdUUID,
-			})
-		}
-	}
-
-	if err := common_mysql.BatchProcessSimple(rows, windowsMDMProfileDeleteBatchSize, func(batch []pendingRemoveRow) error {
-		// Collect the profile_uuid -> cmd_uuid mapping needed by this batch. Most
-		// batches span 1 to N profiles; we only need one CASE arm per distinct
-		// profile in the batch.
-		profileCmds := make(map[string]string)
-		for _, r := range batch {
-			profileCmds[r.profileUUID] = r.cmdUUID
-		}
-		sortedBatchProfUUIDs := slices.Sorted(maps.Keys(profileCmds))
-
-		var sb strings.Builder
-		sb.WriteString(`UPDATE host_mdm_windows_profiles
-			SET operation_type = ?,
-			    status = ?,
-			    detail = '',
-			    command_uuid = CASE profile_uuid`)
-		args := make([]any, 0, 2+2*len(profileCmds)+2*len(batch))
-		args = append(args, fleet.MDMOperationTypeRemove, fleet.MDMDeliveryPending)
-		for _, profUUID := range sortedBatchProfUUIDs {
-			sb.WriteString(" WHEN ? THEN ?")
-			args = append(args, profUUID, profileCmds[profUUID])
-		}
-		// ELSE command_uuid is defensive: WHERE restricts the update to rows
-		// whose profile_uuid is present in profileCmds, so in practice every
-		// updated row matches a WHEN arm.
-		sb.WriteString(` ELSE command_uuid END
-			WHERE (host_uuid, profile_uuid) IN (`)
-		for i, r := range batch {
-			if i > 0 {
-				sb.WriteByte(',')
-			}
-			sb.WriteString("(?,?)")
-			args = append(args, r.hostUUID, r.profileUUID)
-		}
-		sb.WriteByte(')')
-
-		if _, err := tx.ExecContext(ctx, sb.String(), args...); err != nil {
-			return ctxerr.Wrap(ctx, err, "updating host profiles to remove")
-		}
-		return nil
-	}); err != nil {
-		return err
+	if _, err := tx.ExecContext(ctx, stmt, args...); err != nil {
+		return ctxerr.Wrap(ctx, err, "copying windows config profiles to pending delete")
Evidence
Pending-delete retention is written with INSERT IGNORE (no timestamp refresh), DeleteTeam performs
that write before its main deletion transaction, GC is strictly age-based (7-day cutoff), and the
reconciler skips removals when it cannot load profile contents.

server/datastore/mysql/microsoft_mdm.go[1714-1733]
server/datastore/mysql/teams.go[164-170]
server/datastore/mysql/teams.go[228-249]
cmd/fleet/cron.go[1478-1485]
server/datastore/mysql/microsoft_mdm.go[3766-3794]
server/service/microsoft_mdm.go[3943-3947]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`copyWindowsConfigProfilesToPendingDeleteDB` uses `INSERT IGNORE ... SELECT ... NOW(6)` into `mdm_windows_configuration_profiles_pending_delete`. If the row already exists, `created_at` is not updated, so the GC grace-period clock is not refreshed.
This is especially relevant for team deletion, where retention is written in a separate transaction before the main `DeleteTeam` transaction. If `DeleteTeam` fails after the retention write and is retried later, the existing retention row can have a stale `created_at` and be GC’d earlier than intended.
## Issue Context
- Retention copy uses INSERT IGNORE.
- GC cron deletes rows older than cutoff.
- Reconciler explicitly skips building deletes when content is missing.
## Fix Focus Areas
- server/datastore/mysql/microsoft_mdm.go[1718-1733]
- server/datastore/mysql/teams.go[164-170]
- cmd/fleet/cron.go[1481-1485]
- server/datastore/mysql/microsoft_mdm.go[3772-3794]
- server/service/microsoft_mdm.go[3944-3947]
Suggested direction:
- Replace `INSERT IGNORE` with an upsert that refreshes `created_at` (and optionally `team_id`, `name`, `syncml`) on duplicates, e.g.:
- `INSERT INTO ... SELECT ...` 
- `ON DUPLICATE KEY UPDATE created_at = VALUES(created_at), team_id=VALUES(team_id), name=VALUES(name), syncml=VALUES(syncml)`
- Keep the operation idempotent under transaction retries.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses #46993 by making Windows MDM profile removals asynchronous (like installs already are) to prevent POST /api/latest/fleet/mdm/profiles/batch timeouts on large teams. It introduces a retention/tombstone table for deleted Windows profile SyncML so the profile-manager cron can generate <Delete> commands in bounded batches, and updates reconcile logic to keep LocURI protection correct when removals overlap with label-scoped desired profiles.

Changes:

  • Retain deleted Windows profile definitions in mdm_windows_configuration_profiles_pending_delete and defer host fan-out <Delete> work to the Windows profile reconciler.
  • Update Windows profile reconcile execution to fetch “still-desired” profile contents and apply per-host (label-aware) LocURI protection when generating <Delete> commands.
  • Add age-based GC for retained deleted-profile content via a new cron cleanup job and datastore method, with associated migrations/tests.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
server/service/microsoft_mdm.go Passes per-host desired profile UUIDs into the reconcile execute step and updates <Delete> generation to be per-host/label-aware.
server/mdm/microsoft/reconcile.go Adds helper to compute desired (applicable) profile UUIDs per host for LocURI protection.
server/datastore/mysql/microsoft_mdm.go Copies deleted profile contents to pending-delete table, removes synchronous delete fan-out, adds content fallback + GC method.
server/datastore/mysql/teams.go Updates team deletion path to retain Windows profile contents before the cascade removes definitions.
server/fleet/datastore.go Extends the datastore interface with pending-delete GC.
server/mock/datastore_mock.go Updates datastore mock to include the new GC method.
cmd/fleet/cron.go Schedules periodic age-based cleanup of pending-delete profile retention rows.
server/datastore/mysql/migrations/tables/20260609143000_AddWindowsMDMConfigProfilesPendingDelete.go Adds migration creating the pending-delete retention table.
server/datastore/mysql/migrations/tables/20260609143000_AddWindowsMDMConfigProfilesPendingDelete_test.go Verifies the migration creates the table and supports the intended usage pattern.
server/datastore/mysql/schema.sql Updates schema snapshot to include the new pending-delete table and migration entry.
server/datastore/mysql/microsoft_mdm_test.go Adjusts/remodels tests to reflect async deletion and validates retention + GC behavior.
changes/46993-windows-batch-remove-async User-visible change entry describing the fix and new async removal behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread server/service/microsoft_mdm.go
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 959c4b7a-089a-4657-bb5f-8f6461ae66f7

📥 Commits

Reviewing files that changed from the base of the PR and between 86cb1f6 and f300f09.

📒 Files selected for processing (2)
  • changes/46993-windows-batch-remove-async
  • cmd/fleet/cron.go
✅ Files skipped from review due to trivial changes (1)
  • changes/46993-windows-batch-remove-async

Walkthrough

This PR implements asynchronous Windows MDM profile deletion to resolve batch-endpoint timeouts when removing large numbers of profiles from teams with many hosts. The fix introduces a mdm_windows_configuration_profiles_pending_delete retention table to store deleted profile SyncML content after logical deletion. Delete API operations now return quickly, retaining only the deleted profile's SyncML in the pending table. The profile-manager cron then generates and enqueues <Delete> commands asynchronously in bounded batches. The reconciler now computes per-host "desired" profiles using team and label applicability rules, grouping hosts by protected LocURI subsets per removed profile to build correctly-scoped deletion commands. Garbage collection periodically removes retention rows older than 7 days. Tests are updated to enable Windows MDM and drive reconciliation cron execution for async delete command generation.

Possibly related PRs

  • fleetdm/fleet#42206: Main PR continues and refactors the Windows profile deletion/SyncML <Delete> pipeline by deferring <Delete> generation to the cron via pending-delete retention instead of immediate host cleanup.
  • fleetdm/fleet#47032: Windows MDM deletion logic now computes per-host "desired" profiles using label-applicability rules extracted in PR #47032.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 42.86% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: deferring Windows MDM profile removals using a pending-delete retention mechanism, which is the core objective of this PR.
Description check ✅ Passed The PR description is mostly complete, covering the related issue, checklist items (changes file, testing, database migrations), and key context (loadtest improvements and dependency on #47071).
Linked Issues check ✅ Passed The PR implementation directly addresses #46993 by decoupling definition deletion from host fan-out, retaining profile definitions in a pending-delete table, and deferring removals to the cron for asynchronous processing in bounded batches.
Out of Scope Changes check ✅ Passed All code changes are in-scope and directly support the pending-delete retention mechanism, async reconciliation, and related testing updates required to resolve issue #46993.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 46993-pending-delete
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch 46993-pending-delete

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
server/datastore/mysql/microsoft_mdm.go (1)

1684-1690: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Keep verifying remove rows out of phase-0 cleanup.

Lines 1232-1234 only treat remove rows as terminal when they reach verified or failed. Including fleet.MDMDeliveryVerifying here deletes in-flight removals that the new async flow is supposed to leave behind for the cron, so a host can lose the last bit of state Fleet has for finishing or retrying that delete.

Suggested fix
-	terminalStatuses := []fleet.MDMDeliveryStatus{fleet.MDMDeliveryFailed, fleet.MDMDeliveryVerified, fleet.MDMDeliveryVerifying}
+	terminalStatuses := []fleet.MDMDeliveryStatus{fleet.MDMDeliveryFailed, fleet.MDMDeliveryVerified}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/datastore/mysql/microsoft_mdm.go` around lines 1684 - 1690, The
current phase-0 cleanup treats remove operations with status
fleet.MDMDeliveryVerifying as terminal and deletes them; remove in-flight
removals from this deletion set by excluding fleet.MDMDeliveryVerifying from the
terminalStatuses slice used for remove operations (the slice named
terminalStatuses used in the DELETE for host_mdm_windows_profiles with
fleet.MDMOperationTypeRemove), so only fleet.MDMDeliveryFailed and
fleet.MDMDeliveryVerified remain; update the code that builds
delRemStmt/delRemArgs accordingly.
🧹 Nitpick comments (1)
server/datastore/mysql/microsoft_mdm_test.go (1)

5208-5214: ⚡ Quick win

Restore WindowsEnabledAndConfigured after mutating app config in tests.

Both blocks set a global app-config flag but never restore it, which can leak state into later subtests sharing the same datastore and create order-dependent flakes.

Suggested fix
  appCfg, acErr := ds.AppConfig(ctx)
  require.NoError(t, acErr)
+ prevWindowsEnabled := appCfg.MDM.WindowsEnabledAndConfigured
+ t.Cleanup(func() {
+   cfg, err := ds.AppConfig(ctx)
+   require.NoError(t, err)
+   cfg.MDM.WindowsEnabledAndConfigured = prevWindowsEnabled
+   require.NoError(t, ds.SaveAppConfig(ctx, cfg))
+ })
  appCfg.MDM.WindowsEnabledAndConfigured = true
  require.NoError(t, ds.SaveAppConfig(ctx, appCfg))

Also applies to: 5564-5569

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/datastore/mysql/microsoft_mdm_test.go` around lines 5208 - 5214, The
test mutates the global app config flag MDM.WindowsEnabledAndConfigured via
AppConfig() and SaveAppConfig() but never restores it, leaking state across
subtests; capture the original value of appCfg.MDM.WindowsEnabledAndConfigured
before setting it true, then after calling SaveAppConfig(ctx, appCfg) register a
deferred restore that sets the flag back to the original value and calls
SaveAppConfig(ctx, appCfg) to persist it. Apply the same pattern to the other
block around lines noted (the second AppConfig/SaveAppConfig mutation) so both
modifications are reverted after the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@server/datastore/mysql/microsoft_mdm.go`:
- Around line 1684-1690: The current phase-0 cleanup treats remove operations
with status fleet.MDMDeliveryVerifying as terminal and deletes them; remove
in-flight removals from this deletion set by excluding
fleet.MDMDeliveryVerifying from the terminalStatuses slice used for remove
operations (the slice named terminalStatuses used in the DELETE for
host_mdm_windows_profiles with fleet.MDMOperationTypeRemove), so only
fleet.MDMDeliveryFailed and fleet.MDMDeliveryVerified remain; update the code
that builds delRemStmt/delRemArgs accordingly.

---

Nitpick comments:
In `@server/datastore/mysql/microsoft_mdm_test.go`:
- Around line 5208-5214: The test mutates the global app config flag
MDM.WindowsEnabledAndConfigured via AppConfig() and SaveAppConfig() but never
restores it, leaking state across subtests; capture the original value of
appCfg.MDM.WindowsEnabledAndConfigured before setting it true, then after
calling SaveAppConfig(ctx, appCfg) register a deferred restore that sets the
flag back to the original value and calls SaveAppConfig(ctx, appCfg) to persist
it. Apply the same pattern to the other block around lines noted (the second
AppConfig/SaveAppConfig mutation) so both modifications are reverted after the
test.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ed974f17-8d77-43c3-862c-3eea3ed1b1ed

📥 Commits

Reviewing files that changed from the base of the PR and between 1cad37d and 86cb1f6.

📒 Files selected for processing (12)
  • changes/46993-windows-batch-remove-async
  • cmd/fleet/cron.go
  • server/datastore/mysql/microsoft_mdm.go
  • server/datastore/mysql/microsoft_mdm_test.go
  • server/datastore/mysql/migrations/tables/20260609143000_AddWindowsMDMConfigProfilesPendingDelete.go
  • server/datastore/mysql/migrations/tables/20260609143000_AddWindowsMDMConfigProfilesPendingDelete_test.go
  • server/datastore/mysql/schema.sql
  • server/datastore/mysql/teams.go
  • server/fleet/datastore.go
  • server/mdm/microsoft/reconcile.go
  • server/mock/datastore_mock.go
  • server/service/microsoft_mdm.go

@codecov

codecov Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.38424% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.20%. Comparing base (6ab205e) to head (26ec679).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
server/datastore/mysql/microsoft_mdm.go 65.85% 16 Missing and 12 partials ⚠️
server/service/microsoft_mdm.go 84.21% 7 Missing and 5 partials ⚠️
...202649_AddWindowsMDMConfigProfilesPendingDelete.go 68.42% 4 Missing and 2 partials ⚠️
cmd/fleet/cron.go 0.00% 3 Missing ⚠️
server/datastore/mysql/teams.go 57.14% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #47156      +/-   ##
==========================================
- Coverage   67.22%   67.20%   -0.02%     
==========================================
  Files        3394     3395       +1     
  Lines      228363   228307      -56     
  Branches    11908    11908              
==========================================
- Hits       153516   153441      -75     
- Misses      61014    61048      +34     
+ Partials    13833    13818      -15     
Flag Coverage Δ
backend 68.83% <74.38%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…lcheck lint

The Windows profile delete tests (TestDeleteMDMProfilesCancelsInstalls, TestDeleteTeamCancelsWindowsProfileInstalls,
integration TestDeleteMDMProfileCancelsInstalls) asserted the synchronous remove+pending flip that now happens via the
profile-manager cron. They each run/trigger the reconciler after the delete and enable Windows MDM in app config (restored
after, to avoid leaking into sibling subtests).

Lint: BuildDeleteCommandFromProfileBytes now takes a map[string]struct{} set instead of map[string]bool (setboolcheck), with
callers and tests updated.
@getvictor
getvictor marked this pull request as ready for review June 9, 2026 21:35
@getvictor
getvictor requested a review from a team as a code owner June 9, 2026 21:35
Comment thread server/datastore/mysql/microsoft_mdm.go
SyncML []byte `db:"syncml"`
Checksum []byte `db:"checksum"`
}
if err := sqlx.SelectContext(ctx, ds.reader(ctx), &pending, pdStmt, pdArgs...); err != nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this safe to get from the reader?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, since it self-heals on the next tick. But I should not return an error if not found.

Comment thread server/datastore/mysql/microsoft_mdm.go
@getvictor

Copy link
Copy Markdown
Member Author

@ksykulev This one is ready for re-review, but we need to review/merge #47071 first before this one

Base automatically changed from 45635-reconciler to main June 11, 2026 06:03
@getvictor

Copy link
Copy Markdown
Member Author

@ksykulev I merged this one with main and resolved conflicts. Ready to re-review/approve

ksykulev
ksykulev previously approved these changes Jun 11, 2026
@getvictor

Copy link
Copy Markdown
Member Author

@ksykulev regenerated schema.sql again. Please approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch-set MDM profiles endpoint times out for large removals (Windows)

3 participants