Skip to content

Refactor makeAndroidAppAvailable to use staggered job queuing - #47880

Merged
ksykulev merged 4 commits into
mainfrom
47543-android-job-refactor
Jun 25, 2026
Merged

Refactor makeAndroidAppAvailable to use staggered job queuing#47880
ksykulev merged 4 commits into
mainfrom
47543-android-job-refactor

Conversation

@ksykulev

@ksykulev ksykulev commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Related issue: Resolves #47543

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.

Testing

  • Added/updated automated tests
  • QA'd all new/changed functionality manually

Summary by CodeRabbit

  • Refactor
    • Updated Android app availability to use staggered batch jobs instead of processing everything at once, improving throughput and smoothing workload.
  • New Features
    • Added batched handling that can perform per-host managed configuration substitution when variables are present, including scheduling “pending apply config” updates when required.
  • Configuration
    • Reduced the default Android batch size (mdm.android_batch_size) to 100.
  • Bug Fixes / Tests
    • Updated unit and integration tests to verify batching, staggering timing, full host coverage, and order-independent policy application behavior.

@ksykulev
ksykulev requested a review from a team as a code owner June 18, 2026 21:46
Copilot AI review requested due to automatic review settings June 18, 2026 21:46

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@coderabbitai

coderabbitai Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 2f226320-6758-4db5-93e6-d76fdfaadfff

📥 Commits

Reviewing files that changed from the base of the PR and between eaf0f00 and bacf89f.

📒 Files selected for processing (2)
  • server/config/config.go
  • server/worker/software_worker.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • server/worker/software_worker.go

Walkthrough

makeAndroidAppAvailable in server/worker/software_worker.go is refactored to resolve the host set upfront and enqueue staggered background batch jobs via QueueJobWithDelay, replacing the previous approach of sleeping between batches inside a single worker job. A new SoftwareWorkerTask constant (makeAndroidAppAvailableBatchTask) is added, SoftwareWorker.Run is extended to dispatch it, and a new makeAndroidAppAvailableBatch method performs the actual per-batch Android policy updates, with optional per-host fleet-variable substitution detection and branching. A helper queueMakeAndroidAppAvailableBatch constructs and enqueues the job args. The default batch size configuration is reduced from 1000 to 100. Tests are updated: TestMakeAndroidAppAvailableBatching is rewritten to validate job queuing behavior synchronously, two new unit tests validate the batch handler with and without fleet-variable substitution in managed configurations, and the integration test assertion is made order-independent to accommodate async job execution.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically describes the main refactoring objective: converting from sleep-based throttling to staggered job queuing for the makeAndroidAppAvailable function.
Description check ✅ Passed The description includes the required related issue, uses the checklist format appropriately with relevant items completed, and indicates testing was done and a changes file was added.
Linked Issues check ✅ Passed The PR implementation fully addresses issue #47543 by refactoring makeAndroidAppAvailable to use staggered job queuing with batching and delays, eliminating the 10K host limit, preserving idempotency and per-batch appConfigChanged handling, and following the existing queuing pattern.
Out of Scope Changes check ✅ Passed All changes directly support the staggered job queuing refactoring: software_worker.go implements the batching logic, tests validate batching and variable substitution, integration tests verify unchanged behavior, and config.go adjusts the batch size for the new approach.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 47543-android-job-refactor

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
server/worker/software_worker_test.go (1)

265-268: ⚡ Quick win

Assert the full batch-job arg contract in this test.

This verifies only part of the queued args. Please also assert AppTeamID and AppConfigChanged so regressions in phase-2 job payload wiring are caught.

Suggested patch
 		require.Equal(t, makeAndroidAppAvailableBatchTask, args.Task)
 		require.Equal(t, "com.example.app", args.ApplicationID)
+		require.Equal(t, uint(1), args.AppTeamID)
 		require.Equal(t, "enterprises/test", args.EnterpriseName)
+		require.False(t, args.AppConfigChanged)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/worker/software_worker_test.go` around lines 265 - 268, The test
assertions in the block checking Task, ApplicationID, and EnterpriseName are
incomplete and do not verify the full batch job argument contract. Add
require.Equal assertions for the AppTeamID and AppConfigChanged fields on the
args object immediately after the existing assertions to ensure the phase-2 job
payload wiring is properly validated and prevent future regressions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@server/worker/software_worker_test.go`:
- Around line 265-268: The test assertions in the block checking Task,
ApplicationID, and EnterpriseName are incomplete and do not verify the full
batch job argument contract. Add require.Equal assertions for the AppTeamID and
AppConfigChanged fields on the args object immediately after the existing
assertions to ensure the phase-2 job payload wiring is properly validated and
prevent future regressions.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5377a20c-6f07-484e-9209-46643eddcc33

📥 Commits

Reviewing files that changed from the base of the PR and between 567c02c and ce0540f.

📒 Files selected for processing (3)
  • changes/47543-android-staggered-job-queuing
  • server/worker/software_worker.go
  • server/worker/software_worker_test.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

This PR refactors the Android “make app available” worker flow to avoid sleeping inside a single long-running job by splitting work into staggered, delayed sub-jobs. This aligns makeAndroidAppAvailable with the existing staggered queuing pattern used by other Android software worker tasks and helps prevent hitting the integrations worker max runtime on large host sets.

Changes:

  • Added a new worker task (make_android_app_available_batch) and corresponding handler to execute work on a chunk of hosts.
  • Updated makeAndroidAppAvailable to resolve hosts at runtime and queue delayed batch jobs instead of doing in-job batching + sleeping.
  • Updated tests to validate that phase-1 queues the expected number of delayed batch jobs (and removed synctest/atomic usage).

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File Description
server/worker/software_worker.go Adds a new batch task + refactors makeAndroidAppAvailable to queue staggered sub-jobs and implements the batch execution logic.
server/worker/software_worker_test.go Updates batching test to assert job queuing + not_before staggering rather than observing in-job sleeps.
changes/47543-android-staggered-job-queuing User-visible change entry (content excluded from review by policy).
Files excluded by content exclusion policy (1)
  • changes/47543-android-staggered-job-queuing

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread server/worker/software_worker.go Outdated
Comment thread server/worker/software_worker.go Outdated
Comment thread server/worker/software_worker.go
@codecov

codecov Bot commented Jun 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.85714% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.33%. Comparing base (567c02c) to head (bacf89f).
⚠️ Report is 83 commits behind head on main.

Files with missing lines Patch % Lines
server/worker/software_worker.go 72.46% 12 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #47880      +/-   ##
==========================================
+ Coverage   67.23%   67.33%   +0.09%     
==========================================
  Files        3639     3641       +2     
  Lines      230043   230669     +626     
  Branches    11836    11836              
==========================================
+ Hits       154678   155313     +635     
+ Misses      61474    61387      -87     
- Partials    13891    13969      +78     
Flag Coverage Δ
backend 68.96% <72.85%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.

Files excluded by content exclusion policy (1)
  • changes/47543-android-staggered-job-queuing

Comment thread server/worker/software_worker.go
Comment thread server/worker/software_worker.go Outdated
Comment thread server/service/integration_mdm_setup_experience_test.go
Comment thread server/service/integration_mdm_setup_experience_test.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
server/worker/software_worker_test.go (1)

310-325: ⚡ Quick win

Strengthen pending-config assertions to prevent false positives.

This test only checks len(pendingConfigs) == 2; it can still pass if the wrong hosts (or duplicate host IDs) are updated. Capture UUID+policyVersion and assert exact host coverage (and expected version) for this batch.

Suggested assertion upgrade
-	var pendingConfigs []string
+	pendingConfigs := make(map[string]int64)
 	ds.SetAndroidAppInstallPendingApplyConfigFunc = func(ctx context.Context, hostUUID, applicationID string, policyVersion int64) error {
-		pendingConfigs = append(pendingConfigs, hostUUID)
+		pendingConfigs[hostUUID] = policyVersion
 		return nil
 	}
@@
-	require.Len(t, pendingConfigs, 2, "appConfigChanged=true should update both hosts")
+	require.Len(t, pendingConfigs, 2, "appConfigChanged=true should update both hosts")
+	require.Equal(t, int64(42), pendingConfigs["host-1"])
+	require.Equal(t, int64(42), pendingConfigs["host-2"])
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@server/worker/software_worker_test.go` around lines 310 - 325, The test for
makeAndroidAppAvailableBatch only asserts that pendingConfigs has length 2,
which doesn't verify that the correct hosts or policy versions were actually
updated. Modify the SetAndroidAppInstallPendingApplyConfigFunc mock to capture
both the hostUUID and policyVersion parameters (not just hostUUID) into a data
structure that can be verified. Then replace the simple length assertion with a
more thorough assertion that verifies both the exact host IDs that were updated
(matching the expected hosts map keys) and that they were all updated with the
expected policyVersion value (1) to prevent false positives from wrong hosts or
duplicates being processed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@server/worker/software_worker_test.go`:
- Around line 371-374: The test assertion checking for NotContains of the
placeholder variable is insufficient for verifying correct substitution. In the
loop iterating over capturedConfigs, replace the weak assertion with stronger
ones that verify the capturedConfigs count matches the expected number of hosts
and that each iteration of the loop contains the actual expected host UUID value
(not just the absence of the placeholder). This ensures that variable
substitution is actually working correctly with the proper values, not just that
placeholders are removed.

---

Nitpick comments:
In `@server/worker/software_worker_test.go`:
- Around line 310-325: The test for makeAndroidAppAvailableBatch only asserts
that pendingConfigs has length 2, which doesn't verify that the correct hosts or
policy versions were actually updated. Modify the
SetAndroidAppInstallPendingApplyConfigFunc mock to capture both the hostUUID and
policyVersion parameters (not just hostUUID) into a data structure that can be
verified. Then replace the simple length assertion with a more thorough
assertion that verifies both the exact host IDs that were updated (matching the
expected hosts map keys) and that they were all updated with the expected
policyVersion value (1) to prevent false positives from wrong hosts or
duplicates being processed.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 46237e14-6cda-4e85-a3a0-bc1338671926

📥 Commits

Reviewing files that changed from the base of the PR and between ce0540f and 54a8490.

📒 Files selected for processing (3)
  • server/service/integration_mdm_setup_experience_test.go
  • server/worker/software_worker.go
  • server/worker/software_worker_test.go

Comment thread server/worker/software_worker_test.go Outdated
Comment thread server/worker/software_worker.go Outdated
const androidSoftwareInstallStaggerInterval = 60 * time.Second
const (
androidSoftwareInstallStaggerInterval = 60 * time.Second
defaultAndroidBatchSize = 1000

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given current rate limits should this batch size perhaps be smaller? Feels like this leaves little overhead

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, 1000 would take up nearly 100% of our quota. That math only works if we don't share AMAPI quotas for our clients.

@ksykulev
ksykulev merged commit 214619d into main Jun 25, 2026
45 checks passed
@ksykulev
ksykulev deleted the 47543-android-job-refactor branch June 25, 2026 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor makeAndroidAppAvailable to use staggered job queuing instead of in-job sleep

6 participants