Android Load testing with osquery perf - #48535
Conversation
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
There was a problem hiding this comment.
Pull request overview
This PR extends Fleet’s existing osquery-perf load-testing harness to simulate Android MDM behavior (enrollment, periodic status reporting, and command acknowledgements) and adds a lightweight Android Management API (AMAPI) mock/proxy to support realistic policy/command flows at scale.
Changes:
- Add Android-specific stats counters and reporting in
osquery-perf. - Introduce an
androidAgentsimulator that sends Android PubSub push payloads to Fleet and polls a mock AMAPI proxy for policy/command state. - Add
android-amapi-mock, a standalone mock/proxy server for AMAPI endpoints plus a coordination API used byosquery-perf.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| cmd/osquery-perf/osquery_perf/stats.go | Adds Android counters and logs Android activity alongside existing osquery/orbit/MDM metrics. |
| cmd/osquery-perf/android.tmpl | Placeholder template enabling --os_templates=android to work with existing template parsing. |
| cmd/osquery-perf/android_agent.go | Implements a fake Android device loop (enroll → periodic status → command ack) using Fleet’s PubSub endpoint + mock proxy coordination. |
| cmd/osquery-perf/agent.go | Wires Android template selection and adds Android-specific CLI flags for osquery-perf. |
| cmd/android-amapi-mock/middleware.go | Adds middleware for forwarding non-fake-device requests to Google AMAPI when configured. |
| cmd/android-amapi-mock/main.go | Implements the android-amapi-mock command wiring: routes, forwarding, and coordination API endpoints. |
| cmd/android-amapi-mock/handlers.go | Adds mock handlers for device/policy/command endpoints and coordination endpoints. |
| cmd/android-amapi-mock/google_forwarder.go | Implements Google AMAPI forwarding via the official SDK when credentials are provided. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThis change adds a mock Android Management API server with optional Google forwarding, in-memory device and policy state, and HTTP routes for device, policy, enrollment, application, web app, and enterprise operations. It also extends osquery-perf with an Android device simulator that registers with the mock server, sends PubSub enrollment/status/command messages to Fleet, adds Android-specific CLI flags and template support, and records Android counters in load-test stats. 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (2)
cmd/osquery-perf/android_agent.go (1)
198-209: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRegister/enrollment failures aren't counted in Stats.
Unlike the per-iteration status-report/poll/command-ack failures in the loop below (which call
a.stats.IncrementAndroidErrors()), the initialregisterWithProxyandsendEnrollmentfailures just log and return, leaving these failures invisible in the aggregated stats output.🔧 Suggested fix
if err := a.registerWithProxy(); err != nil { log.Printf("Android agent %d: failed to register with proxy: %v", a.agentIndex, err) + a.stats.IncrementAndroidErrors() return } // Step 2: Send ENROLLMENT PubSub to Fleet if err := a.sendEnrollment(); err != nil { log.Printf("Android agent %d: enrollment failed: %v", a.agentIndex, err) + a.stats.IncrementAndroidErrors() return }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cmd/osquery-perf/android_agent.go` around lines 198 - 209, The initial register/enrollment failures in androidAgent.runLoop are only logged, so they never show up in aggregated error stats. Update the registerWithProxy and sendEnrollment error paths in runLoop to also call a.stats.IncrementAndroidErrors() before returning, matching the existing failure handling used for the per-iteration status-report/poll/command-ack paths.cmd/osquery-perf/agent.go (1)
4059-4062: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winValidate Android flags before starting any agents, not mid-loop.
The required-flag check only fires when the loop reaches the first
android.tmplhost. If--os_templatesmixes android with other templates, earlier non-android hosts will already be enrolling against Fleet before this fatal error triggers, wasting the partial run.🔧 Suggested fix: validate right after flag parsing
flag.Parse() rand.Seed(*randSeed) + + if strings.Contains(*osTemplates, "android") && + (*androidPubSubToken == "" || *androidProxyAddress == "" || *androidEnterpriseID == "") { + log.Fatalf("Android template requires --android_pubsub_token, --android_proxy_address, and --android_enterprise_id flags") + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cmd/osquery-perf/agent.go` around lines 4059 - 4062, Move the Android required-flag validation out of the host-processing loop in agent.go and run it immediately after flag parsing, before any enrollment or agent-start logic begins. Use the existing android.tmpl check and the --android_pubsub_token, --android_proxy_address, and --android_enterprise_id flags to fail fast up front so mixed-template runs cannot partially enroll non-Android hosts before the fatal error.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cmd/android-amapi-mock/google_forwarder.go`:
- Around line 104-108: ForwardDevicesList is swallowing Google API list errors
by logging and breaking out of the loop, which can make callers think the
request succeeded with a partial or empty result. Update the error path in
googleForwarder’s device-list loop to return the failure from ForwardDevicesList
instead of continuing, and make the HTTP handler that calls it propagate that
error as a failed response rather than a 200. Use the existing
ForwardDevicesList and call.Do symbols to locate the affected flow.
In `@cmd/android-amapi-mock/handlers.go`:
- Around line 229-232: The asynchronous Google forward in the handler reuses the
live request, so `ForwardPoliciesPatch` may see a canceled context or a closed
body after the handler returns. In the `google != nil &&
hasSeenRealDevice.Load()` branch, clone the incoming request before starting the
goroutine by creating a detached copy with a new context and a copied body, then
pass that cloned request into `google.ForwardPoliciesPatch` instead of `r`.
- Around line 104-107: The request body parsing in the handler that reads r.Body
and unmarshals into reqBody currently ignores malformed JSON, causing bad device
patch requests to succeed as empty patches. Update this logic to check the
json.Unmarshal error in the relevant handler in handlers.go and return a 400
response for invalid JSON instead of proceeding; keep the fix localized around
the request parsing path that handles the patch request.
- Around line 315-323: The catch-all in handleCatchAll currently returns a
successful empty JSON response for unsupported AMAPI paths, which masks missing
coverage. Update handleCatchAll to send an error status for unhandled requests
and return an error payload instead of "{}"; keep the existing logging and use
the googleForwarder context to preserve the current log message behavior.
- Around line 189-202: Validate the parsed pageToken before using it to slice
allDevices in the handler that builds resp. The current offset can become
negative from fmt.Sscanf, so add bounds checks to clamp offset to the valid
range before computing end and before evaluating allDevices[offset:end],
ensuring both offset and end are safe for slicing.
- Around line 170-184: The device list handler is mixing fake devices from all
enterprises because it always uses store.allDeviceNames() in the GET
/v1/enterprises/{eid}/devices flow. Update the handler in handlers.go so the
fake-device lookup is scoped by the requested enterprise ID (the same
r.PathValue("eid") used for enterpriseName), and only append fake devices
belonging to that enterprise before returning allDevices.
In `@cmd/osquery-perf/android_agent.go`:
- Line 268: The outbound HTTP calls in registerWithProxy, pollProxyState, and
sendPubSubMessage currently use http.Post/http.Get with http.DefaultClient and
no timeout, so they can hang indefinitely. Update these paths to use a shared
http.Client with an explicit Timeout (while preserving the existing TLS
transport customization from main in agent.go), and route the proxy
registration, polling, and Fleet PubSub requests through that client instead of
the default helpers.
---
Nitpick comments:
In `@cmd/osquery-perf/agent.go`:
- Around line 4059-4062: Move the Android required-flag validation out of the
host-processing loop in agent.go and run it immediately after flag parsing,
before any enrollment or agent-start logic begins. Use the existing android.tmpl
check and the --android_pubsub_token, --android_proxy_address, and
--android_enterprise_id flags to fail fast up front so mixed-template runs
cannot partially enroll non-Android hosts before the fatal error.
In `@cmd/osquery-perf/android_agent.go`:
- Around line 198-209: The initial register/enrollment failures in
androidAgent.runLoop are only logged, so they never show up in aggregated error
stats. Update the registerWithProxy and sendEnrollment error paths in runLoop to
also call a.stats.IncrementAndroidErrors() before returning, matching the
existing failure handling used for the per-iteration
status-report/poll/command-ack paths.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 420d2d0b-a623-43da-ae9c-78a99b65d9e5
📒 Files selected for processing (8)
cmd/android-amapi-mock/google_forwarder.gocmd/android-amapi-mock/handlers.gocmd/android-amapi-mock/main.gocmd/android-amapi-mock/middleware.gocmd/osquery-perf/agent.gocmd/osquery-perf/android.tmplcmd/osquery-perf/android_agent.gocmd/osquery-perf/osquery_perf/stats.go
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #48535 +/- ##
==========================================
- Coverage 67.97% 67.74% -0.24%
==========================================
Files 3801 3777 -24
Lines 239965 240856 +891
Branches 12656 12437 -219
==========================================
+ Hits 163123 163158 +35
- Misses 62058 62830 +772
- Partials 14784 14868 +84
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@cmd/android-amapi-mock/handlers.go`:
- Around line 353-358: The catch-all response in handleCatchAll is building JSON
with raw r.URL.Path text, which can produce invalid output for paths needing
escaping. Update the handler to construct the error payload as a structured
value and serialize it with a JSON encoder/marshaler instead of fmt.Fprintf,
keeping the existing logging and 501 response behavior intact.
- Around line 257-277: The policy PATCH handler currently advances the version
and may forward the request even when reading or parsing the body fails, so
invalid policy updates can still appear successful. In the PATCH flow in
handlers.go, validate the request body in the policy update handler before
calling policyVersionCounter.Add, store.setPolicyVersion, or
google.ForwardPoliciesPatch; if io.ReadAll fails or the body is not valid JSON,
return an error response immediately and do not update version state. Use the
existing handler path around bodyBytes, policyVersionCounter, and
ForwardPoliciesPatch to keep the rejection logic centralized.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 95457d12-5a3e-45ff-a6f4-6d2d78715edb
📒 Files selected for processing (7)
cmd/android-amapi-mock/google_forwarder.gocmd/android-amapi-mock/handlers.gocmd/android-amapi-mock/main.gocmd/android-amapi-mock/middleware.gocmd/osquery-perf/agent.gocmd/osquery-perf/android_agent.gocmd/osquery-perf/osquery_perf/stats.go
🚧 Files skipped from review as they are similar to previous changes (6)
- cmd/osquery-perf/agent.go
- cmd/osquery-perf/osquery_perf/stats.go
- cmd/android-amapi-mock/google_forwarder.go
- cmd/osquery-perf/android_agent.go
- cmd/android-amapi-mock/main.go
- cmd/android-amapi-mock/middleware.go
There was a problem hiding this comment.
🧹 Nitpick comments (1)
cmd/android-amapi-mock/google_forwarder.go (1)
204-215: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider preserving the underlying Google error status.
writeGoogleErroralways emits502 Bad Gateway, even when the underlyinggoogleapi.Errorcarries a more specific code (e.g.,404 NOT_FOUND,400 INVALID_ARGUMENT). Callers of the mock server can't distinguish "device not found on Google" from "Google API unreachable." Since this is a load-testing/mock tool the impact is limited, but extracting*googleapi.Error(viaerrors.As) would make error semantics more realistic for load-test scenarios that exercise error paths.♻️ Optional refactor
func writeGoogleError(w http.ResponseWriter, err error) { log.Printf("googleForwarder: Google API error: %v", err) w.Header().Set("Content-Type", "application/json") - w.WriteHeader(http.StatusBadGateway) - _ = json.NewEncoder(w).Encode(map[string]any{ - "error": map[string]any{ - "code": 502, - "message": err.Error(), - "status": "BAD_GATEWAY", - }, - }) //nolint:errcheck + code := http.StatusBadGateway + status := "BAD_GATEWAY" + var gErr *googleapi.Error + if errors.As(err, &gErr) { + code = gErr.Code + } + w.WriteHeader(code) + _ = json.NewEncoder(w).Encode(map[string]any{ + "error": map[string]any{ + "code": code, + "message": err.Error(), + "status": status, + }, + }) //nolint:errcheck }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cmd/android-amapi-mock/google_forwarder.go` around lines 204 - 215, Preserve the underlying Google API status in writeGoogleError instead of always returning a generic 502. Update the function to detect *googleapi.Error using errors.As, and when present, use that error’s status/code and message in the JSON response and HTTP status. Keep the existing fallback behavior for non-Google errors so the mock server still returns 502 when no specific upstream code is available.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@cmd/android-amapi-mock/google_forwarder.go`:
- Around line 204-215: Preserve the underlying Google API status in
writeGoogleError instead of always returning a generic 502. Update the function
to detect *googleapi.Error using errors.As, and when present, use that error’s
status/code and message in the JSON response and HTTP status. Keep the existing
fallback behavior for non-Google errors so the mock server still returns 502
when no specific upstream code is available.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 602f40dc-7510-473e-b35a-1fa8d2f851a8
📒 Files selected for processing (4)
cmd/android-amapi-mock/google_forwarder.gocmd/android-amapi-mock/handlers.gocmd/android-amapi-mock/middleware.gocmd/osquery-perf/android_agent.go
🚧 Files skipped from review as they are similar to previous changes (3)
- cmd/android-amapi-mock/middleware.go
- cmd/osquery-perf/android_agent.go
- cmd/android-amapi-mock/handlers.go
There was a problem hiding this comment.
♻️ Duplicate comments (1)
cmd/android-amapi-mock/handlers.go (1)
254-266: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winOnly half the previous fix landed — malformed JSON still advances the policy version.
The
io.ReadAllerror is now handled, but the body is still not validated as JSON beforepolicyVersionCounter.Add(1)/store.setPolicyVersion(name, version)run. A malformed (but readable) body will still be accepted as a successful policy patch, silently corrupting the simulated policy-version state under load.🐛 Proposed fix
if err != nil { http.Error(w, "failed to read request body: "+err.Error(), http.StatusBadRequest) return } + if len(bodyBytes) > 0 && !json.Valid(bodyBytes) { + http.Error(w, "invalid JSON", http.StatusBadRequest) + return + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@cmd/android-amapi-mock/handlers.go` around lines 254 - 266, The policy patch handler in handlers.go still advances state before validating the request body, so malformed but readable JSON is treated as success. In the policy update path inside the handler that reads r.Body, validate bodyBytes as JSON immediately after io.ReadAll and before policyVersionCounter.Add(1) and store.setPolicyVersion(name, version), returning an error response on parse failure. Use the existing request-handling flow and the same handler function to ensure only valid policy patches increment the version.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@cmd/android-amapi-mock/handlers.go`:
- Around line 254-266: The policy patch handler in handlers.go still advances
state before validating the request body, so malformed but readable JSON is
treated as success. In the policy update path inside the handler that reads
r.Body, validate bodyBytes as JSON immediately after io.ReadAll and before
policyVersionCounter.Add(1) and store.setPolicyVersion(name, version), returning
an error response on parse failure. Use the existing request-handling flow and
the same handler function to ensure only valid policy patches increment the
version.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 560e6630-9eb5-4b3e-bb44-5f80ec145ba7
📒 Files selected for processing (1)
cmd/android-amapi-mock/handlers.go
CI Feedback 🧐A test triggered by this PR failed. Here is an AI-generated analysis of the failure:
|
getvictor
left a comment
There was a problem hiding this comment.
I think this is fine to get started. Have you run it on a bunch of devices already?
The main issue is I'm not sure if the certificates flow works. But then again we don't expect it to be the main source of the load.
|
|
||
| // getCertificateTemplate fetches a certificate template from Fleet. | ||
| func (a *androidAgent) getCertificateTemplate(certID uint) (*certTemplateInfo, error) { | ||
| url := fmt.Sprintf("%s/api/v1/fleet/fleetd/certificates/%d", a.serverAddress, certID) |
There was a problem hiding this comment.
I believe the path is /api/fleetd/certificates/{id} with no version prefix
| var req struct { | ||
| Applications []struct { | ||
| ManagedConfiguration json.RawMessage `json:"managedConfiguration"` | ||
| } `json:"applications"` |
There was a problem hiding this comment.
Is this right? I believe we use modifyPolicyApplications which sends something like: {"changes":[{"application":{"managedConfiguration":...}}]}
| defer statusTicker.Stop() | ||
|
|
||
| // Track which certificate templates we've already verified so we don't re-verify | ||
| verifiedCerts := make(map[uint]struct{}) |
There was a problem hiding this comment.
Renewal flow uses the same template ID. But we can worry about renewals later.
| } | ||
|
|
||
| // Process certificate templates from the proxy state | ||
| for _, certID := range state.PendingCertificates { |
There was a problem hiding this comment.
I don't see where/how PendingCertificates are cleared.
There was a problem hiding this comment.
PendingCertificates on the fake device are never cleared. The agent checks Fleet's status on every poll and skips certs that aren't "delivered" or are already verified. We could clear them after the agent processes them. But that would break renewals because the agent wouldn't know about the cert anymore after clearing.
| } | ||
|
|
||
| // sendEnrollment sends an ENROLLMENT PubSub message to Fleet. | ||
| func (a *androidAgent) sendEnrollment() error { |
There was a problem hiding this comment.
Nit. This seems similar to sendStatusReport. Maybe we can use a helper?
| d.mu.Lock() | ||
| resp := map[string]any{ | ||
| "name": name, | ||
| "appliedPolicyVersion": d.PolicyVersion, |
There was a problem hiding this comment.
Are versions numbers or strings?
There was a problem hiding this comment.
| } | ||
| } | ||
|
|
||
| func handleEnterprisesList(store *deviceStore) http.HandlerFunc { |
There was a problem hiding this comment.
Note: Check to make sure this works as expected with server's cron:
// VerifyExistingEnterpriseIfAny checks if there's an existing enterprise in the database
// and if so, verifies it still exists in Google API. If it doesn't exist, performs cleanup.
// Returns fleet.IsNotFound error if enterprise was deleted, nil if no enterprise exists or verification passed.
VerifyExistingEnterpriseIfAny(ctx context.Context) error
There was a problem hiding this comment.
The cleanup cron runs as part of the cleanups_then_aggregation schedule which is every ~25 minutes. osquery-perf's --start_period is 5 minutes. So fake devices will register well before the first cleanup cron fires.
| PendingCommands: d.PendingCommands, | ||
| PendingCertificates: d.PendingCertificates, | ||
| } | ||
| d.PendingCommands = nil |
There was a problem hiding this comment.
Nit. PendingCommands is cleared. Can we get stuck or miss a command if there is an issue on the agent? Maybe we don't want a BYOD agent to support an enterprise command, for example.
There was a problem hiding this comment.
Command consumption is fire-and-forget for load testing purposes. Command reliability and BYOD filtering are Fleet server-side concerns. I think this is acceptable for a load test?
| tr := http.DefaultTransport.(*http.Transport).Clone() | ||
| tr.TLSClientConfig = tlsConfig | ||
| http.DefaultClient.Transport = tr | ||
| http.DefaultClient.Timeout = 30 * time.Second |
There was a problem hiding this comment.
Was the agent hanging for 30+ seconds during load tests?
There was a problem hiding this comment.
There was no actual hang observed. It is a defensive timeout. I chose 30 seconds as an upper bound so requests under load would stop rather than blocking the goroutine forever.
| } | ||
|
|
||
| // Step 2: Send ENROLLMENT PubSub to Fleet | ||
| if err := a.sendEnrollment(); err != nil { |
There was a problem hiding this comment.
If we try to enroll 100 agents and some fail to enroll initially are we okay with them not retrying?
There was a problem hiding this comment.
good catch. Now each enrollment step retries up to 5 attempts (3 for orbit) with linear backoff (5s, 10s, 15s, 20s, 25s).
dantecatalfamo
left a comment
There was a problem hiding this comment.
🦞 L(obster)
🧄 G(arlic)
🌮 T(aco)
🔍 M(agnify)
Related issue: Resolves #26225
go run ./cmd/android-amapi-mock --listen :9999 --google-credentials "$(cat </path/to/credentials/file.json)"go run ./cmd/osquery-perf --server_url https://localhost:8080 --enroll_secret <secret> --os_templates android:2 --host_count 2 --android_pubsub_token '<token>' --android_proxy_address http://localhost:9999 --android_enterprise_id <enterprise id> --android_status_interval 30sTesting
Summary by CodeRabbit
android-amapi-mockHTTP server with coordination endpoints and mocked/forwarded Android Management API routes for device, policy, enrollment token, application, web app, and enterprise flows.