GitOps: "503 Service Unavailable" from Apple causes GitOps pipeline to fail

[jakestenger]

Fleet version: Self-hosted - will update when known (4.74.0 | Updated to 4.76.1: Issue Persists)

When using Fleet GitOps to manage multiple teams with automatic enrollment profiles, Fleet re-uploads the macOS/iOS Setup Assistant profile to Apple on every GitOps run. If Apple’s DEP endpoint intermittently returns 503 Service Unavailable, Fleet immediately fails the upload and terminates the entire GitOps deployment with a 422 Validation Failed error, even when the profile content has not changed.

This results in unstable GitOps runs that frequently fail due to upstream Apple outages.

---

💥  Actual behavior

https://fleetdm.unthread.io/dashboard/inbox/me/conversations/a25e1497-1997-47b4-a537-34461d145385

Fleet always attempts to upload the enrollment profile to Apple.
Any upstream Apple DEP 503 causes the upload to fail.
Fleet returns a 422 and terminates the GitOps run.

See Unthread: https://fleetdm.unthread.io/dashboard/inbox/me/conversations/a25e1497-1997-47b4-a537-34461d145385

🛠️ To fix

Fleet server should ignore requests from the client (e.g. GitOps) if the automatic enrollment profile is not changed (same content).

🧑‍💻  Steps to reproduce

*This has not been reproduced by a Fleet team member. These are the customers reproduction steps.

• Configure multiple teams with enrollment_profiles/automatic defined in GitOps.
• Run Fleet GitOps sync.
• Apple’s https://mdmenrollment.apple.com/profile intermittently returns 503.
• Fleet attempts to POST the profile regardless of whether the GitOps profile content has changed.

🕯️ More info (optional)

Error: uploading macOS setup assistant for team "montamo | macOS": POST /api/latest/fleet/enrollment_profiles/automatic received status 422 Validation Failed: sending profile to Apple failed: Post "https://mdmenrollment.apple.com/profile": DEP auth error: 503 Service Unavailable:

[zayhanlon]

since this bug is still in drafting, requesting p1 from @noahtalerman rather than an EM

[zayhanlon]

We provided customer deebradel this workaround:
"Since Apple’s services are seeing intermittent flakiness, running the apply for all teams in a single batch creates a high statistical probability of failure (even a 1% Apple error rate means the full run will almost always fail).
To unblock you immediately we recommend temporarily stopping the "apply all" process. Instead, please run the gitops command targeting only the specific teams that require configuration changes right now. You can do this by passing the specific team files using the -f flag

Example syntax:
fleetctl gitops -f ~/repositories/gitops/main/default.yml -f ~/repositories/gitops/main/teams/no-team.yml -f ~/repositories/gitops/main/teams/ade.yml --dry-run
(Remove --dry-run when ready to apply)

This reduces the API calls to Apple and allows you to push critical updates for specific customers without the whole pipeline failing due to an unrelated error."