Android certificate template state and team assignment are lost when the host record is deleted and the device re-enrolls

[getvictor]

💥 Actual behavior

When an Android host's record is deleted from Fleet — by an admin, by auto host-expiry, or by any other path — and the same physical device subsequently re-enrolls via Google AMAPI, all certificate template delivery state is lost and the host's team assignment is reset to the enroll-secret's default team. The agent gets stuck returning HTTP 404 for every certificate fetch and never installs any new certs.

The sequence:

1. Fleet deletes the host record. Multiple code paths can do this:

   • DELETE /api/latest/fleet/hosts/{id} (admin action).
   • The hourly cleanups_then_aggregation cron → CleanupExpiredHosts (currently easy to trigger for Android hosts because of the sentinel-mismatch bug Android hosts deleted prematurely by host expiry: EnrollOrbit writes a different "never seen" sentinel than CleanupExpiredHosts recognizes #45258).
   • Android pubsub UNENROLLMENT event (when Google AMAPI reports the device as deleted).

   In all cases the cascade in hosts.go's additionalHostRefs map removes the matching rows from host_certificate_templates WHERE host_uuid = ?. The Android UNENROLLMENT handler at server/datastore/mysql/android.go also explicitly issues DELETE FROM host_certificate_templates WHERE host_uuid = ?.

2. Google AMAPI is not notified of the Fleet-side deletion. From Google's perspective the device is still enrolled in the enterprise.

3. On the agent's next periodic check-in (within seconds), it hits the orbit-enroll endpoint. EnrollOrbit (server/datastore/mysql/hosts.go) creates a brand-new hosts row keyed by the agent's orbit_node_key. The new row's team_id is whatever team the enroll secret used by the agent points at. The team assignment is lost.

4. Because addNewHost did not run, CreatePendingCertificateTemplatesForNewHost is never called for the new host, even though that function is the one that would have inserted host_certificate_templates rows for the new team's templates. The new host record starts life with zero rows in host_certificate_templates regardless of which team it landed on.

5. The Android agent's locally-cached managed config (delivered by Google AMAPI before the host was deleted) still references the old template IDs. The agent calls GET /api/fleetd/certificates/{id} for each one. The JOIN in GetCertificateTemplateByIdForHost (server/datastore/mysql/certificate_templates.go) finds no matching host_certificate_templates row → HTTP 404 "CertificateTemplateForHost was not found in the datastore".

6. The agent retries 5 times then backs off 15 minutes and retries again. The cycle continues indefinitely. No certificate ever installs.

7. If the admin manually re-transfers the host to the correct team (running Service.AddHostsToTeam → software worker → CreatePendingCertificateTemplatesForNewHost), templates do get queued.

From an admin's perspective, this looks like: "I assigned my Android host to a team with certificates configured, and certs never got delivered." The team-assignment UI succeeds and the host appears on the right team.

Concrete reproduction observed on the dev server (timeline abbreviated):

14:14:49  admin transfers host 19047 (Android) to team 19 (SAN QA, has cert template)
14:28:03  admin transfers a DIFFERENT host 21068 (also Android) to team 19
14:28:22  cert template "qa-preflight" (id 67) created on team 19
          CreatePendingCertificateTemplatesForExistingHosts inserts row for 21068
14:28:30  software_worker pushes managed config update for 21068 via AMAPI
14:52:01  host 21069 (the actually-active record for this physical device) auto-deleted
          cascade-wipes its host_certificate_templates rows
14:52:22  agent re-enrolls via orbit-enroll path -> new host 21070
          team_id = 7 (Personal mobile devices, from enroll secret)
          detail_updated_at = 1970-01-02 (from EnrollOrbit; confirms no addNewHost ran)
          host_certificate_templates is empty for this host_uuid
15:10:29  agent (cached managed config from before deletion) calls
          GET /api/fleetd/certificates/67  ->  HTTP 404
15:11:43  retry -> 404
15:12:18  retry -> 404
15:13:30  retry -> 404
15:15:54  retry -> 404 (5 retries exhausted, backs off 15 min)
15:44:14  admin manually transfers host 21070 to team 19 (workaround)
          worker inserts host_certificate_templates row for (21070's UUID, 67)
15:52:01  host 21070 auto-deleted at the next hourly cron tick
          cascade-wipes the row we just inserted
15:52:25  agent re-enrolls -> host 21071 on team 7, repeat

🛠️ Expected behavior

When the same physical Android device re-enrolls (same enterpriseSpecificID / same getAndroidHostKey), Fleet should restore the host's prior state so admins don't have to re-do the team move and so cert delivery resumes automatically. Two options, either or both:

1. Preserve team assignment across host record churn. When addNewHost is about to create a fresh row for a device whose UUID was seen before, look up the previous team via the audit log, retained IdP association, or a soft-delete tombstone row, and restore the team before inserting. Apple MDM does an analogous thing via host_dep_assignments.

2. Re-queue certificate templates after re-enrollment. When a host record is recreated for an existing UUID, automatically run something equivalent to CreatePendingCertificateTemplatesForExistingHosts for the host's current team. The agent's stale managed-config references should be resolvable as long as the templates still exist server-side.

🧑‍💻 Steps to reproduce

These steps:

• Have been confirmed to consistently lead to reproduction. Reproduced via both the auto-expiry path and the manual delete path.

Path A: manual delete

1. Enroll an Android device. Note its host id and UUID.
2. Move the host to a team with one or more certificate templates configured.
3. Wait for the worker to push the managed config (verify via GET /api/latest/fleet/hosts/{id}/certificates).
4. DELETE /api/latest/fleet/hosts/{id}.
5. Within seconds, the Android agent's next orbit-enroll check-in creates a fresh host record. Note the new id; same UUID.
6. Watch the agent's logs for /api/fleetd/certificates/{id}: every fetch returns HTTP 404 with "CertificateTemplateForHost was not found in the datastore".

🕯️ More info (optional)

[AndreyKizimenko]

Templates state & team are now preserved when host re-appears after deletion