Android hosts deleted prematurely by host expiry: `EnrollOrbit` writes a different "never seen" sentinel than `CleanupExpiredHosts` recognizes

[getvictor]

---

💥 Actual behavior

When host_expiry_settings.host_expiry_enabled = true (any value of host_expiry_window), Android hosts that have re-enrolled — after their previous host record was deleted by an admin, by host-expiry, or by any other path — get immediately auto-deleted again at the next cleanups_then_aggregation cron tick. This loops every hour for as long as host expiry is enabled and the device keeps trying to talk to Fleet.

Sentinel mismatch:

• EnrollOrbit (server/datastore/mysql/hosts.go) initializes a new host's detail_updated_at, label_updated_at, policy_updated_at, and last_enrolled_at to:

  zeroTime := time.Unix(0, 0).Add(24 * time.Hour)  // = 1970-01-02 00:00:00 UTC

• CleanupExpiredHosts (server/datastore/mysql/hosts.go) nullifies the other sentinel:

  COALESCE(GREATEST(COALESCE(hst.seen_time, ne.last_seen_at), COALESCE(ne.last_seen_at, hst.seen_time)),
           NULLIF(h.detail_updated_at, '2000-01-01 00:00:00'),
           h.created_at) < DATE_SUB(NOW(), INTERVAL ? DAY)

  where '2000-01-01 00:00:00' is server.NeverTimestamp.

Because 1970-01-02 != 2000-01-01, NULLIF returns the literal 1970-01-02, the COALESCE chain treats the host as "last seen on 1970-01-02," and the comparison 1970-01-02 < NOW - any_window_DAYS is always true. The host is deleted on the next cron tick.

Why first-time Android enrollment is normally safe: the very first Google AMAPI ENROLLMENT pubsub event triggers addNewHost (server/mdm/android/service/pubsub.go) which calls NewAndroidHost → setTimesToNonZero (server/datastore/mysql/android.go). That sets host.DetailUpdatedAt = 2000-01-01, and the subsequent UPDATE (the orbit-enrolled-host-already-exists branch in NewAndroidHost) writes 2000-01-01 over the 1970-01-02 value the orbit-enroll path had just inserted. After that update, the cleanup SQL's NULLIF(..., '2000-01-01 00:00:00') recognizes the sentinel correctly, the COALESCE falls through to h.created_at, and the host survives.

Why re-enrollment after host record deletion is NOT safe: when the Fleet host record is deleted (any cause), Google AMAPI does not get notified and does not send a new ENROLLMENT pubsub event for the same device. From Google's perspective the device is still enrolled. The agent's next periodic check-in hits the orbit-enroll endpoint, EnrollOrbit inserts a fresh hosts row with detail_updated_at = 1970-01-02, and nothing overwrites that value. The cleanup cron deletes it again at the next hourly tick. The next orbit check-in creates another fresh row with 1970-01-02. The cycle repeats every hour indefinitely.

Why osquery / fleetd desktop hosts mostly don't hit this even on re-enrollment: their host_seen_times.seen_time is populated on the first osquery check-in (within seconds). The COALESCE reads hst.seen_time first, so detail_updated_at = 1970-01-02 is never consulted.

Why Android hosts hit it specifically: Android devices do not insert into host_seen_times (that table is osquery-only, per the comment at hosts.go). Android hosts rely entirely on detail_updated_at being refreshed by Google AMAPI pubsub events. On re-enrollment, no such event is generated.

🛠️ Expected behavior

CleanupExpiredHosts should treat both sentinels as "no real timestamp." Two equally valid fixes:

1. Update the cleanup SQL to nullify both sentinels (smallest, safest change, also tolerates any legacy rows that may exist in production):

   COALESCE(
     GREATEST(COALESCE(hst.seen_time, ne.last_seen_at), COALESCE(ne.last_seen_at, hst.seen_time)),
     NULLIF(NULLIF(h.detail_updated_at, '2000-01-01 00:00:00'), '1970-01-02 00:00:00'),
     h.created_at
   ) < DATE_SUB(NOW(), INTERVAL ? DAY)

2. Make EnrollOrbit use server.NeverTimestamp ('2000-01-01 00:00:00') at line 2436 of hosts.go. This also requires a data migration to update existing rows from 1970-01-02 to 2000-01-01.

Option 1 is sufficient and lower risk. Both together is ideal.

🧑‍💻 Steps to reproduce

These steps:

• Have been confirmed to consistently lead to reproduction in multiple Fleet instances. Three consecutive auto-deletion / re-enrollment cycles observed at the same hourly cadence on the dev server.

  1. Host expiry enabled.
  2. Both AMAPI enrollment AND the Fleet Android agent present.
  3. A host record gets deleted once — admin clicking Delete is the obvious trigger. From Google's perspective the device is still enrolled, so AMAPI does not re-emit an ENROLLMENT pubsub event.
  4. The Fleet Android agent re-enrolls via the orbit endpoint — its existing orbit_node_key now fails authentication. The agent reacts by re-running enrollment with the enroll secret it received in its enrollment token. That call hits EnrollOrbit and inserts a fresh hosts row with detail_updated_at = 1970-01-02. This is the step that plants the bad sentinel.
  5. The next hourly cleanup tick fires before AMAPI sends a status report for that device. The 1970-01-02 sentinel survives the NULLIF, the host is flagged as "last seen 1970," and CleanupExpiredHosts deletes it. Loop back to step 4.

🕯️ More info (optional)

[jkatz01]

QA Notes:

Looks good! One extra step that I had to go through was to add a Certificate Authority (followed ./tools/android/cert-auth-server) and a certificate for the Fleet the host is/gets assigned to, otherwise the android app never tries to re-enroll through the orbit endpoint.

Before (4.85.0):

• Host kept re-enrolling while trying to get a certificate, every cleanups_then_aggregation run deleted it.

After (4.86.0):

• Host kept re-enrolling while trying to get a certificate, cleanups_then_aggregation job never deletes it.