Android certificates: Retry 3 times

[getvictor]

Goal

User story
As an IT admin deploying configuration profiles,
I want Fleet to retry up to 3 times
so that I can be confident my end users will get the settings (e.g. certificate) w/o having to manually resend.

Docs: #42780
Demo: https://www.youtube.com/watch?v=K44wRg9_79M

Changes

Product

• UI changes: Enabled host activity card on Android host details page. Added installed_certificate activity item (success/failure). Added details modal for failed certificate installs matching software install failure pattern. Disabled "Upcoming" tab for Android hosts with tooltip. Added resent_certificate to host past activity types.
• CLI (fleetctl) usage changes: No changes
• YAML changes: No changes
• REST API changes: No new endpoints. PUT /api/fleetd/certificates/{id}/status now triggers automatic retry on failure (up to 3 times) and logs installed_certificate activity on each attempt.
• Fleet's agent (fleetd) changes: No changes (agent already reports status; server-side retry is transparent to agent)
• GitOps mode UI changes: No changes
• GitOps generation changes: No changes
• Activity changes: Added installed_certificate activity type with status field ("installed" or "failed_install") and optional detail field. Activity is host-only and fleet-initiated. Documented in audit-logs.md.
• Permissions changes: No changes
• Changes to paid features or tiers: No changes (Premium only, same as existing certificate support)
• My device and fleetdm.com/better changes: No changes
• Usage statistics: No changes
• Other reference documentation changes: Updated docs/Contributing/product-groups/security-compliance/android-certificates.md with retry behavior docs. Updated docs/Contributing/reference/audit-logs.md with installed_certificate activity.
• First draft of test plan added
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed

Engineering

• Test plan is finalized
• Contributor API changes: No changes
• Feature guide changes: No changes
• Database schema migrations: Added retry_count INT UNSIGNED NOT NULL DEFAULT 0 column to host_certificate_templates table (migration 20260331000000).
• Load testing: Not required. Retry logic is per-host-per-certificate and bounded by MaxCertificateInstallRetries = 3.
• Load testing/osquery-perf improvements: Not required
• This is a premium only feature: Yes, because certificate templates are premium only.

ℹ️ Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

Implementation summary

Backend (PR #42734)

Server-side retry: When the Android agent reports status=failed via PUT /api/fleetd/certificates/{id}/status:

1. Activity is logged (installed_certificate with status: "failed_install") on every failure for IT admin visibility.
2. If retry_count < MaxCertificateInstallRetries (3), the certificate is atomically reset to pending with retry_count incremented. The existing cron job re-delivers it on the next cycle.
3. If retry_count >= MaxCertificateInstallRetries, the failure is terminal and persisted via UpsertCertificateStatus.

Manual resend: ResendHostCertificateTemplate sets retry_count = MaxCertificateInstallRetries so the resend gets exactly one attempt with no automatic retry, matching Apple resend behavior.

Certificate renewal: SetAndroidCertificateTemplatesForRenewal resets retry_count = 0 for a fresh retry budget.

Frontend (PR #42608, merged to main)

• Enabled activity card on Android host details (HostDetailsPage.tsx)
• InstalledCertificateActivityItem -- renders success/failure with bold styling, details modal on failure only
• ResentCertificateActivityItem -- renders resent certificate on host activity feed
• CertificateInstallDetailsModal -- error icon, status message, collapsible error detail (matching software install modal pattern)
• Disabled "Upcoming" tab for Android with tooltip

QA

Risk assessment

• Requires load testing: No
• Risk level: Low
• Risk description: Retry logic is bounded (max 3), uses existing cron delivery pipeline, and is tested end-to-end.

Test plan

1. Certificate install success: Deploy a certificate to an Android host. Verify installed_certificate activity appears with status: "installed" on the host activity feed. No details modal.
2. Certificate install failure with auto-retry: Trigger a certificate install failure (e.g., misconfigure SCEP server). Verify:
   • installed_certificate activity with status: "failed_install" appears after each failure
   • Certificate status resets to pending (visible in OS settings)
   • After 3 retries (4 total attempts), status becomes terminally failed
3. Failure details modal: Click on a failed certificate activity. Verify modal shows error icon, failure message with bold cert name and host name, collapsible "Details" section with error text, and "Done" button.
4. Manual resend after terminal failure: Click "Resend" on a terminally failed certificate. Verify:
   • Status resets to pending
   • If delivery fails again, it is immediately terminal (no auto-retry)
5. Upcoming tab disabled: On an Android host details page, verify "Upcoming" tab is grayed out with tooltip "Currently, upcoming activity is only supported for macOS, Windows, Linux, iOS, and iPadOS hosts."
6. Resent certificate activity: After clicking "Resend", verify resent_certificate activity appears on the host activity feed.
7. Non-Android hosts unaffected: Verify activity card, upcoming tab, and all existing activities work normally on macOS/Windows/Linux/iOS hosts.

Testing notes

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan.
2. QA: Added comment to user story confirming successful completion of test plan.

[rachaelshaw]

Original description

macOS profiles have a fixed 1 time server-based retry. In terms of certificates, this catches issues on Fleet server or the terminal CA or infra/network issues between the two. During retry, the server issues a new 1-time dynamic challenge to the client.

We want to support a 1-time server retry for Android certs to match macOS profile behavior

[noahtalerman]

User story
As an IT admin,
I want a 1-time server retry for Android certs
so that it matches macOS profile behavior.

@getvictor we already have retries right?

This is my understanding of the current behavior is (discussed during testing call w/ customer-pingali):

[getvictor]

@noahtalerman We have client retries only. The server issues a one-time Fleet-specific challenge. If the client "wastes" it by hitting some network issue, then it is stuck. In this retry, the server would issue a new Fleet-specific challenge.