Custom SCEP proxy challenge with non-printable characters breaks Windows certificate enrollment

[getvictor]

Fleet versions

• Discovered: 4.86.0

---

💥 Actual behavior

When a custom SCEP proxy certificate authority is configured with a challenge that contains characters which are not valid ASN.1 PrintableString characters (most commonly the underscore _), Windows SCEP certificate enrollment fails on the device with SCEP: Certificate enroll failed. Result: (The string contains a non-printable character.) (Win32 error 0x2ab0003). No certificate is installed, yet Fleet may still report the profile as Verified. The Fleet UI and API accept the offending challenge without warning, so the problem only surfaces later on the device.

This corrects the root-cause analysis in #46990. That issue attributed the failure to the base64url one-time challenge that Fleet packs into the proxy ServerURL path. On-device testing showed that is not the cause: Windows tolerates _ and - in the ServerURL and enrolls successfully. The character that actually breaks enrollment is in the SCEP .../Install/Challenge node, which Windows encodes as the PKCS#10 challengePassword using an ASN.1 PrintableString. The value of that node is the CA's configured challenge (ca.Challenge), not Fleet's one-time challenge. PrintableString permits A-Z a-z 0-9, space, and ' ( ) + , - . / : = ?, so the hyphen - is accepted but the underscore _ (and @ # * ! ~ ^ & $ %, etc.) is rejected.

🛠️ Expected behavior

Fleet should prevent an admin from saving a custom SCEP proxy challenge that contains characters Windows cannot use, and surface a clear validation error instead of letting the certificate silently fail to install on the device.

• UI: validate the custom SCEP proxy challenge field and reject non-printable (non-PrintableString) characters when the admin enters or edits the challenge, with a clear message.
• Backend: run the same character validation, but only when the challenge value is newly provided or changed. Skip validation when the submitted challenge is unchanged (for example, the masked placeholder sent on update) so that challenges stored before this fix continue to work and unrelated CA edits do not start failing. This keeps the change backward compatible with existing stored challenges.

🧑‍💻 Steps to reproduce

These steps:

• Have been confirmed to consistently lead to reproduction in multiple Fleet instances.
• Describe the workflow that led to the error, but have not yet been reproduced in multiple Fleet instances.

1. Configure a custom SCEP proxy certificate authority (Settings > Integrations > Certificates) with a challenge that contains an underscore, e.g. Fleet_SCEP_underscore.
2. Create a Windows SCEP configuration profile that uses $FLEET_VAR_CUSTOM_SCEP_PROXY_URL_<CA> for the ServerURL node and $FLEET_VAR_CUSTOM_SCEP_CHALLENGE_<CA> for the Challenge node, and assign it to a team with a Windows host.
3. Enroll/sync a Windows host so the profile is delivered.
4. Observe on the device (Event Viewer > Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin) that enrollment fails with The string contains a non-printable character. and no certificate is installed in Cert:\LocalMachine\My (or CurrentUser\My for ./User profiles).
5. Repeat with a challenge that is alphanumeric or uses only PrintableString-safe characters (e.g. Fleet-SCEP-dash-2026) and confirm the certificate installs successfully.

🕯️ More info (optional)

• macOS/iOS SCEP tolerates these characters (Apple accepts a broader encoding), so the same challenge works on Apple devices and only fails on Windows. This is why the problem is easy to miss.
• The SCEP standard (RFC 8894) defers challenge encoding to PKCS#9 v2.0 (RFC 2985), which states that challengePassword values SHOULD use PrintableString encoding. Windows implements this literally, so a Windows-compatible challenge must stay within the PrintableString character set. Industry MDMs (Intune, etc.) avoid this entirely by generating alphanumeric challenges.
• Empirically confirmed on a Windows 11 25H2 host: a challenge containing _ fails with "non-printable character"; the same setup with a hyphen-only challenge installs the certificate.
• Related: Custom SCEP challenge (base64url) breaks Windows enrollment with "non-printable character" #46990 (original report with the incorrect root cause), Improve Windows SCEP certificate verification #36995 (Windows SCEP profile shows Verified even though the certificate never installed, which hides this failure), Deploy certificates from custom SCEP certificate authority on Windows #26912 and Enable Okta Verify on Windows using a SCEP configuration profile #34913 (Windows custom SCEP / Okta Verify on Windows).

Risk assessment

• Risk level: High
• Risk description: This fix reverts a previous change, meaning the behavior being reverted may re-introduce the original issue that change was meant to address. The regression surface is SCEP certificate enrollment on Windows — a failure here results in devices silently failing to get certificates, which Fleet may still report as Verified. Both the fix and any re-introduced regression may not surface until devices sync.

Test plan

Make sure to go through [the list](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/ui/design-qa-considerations.md) and consider all events that might be related to this story, so we catch edge cases earlier.

Core flow — verify the fix (profile should FAIL, not Verified, with bad chars)

• Configure a custom SCEP proxy CA in Fleet (Settings → Integrations → Certificate authorities) with a challenge containing an underscore (e.g., Fleet_SCEP_Test_2026) — confirm Fleet accepts the save (no validation error blocking it)
• Create a Windows certificate profile using $FLEET_VAR_CUSTOM_SCEP_PROXY_URL_<CA> and $FLEET_VAR_CUSTOM_SCEP_CHALLENGE_<CA>, assign it to a Windows host, and allow it to sync
• Verify the profile status in Fleet shows Failed (not Verified)
• On the Windows host, open Event Viewer → Applications and Services Logs → Microsoft → Windows → DeviceManagement-Enterprise-Diagnostics-Provider → Admin and confirm the error SCEP: Certificate enroll failed. Result: (The string contains a non-printable character.) / 0x2ab0003 is present
• Confirm no certificate was installed — check certmgr.msc → Personal → Certificates
• Repeat with other non-PrintableString characters in the challenge: @, *, #, !, ~ — confirm Fleet shows Failed each time

Core flow — verify no regression (valid challenges still enroll successfully)

• Configure a custom SCEP proxy CA with a PrintableString-safe challenge (e.g., Fleet-SCEP-2026 — letters, numbers, hyphens only)
• Assign the profile to a Windows host and allow it to sync
• Verify the profile status in Fleet shows Verified
• Confirm a certificate was installed — check certmgr.msc → Personal → Certificates
• Confirm Event Viewer has no enrollment errors

Core flow — verify regression from the reverted fix (previously-blocked valid chars now accepted)

• Configure a custom SCEP proxy CA with a challenge containing a space (e.g., Fleet SCEP 2026) — the reverted fix was incorrectly rejecting spaces; confirm Fleet accepts the save with no error
• Assign the profile to a Windows host, sync, and confirm enrollment succeeds — Fleet shows Verified, cert present in certmgr.msc
• Repeat with other valid PrintableString special chars in the challenge: ', (, ), +, ,, -, ., /, :, =, ? — confirm Fleet accepts the save and enrollment succeeds

Edge cases

• Configure a SCEP CA with a challenge that is entirely numeric — confirm enrollment succeeds (Verified, cert installed)
• Update an existing SCEP CA's challenge from a valid value to one containing an underscore — confirm the profile re-syncs and transitions to Failed on enrolled hosts
• Update an existing SCEP CA's challenge from an invalid value (underscore) back to a valid one — confirm the profile re-syncs and transitions to Verified on enrolled hosts
• Verify that a host showing Failed due to a bad challenge will successfully re-enroll (Verified, cert installed) after the CA challenge is corrected, without manual intervention on the host

[AndreyKizimenko]

QA — verified on 4.89 RC (repro'd on 4.88 first)

Repro on 4.88 (fleet-v4.88.0, fix not present): configured a Custom SCEP CA with challenge secret_1 (underscore) — Fleet accepted it. Deployed a Windows SCEP profile:

• Fleet marked the profile Verified, but no cert was installed on the device.
• SCEP server logs showed GetCACaps → GetCACert but never PKIOperation (device aborted before submitting the CSR).
• Device Event Viewer (DeviceManagement-Enterprise-Diagnostics-Provider → Admin): SCEP: Certificate enroll failed. Result: (The string contains a non-printable character.) (0x2ab0003).

Confirms the bug: invalid challenge accepted + false "Verified" + silent on-device failure.

Fix verified on 4.89 RC — challenge is now validated at save time (both UI and API):

• UI: Invalid characters. Certificate enrollment only supports letters, numbers, and ' ( ) + , - . / : = ?
• API (POST /certificate_authorities): 20/20 invalid characters rejected with 422 (_ @ # * ! ~ ^ & $ % ; < > { | \ [, non-ASCII, tab, newline). Valid specials (' ( ) + , - . / : = ?`) accepted.
• Control: valid CA + secret challenge enrolls cleanly (PKIOperation reached, cert installed).

⚠️ Minor follow-up — space is rejected @getvictor

The validation rejects a space in the challenge (422), but space is a valid ASN.1 PrintableString character (and is listed as permitted in this issue's own description: "A-Z a-z 0-9, space, and ' ( ) + , - . / : = ?"). So Fleet is slightly stricter than necessary — a space-containing challenge that Windows would accept gets blocked. Low severity / fail-closed, but flagging in case it should be allowed.