feat(render) add 10-bit SDR support

[Myles-Trevino]

Summary

Add support for 10-bit SDR output

Motivation

Some monitors support bit depths higher than 8. When using semi-transparent windows with background blur, banding can become a problem, especially with dark gradients on OLED. 10-bit output makes these gradients smooth

Type of Change

• Bug fix
• New feature
• Breaking change
• Refactoring
• Build / packaging
• Documentation

Related Issue

Testing

Ran in a native session. Confirmed changing bit_depth from 8 to 10 improves color quality and reduces banding

Manual Coverage

• Tested in a nested Umbriel session
• Tested in a native Umbriel session
• Tested with multiple monitors
• Tested with a scaled output
• Tested with native Wayland applications
• Tested with X11 applications through xwayland-satellite
• Tested with the scrolling layout
• Tested with the dwindle layout

Screenshots / Videos

Checklist

• This PR is ready for review, or it is marked as Draft.
• This change fits SCOPE.md, or its scope was agreed in an issue or on Discord first.
• I read and followed the relevant guidance in CONTRIBUTING.md.
• I ran just format, or this PR has no C++ changes.
• I ran the relevant build, test, lint, or verification commands, or explained why they were not run.
• I functionally verified compositor behavior where automated checks are insufficient.
• I self-reviewed the changes.
• I checked for new warnings or errors.
• I updated docs/ and examples/config.toml, or this PR does not change user-facing configuration or behavior.
• I used canonical names for config keys, IPC actions, paths, and identifiers.

Additional Notes

[Ly-sec]

Thanks for working on this @Myles-Trevino . The feature and configuration interface look good, but I found two issues that should be fixed before merging.

Blocking issues

1. 10-bit SDR can be overwritten with XR24. In Output::applyConfiguredState(), a successful XR30 or XB30 selection is replaced with XR24 when leaving HDR or when HDR is unavailable. The fallback reason is then cleared, so IPC can report no problem while the output remains 8-bit. Please clear the HDR image description before probing SDR formats, then select SDR10 or XR24 as one coherent state.

2. FP16 effects use an insufficient capability check. ensure_offscreen_buffer() checks whether FP16 is renderable, but the effects paths also use linear filtering. On GLES2 hardware without GL_OES_texture_half_float_linear, this can produce broken or black effects. Please gate FP16 SDR buffers on that extension, or make every sampling path safe for FP16.

Additional concerns

• A tested SDR10 state can still fail during the real commit, but the retry path does not fall back to XR24.
• The tests cover parsing but not the relevant transitions. Please cover SDR8 to SDR10, HDR to SDR10, unavailable HDR, and rejected SDR10 fallback.
• Human-readable umbriel color does not show the new configured depth or fallback reason.

I like the direction, but I would request changes until the output transitions and FP16 capability handling are corrected.

[Myles-Trevino]

@Ly-sec Appreciate your quick review. I've tried to address the concerns you brought up in the follow-up commit

[Ly-sec]

Thanks, this is much closer @Myles-Trevino . Two retry issues remain:

• If the real HDR commit fails with bit_depth = 10, fallback jumps directly to XR24 instead of trying SDR10.
• The SDR10 fallback can leave VRR disabled when retrying XR24.

The new check also does not exercise active HDR or clear a previously set SDR10 fallback reason. Could you cover these before merge?

[Myles-Trevino]

@Ly-sec Added a fix for the fallback issues and did some refactoring. As far as I can tell, on headless, HDR always gets declined and SDR10 always gets accepted, and it looks like overriding that for testing could expand the scope of this PR a decent amount. The existing HDR feature doesn't seem to do this either, so leaving that as-is for now

[Ly-sec]

Thanks, the follow-up fixes the two retry issues from my previous comment @Myles-Trevino . I also agree that adding a general failure-injection system to the headless backend would expand the scope too much. The branches that headless cannot naturally reach can be covered with focused unit tests instead.

I reviewed the updated implementation as a whole, and there are still four correctness issues I think we should resolve before merging.

1. The FP16 capability check can regress HDR

The updated use_fp16 condition applies the half-float filtering requirement to both SDR10 and existing color-transformed passes. If the extension is absent, an HDR or software color-transform pass can now use an 8-bit UNORM intermediate. That can clip linear values above 1.0 and quantize the image before the final transform.

Color-transformed passes need to remain FP16. The filtering capability gate should apply only to the new SDR10 effect-buffer upgrade. The renderer should also distinguish FP16 renderability, FP16 linear filterability, and legacy GLES2 OES upload support. GLES3 provides half-float filtering in core and does not need to expose the legacy OES extension string.

2. Optimized blur caches do not follow bit-depth changes

optimized_buffer_ready() validates only width and height. If the optimized blur cache is populated under SDR8, switching to SDR10 can continue reusing that 8-bit buffer. Switching back can likewise retain the FP16 buffer.

The required DRM format needs to be part of cache validity so the cache is recreated when output precision changes.

3. Output fallback needs to integrate with the current mode fallback

The branch now conflicts with main, which has newer mode-selection and preferred-mode fallback logic. This needs more than a mechanical conflict resolution.

wlr_output_test_state() evaluates the complete pending state, so an invalid configured mode or an incompatible VRR state can make an SDR10 probe fail even when the render format itself is supported. The current helper can then record an SDR10 failure and stage XR24 before the preferred mode is attempted.

Please make the retry order deterministic and restart it for the preferred fallback mode:

1. Requested HDR, with requested VRR and then without it.
2. XR30 and XB30 when bit_depth = 10, with requested VRR and then without it.
3. SDR8, with requested VRR and then without it.
4. If the configured mode fails, repeat that complete sequence on the preferred mode.

The same sequence should continue when a state test succeeds but the real commit fails. Fallback reasons should only be stored after the relevant candidates are exhausted, otherwise a mode or VRR failure can be incorrectly reported as rejected SDR10.

4. IPC can report SDR10 as active when it is not

The current umbriel color logic infers active SDR10 from the configured depth, inactive HDR, and an empty fallback reason. That can report SDR10 as active for a disabled or DPMS-off output, or before XR30 or XB30 has actually committed.

Please keep configured intent separate from runtime state. Active SDR10 should require all of the following:

• the output is enabled
• HDR is inactive
• the committed render format is XR30 or XB30

The human-readable label should also make it clear that bit_depth is the configured SDR depth, not necessarily the active format. A successful later retry should clear a previous failure reason.

Testing without expanding the headless backend

I think a reasonable test split for this PR is:

• Keep the current headless success-path coverage for live SDR8 to SDR10 to SDR8 transitions.
• Extend that check to render an actual blur or animation, including populating the optimized cache before changing bit depth. The current check never exercises the UmbrielFX changes.
• Extract the output candidate selection into a testable helper and unit-test XR30 rejection followed by XB30, complete SDR10 rejection, VRR retry, real-commit failure, HDR failure followed by SDR10, and restarting on the preferred mode.
• Unit-test active IPC state for enabled, disabled, DPMS-off, HDR, XR30 or XB30, and XR24 fallback states.
• Manually verify active HDR to SDR10 and one SDR10 screenshot or capture on native hardware, since headless cannot represent those paths faithfully.

Each new regression check should be proven failure-first as required by the repository testing policy.

The documentation should also clarify that bit_depth = 10 requests a 10-bit compositor render format. It does not guarantee a 10-bit physical display link. Please mention the VRR fallback policy, direct-scanout and capture caveats, and bit_depth = 8 as the recovery setting.

With those points addressed, I think the feature and its user-facing configuration are in good shape.

[Myles-Trevino]

@Ly-sec Updated to fix issues and add tests

[Ly-sec]

Thanks, this is much closer @Myles-Trevino . I reviewed the latest head and verified that all 59 unit tests and focused headless checks 641 and 643 pass. There are still a few things to address before I can approve it:

1. The FP16 capability handling still conflates GLES3 core support with the legacy GL_OES_texture_half_float path. fp16_linear_filter allows the FP16 formats on GLES3, but those formats still use GL_HALF_FLOAT_OES. GLES3 core provides GL_HALF_FLOAT, and a driver is not required to accept the OES token without that extension. Please keep renderability, linear filtering, and the upload type separate.

2. Check 641 does not catch the optimized blur cache regression. I temporarily removed the new cache-format comparison, rebuilt, and the check still passed with the same screenshot luma. A regression check needs to fail when the stale 8-bit or FP16 cache can be reused.

3. The preferred-mode test only covers an XR24 failure followed by an XR24 success. Please cover restarting the complete HDR, SDR10, SDR8 sequence on the preferred mode, including clearing reasons produced by the first mode.

4. The new IPC tests construct JSON with sdr10_active already supplied and then assert that value. They do not exercise the real committed-state calculation. Please test the actual derivation for XR30 and XB30, SDR8 fallback, HDR-active, disabled, and DPMS-off outputs.

5. The documentation currently says effect intermediates are upgraded to FP16 unconditionally, while the implementation correctly gates that upgrade on renderer capabilities. Please qualify that sentence.

Once these are addressed, please also confirm the native active-HDR to SDR10 transition and native SDR10 capture on the updated head.