Stencil support: Change CORE_3D_DEPTH_FORMAT to Depth24PlusStencil8

[raketenben]

What problem does this solve or what need does it fill?

Bevy currently uses Depth32 as the texture format for the depth buffer, necessitating a modified version of bevy to properly utilize stencils with respect to depth of other scene meshes.

This seems like an important feature to have in bevy, it should be considered sooner rather than later, to avoid larger refactors for this functionality down the line.

What solution would you like?

Changing CORE_3D_DEPTH_FORMAT from TextureFormat;:Depth32 to TextureFormat::Depth24PlusStencil8, should get it most of the way there, enabling third party crates to add further stencil functionality.

In the long term, it would be nice to allow materials or meshes to set a stencil reference, as well as the ability to pass a custom DepthStencilState to the mesh render pipeline in some way.

What alternative(s) have you considered?

If this can not be set as the default depth format, there should be an option to change the depth texture format for the specific application.

[ChristopherBiscardi]

It would be great to have an example of something that can only be done by making this change (or at least made easier), instead of using say, a custom render phase.

[raketenben]

This change would ease the creation of non-euclidic effects such as mirrors and portals.

Though both of these are already achievable through custom render phases or rendering to textures (e.g. bevy_portal),
these setups introduce additional overhead and complexity in the form of texture memory and setup.

Additionally, the stencil approach can be used with very little overhead for effects like partially rendering different objects occupying the same space, depending on “occlusion” through a stencil mesh. (Something not easily achievable with both prior mentioned approaches.)

[beicause]

Actually, Depth32FloatStencil8 offers better precision and it is more widely supported than Depth24PlusStencil8 on desktop. However on mobile, the opposite applies and many android devices doesn't support Depth32FloatStencil8. We should check the supported format and include them as fallback options.

[raketenben]

That definitely sounds like a good idea.
Besides the format change, stencil_ops in DepthAttachment would also need to be set, to allow writing to the stencil buffer.

I'm not sure how the StencilState for the Mesh Pipeline would be best handled.
This requires more consideration, but with a default stencil reference of 0 and this as the default StencilState:

StencilFaceState {
    compare: CompareFunction::Equal,
    fail_op: StencilOperation::Keep,
    depth_fail_op: StencilOperation::Keep,
    pass_op: StencilOperation::Zero,
}

-- the render result should not be affected at all, and still allow for conditional rendering depending on the state of the stencil buffer.

A stencil prepass could be introduced.
And, this is just me spitballing, but perhaps there could be an abstraction like:

• a "stencil layer component" (part of prepass, that writes the corresponding stencil values to the buffer)
• a "stencil member component" (for setting which reference value is checked against during stencil testing)

[raketenben]

Alright, so with the discussed changes, I got it working.

I had to use NoAutomaticBatching to make sure the stencil references got applied correctly.
set_stencil_reference can only be set for a whole renderpass, so that would need to be considered.

I hacked in the stencil reference via the ItemQuery inside the render function of DrawMesh.
There is probably a nicer way to do this, but I'm not familiar enough with bevy's rendering system.

Demo:

I feel like overall, the code changes needed to make this work were very minor.
I think there needs to be some further design discussion on this, but with a few pointers, as to the proper way to implement this, I would be happy to work on a pull request for this.

[IceSentry]

I don't really have any guidance on how this should be implemented but supporting this should be fully optional. If users don't need the stencil buffer they shouldn't have to deal with the potential precision loss or any other stencil related overhead.

Also, personally, I'm not convinced using this is that more efficient than doing a custom render phase. I can certainly see how it's probably easier to implement. I'm just not sure it's worth the added api complexity but that's to be determined on how complex it would be on the user side.

[beicause]

I want the built-in stencil support. I think stencil is a must for 3D. Godot recently added support for stencil which is well recieved.

Using D32S8 on desktop doesn't incur precision loss, and even D24S8 on mobile platforms, the trade-off of some precision for stencil functionality is worthwhile.

[raketenben]

The overall overhead for stencil support is minuscule. If this is really a concern, we could feature gate it entirely.

D32S8 is an optional feature, and would need to be queried for. On some platforms, D24 seems to come with performance advantages, so letting the user choose if they require the added precision of the format might be nice in general.

Currently, the CORE_3D_DEPTH_FORMAT is implemented as a constant, which makes this not quite as straightforward as I had hoped, especially since not all usage contexts, have access to resources.