KHR_materials_basisu and independent data channels

[donmccurdy]

Opening a new thread here, based on discussion around #1612.

Storing three independent data channels in compressed textures introduces significant artifacts with current approaches (see #1612). As a result, these two cases in the current glTF material specification require consideration:

• occlusion/rough/metal (RGB) texture
• three-channel normal (RGB) texture

These could be solved with:

• (1) occlusion (R) and rough/metal (RG) textures, and
• (2) two-channel normal (RG) texture

A general-purpose texture-swizzling extension would be an alternative to (1). Currently that is harder to support on the web, but it won't always be so, and might be worth considering today.

Assuming (1) and (2) are the correct solutions here, what are the right mechanisms to bring them into the format? A couple ideas:

• (A) Introduce new extensions, e.g. KHR_packing_normal, KHR_packing_roughness_metallic, analogous to MSFT_packing_normalRoughnessMetallic.
• (B) Include them with the upcoming "PBR Next" material model's extensions; do not backport to existing metal/rough, spec/gloss, or unlit materials.
• (C) Include these changes, and perhaps other KTX2/Basis-related changes, in a glTF 2.1 release to simplify the perceived complexity, as compared to specifying multiple interdependent extensions.

[MarkCallow]

How would general-purpose texture-swizzling solve the occlusion/rough/metal case? In current compressed textures you can only have 2 independent channels: the RGB channel & alpha.

[lexaknyazev]

@donmccurdy
Possible channel mappings from Basis to GPU formats
KhronosGroup/KTX-Specification#100 (comment)

[donmccurdy]

@MarkCallow the format already allows occlusion to be separated from rough/metal, but even when they're separated rough/metal still points to the GB channels in the spec, the R channel isn't used. A texture swizzle extension would allow rough/metal to be read from different channels.

[zeux]

General-purpose swizzling doesn’t fully solve normal map storage - shader code is necessary to reconstruct the third component.

Two-component normal storage would be very welcome otherwise - high quality normal storage requires two separately compressed components, and games have been using DXT5 and, later, two-channel BC5, for >10 years now to solve this. However we should be careful wrt specification. On desktop the only widely supported format on the web that is suitable for high quality normals is DXT5 (BC3) but it requires storing one channel in G and one channel in A. If transcoding uses BC5 instead the mapping is RG. Not sure what the expected implementation strategy looks like.

How severe is the roughness-occlusion-metallic issue? Metallic is often binary and transition regions commonly have specular fringing due to interpolation even on uncompressed data. I have not tried to use BasisU extensively so don’t have a good intuition - would be good to have examples (uncompressed RGB, DXT1, BasisU transcoded DXT1).

[lexaknyazev]

I think the implementation strategy for the web is:

• EAC RG on mobile (WEBGL_compressed_texture_etc)
• BC5 on capable desktops (EXT_texture_compression_rgtc)
• BC3 on less capable desktops (WEBGL_compressed_texture_s3tc)

Basis Universal can be transcoded to all these formats.

Metallic-Roughness question came from customers who (at some point) had to use PNG over JPEG because of perceived artifacts. Basis Universal codec is not optimized for encoding three uncorrelated channels in the same image, so we might need to be proactive here.

[donmccurdy]

General-purpose swizzling doesn’t fully solve normal map storage...

Agreed, it only solves the occlusion/rough/metal problem.

How severe is the roughness-occlusion-metallic issue?

Good question – I don't know. It's been reported to us but I do think we should try to quantify it a little, maybe with something like https://googlewebcomponents.github.io/model-viewer/test/fidelity/results-viewer.html, before doing anything too elaborate to avoid it.

[zeux]

I think the implementation strategy for the web is:

Right, but then do you have to generate permutations for all shaders that need to read this data because you don't know the component mapping, and WebGL doesn't support general swizzles? General swizzles could be simulated with dot products I suppose, e.g. sample = textureFetch(), r = dot(sample, controlVector1), g = dot(sample, ControlVector2).

FWIW when I said that the only widely suported format on web/desktop is DXT5 (BC3) I meant that even though theoretically an extension is available, in practice Chrome doesn't implement it and neither does Edge (not sure about Firefox/Safari).

[lexaknyazev]

permutations for all shaders

Well, that depends. EAC RG11 and BC5 are sampled from red and green; BC3 (DXT5) would use red and alpha. In my opinion, the former should be the default.

BC4 and BC5 formats are mandatory on all D3D10 and newer class hardware, so they should be widely available on desktops. The corresponding WebGL extension is available today in Firefox when using OpenGL. Hopefully, ANGLE will expose it soon as well. This would unlock Chromium-based browsers including Edge Insider.

ETC2/EAC formats are available today in Chrome and Firefox on Android with capable (ES 3.0+) hardware.

Safari will catch up someday...

[donmccurdy]

Any idea what percentage of desktop devices support EXT_texture_compression_rgtc? I was hoping to find that on https://webglstats.com/webgl/extension/WEBGL_compressed_texture_s3tc but didn't. :/

[lexaknyazev]

Oldest desktop hardware that support RGTC is about 12 years old:

• AMD Radeon HD 2000
• Intel GMA 4500
• NVIDIA GeForce 8

[zeux]

Any idea what percentage of desktop devices support EXT_texture_compression_rgtc?

Almost all GPUs support the formats, but almost no browsers do. WebGL commonly works through ANGLE which, when targeting DX9, can't implement this extension because the format is DX10-only; I'm not sure what the status is for ANGLE-DX11.

[MarkCallow]

WebGL commonly works through ANGLE which, when targeting DX9, can't implement this extension because the format is DX10-only; I'm not sure what the status is for ANGLE-DX11.

This kind if thing is precisely why all compressed formats are expressed as extensions in WebGL. In other words just because ANGLE/DX9 can't support RGTC is no reason for browsers to not support it when running on DX10 or above. I am in communication with the author of webglstats.com to find out why it doesn't list EXT_texture_compression_rgtc among the extensions. I.e, is it because it no browser has ever reported it or because it doesn't know about it.

[lexaknyazev]

ANGLE support for RGTC can now be tracked here:
https://bugs.chromium.org/p/angleproject/issues/detail?id=3149

Chromium support for RGTC (and BPTC) here:
https://bugs.chromium.org/p/chromium/issues/detail?id=1013369

[zeux]

Just so that I understand, is this issue separate from the KTX2 PR (aka with the KTX2 PR we'll only get support for three-channel normals)?

Experimenting with Basis and while I like the quality of diffuse textures, normal quality is not very good. This is using normal map encoding mode but it didn't help much. I haven't tried using 2-channel normal maps with BasisU yet - not sure if the idea is to use 2 channel data with the existing ETC1S encoding, or to have two channels individually stored as ETC1S streams.