API for conversion between pixel formats

[chcunningham]

Currently the spec lists one format: I420. We'll want more, including some format to indicate "opaque", which will be used for GPU backed frames to avoid having to always copy to CPU memory.

Particularly for "opaque", we want an API to convert to another format where the planes can be read (performing the copy to CPU along the way). This API could also be used to convert between various non-opaque formats. Something like...

let frame = await gpu_frame.convertTo(I420);

[chcunningham]

The urgency of this API is reduced. Previously we expected this would be primary means to read back pixel data from GPU backed frames into the CPU space. Since then we've managed to add support (in chromium, not yet in the spec) for exposing pixel data from GPU backed video frames coming off of the users camera in the NV12 pixel format.

A conversion API is still interesting to pursue, but there are no urgent use cases that we are aware of (feel free to share some).

[padenot]

(in chromium, not yet in the spec)

#208, for reference, but this would still be good to have.

[chcunningham]

triage note: marking extension, as this would be a new API.

[alvestrand]

this means, of course, that any application that handles only a limited set of formats will be broken every time a new format is encountered. Seems a rough deal for app developers.

[sandersdan]

any application that handles only a limited set of formats will be broken every time a new format is encountered

There is a fallback path; you can createImageBitmap(videoFrame), then read back that ImageBitmap (presumably as RGBA).

This isn't great if the frame was mappable in the first place, but for GPU-backed frames it actually has acceptable performance.

[alvestrand]

So this means that we have a fallback (effectively "read as RGBA"). Apps can choose between using a fast path if they support the current format and a slow path if they don't. For a lot of cases, this will be sufficient.

[chcunningham]

Options for potential chromium impl in https://bugs.chromium.org/p/chromium/issues/detail?id=1139119

[Kaiido]

There is a fallback path; you can createImageBitmap(videoFrame), then read back that ImageBitmap (presumably as RGBA).

Is there anything that would enforce that conversion? Currently the HTML specs for createImageBitmap(VideoFrame) ask to

Set imageBitmap's bitmap data to a copy of image's visible pixel data.

I guess the term "visible pixel data" is open to interpretation, but there doesn't seem to be anything there asking for a conversion of the pixel format.
As I mentioned in whatwg/html#4785 (comment) it seems that current Chrome does convert most formats to RGB[A|X], but doesn't for instance convert BGRA, so authors still have to handle at least that case.
Having a path to get a standard output format would be great and might become a viable alternative to the often asked ImageBitmap.getImageData().

[dalecurtis]

We've had a few requests for this in different forms, so going to mark as p1 to at least add support for converting to RGBA/BGRA which is already required for canvas interactions.