pixelforge

A Vulkan-based video encoding and decoding library for Rust, supporting H.264, H.265 and AV1 encode, and H.264 decode.

Features

• Hardware-accelerated video encoding and decoding using Vulkan Video extensions.
• Multiple codec support: H.264/AVC, H.265/HEVC, AV1 encode; H.264 decode.
• Asynchronous pipelines: both directions submit without waiting. Encoding hands back an [EncodeFuture]; decoding delivers frames through a [DecodeSource] as the GPU finishes with them.
• GPU color conversion: RGB/BGR → YUV via Vulkan compute shaders (BT.709, BT.2020, sRGB→BT.2020+PQ, scRGB-linear→BT.2020+PQ).
• HDR support: 10-bit encoding (P010, YUV444P10), PQ transfer function, BT.2020 color space.
• GPU-native API: Encode directly from Vulkan images (vk::Image).
• Flexible configuration: Rate control (CBR, VBR, CQP), quality levels, GOP settings.
• Multiple input formats: BGRx, RGBx, BGRA, RGBA, ABGR2101010 (10-bit packed), RGBA16F (FP16).
• Utility helpers: [InputImage] for easy YUV data upload to GPU.
• Optional DMA-BUF support: Zero-copy image import from external processes (Linux only).

Note: B-frame support is not yet implemented. Setting b_frame_count > 0 will panic.

Supported Codecs

Codec	Encode	Decode
H.264/AVC	✓	✓
H.265/HEVC	✓
AV1	✓

H.264 decoding is verified byte-identical to ffmpeg -pix_fmt nv12 on AMD (RADV), NVIDIA and Intel (ANV).

Requirements

• A GPU with Vulkan video support (e.g., NVIDIA RTX series, AMD RDNA2+, Intel Arc). Decoding additionally needs a video decode queue; on Intel Arc under Mesa it currently has to be enabled with ANV_DEBUG=video-decode,video-encode.

Installation

Add this to your Cargo.toml:

[dependencies]
pixelforge = "0.1"

Optional Features

Feature	Description
dmabuf	Enable DMA-BUF support for zero-copy image import from external processes (Linux only). Adds Vulkan extensions: VK_KHR_external_memory, VK_KHR_external_memory_fd, VK_EXT_external_memory_dma_buf, VK_EXT_image_drm_format_modifier.

To enable DMA-BUF support:

[dependencies]
pixelforge = { version = "0.1", features = ["dmabuf"] }

Quick Start

Query Capabilities

use pixelforge::{Codec, VideoContextBuilder};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let context = VideoContextBuilder::new()
        .app_name("My App")
        .build()?;

    for codec in [Codec::H264, Codec::H265, Codec::AV1] {
        println!("{:?}: encode={}",
            codec,
            context.supports_encode(codec)
        );
    }
    Ok(())
}

Encoding Video

use pixelforge::{
    Codec, EncodeBitDepth, EncodeConfig, Encoder, InputImage, PixelFormat, RateControlMode,
    VideoContextBuilder,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let context = VideoContextBuilder::new()
        .app_name("Encoder Example")
        .require_encode(Codec::H264)
        .build()?;

    let config = EncodeConfig::h264(1920, 1080)
        .with_rate_control(RateControlMode::Vbr)
        .with_target_bitrate(5_000_000)
        .with_frame_rate(30, 1)
        .with_gop_size(60);

    // Create an InputImage helper for uploading YUV data to the GPU.
    let mut input_image = InputImage::new(
        context.clone(),
        Codec::H264,
        1920,
        1080,
        EncodeBitDepth::Eight,
        PixelFormat::Yuv420,
    )?;
    let mut encoder = Encoder::new(context, config)?;

    // For each frame: upload YUV data and encode.
    // let yuv_data: &[u8] = ...;  // YUV420 frame data
    // input_image.upload_yuv420(yuv_data)?;
    // let packets = encoder.encode(input_image.image())?;

    Ok(())
}

Decoding Video

The decoder is stream-driven: it creates its Vulkan session from the stream's own parameter sets, so nothing has to be configured up front, and a mid-stream resolution change is handled transparently.

Bytes go in through a [DecodeSink], frames come out of a [DecodeSource]. A [Decoder] holds both, so one thread can drive the whole thing; Decoder::split separates them for a producer and a consumer on their own threads.

Frames come out in presentation order and, where the device supports unified image layouts, without ever being copied: the frame is the decoder's own image. Drop each one when done, which returns its storage.

use pixelforge::{Codec, VideoContextBuilder};
use pixelforge::decoder::{DecodeConfig, Decoder, FramePoll};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let context = VideoContextBuilder::new()
        .app_name("Decoder Example")
        .require_decode(Codec::H264)
        .build()?;

    // A file can cut anywhere, so let the decoder frame it. Input that
    // arrives already framed (RTP, a container) skips `with_byte_stream`.
    let config = DecodeConfig::h264().with_byte_stream();
    let mut decoder = Decoder::new(context, config)?;
    let stream: Vec<u8> = std::fs::read("input.264")?;

    for (i, chunk) in stream.chunks(64 * 1024).enumerate() {
        // The status says what happened; an `Err` means something is
        // actually wrong. Joining a stream partway through is not.
        let _status = decoder.decode(chunk, i as u64)?;
        // Take what the GPU has finished with; `Pending` just means "not yet".
        while let FramePoll::Frame(frame) = decoder.try_next_frame()? {
            // `frame.image` is a decoder-owned GPU image, valid until dropped.
            let _ = frame.image;
        }
    }

    // End of stream: decodes the trailing picture, emits what reordering
    // held back, and closes the source.
    decoder.finish()?;
    while let Some(frame) = pollster::block_on(decoder.next_frame())? {
        let _ = frame.image;
    }
    Ok(())
}

A live frame reserves a DPB slot, so DecodeConfig::with_output_depth bounds how many can be outstanding before the decoder starts copying pictures out instead of handing over its own. Reading a frame back to the CPU is the consumer's job; examples/common shows one way.

Color Conversion (RGB → YUV)

PixelForge includes a GPU compute shader for converting RGB input to YUV output, supporting multiple color spaces:

Color Space	Description
Bt709	Standard SDR (BT.709 coefficients)
Bt2020	HDR passthrough (BT.2020 coefficients, PQ-encoded input)
SrgbToBt2020Pq	SDR-in-HDR (sRGB → linear → BT.2020 gamut → PQ OETF)
Bt709LinearToBt2020Pq	scRGB HDR (linear BT.709 → BT.2020 gamut → PQ OETF). sdr_reference_white_nits sets the interpretation of 1.0; per the scRGB spec (IEC 61966-2-2), 80 nits.

Supported input formats: BGRx, RGBx, BGRA, RGBA, ABGR2101010 (10-bit packed), RGBA16F (FP16). Supported output formats: NV12 (8-bit), I420 (8-bit), YUV444 (8-bit), P010 (10-bit), YUV444P10 (10-bit).

use pixelforge::{ColorConverter, ColorConverterConfig, ColorSpace, InputFormat, OutputFormat, VideoContextBuilder};

let context = VideoContextBuilder::new()
    .app_name("Color Converter")
    .build()?;

let mut config = ColorConverterConfig::new(1920, 1080, InputFormat::BGRx, OutputFormat::NV12);
config.color_space = ColorSpace::SrgbToBt2020Pq;

let mut converter = ColorConverter::new(context.clone(), config)?;
// converter.convert(input_image, output_buffer)?;

Benchmarking

Run the encode latency benchmark with:

cargo run --example encode_bench

Examples

Run the examples with:

# Query codec capabilities
cargo run --example query_capabilities

# H.264 decoding to raw YUV
cargo run --example decode_h264 -- input.264 output.yuv

# H.264 encoding example
cargo run --example encode_h264

# H.265 encoding example
cargo run --example encode_h265

# AV1 encoding example
cargo run --example encode_av1

# Verify all codecs and formats
cargo run --example verify_all

Shader Development

The color conversion shader is precompiled to SPIR-V and embedded at build time. See shader/README.md for details on editing and recompiling shaders.

TODO's

1. [] H.265 and AV1 decoding.
2. [] B-frames support (encode).

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Acknowledgement

This project was heavily inspired by the vk_video_samples repository by NVIDIA, which provided invaluable reference for Vulkan Video encoding.

License: BSD-2-Clause