Skip to content

[Bug]: Encode hang on Arc B580 during final encode pass, batch buffer never returns #2025

Description

@leow149

Which component impacted?

Video Processing

Is it regression? Good in old configuration?

No, this issue exist a long time

What happened?

Related issues

Originally filed against drm/xe/kernel: https://gitlab.freedesktop.org/drm/xe/kernel/-/work_items/8732

Possibly related to #6029 there (B580 hangs via
guc_exec_queue_timedout_job, different reporters, various triggers,
not confirmed same root cause).

Matthew Auld (xe maintainer) reviewed the kernel-side coredump and
concluded this isn't a kernel issue. His read: the GPU jumped into
the user-submitted batch buffer and never came back, and there's
nothing in dmesg pointing at the driver. He suggested filing here
and linking back in case the team feels otherwise.

Possibly related to #1889 (closed as not planned). Similar GPU
Hang(-21) symptom on hevc_qsv decode with HDR content, different
hardware. Flagging for pattern overlap, not claiming same root cause.

Which component impacted?

Encode (QSV). Decode runs concurrently throughout, but decode alone
and short concurrent decode+encode bursts both complete fine. Only
the sustained final encode pass hangs. See "Suspected trigger."

Environment

Hardware: Intel Arc B580 (Battlemage G21), PCI ID [8086:e20b]
Running under Proxmox VE / QEMU-KVM via VFIO passthrough (ruled out
as a factor, see below)
Guest OS: Ubuntu 26.04 LTS, kernel 7.0.0-28-generic
Kernel driver: xe, GuC firmware 70.65.0 (also happened on 70.45.2
before I updated)
Userspace: ffmpeg (jellyfin-ffmpeg build) via VA-API, iHD driver

What happened?

During HEVC transcoding, the video engine hangs partway through the
final encode pass. Per the kernel-side coredump, the GPU jumps into
a user-submitted batch buffer and never returns from it. GuC's
engine-level timeout/reset fires repeatedly afterward but the
underlying condition doesn't clear. Every subsequent job submitted
to the context hangs identically until the process is killed.

The encode pipeline runs an optimized mode that performs several
short VMAF-scoring sample passes (each about 20s, each its own
decode+encode cycle against the source) before committing to the
final full-length encode. Both phases run concurrent hardware decode
and encode on the same context. The hang has only been observed
during the final pass. Sampling has completed cleanly on every run.

Suspected trigger

Since both phases run concurrent decode+encode, that alone doesn't
look like the differentiator. Sampling does the same thing repeatedly
without issue. What differs is duration and continuity: sampling
passes are short and bounded (about 20s each, several of them, each
with its own start/stop), while the final pass runs continuous
decode+encode for the length of the whole file, often over an hour
for this content.

This points toward something duration- or frame-count-dependent,
maybe a counter, timestamp, or accumulating internal resource that
only misbehaves after sustained continuous submission, rather than
rapid switching between short bursts (my initial guess, now ruled
out).

Steps to reproduce

ffmpeg via VA-API (iHD driver), HEVC encode targeting QSV, using an
optimized/VMAF-sampling encode mode, hardware decode enabled
concurrently on the source. Reproduces reliably during the final
encode pass after sampling completes. Not observed during sampling
itself. Typically reproduces on the first file of a session.

Still to test

Whether the final pass hangs on its own, with no preceding VMAF
sampling (single-pass encode, sampling disabled entirely).

If it still hangs, that confirms the final pass fails on its own
merits, independent of what preceded it.
If it doesn't hang without prior sampling, that would suggest
sampling leaves some state behind that only causes problems once the
longer final pass begins. A different, more specific lead.

Will update this issue once I've tested it.

Passthrough ruled out

Checked the Proxmox host's own vfio-pci reset log before filing, in
case this was a passthrough/VFIO reset problem instead. It's clean.
All resets complete within 0-1s and only happen at VM start/stop,
not correlated with hang timing. ASPM is structurally disabled on
this platform at the firmware level, and intel_iommu=on iommu=pt has
been active throughout. Neither appears to be a factor.

Attachments

Xe devcoredump (from the kernel-side report, attached here too)

dmesg excerpt covering the hang window

What's the usage scenario when you are seeing the problem?

Transcode for media delivery

What impacted?

No response

Debug Information

Debug Information

  1. Versions (inside the FileFlows container, running jellyfin-ffmpeg):

    • libva2: 2.23.0-1ubuntu1
    • libva-drm2 / libva-x11-2 / libva-wayland2: 2.23.0-1ubuntu1
    • intel-media-va-driver-non-free (iHD): 26.1.2+ds1-1
  2. GPU present, confirmed on both host and container:
    $ ls /dev/dri
    by-path card0 renderD128
    (same card0/renderD128 visible inside the container)

  3. lspci -nn | grep -Ei 'VGA|DISPLAY':
    01:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage
    G21 [Arc B580] [8086:e20b]

  4. vainfo output (clean init, no crash):
    libva info: VA-API version 1.23.0
    libva info: va_openDriver() returns 0
    vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 26.1.2 ()
    HEVCMain10 correctly exposes both VAEntrypointVLD and
    VAEntrypointEncSlice (full profile list attached).

5/6. LIBVA_TRACE and a fresh matched dmesg were not captured for this
report. The dmesg already attached above covers the hang window
from the original report. Happy to provide a trace log if it turns
out to be needed to move this forward.

Do you want to contribute a patch to fix the issue?

None

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions