Skip to content

Remove libopenh&libx264 and Add vp9#1930

Merged
abhinavg4 merged 9 commits into
r1.2.0from
aot/remove_libopenh_add_vp9
May 6, 2026
Merged

Remove libopenh&libx264 and Add vp9#1930
abhinavg4 merged 9 commits into
r1.2.0from
aot/remove_libopenh_add_vp9

Conversation

@suiyoubi
Copy link
Copy Markdown
Contributor

@suiyoubi suiyoubi commented May 6, 2026

Continue work based on #1918 for adding VP9 as the fallback when nvenc is not avaliable.

Drop bundled libopenh264, add libvpx-vp9 CPU fallback, support BYO H.264

Summary

Curator's bundled FFmpeg no longer ships with libopenh264 (Cisco patent-license redistribution model is incompatible with our distribution). To preserve the ability to transcode video clips on hardware without NVENC (notably A100/H100), this PR:

  1. Adds libvpx-vp9 as a built-in CPU encoder fallback in ClipTranscodingStage. VP9 is BSD-licensed, royalty-free under the WebM patent grant, and produces VP9 in .mp4 — no downstream changes required.
  2. Keeps libopenh264 accepted as an opt-in encoder for users who provide their own FFmpeg build. The stage probes at setup and surfaces a clear error pointing to the docs if it's not actually compiled in.
  3. Documents Bring-Your-Own H.264 for both libopenh264 and libx264, including the licensing tradeoffs.

What changed

Code (nemo_curator/stages/video/clipping/clip_extraction_stages.py)

  • New SUPPORTED_ENCODERS = ("h264_nvenc", "libvpx-vp9", "libopenh264") validator.
  • __post_init__ routes libvpx-vp9 to CPU resources (vs GPU for NVENC) and emits a perf advisory recommending NVENC if available.
  • setup() calls _verify_libopenh264_available() when encoder="libopenh264" — uses shutil.which("ffmpeg") + ffmpeg -hide_banner -encoders grep; raises RuntimeError with docs link if the encoder isn't compiled in.
  • New _add_libvpx_vp9_options helper with sensible CRF defaults (-b:v 0 -crf 31 -deadline good -cpu-used 4 -row-mt 1 -tile-columns 2).
  • Rejects use_hwaccel=True with libvpx-vp9 (CPU encoder).
  • Output stays in .mp4 for all encoders.

Tutorial (tutorials/video/getting-started/video_split_clip_example.py)

  • --transcode-encoder choices extended to {h264_nvenc, libvpx-vp9, libopenh264} with help text noting libopenh264 requires a user-installed FFmpeg.

Tests (tests/stages/video/clipping/test_clip_transcoding_stage.py)

  • 12 new tests covering validator behavior, encoder-options branching, hwaccel rejection, libopenh264 probe (available / unavailable / ffmpeg-missing), and VP9 perf warning.

Docs

  • New "Bring-Your-Own H.264 Software Encoder (Advanced)" section in docs/admin/installation.md with two install paths (system FFmpeg vs custom build), licensing caveats, and a note that libopenh264 is auto-accepted while libx264 requires extending SUPPORTED_ENCODERS.
  • Updated docs/curate-video/process-data/transcoding.md, docs/get-started/video.md, docs/curate-video/tutorials/beginner.md, and tutorials/video/getting-started/README.md to describe the new encoder set.

Test plan

  • Unit tests pass (44 total, +12 new)
  • Pre-commit hooks pass (ruff, ruff-format, end-of-file, trailing whitespace, etc.)
  • End-to-end pipeline run on local videos with --transcode-encoder libvpx-vp9 → 37 valid VP9-in-MP4 clips
  • End-to-end regression run with --transcode-encoder h264_nvenc → 37 valid H.264-in-MP4 clips
  • Behavioral smoke tests in two Docker images (with/without libopenh264):
    • Image without libopenh264 + --transcode-encoder libopenh264 → raises RuntimeError with docs link ✓
    • Image without libopenh264 + --transcode-encoder libvpx-vp9 → succeeds + warning emitted ✓
    • Image with libopenh264 + --transcode-encoder libopenh264 → succeeds, output is H.264 ✓

Pipeline-level benchmarks

RTX 6000 Ada × 2, source video with total of 30min. Wall time / output size:

Workflow libopenh264 libvpx-vp9 h264_nvenc
Transcode-only 79 s / 467 MB 795 s / 1.4 GB 79 s / 1.8 GB
+ embedding (cosmos-embed1-224p) 145 s / 468 MB 943 s / 1.4 GB 150 s / 1.8 GB
+ captioning (Qwen2.5-VL-7B, warm cache) 392 s / 468 MB 1068 s / 1.4 GB 444 s / 1.8 GB

VP9's relative slowdown shrinks as the pipeline gets heavier (10× transcode-only → 2.7× full caption pipeline) because downstream GPU work amortizes the encode cost. NVENC remains the right default when available; VP9 is a viable A100/H100 fallback if users accept the throughput hit.

Notes for reviewers

  • A100/H100 are NVENC-less data-center GPUs. Without this PR, those nodes have no working transcode path. VP9 is the only royalty-free, ship-with-Curator option that works there.
  • Output container stays .mp4 for all three encoders. PyAV/FFmpeg consume VP9-in-MP4 fine; downstream stages (embedding, motion filter, captioning, writer) are codec-agnostic.
  • libx264 is intentionally still rejected by the validator. The BYO docs explain how to enable it (it's GPL-tainting).
  • The libvpx-vp9 perf advisory fires on every stage construction (no module-level dedup) — clean and predictable.

thomasdhc and others added 4 commits May 5, 2026 10:17
Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Dong Hyuk Chang <9426164+thomasdhc@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lback encoder alongside `h264_nvenc`. Update installation instructions and verification steps to reflect the new encoder options. Modify `ClipTranscodingStage` to validate encoder selection and handle encoding options for both encoders. Update relevant documentation and examples to guide users on using the new encoder.

Signed-off-by: Ao Tang <aot@nvidia.com>
@suiyoubi suiyoubi requested review from a team and abhinavg4 as code owners May 6, 2026 13:55
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 6, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Ao Tang <aot@nvidia.com>
@suiyoubi
Copy link
Copy Markdown
Contributor Author

suiyoubi commented May 6, 2026

/ok to test 1e85ff8

@suiyoubi
Copy link
Copy Markdown
Contributor Author

suiyoubi commented May 6, 2026

/ok to test c304db3

Copy link
Copy Markdown
Contributor

@jgerh jgerh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed tech pubs review of .md files and ran standard copyedit checks for consistency and adherence to style guidelines.

Comment thread docs/admin/installation.md Outdated
Comment thread docs/admin/installation.md Outdated
Comment thread docs/admin/installation.md Outdated
Comment thread docs/admin/installation.md Outdated
Comment thread docs/admin/installation.md Outdated
Comment thread docs/get-started/video.md
Comment thread docs/get-started/video.md Outdated
Comment thread docs/get-started/video.md
Comment thread docs/get-started/video.md

```{tip}
**Using a config file**: The example script accepts many command-line arguments. For complex configurations, you can store arguments in a file and pass them with the `@` prefix:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
```bash

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already have ```{tip} at L245

Comment thread docs/get-started/video.md Outdated
Signed-off-by: Ao Tang <aot@nvidia.com>
@suiyoubi
Copy link
Copy Markdown
Contributor Author

suiyoubi commented May 6, 2026

/ok to test 288146b

Copy link
Copy Markdown
Contributor

@abhinavg4 abhinavg4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

h264

Copy link
Copy Markdown
Contributor

@lbliii lbliii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

content lgtm, it just needs to migrate into the fern/ folder for 26.04. i can handle that separately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants