Video workflow: HEVC export failure (#127 root cause), silent objectFit, unfinishable frame-step fallback + 7 more #128

Summary

One working session (2026-09-15) on a 9:16 Instagram Reel video-comp canvas turned up 10 problems in the video workflow: embedding, preview, editing, export, transcription, plus one packaging bug. Item 1+2 is the root cause behind the in-app report #127 ("Can't export video to mp4").

Each item has evidence and the code location in the 1.2.0 app bundle. Items are ordered by user impact.

Environment

• Maude.app 1.2.0 (desktop), darwin arm64
• Canvas: kind="video" artboard 1080×1900, <VideoComp> + @remotion/media <Video>, two iPhone clips + a Remotion motion end card + word-by-word captions
• Export via the ⌘E dialog and POST /_api/export-jobs

1. HEVC / VFR / spatial-audio sources: no warning at import, export silently degrades (root cause of #127)

What happened. Clip 1 was an iPhone recording: HEVC Main, rotation 90, VFR (avg 32.45 fps, 1388 frames in 41.8 s). Clip 2 was a .mov with an extra apple_apac (spatial audio) track plus 5 data tracks. Both played fine in the Player. On export:

⚠ renderMediaOnWeb failed (Error: Cannot render video "assets/3b165be8.mp4": The video could not be decoded by the browser.)
falling back to frame-step screenshot capture — video will export WITHOUT audio

Also in the console: Unsupported audio codec (sample entry type 'apac').

Workaround that fixed it. ffmpeg -vf fps=30,format=yuv420p -c:v libx264 -crf 18 for the HEVC clip, and a remux of the .mov keeping only 0:v:0 + 0:a:0. After that the whole-comp path exported in 81 s with audio.

Suggested fix.

• Probe on ingest (probe-footage already exists). For HEVC, VFR or undecodable extra audio tracks, warn in the UI and offer (or create automatically) an H.264 / CFR / single-AAC proxy next to the original.
• When the fast path fails, tell the user before starting the frame-step fallback that the result will have no audio, and name the offending asset and codec.

2. Frame-step fallback can't finish, and the default scale is 2× even for 1080-wide artboards

• bin/_video-playwright.mjs:74: HARD_KILL_MS = Math.max(timeoutMs, 150_000). exporters/video.ts:226 itself warns that frame-step runs ~2.5 s/frame. So any comp longer than about 60 frames can never finish in the fallback. Our 1294-frame comp died at frame 413/1294:
  HARD TIMEOUT after 150000ms — exiting … on headless Linux this is usually the video encoder or GPU init
• That message talks about headless Linux while running on macOS, which is misleading.
• exporters/video.ts:211: scale defaults to 2 regardless of artboard size, so a native 1080×1900 Reel was rendered at 2160×3804. That doubles the frame-step cost and gives no benefit for social formats.

Suggested fix. Scale the hard-kill budget by frame count (or drop it once frames are progressing). Default scale to 1 when the artboard's short side is ≥1080. Make the timeout message platform-aware.

3. style.objectFit on @remotion/media <Video> is silently ignored (falls back to contain)

<Video> takes objectFit as a prop. style.objectFit is overwritten (actualStyle = { ...style, objectFit: objectFitProp }, default "contain"), and the only signal is a console.warn. A 16:9 clip in a 9:16 artboard with style={{ objectFit: "cover" }} stays letterboxed. This cost two rounds with the user.

The pattern is common in canvases: the old OffthreadVideo API accepted it, so existing kits (e.g. a cover() helper returning { objectFit: "cover", ... } spread into <Video style>) now silently letterbox any clip whose aspect ratio doesn't match the artboard.

Suggested fix. Surface the warning in the studio (canvas problems panel / build diagnostics), or map style.objectFit → the objectFit prop in the canvas-lib runtime. Add it as an iron rule to the video-comp skill.

4. Inserting a video into an existing artboard produced an invisible / mis-sized comp

State found in the canvas after inserting a video into a fresh brief-board artboard:

• <VideoComp> was added as a sibling after the board's full-size (width/height: 100%) placeholder <div>, so the video was pushed out of the fixed artboard and nothing was visible.
• <Video style={{ width: "2622px", height: "1673.8px" }}>: absolute pixel sizes derived from the source, not fit to the artboard.
• <VideoComp durationInFrames={150}> while the inner TransitionSeries.Sequence had durationInFrames={1255}.
• Stray top: "0px" on the AbsoluteFill.

Suggested fix. When inserting into a kind="video" artboard, replace placeholder content, emit width/height: 100% + the objectFit="cover" prop, and keep VideoComp.durationInFrames in sync with the clip.

5. No visual resize / reposition of video inside a video artboard (feature)

The user's first instinct was to drag the video's corners in the preview. Pointer events over the Player only control playback, and the Timeline only retimes and trims. Every framing change (fit, crop focus via objectPosition, scale) needs a code edit.

Suggested fix. A transform or crop handle for the selected <Video> layer, or at least inspector fields for fit / focus / scale that write back to the TSX.

6. maude design screenshot can't target a frame of a video-comp

maude design screenshot --screen <id> on a video artboard captures frame 0, which is usually black because the media isn't decoded yet. The seek bridge window.__maude_seek__(frame, { strict: true }) exists (video-comp.tsx), but the helper doesn't expose it. The workaround was driving agent-browser by hand (open → eval __maude_seek__(N) → wait → screenshot), plus enlarging the viewport because the lower part of a tall artboard was clipped.

Suggested fix. maude design screenshot --screen <id> --frame <N> (seek strict, wait for paint), and size the viewport to the target element.

7. Cloud transcription (ElevenLabs / Groq) throws away word timestamps

generation/adapters/elevenlabs.ts:310-314 receives Scribe's word-level words[], but only wordsToSrt(words) (reflowed segments) is persisted. bin/_transcribe.mjs runCloud() ignores the words / --segments flag entirely (it only affects whisper). Word-by-word "karaoke" captions (the standard Reels style) therefore need guessed per-word timings.

Suggested fix. Persist a <sha8>.words.json sidecar ({ text, start, end }[]) next to the SRT for every engine, and make --segments / default behave the same on the cloud path.

8. Player ↔ export (renderMediaOnWeb) visual parity

Visible in the Player, missing or altered in the exported MP4:

• absolutely positioned <img> with low opacity (e.g. opacity: 0.05 brand watermark)
• div with background-image + clip-path (camo corner)
• transform: skewX(-7deg) on caption tags (exported unskewed)

Suggested fix. A parity check or warning list of CSS features the web renderer doesn't support, shown at export time, or a "preview as export" mode in the Player.

9. GET /_api/export-jobs?id=<id> ignores the id filter

It returns all jobs (including other canvases and PDF jobs), so a poller has to filter client-side. Minor, but it made scripted polling error-prone.

10. Bundled kg binary can't load libkuzu.dylib (packaging)

dyld: Library not loaded: @rpath/libkuzu.dylib
  Referenced from: /Applications/Maude.app/Contents/Resources/kgai/kg
  Reason: tried: '/Users/runner/work/kgai/kgai/third_party/go-kuzu/lib/dynamic/darwin/libkuzu.dylib', …, '/Applications/Maude.app/Contents/Resources/kgai/../lib/libkuzu.dylib' (no such file)

libkuzu.dylib is shipped at Resources/kgai/libkuzu.dylib (next to kg), but the binary's LC_RPATH entries are the CI path and @loader_path/../lib. Nothing points at @loader_path. Every maude kg … call fails, e.g. maude kg record-log ("ingest failed … the file is on disk") and the stop-hook kg ingest.

Suggested fix. Add @loader_path to the rpath (install_name_tool -add_rpath @loader_path kg), or ship the dylib at Resources/lib/. Add a smoke test (kg --version) to the app bundle CI.

Related: #127 (in-app report, same session), #126 (media >100 MB can't be dropped; the same iPhone footage workflow).