Skip to content

perf(core): run visual participants (actor, scene) in parallel within a shared decode stream - #129

Open
Mahnoor-Zaffar wants to merge 3 commits into
grayhatdevelopers:mainfrom
Mahnoor-Zaffar:feat/parallel-visual-participants
Open

perf(core): run visual participants (actor, scene) in parallel within a shared decode stream#129
Mahnoor-Zaffar wants to merge 3 commits into
grayhatdevelopers:mainfrom
Mahnoor-Zaffar:feat/parallel-visual-participants

Conversation

@Mahnoor-Zaffar

Copy link
Copy Markdown

Summary

Runs the actor and scene visual participants in parallel over a single shared decode stream: one decode pass, per-participant bounded queues, and sentinel-based shutdown. Actor remains in-order; backpressure stalls decode when a participant falls behind. Also fixes a deadlock where a participant failing mid-stream would hang the pipeline — the participant error now propagates instead.

Internal-only; no public API or storage changes.

Motivation

Refs grayhatdevelopers#96. Video decoding is the common cost shared by the visual participants (scene, actor). Previously each participant ran its own pass over the decoded frames, so N participants meant N decode passes. This change decodes once and feeds both participants concurrently, cutting redundant decode work and letting the slower participant overlap with the faster one.

What changed

  • src/vidxp/capabilities/visual.py — parallel participant scheduling over a shared decode stream:
    • one decode loop produces frames into per-participant bounded queues;
    • scene and actor consumers run concurrently in their own workers;
    • sentinel shutdown terminates consumers cleanly when decode completes or is cancelled;
    • actor remains in-order while scene may lag;
    • backpressure: if a participant falls behind its queue limit, decode stalls so memory stays bounded;
    • mid-stream participant failures are surfaced as errors instead of deadlocking the pipeline (they are joined and re-raised, not swallowed).
  • tests/test_visual_threading.py — 7 unit tests covering concurrent processing, cancellation, decode failures, participant failures, ordering, and backpressure/shutdown.

Design details

  • One decode pass: frames advance once per video; participants subscribe to the shared stream rather than re-decoding.
  • Bounded queues: each participant has a finite queue depth, so a slow consumer cannot grow memory unboundedly; when the queue is full, the decode loop pauses (backpressure).
  • Deterministic shutdown: a sentinel object marks end-of-stream; consumers drain remaining frames then exit. Cancellation and error paths use the same sentinel protocol so no worker leaks.
  • Failure propagation: a failing participant's exception is collected during join and raised from the indexing run — reproducing the previously-hanging mid-stream failure now raises the participant error.

Validation

  • uv run --no-sync python -m pytest -q tests/test_visual_threading.py7 passed.
  • uv run --no-sync python -m pytest -q tests/test_runner.py tests/test_manifest.py tests/test_generation_manifest.py — passed (reported 27 in the fork PR).
  • uv run --no-sync ruff check src/vidxp/capabilities/visual.py tests/test_visual_threading.py — clean.
  • Reproduced the mid-stream participant failure that previously deadlocked; it now raises the participant error.
  • Rebased onto the original repository's main (c17a884); the working tree is clean and the visual-area tests pass on the rebased base.

Notes

Internal-only performance/reliability change. No public API, storage schema, CLI, or MCP surface changes.

… shared decode stream

Refactors _consume_visual_stream so the decode loop runs on a thread
and each visual participant (actor via OpenCV, scene via torch) gets
its own worker thread consuming from a per-participant queue. This lets
actor and scene overlap on CPU since both libraries release the GIL
during computation.

- Decode thread pushes RGB batches to bounded queues (maxsize=4) for
  backpressure
- Each participant worker filters its own samples and calls process()
  independently
- Exceptions from any thread propagate via a shared error list
- Cancellation checked in both decode and participant workers with a
  0.2s polling timeout on the queue get to stay responsive
- The single-decode property (one pass through iter_frame_batches) is
  preserved; actor stays in-order (single FIFO consumer)

No changes to runner.py, IndexConfig, ResourceScheduler, or any
capability's indexing logic. All 5 new threading tests pass.
- Fix a critical shutdown deadlock when a participant queue is full: the
  sentinel drops, so workers now also exit once decoding is done and their
  queue is empty.
- Stop workers promptly after a sibling records an error instead of
  processing remaining queued batches.
- Normalize duplicate modality names in index_visuals before building
  participant/reporting structures.
- Reorder the cancel timer callback so cancellation happens before unblocking
  the stream.
- Prefer addClassCleanup over tearDownClass and drop an unused rebinding.

Internal-only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant