perf(core): run visual participants (actor, scene) in parallel within a shared decode stream - #129
Open
Mahnoor-Zaffar wants to merge 3 commits into
Conversation
… shared decode stream Refactors _consume_visual_stream so the decode loop runs on a thread and each visual participant (actor via OpenCV, scene via torch) gets its own worker thread consuming from a per-participant queue. This lets actor and scene overlap on CPU since both libraries release the GIL during computation. - Decode thread pushes RGB batches to bounded queues (maxsize=4) for backpressure - Each participant worker filters its own samples and calls process() independently - Exceptions from any thread propagate via a shared error list - Cancellation checked in both decode and participant workers with a 0.2s polling timeout on the queue get to stay responsive - The single-decode property (one pass through iter_frame_batches) is preserved; actor stays in-order (single FIFO consumer) No changes to runner.py, IndexConfig, ResourceScheduler, or any capability's indexing logic. All 5 new threading tests pass.
- Fix a critical shutdown deadlock when a participant queue is full: the sentinel drops, so workers now also exit once decoding is done and their queue is empty. - Stop workers promptly after a sibling records an error instead of processing remaining queued batches. - Normalize duplicate modality names in index_visuals before building participant/reporting structures. - Reorder the cancel timer callback so cancellation happens before unblocking the stream. - Prefer addClassCleanup over tearDownClass and drop an unused rebinding. Internal-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runs the
actorandscenevisual participants in parallel over a single shared decode stream: one decode pass, per-participant bounded queues, and sentinel-based shutdown. Actor remains in-order; backpressure stalls decode when a participant falls behind. Also fixes a deadlock where a participant failing mid-stream would hang the pipeline — the participant error now propagates instead.Internal-only; no public API or storage changes.
Motivation
Refs
grayhatdevelopers#96. Video decoding is the common cost shared by the visual participants (scene,actor). Previously each participant ran its own pass over the decoded frames, so N participants meant N decode passes. This change decodes once and feeds both participants concurrently, cutting redundant decode work and letting the slower participant overlap with the faster one.What changed
src/vidxp/capabilities/visual.py— parallel participant scheduling over a shared decode stream:sceneandactorconsumers run concurrently in their own workers;actorremains in-order whilescenemay lag;tests/test_visual_threading.py— 7 unit tests covering concurrent processing, cancellation, decode failures, participant failures, ordering, and backpressure/shutdown.Design details
Validation
uv run --no-sync python -m pytest -q tests/test_visual_threading.py— 7 passed.uv run --no-sync python -m pytest -q tests/test_runner.py tests/test_manifest.py tests/test_generation_manifest.py— passed (reported 27 in the fork PR).uv run --no-sync ruff check src/vidxp/capabilities/visual.py tests/test_visual_threading.py— clean.main(c17a884); the working tree is clean and the visual-area tests pass on the rebased base.Notes
Internal-only performance/reliability change. No public API, storage schema, CLI, or MCP surface changes.