feat(benchmarks): add reproducible indexing-latency benchmark with real DiDeMo corpus

docs/benchmarking/README.md

@@ -18,6 +18,7 @@ installation and product usage, start with the main
 | HiREST transcript localization | Legacy full result + current smoke | The legacy MiniLM stack scored all 193 validation pairs; current Qwen3 passed a two-video real execution smoke; 776 released test predictions remain unscored because their public bounds are placeholders |
 | LongVALE combined evaluation | Next | Build the visual-plus-speech adapter and validate one evaluation archive before scheduling the full run |
 | Actor clustering | Data-gated | The preferred BBT/Buffy evaluation still requires lawful access to the source episodes |
+| Indexing latency benchmark | Ready | `vidxp benchmark index-latency` measures throughput, per-stage timings, and peak memory on synthetic FFmpeg media; supports regression detection against baselines |
 
 Read [current results](results.md) for the scores, plain-language metric
 definitions, honest comparisons, and the next benchmark decision.

docs/benchmarking/performance.md

@@ -0,0 +1,212 @@
+# Latency benchmark protocol
+
+Status: Ready
+
+The latency benchmark (`vidxp benchmark index-latency`) measures indexing
+throughput, per-stage timing, and peak memory. By default it uses
+deterministic synthetic media generated by FFmpeg on the caller's machine;
+with `--corpus` it indexes real, paper-published media that has already been
+prepared by `vidxp benchmark prepare`. It is designed for regression
+detection between VidXP builds and for evaluating the latency impact of
+model or architecture changes.
+
+## Protocol
+
+### Corpus generation (synthetic)
+
+The default benchmark generates deterministic synthetic clips using
+FFmpeg's `lavfi` source filters:
+
+| Parameter | Default | Notes |
+|---|---|---|
+| `--videos` | 1 | Number of synthetic clips |
+| `--duration-seconds` | 8.0 | Wall-clock duration of each clip |
+| `--fps` | 24 | Frame rate |
+| `--resolution` | 320x180 | `WxH` format |
+| `--audio-mode` | `none` | `none`, `sine`, or `flite` |
+| `--input-mode` | `transcript` | `transcript` or `transcribe` |
+
+Video is generated via `testsrc2` (colour bars + timestamp). When
+`--input-mode transcript` and `dialogue` is enabled, a deterministic
+synthetic transcript (seeded PRNG over a fixed English vocabulary) is
+supplied without real transcription. When `input-mode transcribe` is
+used, `--audio-mode flite` must also be set and libflite must be
+available in the ffmpeg build.
+
+### Real media corpus
+
+Pass `--corpus didemo` to index the real DiDeMo videos already prepared by
+`vidxp benchmark prepare didemo` (real Flickr scenes used in the
+Localizing Moments in Video paper). Pass any directory of media files
+(`.mp4`, `.webm`, `.mov`, `.mkv`, `.m4v`, `.avi`) to index arbitrary real
+video. `--videos` caps how many clips are used (default 1; the files are
+taken in sorted name order), so a smoke run uses one clip and a full run
+can sweep the whole prepared split.
+
+| Parameter | Effect in corpus mode |
+|---|---|
+| `--corpus didemo` | Use `<data-dir>/benchmarks/didemo/media`, honoring `media-overrides.json` if present |
+| `--corpus <dir>` | Use the media files in `<dir>` |
+| `--videos` | Maximum number of real clips to index |
+| `--duration-seconds`, `--fps`, `--resolution`, `--audio-mode` | Ignored (describe synthetic generation only) |
+
+Real corpora have no released transcripts: when `dialogue` is selected,
+`--input-mode transcribe` is required so VidXP transcribes the media with
+Whisper. The report and run manifest record the corpus as `kind: "real"`
+with name, source, clip count, total bytes, duration range, and containers;
+baseline comparison rejects a baseline whose corpus signature differs from
+the current run, so synthetic and real results never mix.
+
+### Indexing measurement
+
+Each repetition runs the full indexing pipeline via `run_index()` with
+`reset=True`. The following stages are timed by the existing manifest
+timing infrastructure (`core/manifest.py:record_stage`):
+
+| Stage | Modality | Measures |
+|---|---|---|
+| `frame_stream` | (all visual) | Decode throughput (frames/s) |
+| `scene` | scene | SigLIP2 embedding (frames/s) |
+| `actor` | actor | OpenCV detect + recognise (frames/s) |
+| `visual_indexing` | all visual | Combined group wall time |
+| `dialogue_indexing` | dialogue | Embedding throughput (phrases/s) |
+
+Peak RSS is captured via `resource.getrusage(RUSAGE_SELF).ru_maxrss`
+(POSIX only; `None` on Windows, reported in bytes on macOS, KiB on
+Linux).
+
+### Repetitions
+
+When `--repetitions N` > 1, each repetition runs the full cycle
+(generate once, index each time after `reset`). Results are reported
+as mean, min, and max across all per-video per-repetition samples.
+
+### Baseline comparison
+
+Pass `--baseline <path-to-previous-report.json>` to compare the
+current run against a prior report. For each stage present in both,
+the delta ratio (`new_mean / old_mean - 1`) is computed. A stage with
+a delta exceeding `--baseline-tolerance` (default 0.15 = 15%) is
+flagged as a regression. The verdict is `fail` if any stage regressed,
+else `pass`.
+
+### Output
+
+The benchmark writes its report to `run_directory/report.json` and
+invokes `record_adapter_manifest` (embedding the corpus spec, device,
+and result classification into the run's `manifest.json`).
+
+Report schema:
+
+```json
+{
+  "schema_version": 1,
+  "benchmark": "latency",
+  "run_id": "my-run",
+  "corpus": { "kind": "synthetic", "videos": 1, "duration_seconds": 8.0, ... },
+  # or { "kind": "real", "name": "didemo", "source": "...", "video_count": 2,
+  #      "total_bytes": 12345, "min_duration_seconds": 5.0,
+  #      "max_duration_seconds": 9.0, "containers": [".mp4"],
+  #      "media_overrides": false },
+  "modalities": ["scene", "actor"],
+  "device": "cpu",
+  "repetitions": 1,
+  "git": { "commit": "...", "dirty": false },
+  "environment": { ... },
+  "record_counts": { "scene": 8, "actor": 0 },
+  "processed_frames": 8,
+  "stages": {
+    "scene": {
+      "runs": 1, "mean_seconds": 2.1, "min_seconds": 2.1,
+      "max_seconds": 2.1, "rate_per_second": 3.8
+    }
+  },
+  "summary": {
+    "wall_seconds": { "runs": 1, "mean_seconds": 5.0, ... },
+    "peak_rss": { "unit": "bytes", "samples": 1, "value": 123456789 }
+  },
+  "baseline": null | { "stages": {...}, "regressions": [...], "verdict": "pass" }
+}
+```
+
+## Limitations
+
+- The synthetic video has no semantic scene content, so scene embeddings
+  are representative of throughput but not retrieval quality.
+- Actors are not present in `testsrc2` video; `actor` stage measures
+  the per-frame face-detection overhead with zero detections.
+- When `input_mode=transcript`, no real whisper transcription occurs;
+  dialogue embedding is measured on a synthetic transcript.
+- True transcription latency (`input_mode=transcribe`) requires a
+  speech source (`--audio-mode flite`) and libflite in the FFmpeg
+  build; the generated speech is a short fixed sentence and does not
+  represent naturalistic conversation length or vocabulary.
+- Real corpora give realistic decode, face-detection, and (with
+  `--input-mode transcribe`) speech workloads, but `--corpus didemo`
+  relies on the DiDeMo media already prepared locally; the exact clip
+  set depends on what `vidxp benchmark prepare didemo` downloaded and
+  on any recorded media overrides.
+- Peak RSS measures the whole-process peak, which includes Python
+  overhead, loaded models, and Chroma state; it is not a pure
+  indexing-stage measurement.
+
+## Usage
+
+```bash
+# Default: single 8-second 320x180 clip, scene-only, 1 rep
+vidxp benchmark index-latency --run-id my-baseline
+
+# Scene + actor + dialogue (synthetic transcript), 3 reps, compare with baseline
+vidxp benchmark index-latency \
+  --run-id v2-compare \
+  --modalities scene,actor,dialogue \
+  --videos 2 \
+  --duration-seconds 12 \
+  --repetitions 3 \
+  --json \
+  --baseline benchmark_runs/latency/synthetic/my-baseline/report.json
+
+# Real transcription (requires libflite in ffmpeg)
+vidxp benchmark index-latency \
+  --run-id transcribe-test \
+  --modalities dialogue \
+  --input-mode transcribe \
+  --audio-mode flite \
+  --device cpu
+
+# Real DiDeMo media (prepare it first, then index one clip as a smoke)
+vidxp benchmark prepare didemo --split validation --annotation-indices 0 --yes
+vidxp benchmark index-latency \
+  --run-id didemo-smoke \
+  --corpus didemo \
+  --modalities scene,actor \
+  --videos 1
+
+# Full real-corpus scene sweep over prepared DiDeMo media
+vidxp benchmark index-latency \
+  --run-id didemo-val \
+  --corpus didemo \
+  --modalities scene \
+  --videos 50 \
+  --repetitions 3 \
+  --json
+
+# Real dialogue requires transcription over real audio
+vidxp benchmark index-latency \
+  --run-id didemo-dialogue \
+  --corpus didemo \
+  --modalities dialogue \
+  --input-mode transcribe \
+  --videos 2
+```
+
+## Adding a new performance benchmark
+
+1. Define the corpus parameters and any new modality combinations in
+   the existing `run_latency` entry point.
+2. Run the baseline and save its `report.json`.
+3. Make your change (model swap, concurrency refactor, etc.).
+4. Re-run with `--baseline <baseline-report.json>` and verify no
+   regressions.
+5. Commit the baseline report to a designated location (e.g.
+   `docs/benchmarking/baselines/`) if it serves as a team reference.

src/vidxp/benchmarks/cli.py

@@ -21,6 +21,7 @@
     HIREST_DEFAULT_WINDOW_FRACTION,
     run_hirest,
 )
+from vidxp.benchmarks.latency import run_latency
 from vidxp.benchmarks.prepare import (
     PreparationPlan,
     execute_preparation,
@@ -574,3 +575,159 @@ def hirest_command(
         emit_json(metrics)
     else:
         rich_print(metrics)
+
+
+@app.command("index-latency")
+def index_latency_command(
+    ctx: typer.Context,
+    run_id: Annotated[str, typer.Option(help="Arbitrary label for this run.")],
+    modalities: Annotated[
+        str,
+        typer.Option(
+            help="Comma-separated modality names: scene,actor,dialogue."
+        ),
+    ] = "scene",
+    videos: Annotated[
+        int,
+        typer.Option(
+            min=1,
+            help=(
+                "Number of synthetic clips, or the maximum number of real "
+                "corpus clips to index."
+            ),
+        ),
+    ] = 1,
+    duration_seconds: Annotated[
+        float,
+        typer.Option(min=0.1, help="Duration of each synthetic clip."),
+    ] = 8.0,
+    fps: Annotated[
+        int,
+        typer.Option(min=1, help="Frame rate of synthetic clips."),
+    ] = 24,
+    resolution: Annotated[
+        str,
+        typer.Option(
+            help="Synthetic clip resolution in WxH format (e.g. 320x180)."
+        ),
+    ] = "320x180",
+    repetitions: Annotated[
+        int,
+        typer.Option(min=1, help="Number of times to repeat the run."),
+    ] = 1,
+    input_mode: Annotated[
+        Literal["transcript", "transcribe"],
+        typer.Option(
+            help=(
+                "'transcript' supplies a synthetic transcript for dialogue "
+                "embedding (no real transcription). 'transcribe' runs "
+                "real whisper on audio (requires --audio-mode flite)."
+            )
+        ),
+    ] = "transcript",
+    audio_mode: Annotated[
+        Literal["none", "sine", "flite"],
+        typer.Option(
+            help=(
+                "Audio track for synthetic clips: 'none' (no audio), "
+                "'sine' (tone), or 'flite' (speech synthesis)."
+            )
+        ),
+    ] = "none",
+    corpus: Annotated[
+        str | None,
+        typer.Option(
+            help=(
+                "Real media corpus: 'didemo' (prepared DiDeMo media) or a "
+                "directory of video files. Omit for synthetic media."
+            ),
+        ),
+    ] = None,
+    reset: Annotated[
+        bool,
+        typer.Option(help="Clear any existing index before running."),
+    ] = False,
+    baseline: Annotated[
+        Path | None,
+        typer.Option(
+            exists=True,
+            dir_okay=False,
+            help=(
+                "Path to a previous latency report JSON for regression "
+                "comparison."
+            ),
+        ),
+    ] = None,
+    baseline_tolerance: Annotated[
+        float,
+        typer.Option(
+            min=0.0,
+            max=5.0,
+            help=(
+                "Relative regression tolerance. A stage mean slower by "
+                "more than this ratio flags as regression."
+            ),
+        ),
+    ] = 0.15,
+    json_output: Annotated[
+        bool,
+        typer.Option("--json", help="Emit machine-readable JSON."),
+    ] = False,
+) -> None:
+    """Run a reproducible indexing-latency benchmark on synthetic or real media."""
+
+    selected = [item.strip() for item in modalities.split(",") if item.strip()]
+    if not selected:
+        raise typer.BadParameter(
+            "At least one latency modality is required.", param_hint="--modalities"
+        )
+    for modality in selected:
+        _require_benchmark_dependencies(modality)
+
+    if corpus is not None and "dialogue" in selected:
+        if input_mode != "transcribe":
+            raise typer.BadParameter(
+                "Real corpora have no released transcripts; dialogue "
+                "requires --input-mode transcribe.",
+                param_hint="--input-mode",
+            )
+
+    try:
+        parts = resolution.lower().split("x")
+        if len(parts) != 2:
+            raise ValueError
+        width, height = int(parts[0]), int(parts[1])
+        if width <= 0 or height <= 0:
+            raise ValueError
+    except (IndexError, ValueError, AttributeError):
+        raise typer.BadParameter(
+            f"Invalid resolution: {resolution!r}. Use WxH, e.g. 320x180.",
+            param_hint="--resolution",
+        )
+
+    state = state_from_context(ctx)
+    report = run_latency(
+        run_id=run_id,
+        output_root=state.settings.data_dir / "benchmark_runs",
+        ffprobe=state.settings.ffprobe_executable,
+        ffmpeg=state.settings.ffmpeg_executable,
+        modalities=tuple(selected),
+        videos=videos,
+        duration_seconds=duration_seconds,
+        fps=fps,
+        width=width,
+        height=height,
+        repetitions=repetitions,
+        input_mode=input_mode,
+        audio_mode=audio_mode,
+        device=state.settings.runtime_backend,
+        reset=reset,
+        baseline_path=baseline,
+        baseline_tolerance=baseline_tolerance,
+        corpus=corpus,
+        data_dir=state.settings.data_dir,
+    )
+    if effective_output_format(state, json_output) == OutputFormat.json:
+        emit_json(report)
+    else:
+        rich_print(report)