Skip to content

Latest commit

 

History

History
173 lines (143 loc) · 9.39 KB

README.md

File metadata and controls

173 lines (143 loc) · 9.39 KB

ml-sharp-spatial

Turn 2D videos into Apple Vision Pro Spatial Video — generated, encoded, and published entirely on Windows.

ml-sharp-spatial is a fork of iVideoGameBoss/ml-sharp (itself a fork of apple/ml-sharp) that adds a complete, Windows-native spatial-video pipeline and a production-style API service on top of the original SHARP monocular view-synthesis model.

It integrates with ComfyGallery — my self-hosted media gallery — so generated Spatial MOVs are published back to it as bound spatial-video variants of the source media.

                       upstream (iVideoGameBoss/ml-sharp + apple/ml-sharp)
2D video  ──▶  ML-Sharp SBS  ──▶  ┌─────────────────────────────────────┐  ──▶  ComfyGallery
   (single-image Gaussian        │ this fork: Windows-native MV-HEVC     │       (optional publish,
    splat per frame, rendered    │ Spatial MOV + FastAPI service + CG    │        idempotent variant)
    as side-by-side stereo)      │ orchestration)                        │
                                  └─────────────────────────────────────┘

What this fork adds

The upstream projects regress a 3D Gaussian splat from each frame and render a side-by-side (SBS) stereo video. This fork turns that SBS output into an Apple-compatible Spatial MOV and wraps the whole chain in a service:

  • Windows-native MV-HEVC Spatial Video — single-track, two-layer MV-HEVC produced with a patched x265 4.2 multiview build, muxed with GPAC, carrying Apple stereo sample-entry boxes (vexu / hfov), spatial SEI 176 in hvcC, and QuickTime branding. Structurally matched to a Vision Pro-verified sample (boxes byte-identical; both views decode). See IMPLEMENTATION_STATUS.md.
  • FastAPI batch service (api_service.py) — submit one or more 2D videos, get back original + SBS + Spatial MOV triplets plus a ZIP. FIFO queue (one GPU job at a time), durable on-disk state, crash recovery, pause/resume, delete, and per-file error isolation.
  • Browser batch manager (/batch-manager) — drag-and-drop upload, queue monitoring, cached previews, publish/validate controls, and a gallery accessibility heartbeat.
  • ComfyGallery publishing (comfygallery_publisher.py) — after a Spatial MOV is generated, publish the original + spatial pair back to a ComfyGallery instance as a bound, idempotent spatial variant (hash-first verified, sha-bound, idempotency-keyed). Supports auto-publish on submit, manual/retroactive publish, gallery-existence validation, and a 30s heartbeat that disables the controls when the gallery is unreachable.
  • CG-driven orchestration — ComfyGallery can POST a job (passing its media_id) and MSS generates the spatial and publishes the variant back; CG just polls. See CG_MSS_REQUEST_CONTRACT.md.

Spatial conversion reports success only after layered encoding, muxing, Apple metadata, SEI placement, and container validation all pass — it never copies a non-spatial file and reports success.

Quick start

Requirements: an NVIDIA CUDA GPU, Python 3.10 (conda env sharp), FFmpeg/FFprobe, and the patched x265 multiview build + GPAC (build steps in SPATIAL_PIPELINE_README.md).

conda activate sharp
python -m pip install -r requirements-api.txt
python api_service.py --host 0.0.0.0 --port 8000

Then open:

Submit a spatial batch (the primary call):

curl.exe -X POST http://127.0.0.1:8000/spatial/batches `
  -F "videos=@holiday.mp4" `
  -F "precision=float16" `
  -F "spatial_quality=high"

Poll GET /spatial/batches/{batch_id} until terminal, then fetch the ZIP from download_url. Full request/response contract, defaults, and the CG-driven flow: API_INTEGRATION_GUIDE.md.

For an SSH-safe detached launch (logs + PID under output/):

.\run_api_detached.ps1            # .\stop_api_detached.ps1 to stop

The upstream WebUI (python webui.py) is retained for interactive single-image / SBS use. The detailed PC/Mac/Linux install guides for that flow live in the upstream README.

Project layout

Path Purpose
api_service.py FastAPI service: spatial batches, queue, publish/validate, delete
batch_manager.html Dependency-free browser queue manager
comfygallery_publisher.py Direct ComfyGallery client (publish, validate, test)
spatial_video.py MV-HEVC encoder, GPAC muxer, Apple box writer, validators
spatial_pipeline.py / apple_mv_hevc_converter.py / spatial_metadata_injector.py Compatibility CLIs for the spatial stage
webui.py / src/sharp/ Upstream WebUI + SHARP model (unchanged)
patches/x265-4.2-apple-spatial-compat.patch Reproducible x265 multiview source patch
scripts/build_x265_apple_compat.ps1 Windows x265 configure/build helper
run_api_detached.ps1 / stop_api_detached.ps1 SSH-safe detached launcher / stopper
config/comfygallery.example.json ComfyGallery endpoint + token template
tests/ Spatial, batch, and publisher tests

ComfyGallery integration

ComfyGallery is my self-hosted media gallery and the primary consumer of this pipeline. The two projects work in either direction:

  • MSS → CG (publish) — after a Spatial MOV is generated, MSS publishes the original + spatial pair to a ComfyGallery instance as a bound, idempotent spatial variant of the source media (hash-first verified, sha-bound). Enable with publish_to_gallery=true on the batch request, or trigger it later via POST /spatial/batches/{id}/publish. A 30s accessibility heartbeat disables the publish/validate controls when the gallery is unreachable. Configure the endpoint + token in config/comfygallery.json.
  • CG → MSS (orchestrate) — ComfyGallery POSTs a job to MSS (passing its media_id), MSS generates the spatial and publishes the variant back, and CG just polls. Exact request + read-back: CG_MSS_REQUEST_CONTRACT.md.

What ComfyGallery must provide for the integration: CG_CONFORMANCE_CHECKLIST.md.

Documentation index

Status

  • Spatial pipeline locally validated (structure + dual-view decode + byte-identical reference boxes); 70+ ml-sharp-spatial v0.1 variants live in production ComfyGallery, all passing second-view decode.
  • Final Photos/TV classification on Vision Pro hardware is the remaining certification step.
  • Tests: spatial regression, API batch, and publisher suites all passing.

Attribution & license

This project builds on:

  • Apple SHARP — Sharp Monocular View Synthesis in Less Than a Second, Mescheder et al., arXiv:2512.10685. Model + SBS generation come from apple/ml-sharp.
  • iVideoGameBoss/ml-sharp — the WebUI + SBS-video fork this repo is based on.
  • ComfyGallery — github.com/blkot/ProjectComfyGallery, a separate project by the same author that this pipeline publishes to.

Please see the upstream LICENSE / LICENSE_MODEL and ACKNOWLEDGEMENTS for the original model and code. This fork's additions are provided under the same license.

@inproceedings{Sharp2025:arxiv,
  title   = {Sharp Monocular View Synthesis in Less Than a Second},
  author  = {Lars Mescheder and Wei Dong and Shiwei Li and Xuyang Bai and Marcel Santos
             and Peiyun Hu and Bruno Lecouat and Mingmin Zhen and Ama\"{e}l Delaunoy
             and Tian Fang and Yanghai Tsin and Stephan Richter and Vladlen Koltun},
  journal = {arXiv preprint arXiv:2512.10685},
  year    = {2025},
  url     = {https://arxiv.org/abs/2512.10685},
}