Turn 2D videos into Apple Vision Pro Spatial Video — generated, encoded, and published entirely on Windows.
ml-sharp-spatial is a fork of
iVideoGameBoss/ml-sharp (itself a
fork of apple/ml-sharp) that adds a
complete, Windows-native spatial-video pipeline and a production-style
API service on top of the original SHARP monocular view-synthesis model.
It integrates with ComfyGallery — my self-hosted media gallery — so generated Spatial MOVs are published back to it as bound spatial-video variants of the source media.
upstream (iVideoGameBoss/ml-sharp + apple/ml-sharp)
2D video ──▶ ML-Sharp SBS ──▶ ┌─────────────────────────────────────┐ ──▶ ComfyGallery
(single-image Gaussian │ this fork: Windows-native MV-HEVC │ (optional publish,
splat per frame, rendered │ Spatial MOV + FastAPI service + CG │ idempotent variant)
as side-by-side stereo) │ orchestration) │
└─────────────────────────────────────┘
The upstream projects regress a 3D Gaussian splat from each frame and render a side-by-side (SBS) stereo video. This fork turns that SBS output into an Apple-compatible Spatial MOV and wraps the whole chain in a service:
- Windows-native MV-HEVC Spatial Video — single-track, two-layer MV-HEVC
produced with a patched x265 4.2 multiview build, muxed with GPAC,
carrying Apple stereo sample-entry boxes (
vexu/hfov), spatial SEI 176 inhvcC, and QuickTime branding. Structurally matched to a Vision Pro-verified sample (boxes byte-identical; both views decode). SeeIMPLEMENTATION_STATUS.md. - FastAPI batch service (
api_service.py) — submit one or more 2D videos, get backoriginal + SBS + Spatial MOVtriplets plus a ZIP. FIFO queue (one GPU job at a time), durable on-disk state, crash recovery, pause/resume, delete, and per-file error isolation. - Browser batch manager (
/batch-manager) — drag-and-drop upload, queue monitoring, cached previews, publish/validate controls, and a gallery accessibility heartbeat. - ComfyGallery publishing (
comfygallery_publisher.py) — after a Spatial MOV is generated, publish the original + spatial pair back to a ComfyGallery instance as a bound, idempotent spatial variant (hash-first verified, sha-bound, idempotency-keyed). Supports auto-publish on submit, manual/retroactive publish, gallery-existence validation, and a 30s heartbeat that disables the controls when the gallery is unreachable. - CG-driven orchestration — ComfyGallery can POST a job (passing its
media_id) and MSS generates the spatial and publishes the variant back; CG just polls. SeeCG_MSS_REQUEST_CONTRACT.md.
Spatial conversion reports success only after layered encoding, muxing, Apple metadata, SEI placement, and container validation all pass — it never copies a non-spatial file and reports success.
Requirements: an NVIDIA CUDA GPU, Python 3.10 (conda env sharp),
FFmpeg/FFprobe, and the patched x265 multiview build + GPAC (build steps in
SPATIAL_PIPELINE_README.md).
conda activate sharp
python -m pip install -r requirements-api.txt
python api_service.py --host 0.0.0.0 --port 8000Then open:
- Browser manager: http://127.0.0.1:8000/batch-manager
- API docs (Swagger): http://127.0.0.1:8000/docs
Submit a spatial batch (the primary call):
curl.exe -X POST http://127.0.0.1:8000/spatial/batches `
-F "videos=@holiday.mp4" `
-F "precision=float16" `
-F "spatial_quality=high"Poll GET /spatial/batches/{batch_id} until terminal, then fetch the ZIP from
download_url. Full request/response contract, defaults, and the CG-driven flow:
API_INTEGRATION_GUIDE.md.
For an SSH-safe detached launch (logs + PID under output/):
.\run_api_detached.ps1 # .\stop_api_detached.ps1 to stopThe upstream WebUI (
python webui.py) is retained for interactive single-image / SBS use. The detailed PC/Mac/Linux install guides for that flow live in the upstream README.
| Path | Purpose |
|---|---|
api_service.py |
FastAPI service: spatial batches, queue, publish/validate, delete |
batch_manager.html |
Dependency-free browser queue manager |
comfygallery_publisher.py |
Direct ComfyGallery client (publish, validate, test) |
spatial_video.py |
MV-HEVC encoder, GPAC muxer, Apple box writer, validators |
spatial_pipeline.py / apple_mv_hevc_converter.py / spatial_metadata_injector.py |
Compatibility CLIs for the spatial stage |
webui.py / src/sharp/ |
Upstream WebUI + SHARP model (unchanged) |
patches/x265-4.2-apple-spatial-compat.patch |
Reproducible x265 multiview source patch |
scripts/build_x265_apple_compat.ps1 |
Windows x265 configure/build helper |
run_api_detached.ps1 / stop_api_detached.ps1 |
SSH-safe detached launcher / stopper |
config/comfygallery.example.json |
ComfyGallery endpoint + token template |
tests/ |
Spatial, batch, and publisher tests |
ComfyGallery is my self-hosted media gallery and the primary consumer of this pipeline. The two projects work in either direction:
- MSS → CG (publish) — after a Spatial MOV is generated, MSS publishes the
original + spatial pair to a ComfyGallery instance as a bound, idempotent
spatial variant of the source media (hash-first verified, sha-bound). Enable
with
publish_to_gallery=trueon the batch request, or trigger it later viaPOST /spatial/batches/{id}/publish. A 30s accessibility heartbeat disables the publish/validate controls when the gallery is unreachable. Configure the endpoint + token inconfig/comfygallery.json. - CG → MSS (orchestrate) — ComfyGallery POSTs a job to MSS (passing its
media_id), MSS generates the spatial and publishes the variant back, and CG just polls. Exact request + read-back:CG_MSS_REQUEST_CONTRACT.md.
What ComfyGallery must provide for the integration:
CG_CONFORMANCE_CHECKLIST.md.
API_INTEGRATION_GUIDE.md— the REST contract: submit, poll, download, publish/validate, defaults, the CG-driven flow.SPATIAL_PIPELINE_README.md— native tool setup (x265 multiview build, GPAC) and the spatial stage.IMPLEMENTATION_STATUS.md— what's implemented, verified, and pending.CG_CONFORMANCE_CHECKLIST.md— what ComfyGallery must provide for the integration.CG_MSS_REQUEST_CONTRACT.md— exact request CG should send to MSS, and how to read results back.API_README.md— legacy single-file SBS-only API (pre-batch).
- Spatial pipeline locally validated (structure + dual-view decode + byte-identical
reference boxes); 70+
ml-sharp-spatialv0.1 variants live in production ComfyGallery, all passing second-view decode. - Final Photos/TV classification on Vision Pro hardware is the remaining certification step.
- Tests: spatial regression, API batch, and publisher suites all passing.
This project builds on:
- Apple SHARP — Sharp Monocular View Synthesis in Less Than a Second, Mescheder et al., arXiv:2512.10685. Model + SBS generation come from apple/ml-sharp.
- iVideoGameBoss/ml-sharp — the WebUI + SBS-video fork this repo is based on.
- ComfyGallery — github.com/blkot/ProjectComfyGallery, a separate project by the same author that this pipeline publishes to.
Please see the upstream LICENSE / LICENSE_MODEL and ACKNOWLEDGEMENTS for the original model and code. This fork's additions are provided under the same license.
@inproceedings{Sharp2025:arxiv,
title = {Sharp Monocular View Synthesis in Less Than a Second},
author = {Lars Mescheder and Wei Dong and Shiwei Li and Xuyang Bai and Marcel Santos
and Peiyun Hu and Bruno Lecouat and Mingmin Zhen and Ama\"{e}l Delaunoy
and Tian Fang and Yanghai Tsin and Stephan Richter and Vladlen Koltun},
journal = {arXiv preprint arXiv:2512.10685},
year = {2025},
url = {https://arxiv.org/abs/2512.10685},
}