Date: 2026-07-29 Project: ML-Sharp 3D Gaussian Splatting with REST API Service Repository: https://github.com/YOUR_USERNAME/ml-sharp (fork of apple/ml-sharp)
This project extends Apple's ML-Sharp (Sharp Monocular View Synthesis in Less Than a Second) with a production-ready REST API service for converting 2D videos to 3D Side-by-Side (SBS) format. The service supports AR/VR glasses including XREAL, VITURE, Rokid, RayNeo, Apple Vision Pro, and Meta Quest.
A FastAPI-based service with the following features:
| Feature | Description |
|---|---|
| Async Job Queue | Thread-safe job queue with configurable concurrent workers |
| Progress Tracking | Real-time monitoring with processed_frames / total_frames |
| Dynamic Resolution | Uses input video dimensions (no hardcoded render sizes) |
| Precision Control | float16/float32 computation modes |
| Batch Processing | Configurable batch size (1-8) for GPU optimization |
| Frame Skipping | Process every Nth frame for faster output |
| Auto-Documentation | Built-in OpenAPI/Swagger docs at /docs |
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /jobs/create |
Submit video for SBS conversion |
| GET | /jobs/{job_id} |
Check job status & progress |
| GET | /jobs |
List all jobs (filterable by status) |
| DELETE | /jobs/{job_id} |
Cancel running job |
| GET | /download/{filename} |
Download completed SBS video |
| GET | /status |
Server health & statistics |
| GET | /health |
Health check endpoint |
| GET | /docs |
Interactive API documentation |
| File | Purpose |
|---|---|
requirements-api.txt |
Python dependencies (FastAPI, uvicorn, etc.) |
API_README.md |
Complete API documentation with examples |
test_api.py |
Command-line test client for API testing |
run_api.sh |
Startup script for the service |
ml-sharp-api.service |
systemd service file for production deployment |
The original webui.py uses Flask. We chose FastAPI for the API service because:
- Automatic OpenAPI docs - No manual Swagger configuration needed
- Native async support - Better for handling concurrent requests
- Type validation - Pydantic models enforce request/response schemas
- Better performance - Built on Starlette with modern Python async
- In-memory queue with thread-safe operations
- Configurable concurrent workers (default: 1)
- Jobs persist in memory during server runtime
- Each job has detailed status tracking:
queued→running→completed/failed
Unlike the original webui.py which hardcoded render_w=552, render_h=832, the API:
- Reads the input video dimensions
- Uses those dimensions for rendering
- Outputs SBS video at
input_width * 2xinput_height
This ensures the output matches the input quality.
# Python 3.8+
# CUDA GPU (required for SBS rendering)
# 8GB+ GPU RAM recommended# Clone your fork
git clone https://github.com/YOUR_USERNAME/ml-sharp.git
cd ml-sharp
# Install dependencies
pip install -r requirements-api.txt
# Or install directly
pip install fastapi uvicorn python-multipart# Start with defaults (http://0.0.0.0:8000)
python api_service.py
# With options
python api_service.py --host 0.0.0.0 --port 8000 --preload --workers 2
# Or use the startup script
./run_api.sh --preload# Install as systemd service
sudo cp ml-sharp-api.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable ml-sharp-api
sudo systemctl start ml-sharp-apiRemember to update the User= and WorkingDirectory= paths in the service file.
curl -X POST "http://localhost:8000/jobs/create" \
-F "video=@myvideo.mp4" \
-F "precision=float16" \
-F "batch_size=2" \
-F "frame_skip=1"# Poll job status
curl "http://localhost:8000/jobs/{job_id}"
# Response includes:
# - progress: 0.0 to 1.0
# - processed_frames: current count
# - total_frames: total to process
# - stage: current processing stagecurl "http://localhost:8000/download/001_myvideo_SBS.mp4" -o output.mp4For manual API testing:
- Request Type: POST
- URL:
http://localhost:8000/jobs/create - Body Type: FormData
- Fields:
video(File): Select your videoprecision(Text):float16orfloat32batch_size(Text):1to8frame_skip(Text):1(all frames) or higher
Important: Set video field type to File, not Text.
ml-sharp/
├── api_service.py # NEW: FastAPI service
├── requirements-api.txt # NEW: API dependencies
├── API_README.md # NEW: API documentation
├── test_api.py # NEW: Test client
├── run_api.sh # NEW: Startup script
├── ml-sharp-api.service # NEW: systemd service
├── webui.py # Original: Flask web UI
├── run_webui.sh # Original: Web UI startup
├── src/ # Core ML-Sharp library
│ └── sharp/
│ ├── models/ # Neural network models
│ ├── utils/ # Gaussian splat utilities
│ └── cli/ # Command-line interface
├── output/ # Generated PLY and SBS files
└── data/ # Sample images
| Parameter | Default | Range | Description |
|---|---|---|---|
precision |
float32 |
float32/float16 | Computation precision (float16 = 2x faster) |
batch_size |
1 |
1-8 | Frames processed per batch (more GPU memory) |
frame_skip |
1 |
1+ | Process every Nth frame (1 = all) |
opacity_threshold |
0.0 |
0.0-1.0 | Removes faint splats (halo reduction) |
stereo_offset |
0.015 |
0.0-0.1 | Distance between stereo cameras |
brightness_factor |
1.0 |
0.1-3.0 | Output brightness adjustment |
| Batch Size | Approx GPU Memory | Notes |
|---|---|---|
| 1 | ~4GB | Safest option |
| 2 | ~6GB | Good balance |
| 4 | ~10GB | Faster, needs more GPU |
| 8 | ~16GB | Maximum, may OOM |
| Setting | Effect |
|---|---|
precision=float16 |
2x faster inference |
batch_size=4 |
~3-4x faster processing |
frame_skip=2 |
2x faster, half output FPS |
| Configuration | Use Case |
|---|---|
| float32, batch=1, skip=1 | Highest quality, slowest |
| float16, batch=2, skip=1 | Balanced (recommended) |
| float16, batch=4, skip=2 | Fast preview, lower quality |
Error: "SBS Rendering requires a CUDA GPU"
Solution: Ensure NVIDIA GPU with CUDA drivers installed. CPU/MPS not supported for SBS rendering.
Symptoms: Job fails with "out of memory" error
Solution: Reduce batch_size to 1 or use precision=float16
Check:
- GPU utilization with
nvidia-smi - Batch size setting (higher = faster)
- Precision setting (float16 = faster)
Check:
- Output resolution matches glasses specifications
- File is standard SBS format (not anamorphic)
- Try VLC player first to verify the file
This work extends ML-Sharp by Apple.
Paper: Sharp Monocular View Synthesis in Less Than a Second Authors: Lars Mescheder, Wei Dong, Shiwei Li, et al. License: See LICENSE file in project root
- Current: Single GPU, in-memory queue
- Future: Redis + Celery for multi-GPU or multi-machine scaling
- Current: Jobs lost on server restart
- Future: Database-backed job storage (SQLite/PostgreSQL)
- Current: Open API, no auth
- Future: API keys, OAuth, or JWT tokens
- Current: Polling-based progress updates
- Future: Websocket for real-time push updates
- Current: SBS MP4 only
- Future: VR180, anaglyph, depth map export
- Current: Process entire video first
- Future: Real-time preview while rendering
# Work on feature branch
git checkout -b feature/your-feature
# Make changes and test
python test_api.py convert --video test.mp4
# Commit changes
git add .
git commit -m "Description of changes"
# Push to fork
git push origin feature/your-feature
# Create PR to original if appropriate# Add upstream if not already added
git remote add upstream https://github.com/apple/ml-sharp.git
# Fetch upstream changes
git fetch upstream
# Merge into your branch
git merge upstream/main
# Push updates
git push origin main- Repository: https://github.com/YOUR_USERNAME/ml-sharp
- Original: https://github.com/apple/ml-sharp
- API Docs: http://localhost:8000/docs (when running)
# Start API service
python api_service.py --preload
# Test with video
python test_api.py convert --video test.mp4 --precision float16
# Check server status
curl http://localhost:8000/status
# List all jobs
curl http://localhost:8000/jobs
# Cancel a job
curl -X DELETE http://localhost:8000/jobs/{job_id}Last Updated: 2026-07-29