Skip to content

Latest commit

 

History

History
388 lines (272 loc) · 9.97 KB

HANDOFF.md

File metadata and controls

388 lines (272 loc) · 9.97 KB

ML-Sharp SBS API Service - Handoff Documentation

Date: 2026-07-29 Project: ML-Sharp 3D Gaussian Splatting with REST API Service Repository: https://github.com/YOUR_USERNAME/ml-sharp (fork of apple/ml-sharp)


Overview

This project extends Apple's ML-Sharp (Sharp Monocular View Synthesis in Less Than a Second) with a production-ready REST API service for converting 2D videos to 3D Side-by-Side (SBS) format. The service supports AR/VR glasses including XREAL, VITURE, Rokid, RayNeo, Apple Vision Pro, and Meta Quest.


What Was Built

1. REST API Service (api_service.py)

A FastAPI-based service with the following features:

Feature Description
Async Job Queue Thread-safe job queue with configurable concurrent workers
Progress Tracking Real-time monitoring with processed_frames / total_frames
Dynamic Resolution Uses input video dimensions (no hardcoded render sizes)
Precision Control float16/float32 computation modes
Batch Processing Configurable batch size (1-8) for GPU optimization
Frame Skipping Process every Nth frame for faster output
Auto-Documentation Built-in OpenAPI/Swagger docs at /docs

2. API Endpoints

Method Endpoint Purpose
POST /jobs/create Submit video for SBS conversion
GET /jobs/{job_id} Check job status & progress
GET /jobs List all jobs (filterable by status)
DELETE /jobs/{job_id} Cancel running job
GET /download/{filename} Download completed SBS video
GET /status Server health & statistics
GET /health Health check endpoint
GET /docs Interactive API documentation

3. Supporting Files

File Purpose
requirements-api.txt Python dependencies (FastAPI, uvicorn, etc.)
API_README.md Complete API documentation with examples
test_api.py Command-line test client for API testing
run_api.sh Startup script for the service
ml-sharp-api.service systemd service file for production deployment

Architecture Decisions

Why FastAPI Instead of Flask?

The original webui.py uses Flask. We chose FastAPI for the API service because:

  1. Automatic OpenAPI docs - No manual Swagger configuration needed
  2. Native async support - Better for handling concurrent requests
  3. Type validation - Pydantic models enforce request/response schemas
  4. Better performance - Built on Starlette with modern Python async

Job Queue Design

  • In-memory queue with thread-safe operations
  • Configurable concurrent workers (default: 1)
  • Jobs persist in memory during server runtime
  • Each job has detailed status tracking: queued → running → completed/failed

Dynamic Resolution

Unlike the original webui.py which hardcoded render_w=552, render_h=832, the API:

  1. Reads the input video dimensions
  2. Uses those dimensions for rendering
  3. Outputs SBS video at input_width * 2 x input_height

This ensures the output matches the input quality.


Installation & Setup

Prerequisites

# Python 3.8+
# CUDA GPU (required for SBS rendering)
# 8GB+ GPU RAM recommended

Installation

# Clone your fork
git clone https://github.com/YOUR_USERNAME/ml-sharp.git
cd ml-sharp

# Install dependencies
pip install -r requirements-api.txt

# Or install directly
pip install fastapi uvicorn python-multipart

Running the Service

# Start with defaults (http://0.0.0.0:8000)
python api_service.py

# With options
python api_service.py --host 0.0.0.0 --port 8000 --preload --workers 2

# Or use the startup script
./run_api.sh --preload

Production Deployment

# Install as systemd service
sudo cp ml-sharp-api.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable ml-sharp-api
sudo systemctl start ml-sharp-api

Remember to update the User= and WorkingDirectory= paths in the service file.


API Usage Examples

Create a Conversion Job

curl -X POST "http://localhost:8000/jobs/create" \
  -F "video=@myvideo.mp4" \
  -F "precision=float16" \
  -F "batch_size=2" \
  -F "frame_skip=1"

Monitor Progress

# Poll job status
curl "http://localhost:8000/jobs/{job_id}"

# Response includes:
# - progress: 0.0 to 1.0
# - processed_frames: current count
# - total_frames: total to process
# - stage: current processing stage

Download Result

curl "http://localhost:8000/download/001_myvideo_SBS.mp4" -o output.mp4

Hoppscotch / Postman Setup

For manual API testing:

  1. Request Type: POST
  2. URL: http://localhost:8000/jobs/create
  3. Body Type: FormData
  4. Fields:
    • video (File): Select your video
    • precision (Text): float16 or float32
    • batch_size (Text): 1 to 8
    • frame_skip (Text): 1 (all frames) or higher

Important: Set video field type to File, not Text.


Project Structure

ml-sharp/
├── api_service.py          # NEW: FastAPI service
├── requirements-api.txt     # NEW: API dependencies
├── API_README.md           # NEW: API documentation
├── test_api.py             # NEW: Test client
├── run_api.sh              # NEW: Startup script
├── ml-sharp-api.service    # NEW: systemd service
├── webui.py                # Original: Flask web UI
├── run_webui.sh            # Original: Web UI startup
├── src/                    # Core ML-Sharp library
│   └── sharp/
│       ├── models/         # Neural network models
│       ├── utils/          # Gaussian splat utilities
│       └── cli/            # Command-line interface
├── output/                 # Generated PLY and SBS files
└── data/                   # Sample images

Key Parameters

Parameter Default Range Description
precision float32 float32/float16 Computation precision (float16 = 2x faster)
batch_size 1 1-8 Frames processed per batch (more GPU memory)
frame_skip 1 1+ Process every Nth frame (1 = all)
opacity_threshold 0.0 0.0-1.0 Removes faint splats (halo reduction)
stereo_offset 0.015 0.0-0.1 Distance between stereo cameras
brightness_factor 1.0 0.1-3.0 Output brightness adjustment

Performance Considerations

GPU Memory

Batch Size Approx GPU Memory Notes
1 ~4GB Safest option
2 ~6GB Good balance
4 ~10GB Faster, needs more GPU
8 ~16GB Maximum, may OOM

Processing Speed

Setting Effect
precision=float16 2x faster inference
batch_size=4 ~3-4x faster processing
frame_skip=2 2x faster, half output FPS

Output Quality vs Speed

Configuration Use Case
float32, batch=1, skip=1 Highest quality, slowest
float16, batch=2, skip=1 Balanced (recommended)
float16, batch=4, skip=2 Fast preview, lower quality

Troubleshooting

CUDA Not Available

Error: "SBS Rendering requires a CUDA GPU"

Solution: Ensure NVIDIA GPU with CUDA drivers installed. CPU/MPS not supported for SBS rendering.

Out of Memory Errors

Symptoms: Job fails with "out of memory" error

Solution: Reduce batch_size to 1 or use precision=float16

Slow Processing

Check:

  1. GPU utilization with nvidia-smi
  2. Batch size setting (higher = faster)
  3. Precision setting (float16 = faster)

Video Not Playing in AR Glasses

Check:

  1. Output resolution matches glasses specifications
  2. File is standard SBS format (not anamorphic)
  3. Try VLC player first to verify the file

Original Project Attribution

This work extends ML-Sharp by Apple.

Paper: Sharp Monocular View Synthesis in Less Than a Second Authors: Lars Mescheder, Wei Dong, Shiwei Li, et al. License: See LICENSE file in project root


Ideas Discussed / Future Enhancements

1. Distributed Processing

  • Current: Single GPU, in-memory queue
  • Future: Redis + Celery for multi-GPU or multi-machine scaling

2. Job Persistence

  • Current: Jobs lost on server restart
  • Future: Database-backed job storage (SQLite/PostgreSQL)

3. Authentication

  • Current: Open API, no auth
  • Future: API keys, OAuth, or JWT tokens

4. Websocket Progress

  • Current: Polling-based progress updates
  • Future: Websocket for real-time push updates

5. Output Format Options

  • Current: SBS MP4 only
  • Future: VR180, anaglyph, depth map export

6. Live Preview

  • Current: Process entire video first
  • Future: Real-time preview while rendering

Development Workflow

Making Changes

# Work on feature branch
git checkout -b feature/your-feature

# Make changes and test
python test_api.py convert --video test.mp4

# Commit changes
git add .
git commit -m "Description of changes"

# Push to fork
git push origin feature/your-feature

# Create PR to original if appropriate

Syncing with Upstream

# Add upstream if not already added
git remote add upstream https://github.com/apple/ml-sharp.git

# Fetch upstream changes
git fetch upstream

# Merge into your branch
git merge upstream/main

# Push updates
git push origin main

Contact & Support


Quick Reference Commands

# Start API service
python api_service.py --preload

# Test with video
python test_api.py convert --video test.mp4 --precision float16

# Check server status
curl http://localhost:8000/status

# List all jobs
curl http://localhost:8000/jobs

# Cancel a job
curl -X DELETE http://localhost:8000/jobs/{job_id}

Last Updated: 2026-07-29