A working, browser-local WebGPU engineering preview of TripoSplat, built from the ml-sharp-web application chassis. It selects an image, runs the model pipeline locally in the browser, displays the resulting Gaussian scene, and exports binary PLY or .splat files. The official TripoSplat PyTorch implementation remains the numerical source of truth; the existing SHARP path remains a known-good browser inference baseline.
- Hugging Face Space — static Vite demo, with no Hugging Face inference compute.
- Vercel runner — public browser runner;
/e2e-lab.htmlis the fixture-driven qualification surface. - How TripoSplat works — instructional subsite covering the model, browser port, and wider WebGPU portability pattern.
Both deployments are static front ends: the model package is fetched directly from Yosun/TripoSplat-WebGPU, while model inference and exports run locally in the browser. The model layer does not upload source pixels, although deployment analytics, CDN logs, browser extensions, and an image URL's host are separate privacy considerations.
The instructional subsite explains the same image-to-3D system at three levels. Each track can be read independently:
- Generalist — an intuition-first account of single-image inference, 3D Gaussian splats, the five-stage pipeline, and browser-local execution.
- Undergraduate mathematics — Gaussian covariance and compositing, rectified-flow sampling, classifier-free guidance, Euler integration, octree occupancy, and tensor shapes.
- Researcher and systems engineer — graph contracts, ONNX/WebGPU partitioning, artifact delivery, memory pressure, numerical validation, and current release gates.
The shared generalized platform guide extends beyond TripoSplat. It classifies feed-forward reconstructors, diffusion and flow models, sparse spatial decoders, neural renderers, classical vision pipelines, and mesh generators by WebGPU portability. It also provides an eight-step procedure for partitioning and validating similar Python/CUDA inference systems for browser execution.
The guide is educational documentation, not additional validation evidence. It preserves the limitations recorded in Current status, and the official TripoSplat PyTorch implementation remains the numerical source of truth. For local development, open /learn.html; the production clean URL is /learn. A static Hugging Face mirror is also available at /learn.html.
These are animated WebP browser recordings from the tester results:
Cartoon House |
Women's Shoes |
[Corgi.ceo](https://corgi.ceo) Basket |
The original files are available in public/_testers/results.
The current runner source accepts PNG, WebP, JPEG, and AVIF. Alpha-bearing images are ready to run; opaque photos automatically start the browser-local preparation path, which removes the background, frames the subject, and produces the model-ready 1024px input. Its preparation model is fetched and cached separately on first use. This is a practical input workflow, not a claim of numerical or visual-quality parity for background removal.
- Use desktop Chrome with WebGPU. The only recorded qualification environment is Chrome 150 on an Apple M3 Max with 128 GB unified memory; Edge, 16 GB Apple Silicon, Safari, Firefox, and Windows GPUs are not yet qualified.
- Allow roughly 6.47 GB for the first verified model download and browser cache. Cache persistence depends on browser storage and quota.
- The portable WebGPU runner requests 20 sampling steps / 40 CFG DiT calls and can take several minutes on the recorded high-end hardware. That path executes but fails its recorded qualification and strict final-state gates.
This is an engineering preview, not a production or broad-hardware support claim. The four-step prepared-image path has a structural/export/viewer pass but misses a stricter diagnostic; official whole-scene and rendered-pixel parity, repeated full-generation memory behavior, and production CDN/OPFS qualification remain open. See Current status before integrating or benchmarking.
View the source, benchmarks, and issue tracker on GitHub →
corepack enable
pnpm install --frozen-lockfile
pnpm devOpen http://localhost:<PORT>/e2e-web-debug.html. Test inputs and recorded outputs are stored in public/_testers.
The exact native runner has been extracted into the separate TripoSplatMac
project. It is a native PySide6/PyTorch application and numerical fixture
oracle, not a WebGPU backend or a dependency of this static browser project.
The dedicated huggingface-space/README.md is the Space card and contains the sdk: static metadata. Keeping it separate prevents Hugging Face configuration from leaking into the project README. During deployment, its contents must become the Space repository's root README.md because nested metadata is not interpreted.
Build locally to avoid depending on Hugging Face's custom build environment. The Vite build intentionally excludes local model artifacts and deterministic fixtures; browsers retrieve the versioned model manifest and files directly from the model repository.
corepack enable
pnpm install --frozen-lockfile
pnpm build
hf upload Yosun/TripoSplat-WebGPU-Demo dist . \
--repo-type space \
--commit-message 'Deploy TripoSplat WebGPU demo'
hf repos cp huggingface-space/README.md \
hf://spaces/Yosun/TripoSplat-WebGPU-Demo/README.mdUpload the Space card after the bundle because dist/ does not contain README.md. Do not upload local model artifacts, credentials, caches, or uncommitted experiment files. Hugging Face may require a paid plan or account credits to activate updated Static Space hosting; check current Spaces pricing before deployment.
Current milestone, not a production release. The alpha
@ai3d/triposplat-webgpuworkspace package contains the five-stage browser executor and a complete 6.465 GB fp32 manifest. A measured prepared-image run completes the packaged browser path, exports 262,144 finite Gaussians, and loads the PLY into the retained browser viewer. The WebGPU 20-step loop still fails its final-state gate. Official whole-scene/render parity, bundled BiRefNet, Edge/16 GB qualification, and production memory measurements remain release blockers. See Current status before integrating or benchmarking this work.
| Capability | Result | Evidence |
|---|---|---|
| TripoSplat preprocessing + DINOv3 encoder | STRICT PASS on one recorded fixture | DINOv3 WebGPU validation |
| TripoSplat preprocessing + Flux VAE encoder | PASS on one recorded fixture | VAE WebGPU benchmark |
| One exported fp32 DiT invocation | STRICT PASS against untouched official PyTorch | DiT WebGPU validation |
| Four-step CFG/Euler browser loop | Eight calls complete; qualification PASS, strict diagnostic FAIL | Four-step benchmark |
| Twenty-step guided sampling | Forty calls complete; qualification and strict FAIL | Twenty-step benchmark |
| Eight-level fp32 octree trajectory | PASS for logits, padding independence, resampling, and final points | Octree trajectory validation |
| fp32 Gaussian feature decoder | PASS on WebGPU for raw [1,8192,480] features |
Gaussian decoder WebGPU benchmark; final scene parity remains open |
| Packaged prepared-image end-to-end path | STRUCTURAL/VIEWER PASS: all five stages, 262,144 finite Gaussians, PLY and .splat export, ready viewer canvas |
End-to-end viewer benchmark; whole-scene numerical/render parity is not claimed |
Canonical Gaussian scene + PLY/.splat helpers |
Implemented as @ai3d/gaussian-scene workspace package |
Package tests cover ownership, disposal, validation, PLY, and official-fixture .splat bytes |
@ai3d/triposplat-webgpu package |
Alpha tarball, built-in five-stage executor, integrity cache, and clean consumer checks pass; not published | Exact final tarball installed into fresh Vite and completed the five-stage WebGPU path with 262,144 finite Gaussians and both exports; packed-consumer report. Strict Next-client TypeScript and native ESM/import-map checks also pass |
| SHARP browser path | Preserved | Existing app path remains the baseline; SHARP weights have separate research-only terms |
“PASS” above means the recorded fixture met its declared numerical thresholds. It does not establish general image quality, end-to-end correctness, leak-free repeated generation, or support on untested devices.
Development disclosure: This entire WebGPU port was made using Codex GPT 5.6 Ultra during #OpenAIBuildWeek.
- The packaged browser-local pipeline works end to end at the structural level. The recorded prepared-image run completed DINOv3, Flux VAE, eight DiT calls, eight octree levels, Gaussian decoding, PLY/
.splatexport, and viewer loading. It produced 262,144 finite Gaussians. This is a structural/export/viewer pass, not a whole-scene numerical or rendered-pixel parity claim. - The isolated fp32 stages are strongly validated on the recorded fixture. DINOv3, Flux VAE, one DiT invocation, the complete eight-level octree trajectory, and the raw Gaussian decoder pass their declared browser gates. The four-step CFG/Euler loop completes and passes its qualification envelope, while still failing the separately recorded stricter latent diagnostic.
- The portable WebGPU 20-step fallback completes but is not parity-qualified. All 40 WebGPU DiT calls finish without fallback, but accumulated latent drift fails both qualification and strict final-state gates (
0.0487093maximum absolute error;0.9999996593cosine similarity). A community RTX 3090-class warm-OPFS report completed in 7m 52s, with 5m 2s in DiT inference and only 7ms in DiT readback; the same user found 20-step output visibly acceptable while four steps was too low quality. See the field analysis and optimization direction. - The remaining DiT discrepancy has been localized. The first material conditional/unconditional split occurs in
context_refiner.0attention at probability×V accumulation on the all-zero unconditional branch. ONNX Runtime CPU reproduces the initiating discrepancy, so it is not WebGPU-only; WebGPU introduces additional separation. The official sampler, CFG/Euler arithmetic, standalone octree, and standalone Gaussian decoder are not implicated as the initiating cause. - The best bounded reduction candidate was a no-go. K=256 probability×V chunking improved complete 20-step max/mean/RMSE error by approximately 14.6%/31.5%/30.3%, but remained outside tolerance, ran slower, and lacked a controlled paired visual comparison. The canonical graph and manifest therefore remain unchanged.
- The evidence is deliberately narrow. Headline measurements come from Chrome 150 with ONNX Runtime WebGPU 1.27 on one Apple M3 Max with 128 GB unified memory. Edge, 16 GB Apple Silicon, other Apple and Windows GPUs, Safari, Firefox, repeated-generation behavior, and peak WebGPU/unified-memory use remain unqualified.
See the current status, 20-step quality investigation, attention reduction parity report, and compatibility and benchmarks for evidence, thresholds, timings, and limitations.
All browser measurements below came from one Apple M3 Max Mac with 128 GB unified memory, macOS 26.3, and Chrome 150 using ONNX Runtime WebGPU 1.27.0. WASM fallback was disabled. They are not estimates and must not be extrapolated to the 16 GB target.
| Stage | Measured result |
|---|---|
| DINOv3 encoder | 8,702.1 ms cold model/session load; 4,262.4 ms inference; feature cosine similarity 0.9999999999966; strict gate passed |
| Flux VAE encoder | 1,304.1 ms median inference; 1,601.7 ms median source preprocessing plus encode; tensor cosine similarity 0.9999999997 |
| One fp32 DiT invocation | 12,584.6 ms cold model/session load; 11,844.6 ms inference; latent cosine similarity 0.9999999999996; strict gate passed |
| Four-step fp32 CFG/Euler loop | 11,411.3 ms model load; 102,268.8 ms sampling wall for eight WebGPU calls; latent cosine 0.9999999885; qualification passed, strict diagnostic failed |
| Twenty-step fp32 CFG/Euler loop | 12,510.0 ms model load; 676,669.1 ms sampling wall for forty WebGPU calls; qualification and strict gates failed |
| Full fp32 octree trajectory | 1,327.4 ms model load; 5,713.7 ms primary eight-call inference; final points bit-exact; full trajectory gate passed |
| fp32 Gaussian raw feature decoder | 7,929.6 ms model load; 5,370.6 ms inference; cosine similarity 0.99999999997; browser gate passed |
| Packaged four-step end-to-end structure/export/viewer | 247,901.5 ms generate() including verified temporary artifact staging; 248,951.4 ms through both hashed exports and viewer load; 262,144 finite Gaussians; viewer canvas 1916×954; structural/viewer gate passed |
| Final packed-tarball Vite consumer, full five-stage run | 209,712.2 ms generate() including verified temporary artifact staging; 210,158.7 ms through PLY and .splat export; exact tarball SHA-256 a2ba5ce6…; clean-install structural/export gate passed |
| Reusable fp32 DiT conditioning payloads | 4-step host-to-worker traffic reduced from 184,774,656 to 46,193,664 bytes; 20-step from 923,873,280 to 46,193,664 bytes. This is deterministic payload accounting, not measured wall-time or GPU-upload savings |
Chromium exposes limited main-realm JavaScript heap counters, not worker memory, WebGPU buffer residency, driver allocations, or reliable total GPU/unified memory. No peak-memory or 16 GB compatibility result is claimed. See Compatibility and benchmarks.
This is currently a contributor workflow, not an npm consumer workflow.
Requirements:
- Node.js 22 or newer;
- Corepack and pnpm 11.13.0;
- current desktop Chrome or Edge with WebGPU enabled;
- local or separately hosted model artifacts and deterministic fixtures for the lab being run.
corepack enable
pnpm install --frozen-lockfile
pnpm typecheck
pnpm test
pnpm devpnpm typecheck first builds the @ai3d/gaussian-scene and @ai3d/triposplat-webgpu workspace packages, so the root tests resolve their generated package output without a separate manual build step.
The development server exposes a public runner and engineering validation surfaces:
/— redirects to the public TripoSplat image-to-spatial-scene runner;/e2e-web.html— established browser-local runner and default production route;/e2e-web-debug.html— same-origin diagnostic route with visible file/URL image input, remote model-CDN configuration, compatibility checks, verified browser cache, cause-oriented diagnostics, generation, preview, and exports;/sharp-lab.html— preserved legacy SHARP application path;/encoder-lab.html— preprocessing and encoder vertical-slice comparison;/dit-lab.html— one DiT invocation against a deterministic PyTorch fixture;/dit-profile-lab.html— opt-in warmed invocation-7/invocation-8 profiling and graph-optimization/capture A/B;/flow-lab.html— four-step CFG/Euler execution and final-state comparison;/octree-lab.html— one fp32 occupancy-logit invocation or full eight-level trajectory replay;/gaussian-lab.html— one fp32 raw Gaussian-feature invocation./e2e-lab.html— complete prepared-image package execution, export qualification, and exported-PLY viewer/canvas sanity gate.
The labs remain engineering fixtures rather than the package API. The /e2e-web.html and /e2e-web-debug.html routes use the same built runner and model-cache namespace; the debug route makes its cause-oriented failure reports explicit without requiring a second model download. The runner is still an engineering preview and does not make a production-readiness or numerical-parity claim.
The root application retains the original SHARP browser workflow: local image selection, ONNX Runtime WebGPU inference, SHARP-specific camera/NDC postprocessing, preview, and PLY download. Its normal external-data export needs both sharp_web_predictor.onnx and sharp_web_predictor.onnx.data with the sidecar path expected by the graph.
SHARP is intentionally separate from TripoSplat. Its image stretch, focal-length input, camera-space covariance conversion, opacity filtering, coordinate flips, and PLY metadata are not reused as TripoSplat numerical logic. Apple SHARP weights are restricted to research purposes under LICENSE_MODEL; do not bundle or host them without accepting those terms.
The SHARP path is the known-good browser chassis baseline, not evidence that every TripoSplat release gate has passed.
ONNX graphs may use external data. The graph and every sidecar must be served with the exact location path embedded in the ONNX graph. The current immutable manifest references ten objects totaling 6,465,182,402 bytes. The canonical fp32 DiT pair alone totals 1,643,895,982 bytes; exact object sizes and SHA-256 digests are declared in public/models/triposplat/manifest.json.
Do not route these objects through Vercel functions or another application proxy. Use S3/CloudFront, GCS/Cloud CDN, or an equivalent static object CDN with CORS, byte-range requests, immutable versioned objects, and correct external-data paths. User images must remain in the browser. See Model hosting.
The alpha framework-neutral @ai3d/triposplat-webgpu workspace package exposes the high-level class, typed errors and contracts, built-in five-stage executor, manifest/capability handling, verified OPFS/Cache API/ephemeral artifact delivery, low-level graph runtime, sampler/octree/decode helpers, and Gaussian export surface. It refuses generation when the manifest lacks any required graph. The built-in worker retains the positive and negative DiT conditioning tensors once per session, so each sampling invocation transfers only latent, camera, and timestep payloads. High-level generations are serialized, queued aborts reject promptly, and an in-flight abort tears down the one-shot worker so the same model instance can retry with a clean runtime. Prepared inputs work directly; opaque photographs require a caller-supplied browser-local removeBackground implementation until a qualified BiRefNet artifact is bundled.
Packed tarballs pass clean Vite, strict Next-client TypeScript, and native ESM/import-map consumer checks. The exact final @ai3d/triposplat-webgpu tarball (a2ba5ce6…) was installed without workspace links into a fresh Vite app; its emitted worker first passed a tiny identity graph and then completed the full five-graph path with 262,144 finite Gaussians and both exports. This closes the packed-Vite full-execution gate, but not official whole-scene parity, full browser runs through Next/native ESM, repeated-generation memory, Edge, or 16 GB qualification.
The initial audit separates:
- model-neutral ONNX Runtime worker/session infrastructure;
- SHARP-specific image, camera, and Gaussian postprocessing;
- TripoSplat preprocessing and stochastic VAE conditioning;
- TypeScript-controlled classifier-free-guided flow sampling;
- dynamic octree algorithms;
- canonical Gaussian ownership and export.
Read the browser architecture audit for graph contracts, deployment behavior, attention memory constraints, and inherited-code boundaries.
- Direct and agent integration — copy-ready image input to PLY/
.splatonboarding - Current status and release gates
- Developer guide
- Alpha API reference
- Model hosting, CORS, and caching
- Framework integration examples
- Compatibility and measured benchmarks
- RTX 3090-class 20-step performance and optimization direction
- Gaussian conventions and export
- Privacy and security
- Troubleshooting
- Contribution and release process
- License and provenance notices
- Changelog
The inference design is browser-local: source images are decoded and processed in browser memory and are not sent to an inference server. A deployment still controls its own analytics, error reporting, authentication, and CDN logs. Review those systems separately, avoid logging signed model URLs, and enforce a Content Security Policy appropriate for workers, WebAssembly, WebGPU, and the selected model CDN.
Do not publish this repository or model bundle under a new blanket license yet. TripoSplat source and its released repository material are MIT-licensed at the pinned reference revision. Apple SHARP code uses Apple's repository license, while Apple SHARP model weights are restricted to research purposes. The inherited ml-sharp-web snapshot has no detected root license, so redistribution rights for copied chassis code must be clarified or that code must be reimplemented before a public package release.
See THIRD_PARTY_NOTICES.md. This project does not bundle SHARP or TripoSplat weights into an npm package.
TripoSplat WebGPU was directed, developed, tested, and validated by
Yosun Chang, with substantial implementation
assistance from OpenAI Codex using GPT-5.6 Sol Ultra and GPT-5.6 Sol Max in Kiro.
The repository began from the ml-sharp-web application chassis. This
repository's Git history starts from a clean project snapshot so its contributor
panel reflects work on the TripoSplat WebGPU port. The prior integration history
remains available in the archived repository,
and upstream source references are recorded in THIRD_PARTY_NOTICES.md.
The official TripoSplat implementation remains the numerical source of truth.