Add Depth-Anything-3 reference implementation

examples/rgbd_pose_refine/.gitignore

@@ -0,0 +1,6 @@
+# Local presentation assets and generated reports are not part of the example.
+/docs/
+/visualization/
+/space/
+/showcase_gifs.py
+/benchmark/showcase_report.py

examples/rgbd_pose_refine/README.md

@@ -0,0 +1,82 @@
+# DSLR RGB-D pose refinement
+
+Refine DA3 camera poses from ScanNet++ DSLR photographs using `bae` on CUDA.
+DA3 supplies depth, intrinsics, and initial poses. COLMAP camera calibration is
+used to undistort input images; COLMAP poses are used only for evaluation.
+
+The native solver has two stages:
+
+- **D (odometry):** jointly optimize camera poses over photometric constraints
+  between covisible views.
+- **E (structure):** fuse a fixed point structure, then refine camera poses
+  against it, rebuilding the structure between outer iterations.
+
+`bae_de` chains raw D into E. Each proposal is scored against the initialization
+using a held-out photometric guard. Both raw and guarded results are saved;
+photometric acceptance does not guarantee better ground-truth pose accuracy.
+
+## Run
+
+Commands run from the repository root. For BAE-only refinement of cached
+predictions, install `bae` and OpenCV:
+
+```bash
+python -m pip install --no-build-isolation -v -e .
+python -m pip install opencv-python-headless
+python -m pytest examples/rgbd_pose_refine/tests/ -q
+
+python examples/rgbd_pose_refine/benchmark/showcase_benchmark.py \
+  --scenes 7831862f02 --variants bae_d bae_e bae_de
+```
+
+The benchmark reads `outputs/rgbd_showcase/<scene>/prediction.npz`. To create
+that cache, first install the DA3 dependencies in [SHOWCASE.md](SHOWCASE.md):
+
+```bash
+python examples/rgbd_pose_refine/benchmark/showcase_prepare.py \
+  --scenes 7831862f02 --dataset-root /path/to/scannetpp \
+  --out outputs/rgbd_showcase --device 0
+```
+
+The dataset must contain:
+
+```text
+<dataset-root>/data/<scene>/dslr/colmap.zip
+<dataset-root>/data/<scene>/dslr/resized_images.zip
+```
+
+The COLMAP archive contains `colmap/cameras.txt` and `colmap/images.txt`.
+Preparation writes `prediction.npz` and `input.json`. Refinement writes
+`refined.npz` (poses and optimization histories) and `metrics.json` (AUC,
+timings, and guard verdicts) beside the predictions. Outputs stay under the
+ignored `outputs/` directory.
+
+## Options
+
+| Benchmark flag | Default | Purpose |
+|---|---|---|
+| `--scenes` | `7831862f02` | Scene IDs with cached predictions |
+| `--out` | `outputs/rgbd_showcase` | Prediction/refinement cache root |
+| `--variants` | All six methods | Use `bae_d bae_e bae_de` for BAE-only runs |
+| `--selfcal` / `--no-selfcal` | on | Intrinsics self-calibration |
+| `--neighbors` | 16 | Covisibility neighbors |
+| `--samples` | 512 | Odometry pixels sampled per pair |
+| `--structure-points` | 5000 | Maximum structure points per view |
+| `--device` | 0 | CUDA device index |
+| `--force` | off | Recompute cached refinement |
+
+`form_d`, `form_e`, and `form_de` compare against the separate DA3/Open3D
+reference implementation. They require `da3/refine_poses.py`; BAE-only runs do
+not import it. See [SHOWCASE.md](SHOWCASE.md) for the comparison protocol and
+[VALIDATION.md](VALIDATION.md) for full-split evaluation.
+
+## Code map
+
+| Files | Purpose |
+|---|---|
+| `photometric.py` | BAE residuals and refinement stages |
+| `geometry.py`, `covis.py` | Camera geometry and view pairing |
+| `guards.py`, `selfcal.py` | Held-out scoring and intrinsics self-calibration |
+| `dslr.py`, `eval.py` | DSLR loading and pose accuracy |
+| `benchmark/` | Prediction preparation, refinement, and split evaluation |
+| `tests/` | Synthetic convergence, derivatives, scoring, and loading checks |

examples/rgbd_pose_refine/SHOWCASE.md

@@ -0,0 +1,68 @@
+# DSLR reproduction and comparison protocol
+
+See [README.md](README.md) for BAE-only refinement of cached predictions.
+Generating predictions also requires Depth Anything 3. The experiments used a
+local checkout at `da3/`; reference comparisons additionally need the custom
+`refine_poses.py` in that checkout. Those reference scripts are not bundled
+with this example, and a stock DA3 install does not supply them.
+
+## Recorded environment
+
+The experiments used CUDA PyTorch 2.7.0a0, Warp 1.17.0, and Open3D 0.19.0.
+The PyPose revision below avoids the released wheel's `bae==0.2` dependency:
+
+```bash
+python -m pip install --no-deps git+https://github.com/pypose/pypose.git@afedf8a61927825bb9835e238217dfc36c80cc0e
+python -m pip install 'warp-lang>=1.13.0' 'nvidia-cudss-cu12<0.9'
+python -m pip install --no-build-isolation -v -e .
+python -m pip install opencv-python-headless
+python -m pip install -r examples/rgbd_pose_refine/requirements.txt
+python -m pip install --no-deps moviepy==1.0.3 -e da3
+```
+
+`requirements.txt` supplies DA3 inference/reference dependencies. MoviePy is
+needed by that checkout's eager export imports even though this example does
+not export videos. Cached-prediction BAE runs need only the base installation
+and OpenCV described in the README.
+
+## Inputs
+
+`benchmark/showcase_prepare.py` undistorts DSLR images using COLMAP's fisheye
+calibration before inference. It samples registered image names uniformly
+across the full scene, including both endpoints, and runs one globally
+attended DA3 prediction. Its view-count search backs off on CUDA OOM; it does
+not split scenes into windows or condition DA3 on COLMAP poses.
+
+`input.json` records the checkpoint revision, selected frame names, resolution,
+memory attempts, and successful view count. Reuse a cache only for the same
+scene/model/resolution; choose another `--out` or use `--force` to replace it.
+
+## Comparisons
+
+- Every method consumes identical cached RGB, depth, confidence, intrinsics,
+  and initial poses. Self-calibration and the random seed are shared.
+- Native D solves one joint sparse photometric LM problem. Reference D calls
+  Open3D pairwise RGB-D odometry followed by pose-graph optimization.
+- Native E uses a voxel-averaged point structure rebuilt three times.
+  Reference E uses a TSDF mesh and 100 rigid color-map iterations.
+- D+E chains raw D into E. The whole proposal is compared with the initial
+  poses using a common 512-pair photometric guard. Reference D's internal
+  texture gate and guard are disabled for this comparison.
+- Reference E is re-anchored to camera zero with
+  `w @ inv(w[0]) @ initial[0]`; relative-pose accuracy is unchanged.
+- AUC uses every unordered camera pair and sign-ambiguous translation
+  direction. JSON values are fractions. Raw regressions and rejected proposals
+  are retained; the formulations are not numerically equivalent.
+
+```bash
+# BAE only, using cached predictions:
+python examples/rgbd_pose_refine/benchmark/showcase_benchmark.py \
+  --scenes 7831862f02 --variants bae_d bae_e bae_de
+
+# All six methods, when the reference checkout is available:
+python examples/rgbd_pose_refine/benchmark/showcase_benchmark.py \
+  --scenes 7831862f02
+```
+
+The source example contains no HTML gallery, rendering application, or generated
+media. Numerical outputs and reports are generated locally under `outputs/`.

examples/rgbd_pose_refine/VALIDATION.md

@@ -0,0 +1,84 @@
+# Complete DSLR validation
+
+Install dependencies as described in [SHOWCASE.md](SHOWCASE.md). The split
+runner uses the Hugging Face cache offline. On a fresh machine, cache the pinned
+checkpoint once before starting it:
+
+```bash
+python -c "from depth_anything_3.api import DepthAnything3; DepthAnything3.from_pretrained('depth-anything/DA3-GIANT-1.1', revision='72ee9f89ce4e50d704e9d55ee9c646ec8dc25a19')"
+```
+
+Run all 50 scenes from the mounted `nvs_sem_val.txt` split:
+
+```bash
+python examples/rgbd_pose_refine/benchmark/validate_split.py
+python examples/rgbd_pose_refine/benchmark/validation_report.py
+```
+
+The runner uses both 24 GB RTX 4090 GPUs, with one isolated scene worker per GPU.
+It pins `DA3-GIANT-1.1` to commit
+`72ee9f89ce4e50d704e9d55ee9c646ec8dc25a19`. Each scene's input spans the complete
+registered DSLR sequence. Images are undistorted before inference. The global
+attention view search starts near the measured 156-frame budget, probes upward
+to a cap of 192, and bisects on OOM with eight-frame precision. No sliding
+windows are used. Cached inputs retain their original search attempts.
+The runner enables `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, as in the
+showcase commands, to avoid allocator fragmentation shrinking the usable budget.
+
+`outputs/rgbd_validation_giant11/manifest.json` lists every expected scene and
+its state. Each scene has its own `run.log`, prediction, input metadata, raw
+and guarded optimizer poses, and metrics. Rerunning the command resumes the
+same configuration; it does not count failures as successful scenes.
+
+The generated report in `outputs/rgbd_validation_report/` uses the original DA3
+evaluator to rescore every method on identical unordered camera pairs. It
+reports both macro AUC (equal scene weight) and pooled AUC (equal pair weight),
+per-scene results, native/reference gaps, and differences between the example
+and DA3 metric implementations. Raw and guarded results are kept separate.
+
+## Gradient correctness
+
+The example enables this before importing bae:
+
+```python
+import os
+os.environ.setdefault('BAE_USE_PYPOSE_AMBIENT_GRAD', '1')
+```
+
+`photometric.py` also installs the enabled patch when bae was imported earlier.
+The validation runner rejects an explicitly disabled setting. This is required
+for correct derivatives of composed PyPose Lie operations and their conversion
+to the optimizer's local SE(3) update.
+
+Checks include nonidentity composed-pose sparse Jacobians against finite
+differences, synthetic RGBD convergence, Lie-operation ambient gradients, and
+three PGO convergence problems with final costs calibrated against Ceres:
+
+```bash
+python -m pytest -q examples/rgbd_pose_refine/tests \
+  tests/autograd/test_pypose_ambient_grad.py \
+  tests/autograd/test_pgo_convergence.py
+```
+
+## What can be compared
+
+All six methods use identical cached DA3 images, depths, confidence, predicted
+poses, self-calibrated intrinsics, and frame selections. The benchmark follows
+the [showcase comparison protocol](SHOWCASE.md): raw D, raw E, and raw D then E,
+each followed by the same wide-pair photometric guard. Reference D calls the
+actual Open3D pairwise RGBD odometry and PGO code in `da3/refine_poses.py`.
+Reference E calls its TSDF/color-map implementation. Native D instead solves
+a joint photometric problem; native E uses a voxel-averaged point structure.
+Correct gradients alone do not make these formulations numerically equivalent.
+
+The optional `odometry_workers` argument in the reference evaluates independent
+pairwise odometry problems concurrently and collects them in the original edge
+order. Its default remains one worker. This validation uses eight edge workers,
+with one OpenMP thread each; global PGO and all numerical settings are unchanged.
+Serial control results are retained in `outputs/rgbd_reference_serial_control`.
+
+The saved historical `da3/da3_pose_auc_nvs_test_GIANT11.json` is from
+`nvs_test.txt`, which has **zero scenes in common** with the mounted validation
+split. It also used up to 627 frames per scene. Its absolute AUC is not a
+matching target for this dataset and GPU budget. Fresh reference runs here
+provide the comparison on identical inputs.

examples/rgbd_pose_refine/benchmark/README.md

@@ -0,0 +1,25 @@
+# Benchmarks and reproduction
+
+These scripts use the core refinement modules in the parent directory.
+Run them from the repository root:
+
+| Script | Purpose |
+|---|---|
+| `showcase_prepare.py` | Cache DA3 predictions from scene-wide DSLR photographs |
+| `showcase_benchmark.py` | Run BAE D, E, D+E and optional Open3D reference comparisons |
+| `validate_split.py` | Schedule and resume a complete split across GPUs |
+| `validation_report.py` | Audit split results against the separate DA3 evaluator |
+
+```bash
+python examples/rgbd_pose_refine/benchmark/showcase_benchmark.py \
+  --scenes 7831862f02 --variants bae_d bae_e bae_de
+```
+
+This command consumes `outputs/rgbd_showcase/7831862f02/prediction.npz`.
+Preparation needs DA3 inference dependencies. BAE-only refinement of an existing
+cache does not import the reference implementation. Reference comparisons and
+the independent validation audit require the separate `da3/` checkout.
+
+See [SHOWCASE.md](../SHOWCASE.md) for installation and preparation commands and
+[VALIDATION.md](../VALIDATION.md) for the split protocol. Predictions, refinement
+results, and generated reports stay under ignored `outputs/`.