feat(#1018): photo depth estimation via Depth-Anything-V2-Small (exported, parity-verified, hosted)

THIRD_PARTY_AI_MODELS.md

@@ -20,7 +20,8 @@ the binary). Attribution + licenses for the models and their training data:
 > [`QtMeshEditor-facemesh-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-facemesh-onnx),
 > [`QtMeshEditor-faceblendshapes-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-faceblendshapes-onnx),
 > [`QtMeshEditor-blazepose-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-blazepose-onnx),
-> [`QtMeshEditor-poselandmarks-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-poselandmarks-onnx)
+> [`QtMeshEditor-poselandmarks-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-poselandmarks-onnx),
+> [`QtMeshEditor-depthanything-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-depthanything-onnx)
 > (plus the in-house
 > [`QtMeshEditor-rmib-inbetween`](https://huggingface.co/fernandotonon/QtMeshEditor-rmib-inbetween),
 > [`QtMeshEditor-mesh-segmentation`](https://huggingface.co/fernandotonon/QtMeshEditor-mesh-segmentation),
@@ -184,6 +185,44 @@ the binary). Attribution + licenses for the models and their training data:
   `QTMESH_REMBG_MODEL_BASE_URL` / `QSettings ai/rembgModelBaseUrl`; offline guard
   `QTMESH_REMBG_NO_DOWNLOAD`). Falls back to the raw image when unavailable.
 
+## Depth-Anything-V2-Small — photo depth estimation (issue #1018, epic #818 C4)
+
+- **Model:** `depth-anything/Depth-Anything-V2-Small-hf` exported to ONNX
+  (`depth/da2_small.onnx`, ~99 MB), monocular relative-depth estimation from a
+  single photograph.
+- **Licence: Apache-2.0 — the Small variant ONLY.**
+  **Depth-Anything-V2 Base and Large are `cc-by-nc-4.0` and must NEVER be
+  shipped** (non-commercial, fails the project's permissive-redistribution bar
+  the same way SF3D and Hunyuan3D did). This is not left to reviewer vigilance:
+  `scripts/export-depth-anything-onnx.py` queries the HF model API at export
+  time and **hard-fails** on any licence outside its allow-list, before
+  downloading a single weight.
+- **Mirror repo:**
+  [`QtMeshEditor-depthanything-onnx`](https://huggingface.co/fernandotonon/QtMeshEditor-depthanything-onnx)
+  (standalone model card + I/O contract; the runtime downloads from the
+  aggregate repo under `depth/`). Refresh with
+  `scripts/sync-hf-model-repos.sh depthanything`.
+- **Used by** `src/PhotoDepth.{h,cpp}`: CLI `qtmesh material --photo-depth
+  <img> [-o out.png] [--depth-letterbox]`. Produces an 8-bit grayscale map with
+  **near = bright**, the same convention `MeshDepthRenderer` emits, so a photo
+  and a rendered mesh can feed `SDWorker::generateTextureControlled`
+  interchangeably. Downloads on first use to `AppData/ai_models/depth/`
+  (override `QTMESH_DEPTH_MODEL_BASE_URL` / `QSettings ai/depthModelBaseUrl`;
+  offline guard `QTMESH_DEPTH_NO_DOWNLOAD`).
+- **Export contract:** input `pixel_values` float32 `[1,3,518,518]`
+  ImageNet-normalised; output `predicted_depth` float32 `[1,518,518]`, larger =
+  nearer. The spatial dims are **pinned, not dynamic**: a dynamic-axis export
+  traces cleanly but drifts from the traced resolution (measured 2.3e-02
+  relative error at 462² and 4.9e-02 at 392², versus 2.7e-06 at 518²) because
+  the ViT position-embedding interpolation is only partly captured. The
+  reference preprocessor resizes every input to 518 anyway, so pinning costs
+  nothing and removes a silent-wrong-answer failure mode.
+- **Parity-verified before upload:** the export script asserts torch-vs-ORT
+  agreement on both noise and a smooth structured image (measured 4.1e-06 and
+  6.9e-06 max abs difference) and refuses to write a graph with any non-finite
+  output. This exists because the UniRig export (#1025) shipped a graph
+  emitting 100% NaN latents that went unnoticed for months.
+
 ## PBRify_Remix — PBR map synthesis (issue #404)
 
 - Three SPAN models from https://github.com/Kim2091/PBRify_Remix — **CC0-1.0**,

qml/TexturePropertiesPanel.qml

@@ -278,6 +278,7 @@ GroupBox {
         // apply it via other tools even when in-app application of
         // the texture to certain materials isn't working yet).
         RowLayout {
+            id: depthRow
             Layout.fillWidth: true
             // #404: synthesize normal/roughness/height from the current diffuse.
             // Only shown on an ONNX build; disabled until a real texture is set.
@@ -343,8 +344,44 @@ GroupBox {
                 onClicked: { pbrStatus.text = "Cancelling…"; MaterialEditorQML.cancelUpscale() }
             }
 
+            // #1018: monocular depth from a PHOTO (Depth-Anything-V2-Small).
+            // Unlike the #403 mesh-depth path this needs no selected mesh — it
+            // estimates depth from any 2-D image, so a reference photo can
+            // condition the ControlNet texture generation below. Writes
+            // <stem>_depth.png next to the chosen photo.
+            property bool depthRunning: false
+            ThemedButton {
+                text: "Depth from photo…"
+                visible: MaterialEditorQML.aiPhotoDepthAvailable() && !parent.depthRunning
+                onClicked: {
+                    const src = MaterialEditorQML.openFileDialog()
+                    if (src && src.length > 0) {
+                        parent.depthRunning = true
+                        pbrStatus.text = "Estimating depth…"
+                        MaterialEditorQML.generatePhotoDepth(src)
+                    }
+                }
+            }
+            ThemedLabel {
+                text: "Estimating depth…"
+                visible: parent.depthRunning
+                elide: Text.ElideRight
+            }
+
             Connections {
                 target: MaterialEditorQML
+                // #1018 photo depth
+                function onPhotoDepthStarted() { pbrStatus.text = "Estimating depth…" }
+                function onPhotoDepthCompleted(depthPath) {
+                    depthRow.depthRunning = false
+                    // Show the file name, not the whole path — the panel is narrow.
+                    const name = depthPath.split("/").pop()
+                    pbrStatus.text = "Depth map written: " + name
+                }
+                function onPhotoDepthError(message) {
+                    depthRow.depthRunning = false
+                    pbrStatus.text = "Depth failed: " + message
+                }
                 function onPbrSynthCompleted(result) {
                     pbrStatus.text = result.fromCache ? "PBR maps ready (cached)."
                                                       : "PBR maps generated."

scripts/export-depth-anything-onnx.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""Export Depth-Anything-V2-Small to ONNX for QtMeshEditor (#1018, epic #818 C4).
+
+ONE-TIME, OFFLINE DEV TOOL — not shipped with the app, not run at runtime.
+Mirrors scripts/export-pbrify-onnx.py / export-skintokens-onnx.py.
+
+LICENCE GATE (the reason this script refuses anything but Small):
+    Depth-Anything-V2-Small  apache-2.0    -> redistributable, SHIPPED
+    Depth-Anything-V2-Base   cc-by-nc-4.0  -> non-commercial, REJECTED
+    Depth-Anything-V2-Large  cc-by-nc-4.0  -> non-commercial, REJECTED
+Verified against the HF model API at export time. The project's bar is
+permissive redistribution (Homebrew/Snap/WinGet/Docker), so the NC variants can
+never ship no matter how much better they look. The script hard-fails on them
+rather than leaving that to reviewer vigilance.
+
+Contract the C++ side (src/PhotoDepth.{h,cpp}) relies on:
+    input   pixel_values  float32 [1,3,H,W]   ImageNet-normalised RGB
+    output  predicted_depth float32 [1,H,W]   RELATIVE inverse depth,
+                                              larger = NEARER
+H and W are PINNED to 518x518 — only the batch axis is dynamic, and the C++
+runtime hard-codes the same size (PhotoDepth::kInputSize). A dynamic-axis export
+traces cleanly but drifts away from the traced resolution (measured 2.3e-02
+relative error at 462 and 4.9e-02 at 392, versus 2.7e-06 at 518) because the ViT
+position-embedding interpolation is only partly captured. The C++ side resizes
+the source to this fixed shape and normalises the output to 0..255 itself,
+matching MeshDepthRenderer's near=bright convention.
+"""
+import argparse
+import json
+import sys
+import urllib.request
+
+REPO_SMALL = "depth-anything/Depth-Anything-V2-Small-hf"
+# Must equal PhotoDepth::kInputSize in src/PhotoDepth.h.
+RUNTIME_SIZE = 518
+ALLOWED_LICENSES = {"apache-2.0"}
+PATCH = 14
+
+
+def check_license(repo: str) -> str:
+    """Refuse to export a model whose licence we cannot redistribute."""
+    url = f"https://huggingface.co/api/models/{repo}"
+    with urllib.request.urlopen(url, timeout=60) as r:
+        meta = json.load(r)
+    lic = (meta.get("cardData") or {}).get("license") or "unknown"
+    if lic not in ALLOWED_LICENSES:
+        sys.exit(
+            f"REFUSING to export {repo}: license '{lic}' is not in the "
+            f"permissive allow-list {sorted(ALLOWED_LICENSES)}.\n"
+            "Depth-Anything-V2 Base/Large are cc-by-nc-4.0 and must never be "
+            "shipped (see THIRD_PARTY_AI_MODELS.md). Only the Small variant "
+            "clears the bar."
+        )
+    return lic
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--repo", default=REPO_SMALL,
+                    help="HF model id (must be permissively licensed)")
+    ap.add_argument("-o", "--out", default="da2_small.onnx")
+    ap.add_argument("--opset", type=int, default=18)
+    ap.add_argument("--size", type=int, default=RUNTIME_SIZE,
+                    help=f"pinned input resolution; the C++ runtime "
+                         f"(PhotoDepth::kInputSize) hard-codes {RUNTIME_SIZE}, "
+                         f"so --allow-size-mismatch is required to change it")
+    ap.add_argument("--allow-size-mismatch", action="store_true",
+                    help="export at a --size the shipped runtime cannot consume "
+                         "(for experiments only; do NOT host the result)")
+    ap.add_argument("--tolerance", type=float, default=2e-3,
+                    help="max abs torch-vs-ORT difference accepted")
+    args = ap.parse_args()
+
+    if args.size % PATCH:
+        sys.exit(f"--size must be a multiple of {PATCH} (ViT patch size)")
+    # The graph is PINNED to --size, but PhotoDepth::estimate always builds a
+    # [1,3,518,518] tensor. A 392-pinned artifact hosted under the expected
+    # filename would fail EVERY inference with a dimension mismatch, so refuse
+    # by default rather than produce a parity-checked but unusable model.
+    if args.size != RUNTIME_SIZE and not args.allow_size_mismatch:
+        sys.exit(
+            f"--size {args.size} does not match the runtime's pinned "
+            f"{RUNTIME_SIZE} (PhotoDepth::kInputSize). The exported graph would "
+            f"reject every inference with a dimension mismatch. Pass "
+            f"--allow-size-mismatch only for experiments, and do not host the "
+            f"result; changing this for real means updating kInputSize too."
+        )
+
+    lic = check_license(args.repo)
+    print(f"[licence] {args.repo}: {lic} — OK")
+
+    import numpy as np
+    import torch
+    from transformers import AutoModelForDepthEstimation
+
+    print(f"[load] {args.repo}")
+    model = AutoModelForDepthEstimation.from_pretrained(args.repo)
+    model.eval()
+
+    class Wrap(torch.nn.Module):
+        """Return the bare depth tensor, resized to follow the INPUT size.
+
+        The HF head ends with
+
+            nn.functional.interpolate(
+                ..., (int(patch_height * self.patch_size), int(...)), ...)
+
+        and those `int()` casts turn the target size into a TorchScript
+        constant. The exported graph then declares dynamic height/width but
+        always emits the tracing resolution — measured: a 462x462 input still
+        produced a 518x518 depth map. Callers would silently get a depth map of
+        the wrong resolution for every size but the traced one.
+
+        Re-interpolating here to `pixel_values.shape[-2:]` keeps the size as a
+        traced tensor shape rather than a baked constant, so the output really
+        does follow the input. At the tracing size this is a no-op resize.
+        """
+        def __init__(self, m):
+            super().__init__()
+            self.m = m
+
+        def forward(self, pixel_values):
+            depth = self.m(pixel_values=pixel_values).predicted_depth
+            if depth.dim() == 3:
+                depth = depth.unsqueeze(1)          # [B,1,H,W] for interpolate
+            depth = torch.nn.functional.interpolate(
+                depth, size=pixel_values.shape[-2:],
+                mode="bilinear", align_corners=False)
+            return depth.squeeze(1)                 # back to [B,H,W]
+
+    wrapped = Wrap(model).eval()
+    example = torch.randn(1, 3, args.size, args.size, dtype=torch.float32)
+
+    print(f"[export] opset {args.opset} -> {args.out}")
+    torch.onnx.export(
+        wrapped, (example,), args.out,
+        input_names=["pixel_values"], output_names=["predicted_depth"],
+        # Batch stays dynamic; H/W are PINNED to the tracing size on purpose.
+        # A dynamic-axis export traces clean but drifts badly away from the
+        # traced resolution: the ViT position-embedding interpolation is only
+        # partly captured, measured at 2.3e-02 relative error at 462x462 and
+        # 4.9e-02 at 392x392 (vs 2.7e-06 at 518x518) on smooth images. Since
+        # the reference preprocessor resizes every input to 518x518 anyway
+        # (preprocessor_config.json: size 518, keep_aspect_ratio,
+        # ensure_multiple_of 14), a fixed shape costs nothing and removes a
+        # silent-wrong-answer failure mode. The C++ side letterboxes to 518.
+        dynamic_axes={"pixel_values": {0: "batch"},
+                      "predicted_depth": {0: "batch"}},
+        opset_version=args.opset, do_constant_folding=True, dynamo=False,
+    )
+
+    # ---- PARITY: the step the UniRig export (#1025) skipped ----------------
+    # That export shipped a graph emitting 100% NaN latents and nobody noticed
+    # until the feature was investigated months later. Never upload a graph
+    # whose output has not been compared against torch on real input.
+    import onnxruntime as ort
+    print("[parity] torch vs onnxruntime")
+    with torch.no_grad():
+        ref = wrapped(example).cpu().numpy()
+    sess = ort.InferenceSession(args.out, providers=["CPUExecutionProvider"])
+    got = sess.run(None, {"pixel_values": example.numpy()})[0]
+
+    if got.shape != ref.shape:
+        sys.exit(f"shape mismatch: torch {ref.shape} vs ORT {got.shape}")
+    if not np.isfinite(got).all():
+        bad = int((~np.isfinite(got)).sum())
+        sys.exit(f"ONNX output has {bad}/{got.size} non-finite values — "
+                 "the export is numerically broken (this is exactly the "
+                 "UniRig #1025 failure; do NOT upload)")
+    err = float(np.abs(ref - got).max())
+    rng = float(ref.max() - ref.min())
+    print(f"  shape {got.shape}  max|diff| {err:.3e}  (output range {rng:.3f})")
+    if err > args.tolerance:
+        sys.exit(f"parity FAILED: {err:.3e} > tolerance {args.tolerance:.3e}")
+
+    # Contract check: the input shape must be PINNED. A graph that advertises
+    # dynamic H/W here would be accepted by the C++ side and then return
+    # quietly wrong depth at any other size (see the dynamic_axes note above).
+    decl = sess.get_inputs()[0].shape
+    if decl[2] != args.size or decl[3] != args.size:
+        sys.exit(f"expected pinned {args.size}x{args.size} input, got {decl} — "
+                 "a dynamic-axis export drifts away from the traced size")
+    print(f"  input shape pinned: {decl}")
+
+    # Parity on a SMOOTH image too. White noise is the easiest case for a
+    # resize mismatch to hide in; a structured image is what the model will
+    # actually see.
+    yy, xx = np.mgrid[0:args.size, 0:args.size].astype(np.float32) / args.size
+    smooth = np.stack([np.sin(6 * xx) * np.cos(5 * yy),
+                       np.cos(4 * xx + yy), xx * yy], 0)[None].astype(np.float32)
+    with torch.no_grad():
+        ref3 = wrapped(torch.from_numpy(smooth)).cpu().numpy()
+    got3 = sess.run(None, {"pixel_values": smooth})[0]
+    if not np.isfinite(got3).all():
+        sys.exit("smooth-image output is non-finite — do NOT upload")
+    err3 = float(np.abs(ref3 - got3).max())
+    print(f"[parity] smooth image  max|diff| {err3:.3e}")
+    if err3 > args.tolerance:
+        sys.exit(f"smooth-image parity FAILED: {err3:.3e}")
+
+    print(f"[ok] {args.out} — parity verified, safe to host")
+
+
+if __name__ == "__main__":
+    main()

scripts/sync-hf-model-repos.sh

@@ -61,6 +61,7 @@ declare -A REPOS=(
   [faceblendshapes]=QtMeshEditor-faceblendshapes-onnx
   [blazepose]=QtMeshEditor-blazepose-onnx
   [poselandmarks]=QtMeshEditor-poselandmarks-onnx
+  [depthanything]=QtMeshEditor-depthanything-onnx
 )
 declare -A FILES=(
   [pbrify]="1x-PBRify_NormalV3.onnx 1x-PBRify_RoughnessV2.onnx 1x-PBRify_Height.onnx"
@@ -81,6 +82,7 @@ declare -A FILES=(
   [faceblendshapes]="mocap/face/face_blendshapes.onnx"
   [blazepose]="mocap/pose/pose_detector.onnx"
   [poselandmarks]="mocap/pose/pose_landmarks.onnx"
+  [depthanything]="depth/da2_small.onnx"
 )
 
 sync_one() {