Hard-thresholded logit decisions cause scattered surface holes on non-CUDA platforms (data + analysis from a working ROCm port) #169

Activity

1. abrhamshibeshi1212-star commented on Jun 20, 2026

   

   abrhamshibeshi1212-star

   on Jun 20, 2026

   More actions

   ## Summary
   
   We have TRELLIS.2 image→3D running end-to-end on AMD ROCm (gfx1151,
   Strix Halo; related: #74). After fixing platform bugs (CuMesh
   `cudaMemcpy2D` failing above ~1M rows on HIP — fix PR'd to CuMesh:
   https://github.com/JeffreyXiang/CuMesh/pull/35), output quality is close to reference but not equal: meshes carry
   tens of thousands of micro boundary defects (missing single triangles,
   missing vertex fans) that NVIDIA outputs do not have.
   
   Reference comparison: official HF Space (same input image, seed=0, 512)
   produces a fully watertight mesh after welding UV seams — 0 boundary
   edges. Our ROCm mesh: ~220K boundary edges before fill_holes, ~95% of
   visible artifacts removed by fill_holes, but a floor remains.
   
   ## What it is NOT (all verified directly)
   
   - Not sparse-conv kernels: FlexGEMM submanifold conv unit-tested vs
     brute-force reference at 69K and 1.8M voxels — max err 1e-3 (fp16
     rounding), zero outliers, both algorithms.
   - Not the dual-grid converter: two independent implementations (CUDA/HIP
     kernels and a pure-PyTorch port) produce the same defect pattern.
   - Not NaN/Inf: decoder outputs at the extraction boundary instrumented —
     zero NaN/Inf.
   - Not fp16 range: bf16 mixed-precision decoder produces the same pattern.
   - Not attention: the FDG VAE decoder contains no attention layers.
   
   ## Proposed mechanism
   
   The decoder makes hard binary decisions on learned logits at multiple
   points: `to_subdiv logits > 0` (child voxel creation, per upsample level)
   and intersection logits `> 0` (quad emission at extraction). Logits within
   ~±1e-3 of zero — ordinary cross-platform fp16 accumulation differences —
   flip these decisions per-cell, producing scattered missing cells/quads.
   Consistent with this: globally biasing the decisions positive
   (bias += 0.1) halves the defect count but visibly changes the generation
   (smoother, loses fine detail), while applying the same shift only at
   decode time has no effect — the sensitivity is distributed across the
   sampling/upsample chain, not localized at extraction.
   
   ## Suggestion
   
   If portability across accelerators is a goal, the hard `> 0` thresholds
   are the fragile point. Options that would help downstream ports:
   - a documented/configurable decision margin,
   - training-time margin regularization on these logits (push them away
     from 0), or
   - an official statement of the expected defect rate + the intended
     fill_holes responsibility, so ports know what "correct" looks like.
   
   full package and input and output intermediate glb files available at:
   https://github.com/ron-ytsma/trellis2-rocm-gfx1151/releases/tag/1.0.0
   
2. Igor Shaposhnikov (Jourloy)

   added a commit that references this issue on Jul 24, 2026

   feat: opt-in fp32 decode thresholds (WP9)

   ...

   3aa7c43