Repository navigation
epic: dense prediction (depth, geometry, pose, segmentation, 3D) #2382
Description
Problem / Background
Upstream mlx-vlm now ships dense-prediction models that output depth, point, normal and segmentation maps, human keypoints and 3D meshes rather than text. mlxcel has no output path for any of them. #524 recorded these output kinds as needing their own infrastructure first; the maintainer has asked for depth, pose and 3D support, and this epic builds that infrastructure and the model set on it.
Goal
mlxcel predict writes maps, keypoints, instance masks and meshes to files for Video Depth Anything, MoGe-3, Sapiens2, SAM 3D Objects, SAM 3, SAM 3.1 and SAM 3D Body from their published mlx-community checkpoints, with DINOv2 shared as one backbone, parity against upstream for each model, and every family listed by mlxcel arch. RF-DETR joins the same effort through mlxcel detect (see the last section).
Dependency notes: DINOv2 is imported upstream by Video Depth Anything, MoGe-3 and SAM 3D Objects; Sapiens2 has its own backbone, SAM 3D Body its own DINOv3, RF-DETR its own windowed DINOv2. SAM 3D Objects also uses MoGe-3 to estimate a point map when none is supplied. SAM 3 adds the instance-mask output (instances.json, mask PNGs, --prompt / --boxes) that SAM 3.1 reuses; SAM 3D Objects adds the mesh writer that SAM 3D Body reuses.
Sub-issues
Phase 1
Phase 2
- feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349)
- feat: Port Sapiens2 (sapiens2) human pose and dense human prediction #2371 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349)
- feat: Port SAM 3 (sam3) promptable segmentation and video tracking #2429 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349)
Phase 3
- feat: Port Video Depth Anything (video_depth_anything) video depth #2352 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349, feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350)
- feat: Port MoGe-3 (moge3) monocular geometry: point, depth, normal maps #2355 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349, feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350)
- feat: Port SAM 3.1 (sam3_1) with Object Multiplex video tracking #2430 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349, feat: Port SAM 3 (sam3) promptable segmentation and video tracking #2429)
Phase 4
- feat: Port SAM 3D Objects (sam3d_objects) with mesh and Gaussian output #2381 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349, feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350, feat: Port MoGe-3 (moge3) monocular geometry: point, depth, normal maps #2355)
Phase 5
- feat: Port SAM 3D Body (sam3d_body) monocular human mesh #2432 (depends on feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349, feat: Port SAM 3D Objects (sam3d_objects) with mesh and Gaussian output #2381)
Tracked outside this DAG
#2431 (RF-DETR detection and instance masks) belongs to this effort but rides mlxcel detect beside RT-DETRv2 through the Detector trait from #2412 (or #2413), which is not a sub-issue here, so it is not listed under a phase. Its prerequisites #2412 and #2349 are complete; #2431 is merged through #2581. The authorized #2412 prerequisite was implemented through #2579 as part of this run.
Acceptance Criteria
- All sub-issues completed and merged, and feat(detect): Add RF-DETR (rf-detr) detection and instance masks #2431 merged.
- End-to-end: each model runs through
mlxcel predicton a real checkpoint and writes its documented file set; RF-DETR runs throughmlxcel detect, writing masks with--masks-dir. -
docs/dense-prediction.mdanddocs/supported-models.mdlist all nine families (the eight above plus the standalonedinov2feature family from feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350) with their tasks and output files. - tracking: non-generative vision models present in the VLM reference but out of mlxcel's LLM/VLM-generation scope #524 updated to point at this epic for the depth, pose, segmentation and 3D families.
Activity
[-]epic: dense prediction (depth, geometry, pose)[/-][+]epic: dense prediction (depth, geometry, pose, segmentation, 3D)[/+]docs: record vision model scope for non-generative families (#2508)
inureyes commentedon Oct 24, 2026
Epic implementation complete
Implemented and merged 11 units across five execution waves: the nine sub-issues in the epic DAG, RF-DETR #2431, and its authorized PP-DocLayoutV3 prerequisite #2412.
Each family has actual-checkpoint runtime and CLI validation recorded in its PR and bilingual technical reports. Final Body boxed/default and literal CLI outputs match all reference tensors and faces exactly; the OBJ loads in a standard mesh reader. Final RF-DETR raw outputs meet the fixed error bounds, all four segmentation masks have IoU 1.0, and existing RT-DETRv2 and PP-DocLayoutV3 CLI results remain unchanged. Historical CPU-reference recovery, failed wrapper attempts and source/binary provenance limitations remain explicitly documented rather than replaced with invented successful exits.
The final default branch passes all eight orchestration contract checks. Every implemented issue is closed, each issue and PR has
status:done, and every linked PR is merged. Bothdocs/dense-prediction.mdanddocs/supported-models.mdlist the same nine families, actual model types, tasks and concrete output files; the final executable's architecture registry confirms the aliases. The sharedinstances.jsonoutput and RF-DETR's stdout/optional-mask contract are distinguished explicitly. #524 and the scoped dense-prediction section ofdocs/vision-model-scope.mdnow point to the completed implementation.Validated artifacts, source snapshots and executable caches are retained under
.cache/epic-2382/. All owned worktrees and branches are cleaned up; unrelated worktrees and both original user stashes are preserved.