feat(detect): Add RF-DETR (rf-detr) detection and instance masks #2431

Related to #2382 (listed there; scheduled outside its DAG because it rides mlxcel detect)

Problem / Background

RF-DETR (Roboflow, ICLR 2026) is a real-time closed-vocabulary detector on a windowed DINOv2 backbone with a C2f multi-scale projector, two-stage query selection and a deformable decoder, with an optional instance-segmentation head. Its output is boxes, COCO labels and scores (plus masks for the seg variants), which is exactly the mlxcel detect contract RT-DETRv2 already uses; #524 ranked it as the cheapest family there to re-scope for that reason. mlxcel cannot load it today.

Surface decision: RF-DETR registers in mlxcel detect beside rt_detr_v2 through the Detector trait and load_detector dispatch that #2412 introduces (#2413 adds the same contract if it lands first, and changes threshold to Option<f32>), not in the #2349 predict surface. Masks from the segmentation variants leave detect through a new --masks-dir <DIR> flag (one PNG per detection, path recorded in the JSON output). Rejected alternative: routing the segmentation variants through mlxcel predict, which would give one model two commands and make box-only users depend on the dense-prediction surface.

Upstream reference

• Model directory: https://github.com/Blaizzy/mlx-vlm/tree/main/mlx_vlm/models/rfdetr (README). PRs: Add RF-DETR detection and segmentation model Neywa-Labs/mlx-vlm#884 (model), Use fused SDPA in the RF-DETR backbone attention Neywa-Labs/mlx-vlm#2443 (fused SDPA in backbone attention), Fix off-by-one RF-DETR backbone feature layers Neywa-Labs/mlx-vlm#2450 (off-by-one backbone feature layers), Make RF-DETR sanitize idempotent Neywa-Labs/mlx-vlm#2452 (idempotent sanitize), Fix RF-DETR small variant config Neywa-Labs/mlx-vlm#2453 (small variant config), Use the separable interpolation kernel for the RF-DETR segmentation u… Neywa-Labs/mlx-vlm#2468 (separable interpolation for the segmentation upsample).
• Backbone and projector: vision.py (DINOv2Embeddings.interpolate_pos_encoding L30, DINOv2Attention, DINOv2Layer with layer scale, _window_partition / _window_unpartition L204-L250, DINOv2Backbone L251-L318: layers not in out_feature_indexes attend within num_windows^2 windows (each with its own CLS), the others merge windows for global attention, and normed patch tokens from out_feature_indexes are unpartitioned into feature maps; C2f, Bottleneck, ConvBN, MultiScaleProjector L320-L428).
• Decoder: transformer.py (gen_sineembed_for_position, _gen_encoder_output_proposals, MSDeformableAttention L91-L213, DecoderLayer, Decoder, Transformer.two_stage_select L417-L489; group_detr 13 is a training-time query grouping, inference uses num_queries).
• Heads: rfdetr.py#L56-L106 (bbox_reparam: cxcy = d_xy * ref_wh + ref_xy, wh = exp(d_wh) * ref_wh), segmentation.py (SegmentationHead: bilinear resize of projector features to image / downsample_ratio with align_corners=True, depthwise conv blocks, einsum bhwc,bnc->bnhw plus a learned bias), sanitize at rfdetr.py#L108-L177 (key remaps, in_proj splits, conv transpose only when the shape check says the weight is not already MLX layout).
• Pre- and post-processing: processing_rfdetr.py#L154-L178 (RGB, PIL bilinear resize to resolution x resolution, scale to [0, 1], ImageNet mean and std from preprocessor_config.json), COCO_CLASSES (91 entries starting with N/A); generate.py#L40-L220 (sigmoid, per-query max over classes, top num_select 300, keep score > threshold, cxcywh to xyxy scaled and clipped to the source size, per-class greedy NMS at IoU 0.5 sorted by score, masks for surviving queries resized with cubic interpolation and thresholded > 0); predictor defaults score_threshold 0.5, nms_threshold 0.5 (L222-L240).
• Checkpoints (MLX layout with config.json model_type: "rf-detr", preprocessor_config.json): mlx-community/rfdetr-base-fp32 (detection, resolution 560, patch 14, out_feature_indexes [2, 5, 8, 11], 3 decoder layers, 300 queries), mlx-community/rfdetr-seg-small-fp32 (segmentation, resolution 384, patch 12, out_feature_indexes [3, 6, 9, 12], 4 decoder layers, 100 queries), plus rfdetr-seg-large, -xlarge, -2xlarge variants. Note that the two variants differ in patch size and feature indexes; both must load.

Current Behavior

• src/commands/detect.rs:30 run_detect hard-codes RtDetrV2Predictor::from_pretrained at :55; sort_reading_order is at :141. DetectArgs (src/main.rs:890-906) has --model, --image, --threshold (default 0.3) and --format.
• src/vision/detection/mod.rs:28-31 re-exports only RT-DETRv2 types; Detection (src/vision/detection/rt_detr_v2/predictor.rs:37-47) is bbox, score, label, class_name.
• Reusable RT-DETRv2 pieces: bilinear grid_sample (src/vision/detection/rt_detr_v2/layers.rs), inverse_sigmoid and Mlp (src/vision/detection/rt_detr_v2/transformer.rs:35 and :50), the deformable sampling loop (transformer.rs:88-230), and image decoding plus PIL-equivalent bilinear resize through image::imageops::FilterType::Triangle (src/vision/detection/rt_detr_v2/processor.rs:154-180).
• feat(detect): Add PP-DocLayoutV3 layout detection with reading order #2412 and feat(detect): Add YOLO11 detector for OmniParser v2 icon detection #2413 (both open) each specify the Detector trait, load_detector dispatch on config.json model_type, Detection moved to mod.rs, and per-detector default thresholds; whichever lands second registers into the first one's dispatch.
• The feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 src/vision/dense/interpolate.rs port (open) provides resize_bilinear_nhwc (with align_corners) and resize_bicubic_nhwc, matching upstream mlx_vlm/models/interpolate.py, which RF-DETR's segmentation head imports.
• feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350 ports a standard DINOv2 and explicitly leaves the RF-DETR windowed variant out of scope.

Proposed Solution

New module src/vision/detection/rfdetr/ (config.rs, backbone.rs, projector.rs, transformer.rs, segmentation.rs, sanitize.rs, predictor.rs, tests.rs).

• Dispatch: add an rf-detr arm (also accept rfdetr, upstream's remap target) to load_detector; implement Detector for RfDetrPredictor. Default threshold 0.5 (upstream predictor) when --threshold is unset.
• backbone.rs: the windowed DINOv2 as upstream (own module, not feat: Port DINOv2 backbone (dinov2, dinov2_with_registers) for dense models #2350's Dinov2: per-window CLS tokens and window merge/split around global layers have no counterpart there). Position embedding interpolation uses feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 resize_bicubic_nhwc the way upstream interpolate_pos_encoding does.
• transformer.rs: two-stage selection and deformable decoder, reusing RT-DETRv2's grid_sample, inverse_sigmoid, Mlp and deformable sampling (move any shared function to a common location only if needed, and add // Used by: RT-DETRv2, RF-DETR).
• segmentation.rs: the head, using feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 resize_bilinear_nhwc(.., align_corners: true); if feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 shipped that function without the align_corners parameter, add it there with upstream's weight computation rather than writing a local resize.
• predictor.rs: preprocessing through the RT-DETRv2 processor's decode and Triangle resize with RF-DETR's resolution, mean and std; post-processing as upstream; class names from COCO_CLASSES (ported as a constant) unless config.json carries id2label. NMS IoU defaults to 0.5 and honors --iou when feat(detect): Add YOLO11 detector for OmniParser v2 icon detection #2413 has added it.
• Masks: add mask: Option<image::GrayImage> to Detection (None for every other detector; update their constructors). Masks upsample from image / 4 logits to the source size with feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 resize_bicubic_nhwc (antialias off, matching the upstream cubic upscale) and threshold > 0. New DetectArgs flag --masks-dir <DIR>: creates the directory, writes mask-000.png, mask-001.png, ... (8-bit single channel, 0 or 255, source resolution, numbered in the printed order after sort_reading_order), and adds "mask": "<path>" to each JSON detection. Without the flag no mask is written; with the flag on a model that has no segmentation head, detect errors before inference. The PNG format matches the SAM 3 masks in feat: Port SAM 3 (sam3) promptable segmentation and video tracking #2429.
• sanitize.rs: port the upstream remaps; transpose 4-D convs only when not already (O, H, W, I), idempotent (Make RF-DETR sanitize idempotent Neywa-Labs/mlx-vlm#2452), using the feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 shared guard.
• Registry: add rf-detr to the standalone family table (src/models/registry.rs:185-204) with output kind boxes.

Scope

In scope: the module above, dispatch registration, Detection.mask, --masks-dir, docs (docs/supported-models.md detection section, the detect CLI docs).

Out of scope: conversion from Roboflow .pth checkpoints (needs PyTorch; the MLX checkpoints load directly), video and realtime modes, exclude_classes, annotators, exposing RF-DETR through mlxcel predict or the server.

Implementation Notes

• Reuse: RT-DETRv2 grid_sample, inverse_sigmoid, Mlp, deformable sampling, image decode and resize; feat: Add dense-prediction output surface: mlxcel predict, map and keypoint files #2349 interpolate.rs and conv guard; the feat(detect): Add PP-DocLayoutV3 layout detection with reading order #2412 / feat(detect): Add YOLO11 detector for OmniParser v2 icon detection #2413 Detector trait and dispatch.
• Constraints: RT-DETRv2 (and PP-DocLayoutV3 / YOLO11 if landed) output must not change; their existing tests stay green and their JSON gains no mask field.
• Precision: checkpoints are f32; keep f32 (no bf16 to convert).
• Edge cases: zero detections above threshold (empty list, no files written even with --masks-dir); classes indexing past COCO_CLASSES print class_<id> as upstream; a non-square source image (resize is square, boxes rescale per axis).
• Error handling: unknown variant shapes fail at load naming the tensor; an unwritable --masks-dir fails before loading weights.

Acceptance Criteria

• mlxcel detect -m models/rfdetr-base-fp32 -i street.jpg --format json prints COCO detections in pixel l, t, r, b.
• Raw-output parity against mlx-vlm on mlx-community/rfdetr-base-fp32 for one committed image at 560x560: pred_logits and pred_boxes max abs error within 1e-3 in f32; final detections after NMS match (same labels, boxes within 1 px). It fails on current code.
• Segmentation: mlxcel detect -m models/rfdetr-seg-small-fp32 -i street.jpg --format json --masks-dir out/ writes one PNG per detection and the JSON mask paths; per-mask IoU against upstream at least 0.98.
• Unit tests, each failing on current code: _window_partition / _window_unpartition round-trip; per-class NMS on synthetic boxes against upstream _nms_per_class; bbox_reparam decode on a hand-computed case; sanitize idempotence; --masks-dir on a detection-only model errors.
• RT-DETRv2 detect output is unchanged (existing detect and RT-DETRv2 tests pass).
• mlxcel arch --json lists rf-detr; docs updated.

Verification

cargo build --release --features metal,accelerate
cargo test --workspace --profile test-fast --features metal,accelerate rfdetr
cargo test --workspace --profile test-fast --features metal,accelerate rt_detr
cargo test --workspace --profile test-fast --features metal,accelerate detect
cargo test --workspace --profile test-fast --features metal,accelerate
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --check
./target/release/mlxcel detect -m models/rfdetr-seg-small-fp32 -i /path/to/street.jpg --format json --masks-dir /tmp/rfdetr-masks

Dependencies

Blocked by #2412 (Detector trait and load_detector dispatch; #2413 provides the same contract if it merges first, in which case only that one is required) and #2349 (interpolate.rs resize functions and the conv-permutation guard).