Activity

1. hsliuustc0106

   added

   new modelRequests to support a new modelRequests to support a new model

   help wantedExtra attention is neededExtra attention is needed

   on Oct 11, 2026
2. hsliuustc0106

   changed the title [-][New Model]: Support the D3 / Decision 3.0 series (text, image, and video decisions)[/-] [+][New Model]: Support the D3 / Decision 3.0 series — community help wanted[/+] on Oct 11, 2026
3. MrDongsls commented on Oct 11, 2026

   

   MrDongsls

   on Oct 11, 2026

   Collaborator

   More actions

   Hi, I would like to help on this. The integration plan is as below:

   Plan

   Reference worker first, native follow-on. For the first slice as scoped, BF16, text/JSON state, all three question types, served by a reference worker behind /v1/systemone. Then native Rust/CUDA on the family qwen3_5 stack. I read the pinned runtime and config files, ran the runtime's own code paths on CPU in the pinned environment (transformers==5.17.0, torch==2.14.1+cpu, no flash-linear-attention), and measured the points below — tokenizer, readout.safetensors, and the real noncausal hook on a small random Qwen3.5.

   Design

   src/models/d3/ and recipe/d3/, split as the architecture contracts describe: the processor owns question normalization (choice criteria in insertion order; noul fixed false, true; score's ordered 2–10 levels), prompt rendering, tokenization, media preparation, limit checks, and response finishing; the executor owns weights, language/vision execution, and the readout head.

   The reference worker runs the checkpoint's own pinned runtime files, hash-verified against MODEL_MANIFEST.json — prompt compilation, readout, and error semantics are preserved by construction, and our code owns only the HTTP/JSON adaptation and admission. The serving process needs Python at this stage by design (#136's route); the native stage removes it.

   Natively: the noncausal mode becomes a compile-time template <bool Gated, bool Causal> variant of the family full-attention kernel plus a per-model causality flag flowing config → Mixer → the two live call sites; causal instantiations compile from identical source and stay bit-identical by construction, and the GDN kernel is untouched. Question rows are packed through forward_batch's unpadded row packing — each row keeps its own contiguous token range and boundary mask, so the per-row last-position download reads each row's last real token as-is and no padding is introduced at all — on #98's fixed-algorithm GEMMs, so a row's result does not depend on which rows run with it. Vision goes through #117's shared execution with the video token 248057 mapped onto the image-token boundary. Any kernel signature change is an ABI bump, coordinated with the chains that co-edit family model.rs — #98→#99→#129→#134 and the #117 family — before anything merges.

   PR Sequence and Acceptance Criteria

   1. Reference worker + oracle freeze (no dependencies). The d3 worker serving text/JSON, all three question types, exact refuse/limit semantics, frontend integration behind /v1/systemone. Fixture capture as the oracle: rendered prompts, token ids, readout logits and probabilities for choice / noul / score / mixed multi-question, option-count boundaries (choice 1/255, score 2/10), Unicode, injection guards, malformed and over-limit inputs. Accept: every fixture matches the pinned runtime; one pinned CUDA environment with provenance() recorded; two consecutive GPU runs identical; the flash-linear-attention vs reference-GDN delta and the torchvision vs PIL-resize preprocessing delta measured on the fixture set; numerical gates frozen from those distributions before any native comparison; HTTP checks direct and frontend-proxied, including the first request after readiness.
   2. Multimodal reference (needs PR 1). Multi-image, video, and mixed media with processor settings preserved (patch 16 / temporal 2; video_preprocessor_config.json frozen) and limits documented (16,384 video tokens per request, media budgets). Accept: image, multi-image, video, and mixed fixtures comparable at the processor level; every documented limit has a refusing test; no silent preprocessing substitutions — the PIL-resize fallback is detected and failed loudly.
   3. Export + native eager worker (needs qwen: share vision execution for 4B and 27B models #117 merged, an ABI window). The exporter re-keys prefixes, normalizes the vision string, side-loads the readout and code table with provenance hashes and the pplx-decider attribution; the noncausal kernel variant and a D3-local FP32 readout head over forward_batch rows follow, with warmup readiness. Accept: native answers within the frozen gates on every PR 1/2 fixture, low-margin disagreements reported rather than averaged away; packed-row parity reproduced natively (batched question rows vs individual forwards within gate — the native analog of the 2.6e-06 CPU measurement); sibling consumers (Qwen/Cua-S1/Open-Jev/OmniJev) regression-checked when the ABI changes.
   4. Later, each measured on its own: batched multi-question prefix reuse (its own parity proof), CUDA Graphs where profiles point, on-device readout fusion.

   Things worth attention

   • Questions are plain independent forwards. One full prompt per question (system turn, media, state, question, options, empty-think assistant prefix), one 255-way readout at the last position. No branching, no marker positions, no shared-prefix continuation in the reference math — so the native worker packs question rows through the family forward_batch directly, the same path Open-Jev-27B already serves on this exact Qwen3.8-27B base. None of OmniJev's zq/u readout machinery applies.
   • The code table is exact. Running d3_format.answer_codes with the shipped tokenizer: 255/255 codes are single-token, the ids equal decision_config.token_ids one-for-one, and the sequence is A–Z then AA…JT with the two-token doubles (BQ, BZ, CJ, …) skipped. Options take codes[:N] positionally, so option order is part of the contract.
   • Noncausal is a prefill-kernel property, and only for the full-attention layers. The hook (enable_noncausal_full_attention, docstring credits "Adapted from perplexity-ai/pplx-decider-v1.1-27b, Copyright Perplexity AI, Apache License 2.0" — the attribution should ride into our export docs) forces SDPA, rejects KV caches with its own error, and swaps in two masks. I installed it on a small random Qwen3_5TextModel (2 linear + 2 full layers) and measured: causal→noncausal moves the position-0 hidden by max|Δh| = 3.83 — early positions see the future; the last position also moves (1.01) because later keys/values change through earlier full layers. A causal kernel is not an approximation of this model, it is different execution.
   • The GDN layers need no new mask machinery. Introspected from the layers' own kwargs: the full_attention mask is just the padding broadcast (a row with 8 left pads allows exactly keys 8–19), and the linear_attention "mask" is not a 4-D mask at all — it is a [B, T] int64 state-update gate (1 = update the recurrent state, 0 = padding), with causality inherent to the recurrence. So natively the existing causal GDN kernel with its per-row state reset — already how forward_batch packs rows — is the correct semantics; only the softmax full-attention kernel needs a noncausal variant (open question 3).
   • The readout position is free. Measured on the upstream hook: a left-padded batch (rows of 12 and 20 tokens padded to 20) produces last-column hiddens equal to the individual unpadded forwards' last real token to 2.6e-06 / 3.1e-06 (FP32) — padding is excluded from both attentions and from the recurrence, and batching changes nothing beyond rounding. Upstream pads the same way in both of its paths — the fast path left-pads to the longest prompt exactly like the plain one and only adds masked right padding up to a graph-capture length class, reading every row's last real token at the uniform index width−1. And natively the question is moot in the better direction: forward_batch packs rows unpadded, so each row's last position is its last real token by construction.
   • The readout is a side head with no calibration. FP32 (255, 5120) matmul → codes ≥ option-count masked −inf → /T → softmax, with T = 1.0 shipped in decision_config.json (the division is identity); no calibration mechanism exists upstream, so no calibration loader. About 1.3M multiply-adds per question — CPU first, on-device later.
   • Answer and confidence shapes, verified by execution. confidence = clip(1 − H/ln(n)) over the n option probabilities (choice) or the n renormalized level probabilities (score) — machine-equal to the runtime's own function; noul answers carry only noul: p(true), no probabilities dict, no confidence; choice is the argmax key with first-index tie-break; score is Σ i·p_i over renormalized levels plus a legend. Anything porting Intern-Decision's confidence = max p would silently change the contract.
   • Every question sees all media. The image path re-preprocesses per prompt; the video path computes vision features once per request and scatters them into every prompt. Stage 2 keeps this shape exactly — per-question media subsets do not exist to support. Sharing the media prefix across questions natively would be an optimization, but noncausal attention breaks the capture/continue exactness premise, so it is a separate hypothesis with its own parity proof, not an assumption.
   • Refuse, never truncate. Over-limit or invalid questions refuse the whole request (422 with an invalid_questions map); no truncation path exists in the codebase; max_length ships null (model cap 262,144; 16,384 video tokens per request). usage.input_tokens sums per-question prompt lengths; permutation_average exists but defaults off everywhere.
   • The configs already pass our gates; only naming trips. All language guards pass verbatim (64 layers, head 256, linear k/v 128), and the vision tower [27, 1152, 4304, 16, 2304, 5120, 3, 16, 2, 2] is exactly PR qwen: share vision execution for 4B and 27B models #117's approved Qwen3.8-27B tuple — zero constants differ. Two naming trips only: tensor keys are bare language_model./visual., and vision_config.model_type is qwen3_5_vision. The export re-keys and normalizes; no kernel constant changes anywhere.

   Open questions

   Unless told otherwise, the reference is what the pinned requirements.txt plus flash-linear-attention installed give on CUDA today. If the D3 / vLLM Semantic Router authors can confirm or correct any of these, I'll follow that.

   1. Which kernel set defines the oracle? Default: the requirements-pinned set with flash-linear-attention on CUDA (the README calls it "optional: fast GPU kernels", but it is pinned and the runtime curates kernel selection — it refuses CPU+fla and disables the cuDNN SDPA backend on CUDA). Without fla, transformers' GDN ops fall back to the PyTorch reference implementation — "correct but much slower" in its own words, and numerically different. The choice must be measured, not assumed: PR 1 measures the delta on the fixture set before the gates freeze, and if the checkpoint's published behavior turns out to match the reference path, the gates widen to that distribution instead.
   2. Wire envelope: top-level images/videos arrays, or state.images? Default: top-level, matching system_one(state, questions, images, videos), with model accepted and ignored — the reference signature separates them and the server validates them against separate budgets. To be coordinated with [RFC]: Write down which parts of the /v1/systemone contract all workers share #61 either way; the only real question is which name the shared contract standardizes.
   3. Compile-time causality flag or runtime mask for the kernel? Default: compile-time. The flag is a static per-model property, so a compile-time variant is CUDA-Graph-safe and keeps causal instantiations bit-identical by construction; a runtime mask parameter would key graphs on more than lengths and invites replaying the wrong kernel.

   If open question 3 rejects the compile-time variant, PR 3's kernel work follows the runtime-mask alternative; if the omnijev chain's ABI lands first, PR 3 rebases onto it.

   Related: #136 (this effort), #61 (contract), #133 (Intern-Decision), #117/#129/#134 (open carriers the native stage coordinates with), #85 (observation reuse).

   👍React with 👍1Reacted by Hongsheng Liu
4. hsliuustc0106

   assigned

   psv666

   on Oct 24, 2026
5. hsliuustc0106 commented on Oct 24, 2026

   

   hsliuustc0106

   on Oct 24, 2026 – with Cursor

   ContributorAuthor

   More actions

   @psv666 Owner ask (Hongsheng, 2026-10-24): please drive native/remaining D3 work from this issue (Python reference already exists). Hongsheng reviews daily — prefer small focused PRs.