Repository navigation
[RFC]: Native Rust/CUDA support for OmniJev-4B v1.1 and community help #114
Description
Motivation
Add native Rust/CUDA serving for tinnel123/OmniJev, starting with the released 4B v1.1 decision checkpoint. The goal is lower latency and higher decision throughput for questions about the same visual observation, while preserving the author's inference semantics.
System1-Omni does not currently integrate this checkpoint. At main 9986574, we have a shared Qwen3.5/3.8 CUDA prefill implementation, native Cua-S1 vision code, and serial worker admission. These are useful foundations, but OmniJev's option markers, branching, readout features, learned heads, and calibration need their own implementation.
Community help wanted: contributors are welcome to take one of the bounded tasks below, review the design, or provide reference fixtures and GPU validation. No native OmniJev performance result has been measured yet.
Proposed Change
1. Target and reproducible reference
| Artifact | Inspected revision |
|---|---|
| System1-Omni | 99865743d27316fbe81362dc0f0e6de6fda86284 |
| OmniJev reference code | 14dbec4f71e194852c8d7b88ab36ef639493f400 |
OmniJev 4B adapter, heads, tokenizer, processor; HF tag v1.1 |
ffe5f436eaf22e20e2f041f8e74e121fd057a6cb |
| Qwen3.5-4B backbone | 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a |
The adapter names the base model but does not pin its revision. The Qwen revision above is a proposed baseline inspected for this RFC; confirm it with the authors and freeze known-good dependency versions before generating parity fixtures. Upstream's requirements currently specify minimum versions rather than a complete environment lock.
The checkpoint metadata enables norm=softmax, lm_feats=true, ordinal=true, and branch=true. The adapter has rank 32 and alpha 64. The base has 32 language layers: 24 linear-attention and 8 full-attention layers.
Start with BF16 CUDA inference, one PNG/JPEG image plus textual questions, and all three question types. Native image encoding is part of this first serving milestone. Follow with numbered multi-image panels and the reference video/frame path. Other model sizes, quantization, Metal, text-only requests, experimental v1.2 action outputs, cross-request prefix caches, and dynamic scheduling are separate work.
2. Ownership and serving interface
Follow the architecture contracts and model integration guide:
- Put the native worker, model-specific processing, batching adapter, and executor under
src/models/omnijev/; preparation and deployment instructions belong inrecipe/omnijev/. - Preserve the Rust frontend's
POST /v1/systemoneandGET /healthinterface. Proposed wire input uses the existingmodel / state / questionsenvelope andstate.imagescontaining inline PNG/JPEG data URLs. The reference's filesystem paths become decoded images internally; this transport adaptation must be documented and tested. - Keep question rendering, tokenization, media preparation, calibration, and response assembly in the processor. The executor owns weights, vision/language execution, learned heads, and device buffers. Reusable CUDA operations remain under
src/backends/cuda/. - Reuse
SerialSchedulerwith one complete request as the admission unit. Validate image dimensions, body size, question/candidate counts, expanded token lengths, and branch-work budgets before dispatch; document the chosen limits. Return explicit errors for unsupported inputs or exceeded budgets. - The serving process should require no Python/PyTorch. A one-time Python exporter and a pinned Python reference for fixtures/comparisons are appropriate.
- Readiness requires validated artifacts, device initialization, and a successful real warmup. Preserve resource ownership until device work finishes, including cancellation and error paths.
3. Checkpoint and vision reuse
Prepare a hashed native export containing the language weights, required base vision weights, learned decision and ordinal heads, saved option-token embeddings, tokenizer/processor files, and calibration metadata. Preserve input/output embedding tying after restoring new_tok_emb.pt.
Merge the adapter's actual target tensors; do not assume Cua-S1's adapter layout or constants apply. Qualify the numerical effect of BF16 LoRA merging against the unmerged author's reference before adopting it as the native preparation path. Preserve Apache-2.0 attribution and the base model's license.
Reuse compatible Qwen language kernels and the existing vision tower/3D-position boundary. The current Cua-S1 vision loader is tied to that checkpoint and its adapter validation; extract only the common operations needed by both consumers. Existing Cua-S1 and Open-Jev regression checks must remain intact.
CPU image preparation must match the checkpoint processor: decoding, RGB conversion, resizing, normalization, patch order, placeholder expansion, and rotary positions. Do not substitute an arbitrary resize policy. A deployment that only accepts offline image embeddings is an intermediate validation tool, not completed native image support.
4. OmniJev semantics to preserve
Use the pinned records and rendering code, branching implementation, heads, and response finishing as the reference.
| Boundary | Required behavior |
|---|---|
| Prompts and options | Preserve chat-template rendering, <|opt|> / <|/opt|> tokens, restored embeddings, option-text truncation, region formatting, and caller order. Preserve the author's criteria/option-list Choice forms and ordered levels Score form; specify any additional wire aliases explicitly. |
| Branching | Encode the common visual/text prefix once, then execute isolated question/option suffixes. Preserve causal masks, true lengths, padding, cache positions, and the reference's multimodal prefix and suffix position rules. An attention mask alone cannot isolate the Gated DeltaNet recurrence. |
| Readout | Gather the closing-option hidden state and question-context state at their reference positions. Choice/Noul also require the enabled six-component LM probability features. Returning only one final hidden state per branch is insufficient. |
| Choice and Noul | Run the trained FP32 option scorer, including question gating, feature projections, learned per-type scaling, and Choice abstain logit. Preserve abstention, validity, confidence, rounding, and tie-breaking. |
| Score | Run the trained FP32 cumulative-link ordinal head and preserve ordered-level probabilities and the returned level key. Open-Jev's scalar head and expected-level response are different computations. |
| Calibration | Preserve the reference's finishing order. Saved temperatures are Noul 1.1991, Choice 1.1199, Score 1.1909; apply Noul's saved log-odds bias 0.05310856608843584 before post-hoc temperature scaling. |
| Response grouping | Reconstruct every question and candidate in request order. Preserve whole-question normalization, output fields, reference input-token accounting, and zero generated tokens. Compute timing fields from the native execution. |
The enabled LM features use token log probabilities normalized over the full vocabulary, with an additional normalization across candidates of the same question. A selected-token projection without the full-vocabulary denominator would change the model. Gather only the required hidden rows and compute vocabulary projections/reductions in bounded chunks; fuse or stream this work only after checking the same features against the reference.
For later modalities, reproduce numbered still-image panels. The video helper prepares timestamped frame mosaics, and records.video_content crops the mosaic back into ordered frame images for the prompt. Passing the whole mosaic as an ordinary still image is not the same input. Freeze frame sampling, timestamps, layout, fonts, and encoding; FFmpeg-based decoding can be a documented native dependency.
5. Shared-prefix execution and performance plan
Coordinate with #85, #97, and #99, which already discuss or implement Qwen prefix continuation/reuse, and #96, which proposes a different JEV-VL model. These PRs are open at the inspected baseline; their APIs and validation are not merged OmniJev support.
The shared execution interface needs multimodal embeddings/positions, selected-position readouts, and complete continuation state. Snapshot full-attention KV, FP32 Gated DeltaNet recurrent state, and convolution history; every branch starts from the same immutable prefix snapshot. Bound branch batches and scratch memory, and keep all state request-local initially.
Prefix split points, Gated DeltaNet chunk boundaries, and GEMM algorithm choices can change numerical results. Validate those differences explicitly; text-only or final-hidden-state tests from another model do not establish OmniJev parity.
Proposed delivery sequence:
- Contract/export: pinned reference environment, artifact inventory, exact CPU preparation fixtures, head/feature fixtures, and documented wire input/output.
- Native eager worker: native vision, complete language/readout/head execution, and frontend integration. Independent full-row execution may serve as the correctness baseline while prefix continuation is integrated.
- Request-local reuse: shared multimodal prefix, bounded suffix batches, selected hidden-row gathering, and GPU heads/features. Compare with both the native eager baseline and upstream.
- Further optimization and modalities: CUDA Graphs or fused vocabulary/head reductions where profiling supports them; numbered multi-image and video fixtures/serving.
Native execution, shared-prefix reuse, and individual kernel optimizations are separate hypotheses. Measure each against its appropriate baseline instead of attributing a combined result to one change. A Python worker is useful as the oracle but does not meet the native-serving goal; a second Qwen implementation would duplicate reusable code.
6. Validation and acceptance
- Freeze code/artifact hashes, dependencies, dtypes, preprocessing settings, and proposed numerical gates before GPU work. Suggested initial discussion gates: maximum absolute answer-probability drift 0.01, with matching Choice/Score decisions when the reference top-two margin exceeds 0.02, and matching Noul decisions when its reference probability is more than 0.01 from 0.5. These are proposals, not established parity results. Report low-margin disagreements and near-tied abstention explicitly.
- Exact CPU fixtures cover rendered prompts, token IDs, markers/readout positions, image grids/positions, region options, Unicode, mixed question types, response fields, calibration, usage, limits, and malformed inputs. Keep all tests/helpers/fixtures under root
tests/. - Compare stage outputs: image tensors/embeddings, selected hidden rows, six LM features, decision/ordinal-head outputs, and final calibrated answers. Freeze stage-specific tolerances before execution; aggregate accuracy alone is insufficient.
- Check eager versus prefix/batched execution, candidate/question reordering, branch padding, prefix/chunk boundaries, repeated requests, cache/scratch growth, cancellation, and failures. Cover short/long questions, multiple candidate counts, and empty/special-token option text.
- Run direct-worker and frontend-proxied HTTP checks, including the first request after readiness. Revalidate existing Qwen/Cua-S1/Open-Jev consumers when shared code or ABI changes.
- Performance comparisons begin with a CPU preflight and one excluded feasibility run per configuration, then two measured runs per configuration at predeclared concurrency 1, 8, and 16. Reuse one server per configuration unless measuring startup; stop on a correctness failure or the declared run limit.
- Hold exact reserved GPU IDs, affinity, checkpoint, precision, requests, readiness behavior, and cache policy fixed. Change only the declared implementation/optimization and record necessary consequences. Report preparation, process-to-readiness, first inference, warm p50/p95, requests/s, decisions/s, failures, and peak memory separately. Preserve commands, manifests, raw outputs, variability, and all run results.
- Use the benchmark protocol, with OmniJev-aware response validation for abstention and categorical Score keys. Do not alter served probabilities to satisfy another model's validator.
- Update recipes, model docs, the support matrix, README, navigation, and [Tracking]: Model support status #83 when validated support lands. Record the tested GPU and scope; no general hardware or speedup claim from compilation or narrow fixtures.
Community help wanted
| Contribution | A useful first deliverable |
|---|---|
| Upstream/reference expertise | Confirm the proposed Qwen base revision, known-good dependencies, prompt/readout semantics, and calibration; provide small redistributable image/question fixtures. |
| Rust processing and checkpoint preparation | Native artifact schema/export, image/token fixtures, question compilation, and exact response reconstruction. |
| Qwen/CUDA execution | Coordinate the continuation API with #97/#99; add multimodal prefix state, required hidden-row gathering, and bounded suffix execution without duplicating the backend. |
| Learned heads and LM features | Implement and validate FP32 decision/ordinal heads and bounded full-vocabulary log-probability reductions; identify performance bottlenecks with measurements. |
| Validation and later media support | Run reproducible reserved-GPU parity/performance comparisons, or contribute numbered-panel/video preprocessing fixtures and native helpers. |
If you would like to help, comment with the task you want to take, relevant experience, and available hardware. We can agree on a small implementation PR and its acceptance checks here. Shared-backend contributors should coordinate before changing the CUDA ABI.
Review questions:
- Is the single-image 4B/all-question-types milestone the right first slice?
- Which shared-prefix API should OmniJev build on, including multimodal positions and intermediate readouts?
- Are the proposed numerical gates suitable, and which fixtures best expose LM-feature, ordinal, and abstention regressions?
- Which performance workloads and GPUs can contributors validate reproducibly?
Feedback Period
Initial feedback through 2026-10-21; contributions and follow-up discussion remain welcome afterward.
CC List
@tinnel123666888 — feedback from the upstream authors on checkpoint compatibility, reference dependencies, and inference semantics would be especially helpful.
Anything else
This RFC is a design and contribution request. Preparation inspected pinned public source and metadata; no model weights were loaded, no GPU work or benchmarks were run, and no native speedup is claimed.
Related: #9 (model roadmap), #36 (Cua-S1 multimodal coordination), #83 (support tracker), #85 (observation reuse), #96 (separate JEV-VL integration), #97/#99 (pending Qwen continuation/reuse).
Before submitting a new issue
- Searched existing issues and PRs for OmniJev and its checkpoint ID; read the RFC template, contribution guide, architecture contracts, current native code, and upstream inference/metadata. No dedicated OmniJev integration issue was found.
Activity
hsliuustc0106 commentedon Oct 7, 2026
@twu3202 are you interested in this task?
twu3202 commentedon Oct 8, 2026
Sure, I'd like to take this.
twu3202 commentedon Oct 8, 2026
Plan: native OmniJev-4B v1.1
For the first slice as scoped above (4B, BF16 CUDA, one PNG/JPEG image, all three question types), building on the #85 stack (#97, #98, #99). I read the pinned reference and HF files, checked the points below on CPU, and ran the reference on one GPU with the released weights (hashes checked against
release_manifest.jsonand Qwen3.5-4B851bf6e). No native code yet.What the reference does, and what follows for the port
ask_branchtakes the questions' longest common token prefix, cut at least one token before any question's first<|opt|>, and runs each (question, option) row, the rest of its question plus one option block, as a continuation of that prefix. Rows are padded only at the end, so on transformers 5.7.0 or later each row's output is that of a plain forward over prefix plus row, up to rounding. The native worker can then choose its own split points (qwen3_5: run prompts that share prefixes once per prefix #99's multiples of 64) and run rows unpadded. Input-token usage should still follow the reference's count (its prefix once plus every row), computed from its split rule on the CPU..layers). On a tiny random Qwen3.5 on CPU, continuing a 70-token prefix with 30 tokens differs from the plain forward by up to 0.54 (largest hidden value 4.05) on 5.5.4 and 5.6.2, and by about 1e-6 on 5.7.0 and 5.19.0. On the released checkpoint, with the repository's seven v1.1 demo stills andbench/speed_bench.py's twelve questions (84 answers, transformers' PyTorch Gated DeltaNet path), 5.6.2 changes 12 decisions against 5.19.0, and each answer's largest probability change has a median of 0.11 and a maximum of 0.57; two runs on 5.19.0 give the same output to the reference's four decimals.requirements.txtallowstransformers>=5.0, so the version behind v1.1 matters (open question 1).uat the closing<|/opt|>, the row's last token;zqat the token before<|opt|>(the newline after<think>), the same for all options of a question; and for the LM features, the log-probability of each option-text token from the position before it, plus that of the first text token fromzq. Like the reference, the native split keeps thezqtoken in every branch.add_option_tokensresizes the tied embedding to the tokenizer's 248,079 entries (the option tokens are 248,077 and 248,078), down from the base checkpoint's 248,320 rows, so the log-softmax runs over 248,079 logits. They leavelm_headin BF16 and are cast to FP32 before the softmax.MSO1passesmax_pixels=768*28*28(602,112) toAutoProcessor, which overrides the processor config's 401,408: a 1920×1080 image gives 576 image tokens instead of 364 (5.5.4 to 5.19.0). On the same 84 answers, 401,408 instead of 602,112 changes 12 decisions (median 0.06, maximum 0.60). I'd followMSO1(open question 2). Main's native Cua-S1 preprocessing ports the same image processor; OmniJev needs a different pixel budget and input limits (Cua-S1 accepts at most 2048 pixels per side and 1,048,576 in all).lm_headorin_proj_a/b), so the vision tower is the base one; I'd take it from851bf6e, the Qwen3.5-4B revision Cua-S1 pins (open question 3).head.ptandord.ptcontain every key the heads define, and all file hashes matchrelease_manifest.json.Design
src/models/omnijev/andrecipe/omnijev/, split as the RFC describes. The processor renders prompts, tokenizes, decodes and preprocesses the image, and builds the layout and usage. The executor owns weights, vision, language passes, readouts and heads. Finishing applies the Noul bias, the temperatures and the reference's rounding.model.rs.forward_shared(request prefix, group prefixes, branches) maps onto image prefix, question text and option blocks. It needs three additions:zq,u) and target log-probabilities at given positions;fixed_gemm_rows_do_not_depend_on_mpasses on sm_89 for the backbone shapes; I'd add the vocabulary chunk shape). Batched rows then keep the order invariance the reference reports as 0.0.PR sequence
forward_fixed. Bounded suffix batches and GPU heads follow, each measured on its own.If #85 settles on #96's backend path instead, PR 3 follows that.
Open questions, with the defaults I'd use
Unless told otherwise, the reference is what
pip install -r requirements.txtandMSO1("ckpt", "base")give today. If @tinnel123666888 can confirm or correct these, I'll follow that.branch.py's docstring describes. If v1.1 was trained on 5.5 or 5.6, the rows' linear-attention layers started from zero at the reference's split, and the port would have to reproduce that split.MSO1's 602,112 pixels, the released code's default, rather than the processor's 401,408.851bf6e, on transformers' PyTorch Gated DeltaNet path withoutfla-core; I'd report the difference to fla separately.On the review questions
1. Is the single-image 4B/all-question-types milestone the right first slice?
Yes.
2. Which shared-prefix API should OmniJev build on, including multimodal positions and intermediate readouts?
#97 to #99, with the three additions under Design.
3. Are the proposed numerical gates suitable, and which fixtures best expose LM-feature, ordinal, and abstention regressions?
0.01 seems a fair start for BF16 against BF16; I'd freeze stage tolerances before the first run and report the full distribution. Fixtures: options that differ in one token or share a long start (LM features), a 10-level Score with close cut points (ordinal head), Choice where no option fits, and Noul near 0.5 and near the bias boundary (abstention and calibration).
4. Which performance workloads and GPUs can contributors validate reproducibly?
One RTX 6000 Ada (sm_89, 48 GB): parity runs, and the RFC's concurrency 1, 8 and 16 runs on
bench/speed_bench.py's twelve questions (1, 3, 6 and 12 per request) plus a 32- and a 255-option Choice.twu3202 commentedon Oct 8, 2026
Levius-Fubuki's #117 now moves the vision code into the shared Qwen crate with configurable image limits, so I'll build OmniJev's vision on it instead of moving it myself.
Levius-Fubuki commentedon Oct 8, 2026
Thanks for coordinating. I’m doing another review of #117 before reopening it. I’m happy for OmniJev to reuse the shared vision code and configurable image limits, and I’ll update this thread once it’s ready.
docs: list OmniJev-4B v1.1 in the supported models
omnijev: add the request contract, preparation, heads and export
docs: list OmniJev-4B v1.1 in the supported models
hsliuustc0106 commentedon Oct 24, 2026
@psv666 Owner ask (Hongsheng, 2026-10-24): please drive OmniJev-4B native from this RFC. Hongsheng reviews daily.