[psv666]

Purpose and summary

Reduce the time Open-Jev-27B-v1.1 users wait for a complete decision by removing repeated candidate work while retaining the trained head, calibration, candidate order and question-level normalization. The frozen cumulative candidate reduced mean full-request latency from 216.31 to 123.02 ms (43.13%) on the historical single-request workload.

Accepted tradeoff: for the six Noul (yes/no) requests with 513–2048 tokens, the 95th-percentile latency increased from 116.32 to 119.77 ms (+2.97%). The contributor explicitly accepted this regression for submission on 2026-10-26. It remains recorded as a regression. Four other historical tail checks remain statistically inconclusive; this PR does not relabel them as passes or claim the original all-group performance gate passed.

complete question
  → bounded tokenization, preserving candidate order
  → compatible candidate groups / within-question shared prefix
  → shared native executor + CUDA operations with independent candidate state
  → original scalar head and whole-question normalization
  → existing typed response

Changes include compatible packed projections and batched Gated DeltaNet (the model's recurrent linear-attention computation), compact prefix continuations, bounded projection-plan caches, vectorized convolution, wider independent work assignment, bounded graph/scratch reuse, and skipping unused rows in the final feed-forward layer when its original matrix-multiplication algorithms support that shape. Unsupported optional paths fall back to existing computation. The 27B worker requires the additive packed-projection/batched-state symbols from this checkout; rebuild its worker and CUDA library together. The existing binary interface remains version 9.

Prefix statistics and integration with #179

metadata.prefix_cache now separates configured policy from actual execution. For three compatible candidates sharing 3072 tokens, it reports:

{
  "policy": "default",
  "enabled": true,
  "scope": "within_question",
  "saved_tokens": 6144
}

policy, enabled, scope and saved_tokens are the agreed complete contract. The current implementation emits policy=default and scope=none|within_question. The legacy mode and shared_tokens fields are removed, including their now-unused prefix-length plumbing. saved_tokens sums (group_size - 1) * prefix_length across the compatible groups that completed execution: separate groups compute separate prefixes, singleton fallbacks save nothing, and suffix packing adds no prefix savings. The count comes from the model execution result rather than being inferred from the request's total candidate count. Input-token usage continues to count complete prompts. No reuse reports enabled=false, scope=none, saved_tokens=0.

twu3202 confirmed the contract and default fallback, and MrDongsls supported removing both old fields. This response shape matches #179 at 97d6376b5e4b027d022890a9a8cdf7f19427ee59. That PR implements explicit fixed/shared/auto policies and across_questions scope; those switches and cross-question execution are not implemented here. After integration, auto fallback is agreed to use this PR's default path. Merge order remains a maintainer decision.

The default 27B validation remains against original default full-prompt execution with the same model, inputs and temperature: maximum absolute candidate-probability and ordinal-score error <= 0.001, zero Choice changes, zero Noul flips at >= 0.5, and zero failures. Equality to a separate fixed-algorithm reference does not replace these gates. Future fixed/shared paths can be explicit opt-ins with separately reported numerical behavior. These are comparison-test gates, not a second baseline inference for every served request. The accepted +2.97% latency tradeoff does not relax them.

Required human contributor answers

The following scope was explicitly confirmed in the 2026-10-22 planning exchange (Q1–Q18), recorded in PLAN.md with SHA-256 300625b0c6a2c44bf4b20653bc7e0be953e38c81292d02c621e9e68689a93967. The 2026-10-26 submission instruction accepts the specific +2.97% tail regression and requests this PR. These are references to the contributor's private task record; no private transcript is attached.

1. Outcome: lower steady-service Open-Jev-27B-v1.1 decision waiting time without changing decision semantics.
2. Highest-leverage variable: complete-request latency under one-at-a-time requests, including frontend and response handling, with workload-group tail latency checked separately.
3. How this moves the variable: replace measured redundant candidate computation and repeated host/device work; retain changes supported by measured benefit and numerical checks rather than relying on kernel timings alone.
4. Customer and need: System1-Omni Open-Jev deployers who need faster decisions with unchanged probabilities/decisions within the agreed bounds and preserved input capacity; no specific production service-level target was supplied.

Is a PR needed? Yes. The accepted cumulative result requires maintained Rust/CUDA changes, and the contributor explicitly requested submission on 2026-10-26. This is a performance change, so bug-fix failure reproduction is not applicable; the measured baseline is below.

Review scope and map

Integration merge base: 05075d0f36e399e3a001ffedea400c666990cda9. Current head: 3cac1a74d9de35f1a962ca7cde6b2ad90d86743e. Target main was refreshed to 9466e0edf24e16669c2136fecc0d3dcdad20811e; the later target changes are outside this follow-up's paths.

The full diff contains 3,250 changed authored-code lines and 3,477 changed lines overall, across 32 files. The cumulative change is submitted together because the executable, device operations and optional interfaces are coupled, and the quoted overall gain belongs to the combined frozen candidate. No partial component is assigned that cumulative gain. This is larger than a quick precheck: the full source/callers and error/resource paths were inspected, and component plus integration checks are listed below. A split would need validation at each resulting head.

Review area	Main paths	Invariants and validation
Request preparation	src/models/open_jev/native/src/{contract,processing,main}.rs	Tokenize at most 16 candidates per chunk; retain padding, first-error order, question identity and normalization; CPU and pinned-tokenizer tests.
Candidate adapter	src/models/open_jev/native/src/{batching,executor}.rs	Up to 16 packed sequences / 4096 packed tokens; longer prompts retain independent execution. Prefix sharing is within one question and requires compatible 2–16 long candidates; restore original output order. Full-model tests in both graph modes.
Shared executor and storage	src/models/qwen3_5/native/src/model.rs	Per-candidate state, fresh prefix contents, compact suffix bounds, scratch/graph address lifetime; retain upstream fixed-algorithm, shared-prefix, bidirectional, graph-statistics and synchronization behavior.
Projection plans and interface	src/backends/cuda/qwen3_5/{gemm.cu,ops.h}, src/models/qwen3_5/native/src/cuda.rs	Check original algorithm compatibility and workspace; separate 512-entry caches. Replay after descriptor eviction and changed-input tests. Existing entry points retained.
Device operations	src/backends/cuda/qwen3_5/{attention,elementwise,gdn_prefill}.cu	Independent sequence attention/recurrent state, unchanged accumulator/rounding boundaries; numerical references, dirty-workspace and changed-input replay tests. Added explicit future-key coverage for bidirectional attention.
Last layer	model.rs, tests/qwen3_5/last_mlp.rs	Final-row-only feed-forward/norm computation is gated to eligible single-sequence ordinary text units; both original algorithms must support one row. Prefix/suffix, bidirectional and multimodal paths retain full computation.
Maintained documentation	architecture, model/backend READMEs, native recipe	Describes mechanisms, graph limits, memory, historical scope and accepted regression; strict documentation build.

No generated experiment dumps, binary archives or one-off reports are added to the source tree.

Test plan and results

The current cleanup 3cac1a74d9de35f1a962ca7cde6b2ad90d86743e removes only the two obsolete metadata fields and the unused prefix-length result/argument. Existing response tests now require exactly the four agreed fields for full reuse, partial reuse and no reuse, while preserving answers and complete-input usage. The device algorithms, reuse grouping, savings computation, weights and numerical gates are unchanged.

Current-head check (3cac1a7)	Result
cargo fmt --all --check	Pass
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings	Pass
cargo test --workspace --locked --features omni-laya/serve	370 passed, 0 failed, 82 ignored
cargo build --workspace --release --locked --features omni-laya/serve	Pass
mkdocs build --strict	Pass
Fresh GPU execution / HTTP model replay / performance campaign	Not rerun: this cleanup changes response metadata only; earlier device and timing evidence below remains attributed to its tested revision
GitHub checks on 3cac1a7	Rust, benchmarks, CUDA contract, CUDA compile and documentation build passed; documentation deploy skipped

This increment is 7 files, 22 insertions and 38 deletions. All incremental changes and callers were reviewed; the prior full source review and component coverage remain applicable to unchanged code. Fresh test logs and the exact patch are retained separately from earlier device evidence. The host-local archive is 3cac1a7-prefix-contract-evidence.tar.gz (20,766 bytes, 10 verified members), SHA-256 b6be39a57a451fd585a7037ba9e4c10845550a7c2a5f97cb171efb4b7c99d78d. Current CI run and current documentation run passed on this head.

The earlier metadata head 56dcf44fdc617ac2ff5c68af3711018c6547a86d added executed-prefix accounting and response metadata to the initially validated integration efd02a896886c4693f04576faf382fcc2d9ebc0a. Device math, algorithm selection, compatibility grouping and default dispatch are unchanged. The CUDA library is byte-identical to the initial integration.

Earlier metadata-head check (56dcf44)	Result
Formatting, strict workspace Clippy, release workspace build	Pass
Default workspace Rust tests	370 passed, 0 failed, 82 ignored
Strict documentation build	Pass
Full-model packed candidates / prefix-buffer reuse and savings	2 passed with graph replay disabled, 2 passed enabled
Complete HTTP requests against frozen original responses	508 passed: zero candidate-probability/ordinal-score drift, zero flips, zero failures
Response comparison against initial integration	Exact equality after removing only timing and the three new policy, scope, saved_tokens fields; all legacy prefix fields also equal
New metadata audit	508 checked; 72 responses actually shared prefixes, 436 did not; policy/scope/count invariants pass
Graph capture failures	Zero
GitHub checks on 56dcf44	Rust, benchmarks, CUDA contract, CUDA compile and documentation build passed; documentation deploy skipped

CI run and documentation run were checked on 56dcf44; they are historical evidence for the current cleanup.

At 56dcf44, the CPU cases cover multiple compatible groups, singleton fallback, partial reuse, suffix segmentation, and unchanged answers/token usage when reporting savings. Those full-model tests verify the executed savings for three equal-length candidates through prefix growth, shrinkage and changed contents. The HTTP cases are 231 real requests twice plus 36 primary and 10 capacity/structure synthetic cases, using the pinned model and protocol below. This repeats numerical/integration coverage; it does not rerun or replace the historical balanced timing campaign.

The 56dcf44 device was again GPU-76ec3aee-7fec-b98c-562a-7697a4be04c7, measured CUDA capability 9.0 / 132 streaming multiprocessors / 150,121,283,584 visible bytes, driver label NVIDIA L20X. CUDA 13.0.88 and driver 570.133.20. 56dcf44 worker SHA-256: fbabb4555785c0503933e3d8f7fd85e483f4a4a764c401a1597d012ae2cb7a3c. CUDA library SHA-256: e256ede7524aff3b79d277df428f8b546fd469d637caaf720fe032fa307bd8e2. Raw validation is retained in 56dcf44-prefix-statistics-evidence.tar.gz (8,846,259 bytes, 63 verified members), SHA-256 170dc7e508e846e4899de0cbcd3a77395fdd850ce7eed814ead411fad7262f71. This archive is host-local, with the same public-download limitation noted below.

The following broader checks were run on the initial integration efd02a896886c4693f04576faf382fcc2d9ebc0a. Unchanged kernel, bidirectional, last-layer and fixed-prefix coverage is inherited from that revision; it is not claimed as a fresh rerun on the metadata follow-up. Historical timing uses a separate frozen source.

Check	Result
cargo fmt --all --check	Pass
cargo clippy --workspace --locked --all-targets --features omni-laya/serve -- -D warnings	Pass
cargo test --workspace --locked --features omni-laya/serve	368 passed, 0 failed, 82 ignored; ignored tests are not counted as passes
cargo build --workspace --release --locked --features omni-laya/serve	Pass
CUDA shared-library build, native compute capabilities 8.9 and 9.0 plus 9.0 intermediate code	Pass with CUDA 13.0.88
python3 src/backends/cuda/check_contract.py --repo-root .	Zero errors; existing unrelated Laya ABI-macro warning remains
python3 -m unittest discover -s tests/cuda -p 'test_*.py' -v	26 passed
python3 -m unittest discover -s src/backends/cuda/tests -t . -v	68 passed
mkdocs build --strict	Pass
Pinned 27B tokenizer tests	2 passed
CUDA kernel tests	25 passed; one checkpoint-dependent prefix rejection test explicitly excluded
Full-model packed candidates / prefix-buffer reuse	2 passed with graph replay disabled, 2 passed enabled
Final-layer full-hidden-vector comparisons	124 comparisons in each graph mode; both test executions passed
Current upstream fixed/shared-prefix full-model compatibility	2 passed

Initial-integration HTTP checks on efd02a8 completed on 2026-10-26: 508 responses (231 real requests twice, 36 synthetic primary cases and 10 capacity/structure cases), with zero probability/ordinal-score drift, zero decision flips and zero failures against the frozen original responses. All parsed responses were identical after excluding only metadata.inference_seconds and metadata.prefix_cache; the latter changes only where within-question sharing is used. Graph capture failures: zero. The synthetic cases include 16,384-token capacity, 17/33 candidates, 2/10 score levels and mixed question kinds; they are numerical/structural coverage, not new model-quality evidence.

This initial integration used its frozen frontend and worker, graph replay enabled, and one reserved device with measured CUDA capability 9.0, 132 streaming multiprocessors and 150,121,283,584 visible bytes; driver label NVIDIA L20X, UUID GPU-76ec3aee-7fec-b98c-562a-7697a4be04c7, driver 570.133.20 / CUDA 13.0.88. It is a different physical device from the historical campaign. The single measured traversal validates integration and does not establish a new balanced performance comparison. No extra performance acceptance is inferred from it.

Initial-integration worker SHA-256: 5ec48fb7d452f131475e5e4669fc31e6b32f4ccd62bd0ed24f9e291a6a9aa96f. Initial-integration CUDA library SHA-256: e256ede7524aff3b79d277df428f8b546fd469d637caaf720fe032fa307bd8e2. Initial-integration frontend SHA-256: 5c4d458309ca9ed84983a1857c86afaedbf6d37a7124fe2ac059a31ab5b1d265.

Initial-integration raw validation is retained separately in efd02a8-integration-evidence.tar.gz (13,949,587 bytes, 79 verified archive members), SHA-256 5a5d4f3d53a6636152c5a0608aacb2b005214101c88caa37f2f91c24f94eeeeb. It is host-local; the same public-download limitation described below applies.

Reproduce the device and full-model checks

Prepare the pinned merged checkpoint using recipe/open_jev/native.md, adding --max-length 16384 during export. Build the CUDA library and release tests. All device commands must run inside the host's GPU reservation. Set OPEN_JEV_MODEL to the merged checkpoint, QWEN3_5_MODEL to the same directory, and both OPEN_JEV_CUDA_LIB and CUA_S1_CUDA_LIB to the newly built library.

cargo test --release --locked -p omni-qwen3-5-native -p omni-open-jev-native --no-run
cargo test --release --locked -p omni-open-jev-native --lib -- --ignored --test-threads=1 tokenization_tests::
cargo test --release --locked -p omni-qwen3-5-native --test kernels -- --ignored --test-threads=1 --skip prefix_state_rejects_uncaptured_foreign_and_oversized_prompts
for mode in 0 1; do
  CUA_S1_GRAPH=$mode cargo test --release --locked -p omni-open-jev-native --test prefill_batch -- --ignored --test-threads=1
  CUA_S1_GRAPH=$mode cargo test --release --locked -p omni-qwen3-5-native --lib -- --ignored --test-threads=1 model::last_mlp_tests::last_mlp_matches_complete_forward_at_boundaries_and_replay --exact
done
CUA_S1_GRAPH=0 cargo test --release --locked -p omni-qwen3-5-native --test prefix -- --ignored --test-threads=1

The validation used these release test executables after compilation, with one reserved GPU and serial tests. The shared-prefix rejection test needs a different complete checkpoint and was excluded explicitly. Other complete model checkpoints, multi-request concurrency, other device architectures, and compute-sanitizer remain unverified. The previous candidate's old-library fallback experiment is historical only and was not rerun against the current interface.

Demo / evidence — historical full-request comparison

Before: original source d577bb824a5dbf0f5e385cf829bb6b9025dabf8d, eager execution.
After: frozen cumulative P19 patch on that base, SHA-256 5bb08cafffcf1825c1467de36705ae78baf07e0f7b24662c67ed7237b6504799, graph replay enabled. P18 is the parent experimental candidate. This comparison includes the declared graph-mode treatment; it is not an isolated last-layer gain or a comparison against today's main branch.

Full-request metric	Original	P19	Change
Mean	216.31 ms	123.02 ms	-43.13%
Median	61.04 ms	44.67 ms	-26.82%
95th percentile	1081.56 ms	471.11 ms	-56.44%
Noul 513–2048 tokens, 95th percentile	116.32 ms	119.77 ms	+2.97%, accepted tradeoff
Sampled whole-invocation peak device memory	50,949 MiB	52,035 MiB	+1,086 MiB, approximately 1.06 GiB

231 unique public JevBench requests: 139 Choice, 74 Noul, 18 Score, comprising 775 candidate inputs. Maximum candidate length in the primary workload is 4096 tokens. The persistent client sends the next request only after the previous one finishes. Timing includes request serialization, Rust frontend forwarding, model work, full response receipt, parsing and validation; evidence writes are outside the timer. Loading is outside this request timer. There is no cross-request reuse of prefix contents.

The fixed protocol used all six three-arm orders twice: ABC, BCA, CAB, CBA, ACB, BAC, repeated. A is original, B is P18, C is P19. Each of 36 service starts ran one separate feasibility traversal and eight measured traversals, alternating original order and shuffle seed 20261022. This gives 74,844 responses: 66,528 measured and 8,316 feasibility, or 22,176 measured per arm. No observations were excluded. First traversals mix new-shape captures and existing-shape hits and are retained separately, not called pure cold-start latency.

All 74,844 historical responses had zero observed probability/ordinal-score error, decision flips or request failures against the frozen original responses; the predefined absolute-error tolerance was 0.001. Parsed responses were equal apart from timing and prefix diagnostics. Graph capture failures: zero. These are empirical checks on the stated inputs, not a universal bitwise-equivalence guarantee.

The six-request regressing group has 576 measured responses per arm. Its mean decreased from 75.96 to 75.22 ms, while its 95th percentile increased in all 12 paired blocks. The approximate 95% paired-block interval for the tail change is +2.32% to +3.70%. The original +2% all-group guard therefore failed; the contributor's later acceptance does not rewrite that result. Four additional 95th-percentile intervals remain inconclusive:

Comparison	Group	Approximate 95% interval for change
P19 / original	Score, 129–512 tokens	-12.95% to +8.02%
P19 / P18	Noul, 1–128 tokens	-8.20% to +3.23%
P19 / P18	Score, 1–128 tokens	-7.23% to +4.57%
P19 / P18	Score, 129–512 tokens	-7.60% to +17.54%

The conditional final synthetic timing stage did not run after the real-workload guard failed. Earlier synthetic qualification and current integration checks do not replace that missing timing stage. P19's overall mean was only 0.095% below P18, with an interval spanning zero; the 43.13% cumulative gain must not be attributed to the final increment.

Historical workload-group results and paired-block observations

Groups use the longest candidate token count in a request. Each unique request was measured 96 times per arm.

Task	Tokens	Unique requests	Mean, original → P19 (ms)	95th percentile, original → P19 (ms)	Tail change
Choice	1–128	72	60.51 → 44.07	76.22 → 56.72	-25.58%
Choice	129–512	23	116.38 → 109.27	147.85 → 144.58	-2.22%
Choice	513–2048	17	321.94 → 310.59	579.56 → 561.45	-3.12%
Choice	2049–4096	27	1081.71 → 436.68	1772.95 → 771.10	-56.51%
Noul	1–128	36	21.16 → 19.52	22.39 → 20.43	-8.72%
Noul	129–512	26	33.89 → 32.68	47.66 → 45.55	-4.43%
Noul	513–2048	6	75.96 → 75.22	116.32 → 119.77	+2.97%
Noul	2049–4096	6	253.70 → 245.36	326.69 → 320.41	-1.92%
Score	1–128	12	52.42 → 40.42	56.57 → 43.56	-22.99%
Score	129–512	2	106.93 → 99.65	126.98 → 115.75	-8.84%
Score	513–2048	1	219.74 → 208.15	224.16 → 213.58	-4.72%
Score	2049–4096	3	1190.83 → 430.64	1204.98 → 638.89	-46.98%

Paired-block relative changes (P19 / original minus one; negative is faster). Intervals use the 12 block ratios, Student t with 11 degrees of freedom; they do not treat every repeated request as an independent dataset sample.

Block	Overall mean change	Overall tail change	Noul 513–2048 tail change
1	-43.2193%	-56.4603%	+1.5449%
2	-43.1785%	-56.3364%	+2.9983%
3	-42.9397%	-56.3955%	+3.3214%
4	-43.2351%	-56.5883%	+4.1576%
5	-43.2452%	-56.7117%	+1.3254%
6	-43.2171%	-56.5004%	+2.2788%
7	-43.1473%	-56.3125%	+4.8416%
8	-43.1789%	-56.7328%	+4.1490%
9	-43.2103%	-56.4715%	+2.7791%
10	-43.0929%	-56.5329%	+1.9975%
11	-43.2191%	-56.5996%	+3.4346%
12	-42.6823%	-55.9531%	+3.3395%

Overall mean-change interval: -43.23% to -43.03%. These paired ratios are included for review; they are not the per-request raw response archive.

Historical provenance, controls and evidence availability

• Model: Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 plus ZefanCai/Open-Jev-27B-v1.1 revision 28cf73067d5b337860bbef3c85b8b82ba8730956. Merged bfloat16 weights, unchanged 32-bit trained scalar head, temperature 2.5343690298472983, maximum supported length 16,384.
• One reserved device: driver label NVIDIA L20X, but measured CUDA capability 9.0 / 132 streaming multiprocessors / 150,121,283,584 visible bytes. The conflicting label does not establish a commercial GPU model. UUID GPU-46890c79-bf10-5b49-3d4a-a6b78f956122. CUDA 13.0.88, driver 570.133.20, Rust 1.99.1. The library contains native 8.9 and 9.0 code. Other architectures were not executed in the historical campaign.
• Same model, frontend, corpus, runner, device and CPU affinity 56–71 across arms; Rayon threads 4, Tokio worker threads 2. CPU affinity was not exclusive ownership of those CPUs. Graph mode was the intentional treatment difference. Device memory was sampled approximately every 0.5 seconds across loading, warmup and requests; brief peaks may be missed. Throughput under concurrent clients was not measured.
• Corpus source: local JevBench revision 869260649f7dd5b56cac4d91eaae9087f2e985ee, benchmarks/e2e/text.jsonl, pinned upstream corpus f8ce713. Frozen request manifest SHA-256 1f1376b99b12a3eca10f9e0726936585c8875e0eccf54de82ab110fd4e21867f; token-shape manifest 6b6bb344513e9b3024a37d5df0dc4b1442a77d5e62ccc03f00fad8dcf469c7d5.
• Original worker SHA-256 4ef036988633a4e2750d98a7ec916c67bd585e48039bab1028555ce4d8b76b32; P19 worker 2b8ddb4b75ff5a95076c1d6f63c753431e10b3af1f973ed5967e9e9765b93ff3; P19 CUDA library 67e445cfe71f13d5331091878d0a71f47defe6242118621cb08e67ac3cefbe1d.
• Frozen campaign runner SHA-256 6d5eadc2a505e5e93f3d550c23ff70d14084216fa4deafa0a2d795576c6eb0ef; imported benchmark runner 931313c3dc51bca9d3520ac2abdb98ec28a3e45d2c016bf402c8e5bac04d8d56. Each archived invocation records its arguments and binary/model hashes. Reproduction uses campaign.py --worker <arm-worker> --library <arm-library> --passes 8 --graphs <0-for-A-or-1-for-B/C> --reference <frozen-original-responses> --out <new-directory> in the declared 12-block order, followed by the frozen comparison/response-audit scripts.
• All raw historical responses, timing controls, sources, binaries and reports are retained in L4-fixed-confirmation-evidence.tar.gz (42,738,920 bytes, 964 verified members), SHA-256 c2ef5ab2a9a5655386c4f6416514f9df1ae5cd68a74f925d49803c08abde85e2. Earlier failures and qualifications remain archived separately.
• Evidence availability limitation: the complete raw archive and exact campaign driver currently live on the experiment host, not at a public download URL. A repository checkout alone can run the maintained regression tests but cannot replay this exact historical campaign without that archive. The tables above provide the public numeric evidence; independent raw-response audit still needs access to the archive. No weights, credentials, private requests or generated experiment archives are committed here. No video: this is a nonvisual request-latency change.

Merge Danger

Door: two-way. Revert the commit and rebuild the worker/library together; there is no stored-data migration. Prefix metadata now contains exactly policy, enabled, scope and saved_tokens; the legacy mode and shared_tokens fields are removed. Consumers that read those old fields must update to the agreed contract.

Blast Radius: shared-executor. Open-Jev-27B opts into the optimizations, but its executor and CUDA code are shared with other models. The integration preserves the current bidirectional and fixed/shared-prefix interfaces and tests their affected operators/paths. Complete other-model checkpoint runs are unverified. Larger retained graph/scratch/prefix capacity costs device memory; new shapes still incur capture cost, so graph mode stays opt-in. The measured tail tradeoff and four inconclusive guards remain visible above.

Self-review

Full source review and executed validation are reported above. The contributor confirmed completion of the four self-review items on 2026-10-26 and requested moving this PR out of draft. The disclosed validation limitations remain part of the review scope. Contributor self-review and passing checks do not replace maintainer review.

• I have reviewed the full diff and addressed the issues I found.
• I have checked that the change follows the project's architecture and stays focused on the stated purpose.
• I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
• I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

[twu3202]

Heads-up: #179 (for #85) also runs Open-Jev's shared candidate prefixes once, so the two overlap. A trial merge conflicts in executor.rs, processing.rs, main.rs, tests/open_jev/processing.rs and recipe/open_jev/native.md, and both add finish_with_prefix with different metadata.prefix_cache fields (saved_tokens with modes fixed/shared/auto in #179, shared_tokens with within_question here).

They cover different cases. #179 is opt-in (OPEN_JEV_PREFIX), works for 9B and 27B, and also shares the page state across questions in one request, using forward_shared from #99. Here it is on by default for 27B, for single-question requests whose candidates are all over 2,048 tokens, and it keeps the default GEMM algorithms so responses don't change. Happy to rebase #179 onto whichever lands first. It would help to settle on one prefix_cache shape.

[psv666]

@twu3202 Thanks for checking the overlap. I propose separating configured policy from actual execution in metadata.prefix_cache:

{
  "policy": "default",
  "enabled": true,
  "scope": "within_question",
  "saved_tokens": 6144
}

• policy: default, fixed, shared, or auto. default means no explicit prefix-policy override; Optimize Open-Jev-27B candidate execution #186 can still reuse a compatible long prefix automatically.
• enabled: whether this request actually reused a prefix.
• scope: none, within_question, or across_questions (which can also include within-question reuse).
• saved_tokens: prefix tokens whose computation was actually avoided, after compatibility checks and fallback. For three compatible candidates sharing 3072 tokens this is 6144; if only two can share and the third runs independently it is 3072. Suffix packing does not count as extra prefix savings. usage.input_tokens still counts the complete candidate inputs.

I'll add this accounting to #186, retaining its existing mode and shared_tokens fields as compatibility fields. shared_tokens remains a prefix length, not a synonym for savings. The other policies and cross-question execution remain #179's scope; this is a proposed common contract, pending your feedback.

For the default 27B path, I want to retain this PR's comparison against the original default full-prompt execution: maximum absolute candidate-probability and ordinal-score error <= 0.001, zero Choice changes, zero Noul flips at the >= 0.5 threshold, and zero request failures. The current tests observed zero error; that is not a universal bitwise-equality claim. The accepted +2.97% Noul tail-latency tradeoff does not relax these numerical gates.

Your fixed/shared paths can remain explicit opt-ins with their own numerical evidence. Shared-vs-fixed equality should not be substituted for the default-path comparison when integrating them. In particular, an auto fallback should retain #186's default path after the branches are combined.

I'm preparing the metadata-only follow-up here so there is a concrete implementation to compare. If #186 lands first, #179 can build on it as you offered; the final ordering can follow maintainer preference. This does not require changing the scoring algorithms or rerunning the historical timing campaign just to rename/report statistics.

[twu3202]

Sounds good, and 56dcf44 matches it. One tweak: neither PR is merged, so the only prefix_cache on main is the off case, {"enabled": false, "mode": "independent_candidates"}, which only Open-Jev's own test reads. I'd drop mode and shared_tokens rather than keep them for compatibility, since mode already means different things in our two branches and scope: "none" covers the off case. Then both use policy, enabled, scope and saved_tokens.

I'll switch #179 to that: policy is the switch (default with none set), scope is across_questions when more than one question shares the request prefix, and saved_tokens already counts what you describe.

Agreed on the default path: auto falls back to whatever the default path is, so after combining it would be #186's.

[psv666]

The concrete follow-up is now in 56dcf44. It adds policy, scope and actual saved_tokens, preserving enabled, mode and active shared_tokens. Savings come from the executed compatible groups; singleton fallback and separate suffix launches are covered. The worker still implements only its default within-question policy; #179's explicit policies and cross-question reuse remain separate.

Validation on this head: 370 default Rust tests passed; full-model packing/prefix tests passed with graph replay both off and on; 508 complete HTTP responses had zero probability/score drift, zero decision flips and zero failures against the original default responses. After removing timing and only the three added fields, every response also matched the prior integration, including the legacy prefix metadata. There were 72 responses with actual prefix reuse and zero graph-capture failures. All five GitHub validation checks passed; documentation deployment was skipped.

The PR description and native recipe now spell out the numerical gates and field semantics. This is ready to compare with #179's metadata handling; the proposed common contract and merge order are still open for your feedback. Historical timing results, the accepted +2.97% tail tradeoff and the four inconclusive checks remain unchanged.

[psv666]

@MrDongsls @Levius-Fubuki PTAL

[psv666]

@hsliuustc0106 PTAL

[MrDongsls]

Thanks @psv666 for this PR. I've read the discussion above, most of entries are settled. On the open item I tend to side with @twu3202: Drop mode and shared_tokens fields. The only reader of prefix_cache on main is our own test, which can be updated in the same change. mode already means different things on main and on both branches, and policy: "default" + scope: "none" covers the off case. If either of them stays, keep only shared_tokens, since it's the one field the new four don't already cover.

[psv666]

@twu3202 @MrDongsls Agreed; both mode and shared_tokens are removed in 3cac1a7, along with the now-unused prefix-length result/argument. metadata.prefix_cache now contains exactly policy, enabled, scope and saved_tokens, matching #179 at 97d6376b5e4b027d022890a9a8cdf7f19427ee59.

The response tests require the exact four-field object for full reuse, partial reuse and no reuse, and verify unchanged answers and input-token usage. The default 27B numerical contract is unchanged: maximum absolute probability/ordinal-score error <= 0.001 against original default execution, zero Choice changes, zero Noul flips at >= 0.5, and zero request failures. The agreed auto fallback after integration remains #186's default path; explicit policies and cross-question execution stay in #179.

Fresh local validation: formatting, strict workspace Clippy, release build and strict documentation build passed; workspace tests: 370 passed, 0 failed, 82 ignored. GPU execution, HTTP replay and performance timing were not rerun for this metadata-only cleanup; the PR now explicitly attributes those earlier results to their tested revisions. The accepted +2.97% tail-latency regression and four inconclusive timing checks remain disclosed.

All five validation checks passed on 3cac1a7: Rust, benchmarks, CUDA contract and CUDA compilation, plus documentation build. Documentation deployment was skipped.

The recipe and PR description are updated with the agreed contract. Merge order remains a maintainer decision.

[Levius-Fubuki]

Thanks @psv666, the updated metadata contract looks good. One small doc nit: “Merge Danger” still says the old fields are retained—please update that to match the cleanup.

[psv666]

@Levius-Fubuki Thanks for catching that. Updated “Merge Danger” to state that prefix_cache contains exactly policy, enabled, scope and saved_tokens, and that mode and shared_tokens are removed. It also notes that consumers of the old fields need to update. This corrects the PR description only; the tested code remains at 3cac1a7.