[RFC]: Next-phase roadmap: prefill-only execution, reuse and scheduling #150

Motivation

Decision models here only prefill. A request is one forward pass per prompt with no decoding, and the answer comes from a few label rows or a small head.

vLLM and SGLang now serve /v1/systemone too. They are built for hundreds of model architectures, and they answer decisions through the same engine they use for text generation (details under Anything else). A server that only does decisions has much more room at the system level.

Batching across requests helps less here than in a generation server:

• Agent clients send one request at a time. System1-Agents does this, with a 5 s deadline per decision by default. Bulk users, such as tagging at ingest time, are the exception.
• Batching changes the numbers unless every kernel is batch-invariant.
  • cuBLASLt picks a GEMM algorithm per row count. In 9B's GDN input projection, on synthetic inputs, row 0 differs in 6,768 of 12,352 outputs between 1 and 2 rows ([RFC]: Reuse shared observations across decision questions — community help wanted #85).
  • vLLM's and SGLang's docs both say probabilities depend on which requests share a batch and on prefix-cache hits (vLLM, SGLang).
  • qwen3_5: add a GEMM handle with one algorithm per weight shape #98's fixed algorithms remove the GEMM part, at 8–17% on long single passes.
• Models differ in heads, prompt layouts and limits, and some define how their questions are grouped in one prompt. A batcher has to keep each model's layout and limits.

What helps more is to skip work that repeats, make each prefill fast, and schedule with the cost known before running. Requests repeat in three ways:

1. One state, many questions.
   • A System1-Agents browser step asks 2–6 questions, with 58–110 click targets on the larger pages, at 2K–17K input tokens.
   • Open-Jev scores each candidate as its own sequence, so every candidate repeats the context and the question.
   • TypeSafe's API allows up to 255 options per question.
2. The same questions over many states. Guardrails, routers and taxonomies apply one fixed block of instructions and options to every message or document.
3. The same state again. An agent asks new questions about the page it just looked at, or a voice agent asks again as the transcript grows.

Where things stand:

• Request-local reuse is in review.
  • qwen3_5: add a GEMM handle with one algorithm per weight shape #98 and qwen3_5: run prompts that share prefixes once per prefix #99 build on jev_vl: add native inference and prefix caching #96's prefix continuation, and decider: reuse request prefixes with conservative auto selection #113 and omnijev: reuse the prefix within a request #134 build on them.
  • Each branch is bit-identical to a full pass with the same fixed GEMMs.
  • On 9B, 255 branches of 38 tokens after a 3,072-token prefix take 6.24 s instead of 124.99 s (random tokens, qwen3_5: run prompts that share prefixes once per prefix #99). The branches still run one at a time.
• Open-Jev-9B still runs every candidate as its own prompt. In System1-Agents' A40 runs, a decision took 6.8–8.7 s against the 5 s default (results).
• Only JEV-VL keeps state across requests. On one H800, its cache answers a repeated image in 37 ms vs 103 ms p50 (validation).
• The scheduler is FIFO and serial per executor. Bounded admission and deadlines are in review (runtime: optional bounded admission #109, decider: bound pending requests and enforce HTTP deadlines #149).

This proposes an order for the execution, reuse and scheduling work in #125, plus items past its milestones: branch batching, scheduling by cost, and a cross-request cache. The items are directions with a first step, not commitments. Each one gets its own issue or PR, with measurements, when it starts.

Proposed Change

1. Find the repeated work in real interactions

We don't yet know how much repeats in real agent use:

• how often a state is asked about again;
• how much of it changes between steps;
• how often an image comes back;
• how many questions share a state.

Record episodes from System1-Agents' browser, desktop and tool fronts, and measure these from the requests themselves. The served HTTP path carries no images yet, so screenshot reuse is measured on the in-process paths. The traces become frozen request plans for #116's serving matrix, and they decide which caches in section 4 are worth building.

First step: a trace format and a replay script under benchmarks/. Related: F6, A7, #87.

2. Run each request's repeated work once

This is case 1, in two stages.

1. Land qwen3_5: add a GEMM handle with one algorithm per weight shape #98 → qwen3_5: run prompts that share prefixes once per prefix #99, which decider: reuse request prefixes with conservative auto selection #113 and omnijev: reuse the prefix within a request #134 build on, and move Open-Jev onto forward_shared ([RFC]: Reuse shared observations across decision questions — community help wanted #85).
2. Run all branches of a prefix in one pass instead of one at a time:
   • the GDN layers start each branch from the shared snapshot;
   • attention reads the prefix K/V;
   • the readout runs on the GPU for all rows.

The goal is the same bits as running each branch alone. That needs:

• qwen3_5: add a GEMM handle with one algorithm per weight shape #98's fixed GEMMs;
• splits on GDN chunks and attention query blocks;
• the same key order in attention;
• the same readout on both paths.

It adds backend operations, so the design goes to #85 first, with measured numbers. Related: F3, within-request F4.

3. Scheduling for prefill-only work

A request's cost can be estimated once it is prepared, mostly from its uncached tokens and images. That allows:

• Per-request admission. Admit, queue and account per request, with all its questions, rather than per question. Today Cua-S1 admits one question at a time.
• Early refusal. When the client's deadline can't be met, refuse at admission with 503 and Retry-After, so the client finds out at once.
  • A decision that has started runs to the end even after its client has gone. System1-Agents stops waiting after 5 s by default, while the frontend waits 60 s.
  • This needs the client to send its deadline, for example in a header, which is an API addition for [RFC]: Write down which parts of the /v1/systemone contract all workers share #61.
• Shortest job first, with aging, when a queue forms.
• Cache-aware order. Once section 4 exists, prefer requests whose prefix state is already cached.

This builds on #109 and #149. Batching across requests stays opt-in per model, needs section 5, and gets its own issue as #125 asks. Related: F2, cross-request F4.

4. Keep state across requests

This is case 3, and case 2 for models whose prompt starts with the fixed block. It generalizes JEV-VL's cache into a prefix-state cache in the shared executor, which owns retained prefix state for request-local reuse (#85). Where cross-request budgets live is part of the design.

• Checkpoints sit where the state or a fixed leading block ends, at a 64-token boundary, so a state that grows can reuse the old one.
• Keys cover:
  • the token IDs up to the checkpoint and their positions;
  • the model and adapter revision;
  • the processor settings;
  • an image identity computed from decoded pixels;
  • the precision and GEMM mode.
• Memory has byte budgets, with eviction by compute saved per byte.
  • Each checkpoint holds 49 MiB of GDN and conv state on 4B/9B (147 MiB on 27B), plus 32 KiB (64 KiB) of K/V per token.
  • That is about 146 MiB (340 MiB) for a 3K-token state.
• Only exact hits are used.

Section 1's traces decide its size. It gets a design issue before any code. Related: M7, F3, F5.

5. Repeatable answers

Reuse, packing, caching and scheduling should not change a model's output.

• Batch-invariant execution becomes a declared setting: fixed GEMM algorithms from qwen3_5: add a GEMM handle with one algorithm per weight shape #98, a fixed attention order and 64-aligned splits. Its cost is measured per model. On 9B, the fixed handle costs 8% more at 1,024 tokens and 17% more at 3,072 (qwen3_5: run prompts that share prefixes once per prefix #99).
• Each model states whether it matches its reference exactly or gives the same bits under reuse. fix(cua-s1): match text reference and preserve merged compatibility #143 already adds such a choice for Cua-S1: an exact-reference profile that rejects prefix continuation.
• Comparisons count flips grouped by the reference's top-two margin, not only accuracy.

For comparison, the pplx-decider v1.2 card says its results are bit-stable only with one request per call. Related: F1, F6.

6. Easy to adopt

• Publish the scorecard on section 1's workloads, building on bench: add frozen HTTP workload and concurrency matrix #116, bench: preserve complete serving evidence and offline summaries #131 and test: add weekly local GPU accuracy and latency checks #141. Include vLLM and SGLang on the same checkpoints where they can run them.
• Keep a one-command recipe for each model.
• Compile every CUDA backend in CI (ci: compile every declared CUDA backend #101), and build the qwen3_5 backend for sm_90 and sm_120 as well as sm_89.
• Run an FP8 pilot, which would fit 27B on a 48 GB card (its own issue, under F9).

Related: F6, F7, F9.

Order

Not planned

• Decoding or chat.
• Lossy reuse, such as CacheBlend-style blending or approximate GDN state rebuilds.
• Generic image-token pruning. On Qwen2.5-VL-3B, keeping 30% of tokens with FastV takes ScreenSpot-V2 from 81.5 to 38.6 (FocusUI).
• Perceptual-hash caching. A checked checkbox can change the answer while barely changing a perceptual hash.

Questions

1. Should this become the next phase of [Tracking]: System1-Omni roadmap: models, features, and agent integration #125, or stay a separate issue linked from it?
2. Which comes first: branch batching (section 2) or the cross-request cache (section 4)? I'd let section 1's traces decide.
3. Should batch-invariant execution be declared per model or per deployment?
4. As backend operations are added, should the ABI keep one global version?

I'll start with section 1, and with Open-Jev in section 2. The rest is open.

Feedback Period

One week, until 2026-10-26.

CC List

• @hsliuustc0106
• @MrDongsls
• @Levius-Fubuki: reuse in decider: reuse request prefixes with conservative auto selection #113, admission in decider: bound pending requests and enforce HTTP deadlines #149, the Cua-S1 profile in fix(cua-s1): match text reference and preserve merged compatibility #143
• @linear3735: section 4 generalizes JEV-VL's cache from jev_vl: add native inference and prefix caching #96
• @xiaoyu-xyz: bounded admission in runtime: optional bounded admission #109, CUDA CI in ci: compile every declared CUDA backend #101

Anything else

Before submitting a new issue

• Searched existing issues and PRs for prefill-only execution, scheduling, branch batching and cross-request prefix caching. Besides [RFC]: Reuse shared observations across decision questions — community help wanted #85 (request-local reuse) and [Tracking]: System1-Omni roadmap: models, features, and agent integration #125 (tracking), no dedicated issue was found.