refactor: share native execution admission and dispatch

.agents/skills/add-new-model/SKILL.md

@@ -44,16 +44,18 @@ redesign serving infrastructure or introduce a general model framework.
 | `src/models/<model>/` | Separate model-specific processors and batch adapters, plus executor checkpoint loading, weights, forward passes, learned heads, device state, and kernel selection. A native Rust worker may live in `native/`, following existing models. |
 | `src/models/<family>/` | Model-family execution shared by implementations that actually need it. Reuse compatible code; extract a common implementation when there is a second user. |
 | `src/backends/cuda/` or `src/backends/metal/` | Hardware operations and kernels, native host glue, build scripts, and backend integration. GPU processing transforms or tensor packing retain their processor/batcher ownership. |
+| `src/runtime/` | Shared native admission and dispatch policy. Reuse the serial scheduler for single-executor work; keep model layouts and device state in model adapters/executors. |
 | `src/frontend/` | Transport and existing worker adapters when integration requires them. Keep model-specific input/output semantics in processors rather than the transport layer. |
 | Root `Cargo.toml` and affected crate manifests | Workspace membership, dependencies, binaries, and explicit test registration. Update `Cargo.lock` when dependencies change. |
 | `recipe/<model>/` | Checkpoint preparation/export scripts, setup and launch instructions, and example requests. Reusable runtime implementation belongs in `src/`. |
 | `tests/<model>/` and `tests/<shared-component>/` | All test bodies, test helpers, and fixtures, including contract, tokenizer, checkpoint, HTTP, and kernel tests. |
 
-The shared processing/scheduling runtime is planned. Integrate with available
-worker interfaces while separating processors and batch adapters from forward
-code; adding a model alone does not require implementing that runtime. When
-shared scheduling is implemented, it owns queues and admission policy, while
-models declare supported layouts and compatibility constraints. The native
+The [native runtime](../../../src/runtime/README.md) provides serial FIFO admission
+and blocking dispatch per loaded executor. Integrate with available worker
+interfaces while separating processors and batch adapters from forward code.
+Models declare their execution unit, layouts and compatibility constraints;
+shared runtime code owns admission policy. Processing orchestration, queue/token
+budgets and dynamic batching remain planned. The native
 target uses Rust host orchestration and bindings with CUDA/Metal device kernels.
 
 **Do not add test code or fixtures under `src/`.** This includes inline Rust test

.agents/skills/precheck-pr/SKILL.md

@@ -33,9 +33,9 @@ user instructions for those actions.
 - Check ownership against the architecture contracts: transport, independent
   processing and scheduling, model-specific adapters and executors, and hardware
   operations. In the native target, Rust orchestrates host work and dispatch;
-  CUDA/Metal execute device kernels. Shared runtime layers remain planned until
-  implemented; existing worker pipelines are not proof of dynamic batching.
-  Consult the relevant `src/frontend/`, `src/models/`, `src/backends/`, or
+  CUDA/Metal execute device kernels. Serial runtime admission and dispatch are
+  implemented; further orchestration and dynamic batching remain planned.
+  Consult the relevant `src/frontend/`, `src/runtime/`, `src/models/`, `src/backends/`, or
   `recipe/` documentation for the changed component's contract and current status.
 - For frontend changes, inspect affected request validation, error handling,
   cancellation, and response behavior, including the model-worker boundary.

.agents/skills/self-review/SKILL.md

@@ -18,7 +18,10 @@ semantics and distinguish target layers from current worker capabilities.
 
 For processing or batching changes, trace prepared inputs through executor
 layouts and output reconstruction. Apply the architecture contract's batching,
-numerical, and lifetime invariants to the affected model.
+numerical, and lifetime invariants to the affected model. For shared runtime
+admission changes, check FIFO execution units, cancellation before dispatch,
+permit/resource retention after dispatch, and release on errors or panics against
+the [runtime contract](../../../src/runtime/README.md).
 
 ## Conditional A/B checks
 

.agents/skills/system1-omni-review/SKILL.md

@@ -20,10 +20,10 @@ baseline; target code and configuration determine current paths and commands.
 
 ## Review contracts
 
-- Check separate transport, processing, scheduling, model-executor, and backend ownership against the architecture contracts. Model-specific processors and batch adapters preserve semantics; shared runtime code owns orchestration and scheduling when implemented. Rust owns native host orchestration and bindings/dispatch; CUDA/Metal supply device operations, including optional processing transforms and packing. Shared runtime layers are planned, so assess current workers against their implemented contracts. CUDA and Metal need not have identical internals. A directory or pass-through HTTP support does not prove native support or batching.
+- Check separate transport, processing, scheduling, model-executor, and backend ownership against the architecture contracts. Model-specific processors and batch adapters preserve semantics; shared runtime code owns orchestration and scheduling when implemented. Rust owns native host orchestration and bindings/dispatch; CUDA/Metal supply device operations, including optional processing transforms and packing. Serial runtime admission and dispatch are implemented; further orchestration and batching remain planned. CUDA and Metal need not have identical internals. A directory or pass-through HTTP support does not prove native support or batching.
 - For `src/frontend/`, inspect exact envelope/body and end-to-end header preservation, hop-by-hop filtering, backend path prefix/query handling, timeout/status mapping, health forwarding, cancellation and shutdown. The configured backend bypasses proxies, follows no redirects and retries no requests; changes need transport regression evidence, not a model benchmark.
 - For models, trace serialization → prompt/template → token IDs → one-pass prefill → scoring → response. Preserve the specific model's contract; do not impose Cua-S1's 26-letter choice restriction on a new model that implements score/noul. Verify pinned checkpoint/adapter/tokenizer/configuration identity, shapes/dtypes and readiness after real initialization.
-- For processing or batching changes, apply the architecture contract's compatibility, reconstruction, numerical, and lifetime invariants to the affected model. Require executor/kernel evidence for batch capability.
+- For processing or batching changes, apply the architecture contract's compatibility, reconstruction, numerical, and lifetime invariants to the affected model. For admission changes, inspect the [runtime contract](../../../src/runtime/README.md), execution-unit granularity, cancellation before/after dispatch and permit/resource lifetimes. Require executor/kernel evidence for batch capability.
 - For native/FFI/CUDA changes, inspect Rust and C ABI in lockstep, buffer and graph lifetimes, stream/device ownership, sizes/overflow, BF16 rounding locations, and error propagation/recovery. Compile success and ignored GPU tests do not validate kernels. Test changed inputs, reuse, growth, eviction and failure recovery when graph/cache behavior changes.
 - For performance or numerical claims, freeze baseline/candidate, hardware/toolchain, checkpoint, input manifest, precision, variables, tolerances and budget before comparing. Separate quality/parity from speed, native inference from frontend overhead, and cold/capture costs from warmed latency. Preserve errors and denominators; never relax tolerances or discard failures to produce a winner.
 

CONTRIBUTING.md

@@ -14,16 +14,17 @@ implementations need it.
 Keep processors and batch adapters separate from the model's forward
 implementation, even when they live in the same model directory. The README's
 target architecture shares processing orchestration and scheduling across
-models; these layers are planned, while current worker pipelines remain
-model-specific. Model executors own weights, forward passes, learned heads,
-and device state.
+models. Native workers already reuse
+[serial admission and dispatch](src/runtime/README.md); processing orchestration
+and GPU batching remain planned. Model executors own weights, forward passes,
+learned heads, and device state.
 
 For native implementations, Rust owns host processing, scheduling, model
 orchestration, and backend bindings/dispatch. CUDA/Metal provide device
 operations used by models or processing modules. Document processor inputs,
 executor layouts and batch constraints, output reconstruction, and state/buffer
 lifetimes alongside the existing API and numerical contract. Keep current worker
-integration usable while shared runtime components remain unimplemented.
+integration usable while further shared runtime components remain unimplemented.
 
 All test bodies, test helpers and fixtures belong in the repository-level
 `tests/` tree. Do not add inline test bodies or crate-local `tests/` directories

Cargo.toml

@@ -1,3 +1,3 @@
 [workspace]
-members = ["src/frontend", "src/models/cua_s1/native", "src/models/qwen3_5/native", "src/models/open_jev/native", "src/models/laya"]
+members = ["src/frontend", "src/runtime", "src/models/cua_s1/native", "src/models/qwen3_5/native", "src/models/open_jev/native", "src/models/laya"]
 resolver = "3"

README.md

@@ -47,8 +47,8 @@ shared CUDA kernels in this repository.
   model workers.
 - **Model-owned execution.** Model executors own weights, forward passes,
   learned heads, device state, and kernel selection. Native workers have separate
-  processing and executor modules; shared processing orchestration and scheduling
-  are planned.
+  processing and executor modules, with shared FIFO admission and blocking
+  dispatch in the [native runtime](src/runtime/README.md).
 - **Native CUDA workers.** The Cua-S1 4B 0.2 `text` adapter and
   Open-Jev-27B-v1.1 run as native workers with shared CUDA kernels. Cua-S1
   also has a Python worker that serves as the correctness reference.
@@ -65,10 +65,11 @@ Share processing and scheduling; let each model own its execution.
 
 ![System1-Omni target architecture: Rust frontend, independent processing and batching layers, model executors, and CUDA and Metal backends](docs/assets/architecture.svg)
 
-The diagram shows the **target architecture**, not an implemented shared runtime.
-Today the frontend forwards HTTP requests to separately running workers, whose
-model-specific handlers coordinate independent processors and executors. Shared
-processing orchestration, scheduling, and dynamic batching are planned.
+The diagram shows the **target architecture**. Today the frontend forwards HTTP
+requests to separately running workers, whose handlers coordinate independent
+processors and executors. Both native workers use shared FIFO admission and
+blocking dispatch per loaded executor. Processing orchestration, batch budgets,
+compatibility grouping and dynamic batching remain planned.
 
 | Layer | Responsibility | Native target implementation |
 | --- | --- | --- |
@@ -105,14 +106,15 @@ repository root.
 | Directory | Responsibility |
 | --- | --- |
 | [`src/frontend/`](src/frontend/) | Rust serving code, Python worker adapters, and the small engine interface. |
+| [`src/runtime/`](src/runtime/README.md) | Shared native execution admission and blocking dispatch. |
 | [`src/models/`](src/models/) | Model contracts, existing worker pipelines, and model executors, including the shared Qwen3.5/3.8 prefill implementation. |
 | [`src/backends/cuda/`](src/backends/cuda/) | NVIDIA GPU operations and kernel integration. |
 | [`src/backends/metal/`](src/backends/metal/) | Apple GPU operations and kernel integration. |
 | [`recipe/`](recipe/) | Model setup instructions, launch commands, configuration examples, and example requests. |
 | [`docs/`](docs/) | Project documentation and architecture assets. |
 
-The frontend, both native workers, their shared Qwen3.5/3.8 prefill
-implementation and the Laya checkpoint reader are Cargo workspace members.
+The frontend, native runtime, both native workers, their shared Qwen3.5/3.8
+prefill implementation and the Laya checkpoint reader are Cargo workspace members.
 The other model and backend directories currently document planned work;
 they do not prescribe process boundaries.
 
@@ -158,8 +160,9 @@ requests and a matched comparison with raw HF Transformers and OpenJev-Fast.
 ## Roadmap
 
 The current focus is the native Cua-S1 and Open-Jev CUDA workers and the serving
-benchmark harness. Planned work includes the shared processing and scheduling
-layer shown above, bounded dynamic batching with batch-capable executors,
+benchmark harness. Planned work extends the shared runtime with processing
+orchestration, admission budgets, compatibility grouping and bounded dynamic
+batching with batch-capable executors,
 the in-repository LAYA model engine, additional model engines and GPU backends
 including Metal, and per-model
 performance measurements as implementations are added and validated.

docs/architecture.md

@@ -11,8 +11,10 @@ to separately running workers. Cua-S1 and Open-Jev have native Rust/CUDA workers
 that share the [Qwen3.5/3.8 executor](../src/models/qwen3_5/native/). Their
 model-specific workers coordinate independent processor and executor modules
 through `prepare` → `execute` → `finish`. The shared Qwen executor accepts one
-prompt per forward call; shared processing orchestration, scheduling, and
-dynamic batching are planned.
+prompt per forward call. Both workers use the
+[native runtime](../src/runtime/README.md) for FIFO admission and blocking dispatch
+per loaded executor. Shared processing orchestration, batch budgets,
+compatibility grouping and dynamic batching are planned.
 
 The native workers currently compute their decision heads on the CPU after
 downloading the final hidden state. GPU head execution belongs to the target
@@ -22,19 +24,22 @@ also planned; Python workers retain their documented reference/serving roles.
 ## Native worker boundaries
 
 Both native workers separate `processing.rs` from `executor.rs`; `engine.rs`
-assembles them from the checkpoint, and the HTTP handler coordinates the three
-stages. Preparation validates the entire request before any forward call and
-returns executor inputs plus a response context. The context retains question
+assembles them with a `SerialScheduler` per loaded executor, and the HTTP handler
+coordinates the three stages. Preparation validates the entire request before
+any forward call and returns executor inputs plus a response context. The context retains question
 and candidate identity, usage, and response metadata outside the executor.
 
 | Worker | Prepared executor inputs | Executor outputs | Response finishing |
 | --- | --- | --- | --- |
 | Cua-S1 | One unpadded token-ID vector and option count per question, in request order. | One FP32 answer-letter logit vector per question. | Per-question softmax, choice/confidence, ordered answers, and token usage. |
 | Open-Jev | Token-ID vectors grouped by question, then independent candidate, in request order. | One FP32 learned scalar per candidate in the same grouping. | Add the `noul` false logit of zero, calibrate across each complete question, and restore typed answers, usage, and metadata. |
 
-These input collections are serial work, not GPU batches. Cua-S1 retains its
-per-question model lock; Open-Jev retains its request-wide lock. Executors own
-the loaded Qwen model and CPU head weights, preserving FP64 accumulation and
+These input collections are serial work, not GPU batches. Shared runtime
+admission precedes blocking dispatch: Cua-S1 admits one question forward at a
+time; Open-Jev admits one complete request. Cua-S1's CPU letter projection stays
+outside admission; Open-Jev's scalar heads remain inside its request unit. The
+model mutexes guard mutable state, retaining per-question/request granularity.
+Executors own the loaded Qwen model and CPU head weights, preserving FP64 accumulation and
 the existing FP32 rounding and bias order. Finishing checks output cardinality
 before reconstruction. HTTP validation, error status/body conventions, and real
 warmup before readiness remain model-specific and unchanged.
@@ -82,6 +87,13 @@ probabilities must not be normalized across unrelated questions or requests.
 
 ## Scheduling and batching contracts
 
+The implemented serial scheduler limits admitted execution to one unit per loaded
+executor, with FIFO waiting. Engines own scheduler instances; model-specific
+execution adapters declare the unit and submit owned closures. Waiting is async,
+so queued requests do not occupy blocking threads waiting for a model lock.
+Schedulers are local to each worker/executor; this does not coordinate separate
+processes. Queue-length limits and token budgets are not implemented yet.
+
 - Batch only work accepted by the same loaded executor, with compatible
   checkpoint/adapter identity, device, dtype, and input layout. Models declare
   additional constraints and limits; the scheduler owns queue and admission
@@ -107,15 +119,20 @@ ownership even when their kernels live under `src/backends/`.
 Keep Rust declarations and the native ABI in sync, and rebuild all consumers
 when the ABI changes. Buffer handles and captured graphs must outlive queued
 device operations; synchronize before reading host results or reclaiming
-storage. Cancellation or timeout does not authorize freeing buffers still in
-use by submitted work. Keep readiness tied to successful initialization and
-the worker's real warmup; extraction into shared layers preserves that behavior.
+storage. Cancellation while waiting for admission removes the queued caller
+without dispatch. Once dispatched, the runtime retains admission and captured executor
+resources until the blocking closure completes, even if its caller is cancelled.
+The executor must synchronize device work before returning; cancellation or
+timeout does not authorize freeing buffers still in use by submitted work. Keep
+readiness tied to successful initialization and the worker's real warmup;
+extraction into shared layers preserves that behavior.
 
 ## Integration and validation
 
 Keep model-specific processors and batch adapters separate from forward
 implementations, even when colocated under `src/models/<model>/`. Reuse the
-current worker interface while shared runtime components are unimplemented.
+current worker interface and serial runtime while further runtime components
+remain unimplemented.
 Extract common orchestration when implementing that layer, with explicit scope
 and current consumers; adding a model alone does not require a new framework.
 

mkdocs.yml

@@ -27,6 +27,7 @@ exclude_docs: |
   !/docs/assets/**
   !/recipe/**/*.md
   !/src/frontend/README.md
+  !/src/runtime/README.md
   !/assets/**
   **/.*/**
 
@@ -49,6 +50,7 @@ nav:
   - Architecture and contracts: docs/architecture.md
   - Supported models and hardware: docs/supported-models.md
   - Frontend: src/frontend/README.md
+  - Native runtime: src/runtime/README.md
   - Recipes:
       - recipe/README.md
       - Laya text worker: recipe/laya/README.md