Skip to content

cua-s1: replay language Graphs with GPU-resident vision features - #183

Draft
Levius-Fubuki wants to merge 2 commits into
InferMatrixLab:mainfrom
Levius-Fubuki:codex/cua-device-language-graph
Draft

Levius-Fubuki wants to merge 2 commits into
InferMatrixLab:mainfrom
Levius-Fubuki:codex/cua-device-language-graph

Conversation

@Levius-Fubuki

@Levius-Fubuki Levius-Fubuki commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Required human contributor answers

The contributor explicitly reviewed and confirmed these answers and PR necessity in the implementation task on 2026-10-25.

  1. What outcome are you trying to achieve?

    Enable language CUDA Graphs when Cua-S1 vision features stay on the GPU, preserving screenshot preparation, prompts, probabilities and decision semantics. The goal is lower repeated-decision latency where measurements justify it; capture cost and fallback remain explicit.

  2. What is the highest-leverage variable for that outcome?

    CPU operator submission and kernel-launch overhead in language forward. The primary outcome is warmed complete-request p50/p95, with first-capture cost and GPU memory reported separately. Correct current-input execution is a prerequisite.

  3. How will this work move that variable rather than create busywork?

    Combine the existing resident-feature path and existing multimodal Graph path by staging current input into owned scratch before replay. Use matched eager/Graph evidence; keep the opt-in and do not claim every shape improves.

  4. Who is the customer or user, and what do they actually need?

    GUI/browser-agent operators serving Cua-S1 and native-worker maintainers. They need predictable screenshot decisions with unchanged API behavior, current data after screenshot/shape changes, safe cache retirement and recoverable capture failure, backed by focused tests and reproducible evidence.

After answering the four questions: is a PR needed at all to achieve this outcome?

Yes. The contributor confirmed this scope and requested submission after validation. It requires the shared execution change, root regressions and recipe updates.

For a bug fix: link actual observed failure evidence.

This is a capability extension. The new GPU test fails on main because resident execution makes no capture, then passes here. The exact red/green logs are in the qualification package below.

Purpose

CUA_S1_DEVICE_FEATURES=1 previously always used language eager even with CUA_S1_GRAPH=1. Route host/device staging through one existing multimodal capture/replay helper. Current IDs, T/H/W positions and image rows are staged into owned scratch before replay; external producer pointers remain outside captures. Scratch growth, synchronization, bounded-cache retirement and eager fallback are preserved. No kernel, ABI, response, scheduler or default change.

The latest update is documentation only: explain complete-working-set warmup, language scratch invalidation, vision entry/byte limits, and the actual scope of the measurements. No universal acceleration or default-on recommendation.

Test Plan

Base: 05075d0f36e399e3a001ffedea400c666990cda9. Current head: 911faa0d22753be56c8bf513a23741dfa6d00593. Runtime is unchanged from a14e1027e7de7507e6ff82b3844a916ff19a26b9.

  • Current documentation update: format and strict MkDocs builds passed; full diff reviewed.

  • Historical runtime qualification on RTX4090/SM8.9, driver595.71.05,CUDA13.0,Rust1.98,ABI9: Graph tests cover changed IDs/features/T-H-W positions, producer release, host/device interleave, scratch growth and recording failure; Cua device checks with visionGraph0/1; real queued-copy drain/release and recovery; frontend forwarding/failure spots. Pinned BF16 merged language and original BF16 vision/FP32 adapter; all artifact hashes verified.

  • Original runtime local checks:360 workspace tests passed,0 failed,71 ignored, plus strict Clippy/release/docs/contract checks. These are historical checks of the runtime source, not newly executed GPU tests of the documentation commit.

Test Result

All normal responses match the eager control byte-for-byte. The original mixed matrix measured resident eager p50/p95=121.967/275.370ms, Graph124.894/276.195ms. Follow-up log alignment found5 timed language captures after scratch growth in its first round and15 timed vision captures(3/6/6). Those samples remain preserved and are not pure warmed replay.

The complete-warming follow-up excludes8 growth+8 all-shape stabilization+5 ordinary warmups, then measures3×8 requests on the same worker,concurrency1. Both entry and byte budgets must hold the working set: the1024-square image alone needs262537216 requested scratch bytes, so4 entries/256MiB churns this six-grid matrix. The explicitly configured8 entries/1024MiB fits it and removes timed captures; these are workload-specific controls, not changed defaults.

Same full-warming protocol, cache8/1024MiB p50 ms p95 ms sampled process maximum MiB
resident eager 116.891 271.109 10138
resident Graph 116.399 269.011 10202

The ~0.4% p50 difference in this bounded sample does not establish a stable general win. Graph removes submission work; GEMMs, CPU preprocessing/readout, feature checks/staging and synchronization remain. The feature supplies an opt-in execution capability; defaults remain off. Clock=POST through complete JSON/schema validation; p95=pooled nearest rank; memory is sampled nvidia-smi process maximum, not allocator-exact peak.

Demo / evidence

Follow-up qualification: SHA256 e63be7ff5d2b105d9e19d9a1d88e8ee5283d45c7d46c1a9933d4cf258ca507da. Contains original-cache/full-cache raw and excluded follow-up HTTP samples, source pins and immutable source-archive links, cache/capture traces, protocol and reproduction controls. The unchanged a14e102 runtime was measured before the documentation commit; no fresh GPU run of this SHA is claimed. Expanded prefix numerical records in that package concern PR184 and are explicitly separate.

Initial qualification remains unchanged: original archive SHA256 933ee5749a2f1e02041f1087a992301e9206da44c7bf5a7ba59daf9db7db6e15, with runtime red/green/failure logs, source snapshots and offline audit. Original screenshot pixels/operator details are omitted from the new public package. Video:N/A. General GUI task quality and other hardware support are not established.

Self-review

Full diff and claims reviewed; no remaining actionable runtime finding. Current PR:110 authored Rust/test changed lines,169 total changed lines. The update changes31 documentation lines only; raw run payload remains external. Agent review does not replace the contributor's human attestation; keep draft and leave the checklist for that review.

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant