Repository navigation
cua-s1: replay language Graphs with GPU-resident vision features - #183
Draft
Levius-Fubuki wants to merge 2 commits into
Draft
Levius-Fubuki wants to merge 2 commits into
Levius-Fubuki wants to merge 2 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
The contributor explicitly reviewed and confirmed these answers and PR necessity in the implementation task on 2026-10-25.
What outcome are you trying to achieve?
Enable language CUDA Graphs when Cua-S1 vision features stay on the GPU, preserving screenshot preparation, prompts, probabilities and decision semantics. The goal is lower repeated-decision latency where measurements justify it; capture cost and fallback remain explicit.
What is the highest-leverage variable for that outcome?
CPU operator submission and kernel-launch overhead in language forward. The primary outcome is warmed complete-request p50/p95, with first-capture cost and GPU memory reported separately. Correct current-input execution is a prerequisite.
How will this work move that variable rather than create busywork?
Combine the existing resident-feature path and existing multimodal Graph path by staging current input into owned scratch before replay. Use matched eager/Graph evidence; keep the opt-in and do not claim every shape improves.
Who is the customer or user, and what do they actually need?
GUI/browser-agent operators serving Cua-S1 and native-worker maintainers. They need predictable screenshot decisions with unchanged API behavior, current data after screenshot/shape changes, safe cache retirement and recoverable capture failure, backed by focused tests and reproducible evidence.
After answering the four questions: is a PR needed at all to achieve this outcome?
Yes. The contributor confirmed this scope and requested submission after validation. It requires the shared execution change, root regressions and recipe updates.
For a bug fix: link actual observed failure evidence.
This is a capability extension. The new GPU test fails on main because resident execution makes no capture, then passes here. The exact red/green logs are in the qualification package below.
Purpose
CUA_S1_DEVICE_FEATURES=1previously always used language eager even withCUA_S1_GRAPH=1. Route host/device staging through one existing multimodal capture/replay helper. Current IDs, T/H/W positions and image rows are staged into owned scratch before replay; external producer pointers remain outside captures. Scratch growth, synchronization, bounded-cache retirement and eager fallback are preserved. No kernel, ABI, response, scheduler or default change.The latest update is documentation only: explain complete-working-set warmup, language scratch invalidation, vision entry/byte limits, and the actual scope of the measurements. No universal acceleration or default-on recommendation.
Test Plan
Base:
05075d0f36e399e3a001ffedea400c666990cda9. Current head:911faa0d22753be56c8bf513a23741dfa6d00593. Runtime is unchanged froma14e1027e7de7507e6ff82b3844a916ff19a26b9.Current documentation update: format and strict MkDocs builds passed; full diff reviewed.
Historical runtime qualification on RTX4090/SM8.9, driver595.71.05,CUDA13.0,Rust1.98,ABI9: Graph tests cover changed IDs/features/T-H-W positions, producer release, host/device interleave, scratch growth and recording failure; Cua device checks with visionGraph0/1; real queued-copy drain/release and recovery; frontend forwarding/failure spots. Pinned BF16 merged language and original BF16 vision/FP32 adapter; all artifact hashes verified.
Original runtime local checks:360 workspace tests passed,0 failed,71 ignored, plus strict Clippy/release/docs/contract checks. These are historical checks of the runtime source, not newly executed GPU tests of the documentation commit.
Test Result
All normal responses match the eager control byte-for-byte. The original mixed matrix measured resident eager p50/p95=121.967/275.370ms, Graph124.894/276.195ms. Follow-up log alignment found5 timed language captures after scratch growth in its first round and15 timed vision captures(3/6/6). Those samples remain preserved and are not pure warmed replay.
The complete-warming follow-up excludes8 growth+8 all-shape stabilization+5 ordinary warmups, then measures3×8 requests on the same worker,concurrency1. Both entry and byte budgets must hold the working set: the1024-square image alone needs262537216 requested scratch bytes, so4 entries/256MiB churns this six-grid matrix. The explicitly configured8 entries/1024MiB fits it and removes timed captures; these are workload-specific controls, not changed defaults.
The ~0.4% p50 difference in this bounded sample does not establish a stable general win. Graph removes submission work; GEMMs, CPU preprocessing/readout, feature checks/staging and synchronization remain. The feature supplies an opt-in execution capability; defaults remain off. Clock=POST through complete JSON/schema validation; p95=pooled nearest rank; memory is sampled nvidia-smi process maximum, not allocator-exact peak.
Demo / evidence
Follow-up qualification: SHA256
e63be7ff5d2b105d9e19d9a1d88e8ee5283d45c7d46c1a9933d4cf258ca507da. Contains original-cache/full-cache raw and excluded follow-up HTTP samples, source pins and immutable source-archive links, cache/capture traces, protocol and reproduction controls. The unchanged a14e102 runtime was measured before the documentation commit; no fresh GPU run of this SHA is claimed. Expanded prefix numerical records in that package concern PR184 and are explicitly separate.Initial qualification remains unchanged: original archive SHA256
933ee5749a2f1e02041f1087a992301e9206da44c7bf5a7ba59daf9db7db6e15, with runtime red/green/failure logs, source snapshots and offline audit. Original screenshot pixels/operator details are omitted from the new public package. Video:N/A. General GUI task quality and other hardware support are not established.Self-review
Full diff and claims reviewed; no remaining actionable runtime finding. Current PR:110 authored Rust/test changed lines,169 total changed lines. The update changes31 documentation lines only; raw run payload remains external. Agent review does not replace the contributor's human attestation; keep draft and leave the checklist for that review.