Skip to content

[2/4] offline: prepare target features through the capture plugin - #1064

Open
jiapingW wants to merge 4 commits into
mainfrom
split/capture-02-offline-client
Open

jiapingW wants to merge 4 commits into
mainfrom
split/capture-02-offline-client

Conversation

@jiapingW

@jiapingW jiapingW commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Offline preparation currently loads SGLang's ModelRunner and maintains private scheduler/model patches inside the client. Replace that implementation with a CPU client using HTTP capture requests and Mooncake, so prepare_hidden_states.py no longer imports SGLang or loads target model weights.

--server-urls identifies independently launched plugin servers. Before sending prompts, the client validates their model, capture method and ordered layer IDs using part 1's metadata. Preserve the algorithm-owned feature schemas, compression, stable sample IDs, resume and CPU sharding; publish files atomically and release feature objects after writers finish. Remove the old local inference backend and replace its tests. Move TP/attention settings to the server in the offline examples and AMD walkthrough.

Validation: full pre-commit run --all-files passed. H20 offline/client/schema/architecture regression: 44 passed, 1 skipped (standalone Mooncake-master opt-in). Adds a live plugin-to-files numerical/resume test to the existing GPU gate. The complete stack previously passed that live test and the three other GPU capture tests.

Includes the shared CI dependency-test fix from #1063 via forward merges. All 8 CI dependency tests passed on Linux for this layer; the feature diff is unchanged.

Review and merge order:

  1. [1/4] capture: prepare the native plugin runtime and CI server
  2. [2/4] offline: prepare target features through the capture plugin
  3. [3/4] capture: remove legacy SGLang patch support
  4. [4/4] build: align training with the pinned SGLang revision

Each PR compares against the previous layer. Shared follow-up fixes are merged forward without rewriting published history. After merging a predecessor, retarget the next PR to main; if using squash merges, rebase the remaining stack before retargeting. The GPU workflow filters PRs targeting main, so later layers receive that CI run when retargeted. Lint runs on every layer.

This stack replaces #1062. Its combined feature changes match #1062 plus the lint fix; subsequent CI regression fixes are shared across the stack.

Base automatically changed from split/capture-01-plugin-runtime to main November 5, 2026 09:04
@curnane-lab

Copy link
Copy Markdown
Collaborator

If HTTP times out before publication finishes, cleanup can run before the objects exist. The server can then publish them with nobody left to free them. Can we handle this on the server and add a delayed-publication test?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants