Repository navigation
feat(server): implement multi-adapter LoRA request and hot-swap semantics #1439
Copy link
Copy link
Closed
Labels
area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadatapriority:mediumMedium priorityMedium prioritystatus:doneCompletedCompletedtype:enhancementNew features, capabilities, or significant additionsNew features, capabilities, or significant additions
Description
Parent
- epic(server): close llama-server b10621 compatibility gaps #1431
- Related adapter validation work: fix(lora): refuse adapters whose tensors do not map onto the base model instead of serving the base weights #1328
Problem
b10621 supports multiple LoRA adapters, per-request scaling, adapter inventory, and /lora-adapters hot-swap operations. mlxcel's current single-adapter surface does not provide equivalent loading, isolation, or runtime management semantics.
Scope
- Support multiple named adapters with deterministic load order and scale parsing.
- Add per-request adapter selection/scaling without mutating other concurrent requests. (The inert value that resolves to the fused configuration is accepted; a different per-request set is refused with a diagnostic. True per-request selection needs the unfused runtime-LoRA forward path.)
- Implement compatible GET and POST
/lora-adaptersinventory/update behavior. (GET is full parity; POST carries upstream's request contract and acknowledges the configuration in force, refusing scale changes.) - Define atomic activation, rollback, memory accounting, and active-request behavior for hot swaps. (Deferred with the runtime-swap path; a refused swap trivially leaves the prior set active, which the tests pin.)
- Validate base-model compatibility and reject duplicate names, incompatible tensors, NaN/inf scales, and unsafe paths. (NaN/inf/non-numeric scales and unreadable adapter directories fail startup; incompatible-tensor strictness beyond today's shape errors remains fix(lora): refuse adapters whose tensors do not map onto the base model instead of serving the base weights #1328's scope.)
- Integrate with the model-pool/router design without duplicating adapter state. (The specification lives once in
ServerConfig; router-mode children deliberately do not inherit adapters, since an adapter is base-model-specific.)
Acceptance criteria
- Two adapters can be loaded, independently selected, combined, rescaled, disabled, and inspected. (Loaded, combined, rescaled via
--lora-scaled, disabled via scale 0 or--lora-init-without-apply, inspected via GET; independent per-request selection stays deferred with the runtime-swap path.) - Concurrent requests with different adapter sets do not leak configuration or outputs into each other. (By construction: only one fused set exists and any request asking for a different one is refused, never served on other weights.)
- Failed hot swaps leave the prior adapter set active.
- Route schemas, status codes, authentication, and errors match the claimed b10621 surface.
- Adapter paths cannot escape configured roots and memory limits are enforced before activation. (Adapter paths are operator CLI input validated at startup; no API accepts a path, so there is no client-controlled path surface. Activation happens inside the model load, covered by the existing memory preflight.)
Validation
- Run real-checkpoint smoke tests with at least one compatible LoRA and compare base, enabled, disabled, and scaled outputs.
Remaining after the first implementation PR
Deferred manifest entries staying on this issue: POST /lora-adapters (runtime scale changes are refused; b10621 swaps at runtime), field:lora (only the inert value is accepted), and --lora-init-without-apply (adapters are validated and reported at scale 0.0 but can never be applied afterwards). All three trace to one root: mlxcel fuses adapters into the weights at load, so a true hot swap needs an unfused runtime-LoRA forward path (per-family adapter layers) or a model-pool reload flow built on #1438.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)Generation, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadataModel architectures, weights, loading, metadatapriority:mediumMedium priorityMedium prioritystatus:doneCompletedCompletedtype:enhancementNew features, capabilities, or significant additionsNew features, capabilities, or significant additions
Activity
test(server): add pinned b10621 compatibility manifest and gates
docs: add technical report for PR #1497
feat(server): implement multi-adapter LoRA request and hot-swap seman…
inureyes commentedon Aug 28, 2026
PR #1497 merged the multi-adapter surface to the fused-weights boundary:
--loracomma lists and--lora-scaled FNAME:SCALE,...fuse in deterministic order with per-adapter scaling and startup NaN/inf/path validation,GET /lora-adaptersreports b10621's inventory shape, andPOST /lora-adaptersplus the native per-requestlorafield carry upstream's resolution rule while acknowledging only the configuration in force. Manifest entries flipped tosupported:--lora,--lora-scaled,GET /lora-adapters.This issue stays open for the deferred remainder, all tracing to one root: mlxcel fuses adapters into the weights at load, so runtime scale changes (
POST /lora-adapters), non-inert per-requestloravalues, and applying--lora-init-without-applyadapters later all need an unfused runtime-LoRA forward path or a model-pool reload flow built on #1438. #1328 (strict adapter-tensor validation) remains open and untouched.2 remaining items
feat(server): add the unfused runtime-LoRA forward path
feat(compat): serve b10621 LoRA adapters unfused by default
feat(server): add the unfused runtime-LoRA forward path
feat(compat): serve b10621 LoRA adapters unfused by default
fix(server): report b10621 tokens_cached and accept verbose on the na…
feat(server): serve b10621 LoRA adapters unfused with live scales (la…
test(server): add pinned b10621 compatibility manifest and gates
feat(server): implement multi-adapter LoRA request and hot-swap seman…