Skip to content

feat(server): implement multi-adapter LoRA request and hot-swap semantics #1439

Closed

Description

@inureyes

Parent

Problem

b10621 supports multiple LoRA adapters, per-request scaling, adapter inventory, and /lora-adapters hot-swap operations. mlxcel's current single-adapter surface does not provide equivalent loading, isolation, or runtime management semantics.

Scope

  • Support multiple named adapters with deterministic load order and scale parsing.
  • Add per-request adapter selection/scaling without mutating other concurrent requests. (The inert value that resolves to the fused configuration is accepted; a different per-request set is refused with a diagnostic. True per-request selection needs the unfused runtime-LoRA forward path.)
  • Implement compatible GET and POST /lora-adapters inventory/update behavior. (GET is full parity; POST carries upstream's request contract and acknowledges the configuration in force, refusing scale changes.)
  • Define atomic activation, rollback, memory accounting, and active-request behavior for hot swaps. (Deferred with the runtime-swap path; a refused swap trivially leaves the prior set active, which the tests pin.)
  • Validate base-model compatibility and reject duplicate names, incompatible tensors, NaN/inf scales, and unsafe paths. (NaN/inf/non-numeric scales and unreadable adapter directories fail startup; incompatible-tensor strictness beyond today's shape errors remains fix(lora): refuse adapters whose tensors do not map onto the base model instead of serving the base weights #1328's scope.)
  • Integrate with the model-pool/router design without duplicating adapter state. (The specification lives once in ServerConfig; router-mode children deliberately do not inherit adapters, since an adapter is base-model-specific.)

Acceptance criteria

  • Two adapters can be loaded, independently selected, combined, rescaled, disabled, and inspected. (Loaded, combined, rescaled via --lora-scaled, disabled via scale 0 or --lora-init-without-apply, inspected via GET; independent per-request selection stays deferred with the runtime-swap path.)
  • Concurrent requests with different adapter sets do not leak configuration or outputs into each other. (By construction: only one fused set exists and any request asking for a different one is refused, never served on other weights.)
  • Failed hot swaps leave the prior adapter set active.
  • Route schemas, status codes, authentication, and errors match the claimed b10621 surface.
  • Adapter paths cannot escape configured roots and memory limits are enforced before activation. (Adapter paths are operator CLI input validated at startup; no API accepts a path, so there is no client-controlled path surface. Activation happens inside the model load, covered by the existing memory preflight.)

Validation

  • Run real-checkpoint smoke tests with at least one compatible LoRA and compare base, enabled, disabled, and scaled outputs.

Remaining after the first implementation PR

Deferred manifest entries staying on this issue: POST /lora-adapters (runtime scale changes are refused; b10621 swaps at runtime), field:lora (only the inert value is accepted), and --lora-init-without-apply (adapters are validated and reported at scale 0.0 but can never be applied afterwards). All three trace to one root: mlxcel fuses adapters into the weights at load, so a true hot swap needs an unfused runtime-LoRA forward path (per-family adapter layers) or a model-pool reload flow built on #1438.

Activity

  1. added
    status:backlogIn the backlog, not yet ready
    type:enhancementNew features, capabilities, or significant additions
    area:modelsModel architectures, weights, loading, metadata
    area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)
    on Aug 26, 2026
  2. added
    status:backlogIn the backlog, not yet ready
    and removed on Aug 28, 2026
  3. inureyes commented on Aug 28, 2026

    @inureyes
    MemberAuthor

    PR #1497 merged the multi-adapter surface to the fused-weights boundary: --lora comma lists and --lora-scaled FNAME:SCALE,... fuse in deterministic order with per-adapter scaling and startup NaN/inf/path validation, GET /lora-adapters reports b10621's inventory shape, and POST /lora-adapters plus the native per-request lora field carry upstream's resolution rule while acknowledging only the configuration in force. Manifest entries flipped to supported: --lora, --lora-scaled, GET /lora-adapters.

    This issue stays open for the deferred remainder, all tracing to one root: mlxcel fuses adapters into the weights at load, so runtime scale changes (POST /lora-adapters), non-inert per-request lora values, and applying --lora-init-without-apply adapters later all need an unfused runtime-LoRA forward path or a model-pool reload flow built on #1438. #1328 (strict adapter-tensor validation) remains open and untouched.

  4. 2 remaining items

  5. self-assigned this
    on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadatapriority:mediumMedium prioritystatus:doneCompletedtype:enhancementNew features, capabilities, or significant additions

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions