How save Lora Weight into normal format

[hanguangmic]

I used https://github.com/NVIDIA-NeMo/RL/blob/main/examples/configs/recipes/llm/grpo-nanov3-30BA3B-2n8g-megatron-lora.yaml here to train the model and got the dist torch lora weight. but how to make it works for the inference?
The following is my peft config

{'target': 'megatron.bridge.peft.lora.LoRA',
'a2a_experimental': False,
'alpha': 64,
'canonical_mapping': {},
'dim': 32,
'dropout': 0.0,
'dropout_position': 'post',
'exclude_modules': ['out_proj'],
'lora_A_init_method': 'xavier',
'lora_B_init_method': 'zero',
'lora_dtype': 'None',
'params_to_save': {'decoder.layers.0.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.0.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.1.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.1.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.1.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.1.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.10.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.10.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.10.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.10.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.11.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.11.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.12.self_attention.linear_proj.adapter.linear_in.weight',
'decoder.layers.12.self_attention.linear_proj.adapter.linear_out.weight',
'decoder.layers.12.self_attention.linear_qkv.adapter.linear_in.weight',
'decoder.layers.12.self_attention.linear_qkv.adapter.linear_out.weight',
'decoder.layers.13.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.13.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.13.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.13.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.14.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.14.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.15.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.15.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.15.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.15.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.16.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.16.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.17.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.17.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.17.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.17.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.18.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.18.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.19.self_attention.linear_proj.adapter.linear_in.weight',
'decoder.layers.19.self_attention.linear_proj.adapter.linear_out.weight',
'decoder.layers.19.self_attention.linear_qkv.adapter.linear_in.weight',
'decoder.layers.19.self_attention.linear_qkv.adapter.linear_out.weight',
'decoder.layers.2.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.2.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.20.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.20.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.20.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.20.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.21.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.21.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.22.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.22.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.22.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.22.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.23.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.23.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.24.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.24.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.24.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.24.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.25.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.25.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.26.self_attention.linear_proj.adapter.linear_in.weight',
'decoder.layers.26.self_attention.linear_proj.adapter.linear_out.weight',
'decoder.layers.26.self_attention.linear_qkv.adapter.linear_in.weight',
'decoder.layers.26.self_attention.linear_qkv.adapter.linear_out.weight',
'decoder.layers.27.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.27.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.27.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.27.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.28.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.28.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.29.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.29.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.29.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.29.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.3.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.3.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.3.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.3.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.30.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.30.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.31.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.31.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.31.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.31.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.32.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.32.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.33.self_attention.linear_proj.adapter.linear_in.weight',
'decoder.layers.33.self_attention.linear_proj.adapter.linear_out.weight',
'decoder.layers.33.self_attention.linear_qkv.adapter.linear_in.weight',
'decoder.layers.33.self_attention.linear_qkv.adapter.linear_out.weight',
'decoder.layers.34.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.34.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.34.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.34.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.35.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.35.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.36.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.36.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.36.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.36.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.37.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.37.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.38.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.38.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.38.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.38.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.39.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.39.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.4.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.4.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.40.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.40.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.40.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.40.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.41.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.41.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.42.self_attention.linear_proj.adapter.linear_in.weight',
'decoder.layers.42.self_attention.linear_proj.adapter.linear_out.weight',
'decoder.layers.42.self_attention.linear_qkv.adapter.linear_in.weight',
'decoder.layers.42.self_attention.linear_qkv.adapter.linear_out.weight',
'decoder.layers.43.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.43.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.43.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.43.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.44.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.44.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.45.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.45.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.45.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.45.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.46.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.46.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.47.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.47.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.47.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.47.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.48.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.48.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.49.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.49.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.49.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.49.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.5.self_attention.linear_proj.adapter.linear_in.weight',
'decoder.layers.5.self_attention.linear_proj.adapter.linear_out.weight',
'decoder.layers.5.self_attention.linear_qkv.adapter.linear_in.weight',
'decoder.layers.5.self_attention.linear_qkv.adapter.linear_out.weight',
'decoder.layers.50.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.50.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.51.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.51.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.51.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.51.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.6.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.6.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.6.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.6.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.7.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.7.mixer.in_proj.adapter.linear_out.weight',
'decoder.layers.8.mlp.shared_experts.linear_fc1.adapter.linear_in.weight',
'decoder.layers.8.mlp.shared_experts.linear_fc1.adapter.linear_out.weight',
'decoder.layers.8.mlp.shared_experts.linear_fc2.adapter.linear_in.weight',
'decoder.layers.8.mlp.shared_experts.linear_fc2.adapter.linear_out.weight',
'decoder.layers.9.mixer.in_proj.adapter.linear_in.weight',
'decoder.layers.9.mixer.in_proj.adapter.linear_out.weight',
'output_layer.adapter.linear_in.weight',
'output_layer.adapter.linear_out.weight'},
'target_modules': []}

[hanguangmic]

can anyone help, how to convert torch dist model into lora format which can be used in vllm inference

[hanguangmic]

wait for solution

[yuki-97]

hi @RayenTian , could you help on this? and FYI there's some related discussion at #2173.

[RayenTian]

Hi, @hanguangmic. Thank you for raising this issue and for your patience — this is really helpful feedback!

The bug mentioned in this comment (KeyError: decoder.layers.0.mixer.dt_bias from model not in state dict) does exist, and we have addressed it in PR #2234. The PR is still being polished before merging, but you are welcome to try it out in the meantime. Below is a complete end-to-end workflow to get you started.

---

Step 1: Cherry-pick the NeMo-RL fix

  git cherry-pick main..ruit/convert_lora_ckpt

Step 2: Cherry-pick the Megatron-Bridge fix

Our official recipe uses exclude_modules, which is not yet fully supported in the current version of Megatron-Bridge. Until
Megatron-Bridge PR #3218 is merged and bumped, please apply it manually to your local 3rdparty copy:

cd 3rdparty/Megatron-Bridge-workspace/Megatron-Bridge
git fetch origin
git cherry-pick main..ruit/add_exclude_key

Step 3: Convert the LoRA checkpoint

Choose one of the following options depending on your use case.

Option A — Merge LoRA weights into the base model (single merged HF checkpoint)

  uv run --extra mcore python examples/converters/convert_lora_to_hf.py \
      --base-ckpt ${HF_HOME}/nemo_rl/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16/iter_0000000 \
      --adapter-ckpt <path/to/your/lora/checkpoint>/iter_0000000 \
      --hf-model-name nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
      --hf-ckpt-path <path/to/your/merged/hf/checkpoint>//merged_hf_model

Option B — Export adapter only (HuggingFace PEFT format, for use with vLLM)

  uv run --extra mcore python examples/converters/convert_lora_to_hf.py \
      --adapter-only \
      --adapter-ckpt <path/to/your/lora/checkpoint>/iter_0000000 \
      --hf-model-name nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 \
      --hf-ckpt-path <path/to/your/converted/lora/hf/checkpoint>/hf_lora_adapter

Step 4: Start the vLLM inference server

If you used Option A (merged weights)

The output is a standard HuggingFace model checkpoint. You can serve and query it directly using the normal vLLM workflow without any special LoRA configuration.

If you used Option B (adapter only)

Below is an example script that loads the base model with LoRA support enabled and serves both base and LoRA inference via a simple FastAPI server:

# server_lora_engine.py
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

# 1. Start the vLLM engine (load only the base model, but enable LoRA support)
llm = LLM(
    model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    dtype="auto",
    gpu_memory_utilization=0.95,
    enable_lora=True,
    max_loras=1,
    max_lora_rank=128,   # Adjust based on your LoRA rank
    max_model_len=8192,  # Limit KV cache length to save GPU memory; adjust as needed
)

# Newer vLLM versions do not require pre-registering LoRA; generate() loads it on demand via LoRARequest
LORA_NAME = "<change to your name>"
LORA_PATH = "<path/to/your/converted/lora/hf/checkpoint>"

LORA_REQUEST = LoRARequest(lora_name=LORA_NAME, lora_int_id=1, lora_path=LORA_PATH)

class ChatRequest(BaseModel):
    prompt: str
    max_tokens: int = 128
    temperature: float = 0.0
    use_lora: bool = True

class ChatResponse(BaseModel):
    text: str

app = FastAPI()

@app.post("/chat", response_model=ChatResponse)
def chat(req: ChatRequest):
    params = SamplingParams(
        max_tokens=req.max_tokens,
        temperature=req.temperature,
    )

    outputs = llm.generate(
        [req.prompt],
        params,
        lora_request=LORA_REQUEST if req.use_lora else None,
    )

    text = outputs[0].outputs[0].text
    return ChatResponse(text=text)


@app.get("/verify_lora")
def verify_lora():
    """Generate outputs for the same prompt with base and LoRA separately, then return a logprobs comparison.
    Different logprobs indicate that the LoRA weights are actually taking effect."""
    prompt = "The capital of France is"
    params = SamplingParams(max_tokens=1, temperature=0.0, logprobs=5)

    out_base = llm.generate([prompt], params, lora_request=None)[0].outputs[0]
    out_lora = llm.generate([prompt], params, lora_request=LORA_REQUEST)[0].outputs[0]

    def top_logprobs(output):
        if not output.logprobs:
            return {}
        return {tok: round(lp.logprob, 4) for tok, lp in output.logprobs[0].items()}

    base_lp = top_logprobs(out_base)
    lora_lp = top_logprobs(out_lora)
    return {
        "base_token": out_base.text,
        "lora_token": out_lora.text,
        "base_logprobs": base_lp,
        "lora_logprobs": lora_lp,
        "lora_active": base_lp != lora_lp,
    }

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=10240)

Then start the server: python vllm_nanov3_lora.py
The server will listen on port 10240.

Step 5: Send a test request

  curl -X POST http://localhost:10240/chat \
      -H "Content-Type: application/json" \
      -d '{"prompt": "What is 1+1?", "max_tokens": 128, "temperature": 0.0, "use_lora": true}'

Step 6: Verify the LoRA adapter is active

  curl http://localhost:10240/verify_lora

This endpoint runs the same prompt with and without the LoRA adapter and compares the top logprobs. If the LoRA weights are correctly loaded, you should see different logprobs between the base model and the LoRA model, and "lora_active": true in the response.

---

Please feel free to let us know if you run into any further issues!

[hanguangmic]

Hi, @hanguangmic. Thank you for raising this issue and for your patience — this is really helpful feedback!

The bug mentioned in this comment (KeyError: decoder.layers.0.mixer.dt_bias from model not in state dict) does exist, and we have addressed it in PR #2234. The PR is still being polished before merging, but you are welcome to try it out in the meantime. Below is a complete end-to-end workflow to get you started.

Step 1: Cherry-pick the NeMo-RL fix

git cherry-pick 43f4642

Step 2: Cherry-pick the Megatron-Bridge fix

Our official recipe uses exclude_modules, which is not yet fully supported in the current version of Megatron-Bridge. Until Megatron-Bridge PR #3218 is merged and bumped, please apply it manually to your local 3rdparty copy:

cd 3rdparty/Megatron-Bridge-workspace/Megatron-Bridge
git fetch origin
git cherry-pick dfbb1c4

Step 3: Convert the LoRA checkpoint

Choose one of the following options depending on your use case.

Option A — Merge LoRA weights into the base model (single merged HF checkpoint)

uv run --extra mcore python examples/converters/convert_lora_to_hf.py
--base-ckpt ${HF_HOME}/nemo_rl/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16/iter_0000000
--adapter-ckpt <path/to/your/lora/checkpoint>/iter_0000000
--hf-model-name nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
--hf-ckpt-path <path/to/your/merged/hf/checkpoint>//merged_hf_model

Option B — Export adapter only (HuggingFace PEFT format, for use with vLLM)

uv run --extra mcore python examples/converters/convert_lora_to_hf.py
--adapter-only
--adapter-ckpt <path/to/your/lora/checkpoint>/iter_0000000
--hf-model-name nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
--hf-ckpt-path <path/to/your/converted/lora/hf/checkpoint>/hf_lora_adapter

Step 4: Start the vLLM inference server

If you used Option A (merged weights)

The output is a standard HuggingFace model checkpoint. You can serve and query it directly using the normal vLLM workflow without any special LoRA configuration.

If you used Option B (adapter only)

Below is an example script that loads the base model with LoRA support enabled and serves both base and LoRA inference via a simple FastAPI server:

server_lora_engine.py

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

1. Start the vLLM engine (load only the base model, but enable LoRA support)

llm = LLM(
model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
dtype="auto",
gpu_memory_utilization=0.95,
enable_lora=True,
max_loras=1,
max_lora_rank=128, # Adjust based on your LoRA rank
max_model_len=8192, # Limit KV cache length to save GPU memory; adjust as needed
)

Newer vLLM versions do not require pre-registering LoRA; generate() loads it on demand via LoRARequest

LORA_NAME = ""
LORA_PATH = "<path/to/your/converted/lora/hf/checkpoint>"

LORA_REQUEST = LoRARequest(lora_name=LORA_NAME, lora_int_id=1, lora_path=LORA_PATH)

class ChatRequest(BaseModel):
prompt: str
max_tokens: int = 128
temperature: float = 0.0
use_lora: bool = True

class ChatResponse(BaseModel):
text: str

app = FastAPI()

@app.post("/chat", response_model=ChatResponse)
def chat(req: ChatRequest):
params = SamplingParams(
max_tokens=req.max_tokens,
temperature=req.temperature,
)

outputs = llm.generate(
    [req.prompt],
    params,
    lora_request=LORA_REQUEST if req.use_lora else None,
)

text = outputs[0].outputs[0].text
return ChatResponse(text=text)

@app.get("/verify_lora")
def verify_lora():
"""Generate outputs for the same prompt with base and LoRA separately, then return a logprobs comparison.
Different logprobs indicate that the LoRA weights are actually taking effect."""
prompt = "The capital of France is"
params = SamplingParams(max_tokens=1, temperature=0.0, logprobs=5)

out_base = llm.generate([prompt], params, lora_request=None)[0].outputs[0]
out_lora = llm.generate([prompt], params, lora_request=LORA_REQUEST)[0].outputs[0]

def top_logprobs(output):
    if not output.logprobs:
        return {}
    return {tok: round(lp.logprob, 4) for tok, lp in output.logprobs[0].items()}

base_lp = top_logprobs(out_base)
lora_lp = top_logprobs(out_lora)
return {
    "base_token": out_base.text,
    "lora_token": out_lora.text,
    "base_logprobs": base_lp,
    "lora_logprobs": lora_lp,
    "lora_active": base_lp != lora_lp,
}

if name == "main":
uvicorn.run(app, host="0.0.0.0", port=10240)
Then start the server: python vllm_nanov3_lora.py The server will listen on port 10240.

Step 5: Send a test request

curl -X POST http://localhost:10240/chat
-H "Content-Type: application/json"
-d '{"prompt": "What is 1+1?", "max_tokens": 128, "temperature": 0.0, "use_lora": true}'

Step 6: Verify the LoRA adapter is active

curl http://localhost:10240/verify_lora
This endpoint runs the same prompt with and without the LoRA adapter and compares the top logprobs. If the LoRA weights are correctly loaded, you should see different logprobs between the base model and the LoRA model, and "lora_active": true in the response.

Please feel free to let us know if you run into any further issues!

May I know if i use the leatest code to build a new docker image, can solve the above issues?

[RayenTian]

Hi @hanguangmic. If you only build a new docker but don't cherry-pick the commits, I'm afraid it can't solve the issue, since the PR has not been merged.

[hanguangmic]

Hi @hanguangmic. If you only build a new docker but don't cherry-pick the commits, I'm afraid it can't solve the issue, since the PR has not been merged.

I got it, so nice you can solve my issue.