[RL] add pause, update_weights, resume interface for async RL by mitu626 · Pull Request #6052 · PaddlePaddle/FastDeploy

mitu626 · 2026-01-15T11:39:22Z

Motivation

RL异步异置场景通过RDMA权重原地更新能力。
在FastDeploy内部，支持 api-server <--> engine <--> worker 之间控制信号通信机制。

💡 If this PR is a Cherry Pick, the PR title needs to follow the format by adding the [Cherry-Pick] label at the very beginning and appending the original PR ID at the end. For example, [Cherry-Pick][CI] Add check trigger and logic(#5191)

💡 如若此PR是Cherry Pick，PR标题需遵循格式，在最开始加上[Cherry-Pick]标签，以及最后面加上原PR ID，例如[Cherry-Pick][CI] Add check trigger and logic(#5191)

Modifications

控制信号通信：
- api-server <--> engine：复用当前zmq通信机制，新增ControlRequest、ControlResponse类型。
- engine ---> worker：复用当前通信机制（engine_worker_queue)，插入 ControlRequest 请求
- engine <--- worker：基于 fmq.Queue 新增每个TP rank和engine的通信queue，支持每个TP rank返回自己的执行结果 ControlResponse
新增接口
- /v1/pause: 推理引擎不再响应推理请求，当前正在执行的推理请求会被打断。
- /v1/update_weights: 基于RDMA远程同步权重，并原地更新权重
- /v1/resume：回归推理引擎继续相应推理请求
- /v1/is_paused: 查看当前推理引擎状态

其他说明：

当前控制信号执行会阻塞推理请求的处理，在权重更新场景下本身需要pause，因此无影响，后续需要升级支持可选异步，不阻塞推理请求。
当前worker ---> engine 控制信号返回信道和推理请求输出结果的返回信道独立，后续需要考虑是否可以合并以简化系统。

Usage or Command

服务启动时，新增参数 --load-strategy "rsync" --rsync-config '{"xxxx":"xxxx"}'
接口调用：
- pause：curl -XPOST http://host:port/v1/pause
- update_weights curl-XPOST http://host:port/v1/update_weights -d '{"version": "xxx", "rsync_config": {}}'
- resume: curl -XPOST http://host:port/v1/resume
- is_paused: curl -XGET http://host:port/v1/is_paused

说明：

rsync需要的模型版本，默认从模型路径下的文件内读取 {model}/version.txt，支持调用接口时通过参数指定
rsync依赖的config支持在启动时传入默认配置参数，也支持调用接口时通过参数指定

Accuracy Tests

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

…mon_engine

…gine by common run_control_method

CLAassistant · 2026-01-15T11:39:28Z

All committers have signed the CLA.

paddle-bot · 2026-01-15T11:39:28Z

Thanks for your contribution!

…_requests, resource_manager_v1.preempted_all

Copilot

Pull request overview

这个PR为异步RL场景添加了pause、update_weights、resume接口，支持通过RDMA进行权重原地更新。实现了api-server、engine和worker之间的控制信号通信机制。

Changes:

新增4个REST API接口：/v1/pause、/v1/resume、/v1/is_paused、/v1/update_weights
实现ControlRequest和ControlResponse类以支持控制消息的序列化和传输
在engine层添加pause/resume状态管理和权重更新协调逻辑
在worker层实现RDMA权重同步和原地更新能力

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 29 comments.

Show a summary per file

File	Description
fastdeploy/entrypoints/openai/api_server.py	新增4个控制接口端点
fastdeploy/entrypoints/engine_client.py	添加run_control_method方法处理控制请求
fastdeploy/engine/request.py	定义ControlRequest和ControlResponse类
fastdeploy/engine/common_engine.py	实现pause/resume/update_weights核心逻辑
fastdeploy/engine/sched/resource_manager_v1.py	添加preempted_all和wait相关方法
fastdeploy/worker/worker_process.py	添加控制请求处理和control output queue
fastdeploy/worker/gpu_worker.py	添加update_weights方法
fastdeploy/worker/gpu_model_runner.py	转发update_weights到dynamic_weight_manager
fastdeploy/rl/dynamic_weight_manager.py	实现RDMA权重同步和参数验证
fastdeploy/scheduler/local_scheduler.py	添加get_inflight_requests方法
fastdeploy/config.py	添加rsync_config配置项
fastdeploy/engine/args_utils.py	添加rsync_config命令行参数
fastdeploy/inter_communicator/fmq.py	改进日志输出
fastdeploy/output/token_processor.py	修复metrics访问方式
tests/scheduler/test_local_scheduler.py	添加get_inflight_requests测试
tests/engine/test_resource_manager_v1.py	添加preempted_all测试
tests/engine/test_control_request_response.py	添加ControlRequest/Response单元测试

Copilot · 2026-01-19T10:32:46Z

+            asyncio.run(self._ctrl_output.put(succ_result, shm_threshold=100 * 1024 * 1024))
+        except Exception as e:
+            error_msg = f"Rank-{self.local_rank} Failed run control method {method}: {str(e)}"
+            logger.info(f"{error_msg}\n{traceback.format_exc()}")


日志级别使用不当。这里使用的是logger.info记录失败信息，应该使用logger.error或logger.warning来记录错误信息。

Suggested change

logger.info(f"{error_msg}\n{traceback.format_exc()}")

logger.error(f"{error_msg}\n{traceback.format_exc()}")

Copilot · 2026-01-19T10:32:46Z

+    def _control_pause(self, control_request: ControlRequest) -> dict | None:
+        """暂停请求生成
+
+        Args:
+            args: 控制参数字典，暂停相关的配置参数
+
+        Returns:
+            tuple: (error_code, error_msg) 元组
+                - error_code: 错误代码，0表示成功，非0表示失败
+                - error_msg: 错误信息，成功时为空字符串


注释应该使用英文。根据代码规范，代码中的注释应该使用英文，而不是中文。

Copilot · 2026-01-19T10:32:47Z

+    def _control_update_weights(self, control_request: ControlRequest) -> dict | None:
+        """更新模型权重
+
+        Args:
+            args: 控制参数字典，更新权重相关的配置参数
+
+        Returns:
+            dict | None: 返回结果字典或None，包含更新权重的操作结果信息


注释应该使用英文。根据代码规范，代码中的注释应该使用英文，而不是中文。

Copilot · 2026-01-19T10:32:53Z

+    def _control_resume(self, control_request: ControlRequest) -> dict | None:
+        """恢复暂停的请求生成
+
+        Args:
+            args: 控制参数字典，恢复生成相关的配置参数
+
+        Returns:
+            dict | None: 返回结果字典或None，包含恢复操作的状态信息


注释应该使用英文。根据代码规范，代码中的注释应该使用英文，而不是中文。

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

liyonghua0910 · 2026-01-22T05:31:02Z

+        tp_size = cfg.parallel_config.tensor_parallel_size
+        dp_index = cfg.parallel_config.local_data_parallel_id
+        for rank in range(tp_size):
+            name = f"ctrl_w2e_rank{rank+tp_size*dp_index}"


如果一台机器上部署多个实例，不同实例之间的 fmq 命名会冲突吗

这个确实有可能，命名上我把engine_worker_queue的port加进去吧，这样不同fd实例启动时指定的engine_worker_queue port不同可以独立区分开

liyonghua0910 · 2026-01-22T05:47:39Z

+                        control_reqs.append(req_dict[0])
+                    else:
+                        max_occupied_batch_index = int(bsz)
+                        req_dicts.extend(req_dict)


如果这个 batch 能同时包含控制请求和普通请求，更新权重之后，普通请求还需要执行 execute_model 吗？如果执行的话那本次生成 token 所使用的权重是不是和已生成的 token 所使用的权重不一致

对于权重更新的场景：这里预期是不会既包含控制请求，又包含推理请求的，这个是在engine（作为worker的client）来控制的，可以参考common_engine里的逻辑。

这里更多的是对更通用的控制信号通信时留得一点buffer，比如某些控制信号可能只是想知道当前worker的状态是不是health的，或者想知道worker当前内部的一些动态的状态。不过通用的控制信号通信逻辑也还有一些todo，比如支持可选的异步执行等。

liyonghua0910 · 2026-01-22T06:02:56Z

+            self.llm_logger.info(f"Total {len(running_reqs)} requests need to be aborted.")
+            self.resource_manager.get_real_bsz()
+            self.engine_worker_queue.put_tasks((running_reqs, self.resource_manager.real_bsz))
+            self.resource_manager.wait_worker_inflight_requests_finish(timeout=60)


这里是需要等待推理自然结束还是中止推理呢

因为在上一步将当前所有running请求都调度成了抢占请求，因此worker会在下一个step将所有正在推理请求按照抢占逻辑打断，这里虽然是等待，但其实不是等待正常推理结束，而是在等待worker执行抢占操作。

… queue

liyonghua0910

LGTM

codecov-commenter · 2026-01-22T12:14:42Z

Codecov Report

❌ Patch coverage is 32.48731% with 266 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@955785e). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
fastdeploy/engine/common_engine.py	14.81%	114 Missing and 1 partial ⚠️
fastdeploy/rl/dynamic_weight_manager.py	6.57%	71 Missing ⚠️
fastdeploy/entrypoints/openai/api_server.py	26.31%	28 Missing ⚠️
fastdeploy/worker/worker_process.py	37.20%	25 Missing and 2 partials ⚠️
fastdeploy/engine/sched/resource_manager_v1.py	51.72%	13 Missing and 1 partial ⚠️
fastdeploy/entrypoints/engine_client.py	46.66%	7 Missing and 1 partial ⚠️
fastdeploy/worker/gpu_model_runner.py	50.00%	2 Missing ⚠️
fastdeploy/worker/gpu_worker.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #6052   +/-   ##
==========================================
  Coverage           ?   67.63%           
==========================================
  Files              ?      381           
  Lines              ?    50321           
  Branches           ?     7872           
==========================================
  Hits               ?    34036           
  Misses             ?    13802           
  Partials           ?     2483

Flag	Coverage Δ
GPU	`67.63% <32.48%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…Paddle#6052) * support dynamic run_control_request through zmq from apiserver to common_engine * support pause/resume/is_paused/update_weights in apiserver->common_engine by common run_control_method * change /is_puased from HTTP POST method to GET method * add pause、resume、is_paused implementation * support engine <==> worker communication(request&response) * support sync weights through RDMA from checkpoint_transfer * support specified version, rsync_config in update_weights rpc call * add pause, update_weights, resume interface for async RL * bug fix: update_weights support using default arguments * fix typo * typo fix * typo fix * typo fix * add unitest for control request/response, localscheduler.get_inflight_requests, resource_manager_v1.preempted_all * add "rsync" to LoadConfig.load_strategy Literal type hints Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * typo fix * typo fix * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * check version/rsync params * add error log when version.txt not exists Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * raise specified ValueError when paramters check failed Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * tp barrier after run_control_method * encode 'engine_worker_queue_port' to unique name of worker2engine fmq queue * typo fix * typo fix --------- Co-authored-by: Jiang-Jia-Jun <163579578+Jiang-Jia-Jun@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mitu626 added 12 commits January 4, 2026 19:38

support dynamic run_control_request through zmq from apiserver to com…

d927adf

…mon_engine

support pause/resume/is_paused/update_weights in apiserver->common_en…

b1e1f0e

…gine by common run_control_method

change /is_puased from HTTP POST method to GET method

c877720

add pause、resume、is_paused implementation

c0bf665

support engine <==> worker communication(request&response)

004c57f

support sync weights through RDMA from checkpoint_transfer

3f1115c

support specified version, rsync_config in update_weights rpc call

ae306bc

add pause, update_weights, resume interface for async RL

1b1a348

bug fix: update_weights support using default arguments

9d2178f

fix typo

2b0b7fc

typo fix

85ce074

typo fix

7691b45

mitu626 had a problem deploying to Metax_ci January 15, 2026 11:39 — with GitHub Actions Failure

Merge branch 'develop' into rlhf

ae2acfe

mitu626 temporarily deployed to Metax_ci January 15, 2026 11:46 — with GitHub Actions Inactive

mitu626 added 3 commits January 15, 2026 20:05

typo fix

6ef8f97

Merge remote-tracking branch 'refs/remotes/origin/rlhf' into rlhf

6f61173

Merge branch 'develop' into rlhf

05d0617

mitu626 temporarily deployed to Metax_ci January 19, 2026 02:31 — with GitHub Actions Inactive

mitu626 added 2 commits January 19, 2026 18:13

add unitest for control request/response, localscheduler.get_inflight…

c7b7899

…_requests, resource_manager_v1.preempted_all

Merge remote-tracking branch 'refs/remotes/origin/rlhf' into rlhf

c94d8de

mitu626 had a problem deploying to Metax_ci January 19, 2026 10:15 — with GitHub Actions Error

Jiang-Jia-Jun requested a review from Copilot January 19, 2026 10:24

Copilot started reviewing on behalf of Jiang-Jia-Jun January 19, 2026 10:24 View session

Merge branch 'develop' into rlhf

f643e95

Jiang-Jia-Jun had a problem deploying to Metax_ci January 19, 2026 10:25 — with GitHub Actions Error

Copilot AI reviewed Jan 19, 2026

View reviewed changes

add "rsync" to LoadConfig.load_strategy Literal type hints

d6a4f67

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Apply suggestion from @Copilot

32ab88f

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mitu626 had a problem deploying to Metax_ci January 19, 2026 12:06 — with GitHub Actions Error

check version/rsync params

277c1b5

mitu626 had a problem deploying to Metax_ci January 19, 2026 12:13 — with GitHub Actions Error

add error log when version.txt not exists

0ab17a6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mitu626 had a problem deploying to Metax_ci January 19, 2026 12:18 — with GitHub Actions Error

raise specified ValueError when paramters check failed

d4f7690

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mitu626 had a problem deploying to Metax_ci January 19, 2026 12:20 — with GitHub Actions Error

tp barrier after run_control_method

8845b82

mitu626 temporarily deployed to Metax_ci January 19, 2026 12:23 — with GitHub Actions Inactive

liyonghua0910 reviewed Jan 22, 2026

View reviewed changes

encode 'engine_worker_queue_port' to unique name of worker2engine fmq…

96a85d1

… queue

mitu626 had a problem deploying to Metax_ci January 22, 2026 06:35 — with GitHub Actions Failure

Merge branch 'develop' into rlhf

4a4b053

mitu626 temporarily deployed to Metax_ci January 22, 2026 08:51 — with GitHub Actions Inactive

liyonghua0910 previously approved these changes Jan 22, 2026

View reviewed changes

typo fix

76f10fd

mitu626 dismissed liyonghua0910’s stale review via 76f10fd January 22, 2026 09:23

mitu626 had a problem deploying to Metax_ci January 22, 2026 09:23 — with GitHub Actions Error

liyonghua0910 previously approved these changes Jan 22, 2026

View reviewed changes

typo fix

0aaede5

mitu626 dismissed liyonghua0910’s stale review via 0aaede5 January 22, 2026 09:34

mitu626 temporarily deployed to Metax_ci January 22, 2026 09:35 — with GitHub Actions Inactive

liyonghua0910 approved these changes Jan 22, 2026

View reviewed changes

Jiang-Jia-Jun approved these changes Jan 23, 2026

View reviewed changes

Jiang-Jia-Jun merged commit b7c5daa into PaddlePaddle:develop Jan 23, 2026
23 of 28 checks passed

	logger.info(f"{error_msg}\n{traceback.format_exc()}")
	logger.error(f"{error_msg}\n{traceback.format_exc()}")

Conversation

mitu626 commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

CLAassistant commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paddle-bot Bot commented Jan 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liyonghua0910 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

mitu626 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

liyonghua0910 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

mitu626 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liyonghua0910 Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

mitu626 Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liyonghua0910 left a comment

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Jan 22, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mitu626 commented Jan 15, 2026 •

edited

Loading

CLAassistant commented Jan 15, 2026 •

edited

Loading

mitu626 Jan 22, 2026 •

edited

Loading