[BugFix] fix cache transfer tasks failure after cache cleared by liyonghua0910 · Pull Request #6202 · PaddlePaddle/FastDeploy

liyonghua0910 · 2026-01-24T14:06:15Z

Motivation

为 CacheTransferManager 引入 pause/resume 机制，由 kv_cache_status_signal 触发 pause：

pause 时，会等待目前已经提交线程池的任务执行完毕， pause 后才开始清除缓存；
pause 后，do_data_transfer 线程不再从上游拉取任务，已经拉取的任务本次仍可继续提交；
resume 后，do_data_transfer 线程可以继续提交任务

Modifications

File	Modification
fastdeploy/cache_manager/cache_transfer_manager.py	新增 pause, resume 方法，各 rank 进程同步暂停和恢复；维护 inflight 任务计数，工作线程等待已经取出的任务结束再暂停
fastdeploy/cache_manager/prefix_cache_manager.py	task_swapping_event 设置 0.1 秒超时，若在非推理状态超时，则不必强等；给其他错误增加非推理状态时报错的容错
fastdeploy/inter_communicator/engine_cache_queue.py	支持清空 engine_cache_queue 队列
fastdeploy/entrypoints/engine_client.py	调整 prefix tree 的清除顺序，在 weight/cache 清除之后再清除，增加日志提示

Usage or Command

无

Accuracy Tests

无

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

paddle-bot · 2026-01-24T14:06:20Z

Thanks for your contribution!

codecov-commenter · 2026-01-24T15:54:53Z

Codecov Report

❌ Patch coverage is 45.32020% with 111 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@85db063). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
fastdeploy/entrypoints/engine_client.py	59.09%	28 Missing and 8 partials ⚠️
fastdeploy/cache_manager/prefix_cache_manager.py	34.04%	29 Missing and 2 partials ⚠️
fastdeploy/cache_manager/cache_transfer_manager.py	9.67%	28 Missing ⚠️
...astdeploy/inter_communicator/engine_cache_queue.py	46.66%	16 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #6202   +/-   ##
==========================================
  Coverage           ?   68.15%           
==========================================
  Files              ?      391           
  Lines              ?    52389           
  Branches           ?     8177           
==========================================
  Hits               ?    35705           
  Misses             ?    14071           
  Partials           ?     2613

Flag	Coverage Δ
GPU	`68.15% <45.32%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…e_transfer_0124

Copilot

Pull request overview

这个PR为CacheTransferManager引入了pause/resume机制，旨在修复缓存清除后缓存传输任务失败的问题。当kv_cache_status_signal触发pause时，系统会等待所有正在执行的任务完成后再清除缓存，避免了缓存清除和传输任务之间的竞争条件。

Changes:

在CacheTransferManager中添加了pause/resume方法和inflight任务跟踪机制
修改了PrefixCacheManager的reset方法，移除了清除缓存前阻塞等待任务结果的限制
在EngineCacheQueue中添加了clear_transfer_task和result_queue_empty方法
重构了engine_client.py中的update/clear模型权重逻辑，改进了日志和流程控制
增强了调试日志输出，便于问题追踪

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
fastdeploy/cache_manager/cache_transfer_manager.py	实现核心的pause/resume机制，包括inflight任务计数、跨rank同步以及暂停/恢复传输任务的逻辑
fastdeploy/cache_manager/prefix_cache_manager.py	修改sync_swap_task使用超时机制，在非正常状态下提前退出；增强reset方法以等待inflight任务和结果队列清空；增加错误处理以忽略非正常状态下的异常
fastdeploy/inter_communicator/engine_cache_queue.py	新增barrier0、pause_barrier和resume_barrier用于跨rank同步；添加clear_transfer_task清空任务队列；添加result_queue_empty检查结果队列状态
fastdeploy/entrypoints/engine_client.py	重构update_model_weight和clear_load_weight方法，改进控制流程和日志标记，调整prefix tree操作的执行顺序
fastdeploy/engine/common_engine.py	在clear_data方法中调用cache_task_queue.clear_transfer_task()清空传输任务
fastdeploy/engine/sched/resource_manager_v1.py	在关键位置添加verbose日志输出，改进错误信息
tests/cache_manager/test_prefix_cache_manager.py	更新测试以支持新的signal和方法，包括result_queue_empty、wait timeout参数等

Copilot · 2026-02-06T02:30:03Z

@@ -862,22 +894,46 @@ def do_data_transfer(self):

        while True:
            try:
+                if self.rank == 0:
+                    self.cache_task_is_paused_signal.value[0] = 1 if self.is_paused else 0
+                if self.n_ranks > 1:
+                    self.cache_task_queue.barrier0.wait()
+                    if self.rank == 0:
+                        self.cache_task_queue.barrier0.reset()
+
+                # Ensure all ranks synchronically do one of the following things:
+                # (1) If rank#0 is paused, wait for a short time and check out rank#0 status again;
+                # (2) otherwise, all ranks are allowed to pull tasks from cache task queue
+                if self.cache_task_is_paused_signal.value[0] == 1:
+                    # wait for inflight tasks to finish first
+                    while self.inflight != 0:
+                        time.sleep(0.1)
+                    # mark the current rank as not having inflight tasks
+                    self.cache_task_inflight_signal.value[self.rank] = 0
+                    time.sleep(1)
+                    continue
+                else:
+                    self.cache_task_inflight_signal.value[self.rank] = 1
+
                if self.rank == 0:
                    if not self.cache_task_queue.empty():
                        self.cache_task_broadcast_signal.value[0] = 1
                if self.n_ranks > 1:
                    self.cache_task_queue.barrier1.wait()
                    if self.rank == 0:
                        self.cache_task_queue.barrier1.reset()
+
                if self.cache_task_broadcast_signal.value[0] == 1:
+                    self.inflight += 1


The inflight counter (lines 277, 881, 927) is not protected by a lock. Multiple threads could potentially read and modify it concurrently, leading to race conditions. For example, the decrement in line 881 (self.inflight -= 1) is not atomic. In high-concurrency scenarios, this could result in incorrect counts, causing the pause() method to either wait indefinitely or proceed prematurely. Consider using threading.Lock or an atomic counter (e.g., threading.Event or multiprocessing.Value with lock=True) to protect access to this counter.

…Paddle#6202) * [fix] fix cache transfer tasks failure after cache cleared * [fix] fix submit_task * [fix] fix cache manager hang when clearing prefix cache * [fix] fix list_proxy has no clear method * [fix] fix barrier * [fix] add barrier0 * [fix] add cache_task_is_paused_signal * [fix] fix condition * [fix] fix cache transfer sync and delay prefix cache tree clearing * [fix] fix typo * [chore] polish code * [fix] revert only rank0 write kv_cache_status_signal * [fix] fix thread pool and prefix cache manager hang * [fix] add timeout for task_swapping_event * [fix] tolerate prefix cache manager error while prefix tree is cleared * [chore] add more log * [fix] fix test_prefix_cache_manager * [fix] fix prefix_cache_status_signal usage

[fix] fix cache transfer tasks failure after cache cleared

f6e12a2

liyonghua0910 had a problem deploying to Metax_ci January 24, 2026 14:06 — with GitHub Actions Error

[fix] fix submit_task

d6c9f61

liyonghua0910 temporarily deployed to Metax_ci January 24, 2026 14:18 — with GitHub Actions Inactive

[fix] fix cache manager hang when clearing prefix cache

a8674dd

liyonghua0910 had a problem deploying to Metax_ci January 27, 2026 06:46 — with GitHub Actions Failure

liyonghua0910 added 2 commits January 27, 2026 10:14

[fix] fix list_proxy has no clear method

eca85f4

[fix] fix barrier

f58c564

liyonghua0910 had a problem deploying to Metax_ci January 27, 2026 12:37 — with GitHub Actions Failure

[fix] add barrier0

a2d3887

liyonghua0910 had a problem deploying to Metax_ci January 28, 2026 02:46 — with GitHub Actions Failure

Merge remote-tracking branch 'upstream/develop' into develop+fix_cach…

ee57758

…e_transfer_0124

liyonghua0910 temporarily deployed to Metax_ci January 28, 2026 03:26 — with GitHub Actions Inactive

liyonghua0910 added 5 commits January 29, 2026 11:42

[fix] add cache_task_is_paused_signal

8611708

[fix] fix condition

13f6ac7

[fix] fix cache transfer sync and delay prefix cache tree clearing

b68accc

[fix] fix typo

7675727

[chore] polish code

ba0b306

liyonghua0910 temporarily deployed to Metax_ci January 30, 2026 09:58 — with GitHub Actions Inactive

liyonghua0910 added 5 commits February 2, 2026 03:59

[fix] revert only rank0 write kv_cache_status_signal

f24f1b9

[fix] fix thread pool and prefix cache manager hang

429e448

[fix] add timeout for task_swapping_event

18d3086

[fix] tolerate prefix cache manager error while prefix tree is cleared

c5a8736

[chore] add more log

7d17410

liyonghua0910 temporarily deployed to Metax_ci February 5, 2026 06:21 — with GitHub Actions Inactive

[fix] fix test_prefix_cache_manager

ffdae9a

liyonghua0910 temporarily deployed to Metax_ci February 5, 2026 13:41 — with GitHub Actions Inactive

Jiang-Jia-Jun requested a review from Copilot February 6, 2026 02:24

Copilot started reviewing on behalf of Jiang-Jia-Jun February 6, 2026 02:24 View session

Copilot AI reviewed Feb 6, 2026

View reviewed changes

[fix] fix prefix_cache_status_signal usage

33541ca

liyonghua0910 temporarily deployed to Metax_ci February 6, 2026 07:06 — with GitHub Actions Inactive

Jiang-Jia-Jun approved these changes Feb 8, 2026

View reviewed changes

Jiang-Jia-Jun merged commit 5ac5ecd into PaddlePaddle:develop Feb 8, 2026
20 of 24 checks passed

Navigation Menu

[BugFix] fix cache transfer tasks failure after cache cleared - #6202

[BugFix] fix cache transfer tasks failure after cache cleared#6202
Jiang-Jia-Jun merged 19 commits into
PaddlePaddle:developfrom
liyonghua0910:develop+fix_cache_transfer_0124

liyonghua0910 commented Jan 24, 2026 •

edited

Loading

Uh oh!

paddle-bot Bot commented Jan 24, 2026

Uh oh!

codecov-commenter commented Jan 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Feb 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

liyonghua0910 commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

paddle-bot Bot commented Jan 24, 2026

Uh oh!

codecov-commenter commented Jan 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liyonghua0910 commented Jan 24, 2026 •

edited

Loading

codecov-commenter commented Jan 24, 2026 •

edited

Loading