[Feature] [KVCache] support file_store kv cache backend by Moonchild1227 · Pull Request #6188 · PaddlePaddle/FastDeploy

Moonchild1227 · 2026-01-23T07:48:37Z

Motivation

add FileStore persistence backend for KVCache disk caching
wait cache_transfer_manager inited
Adds a third KVCache storage backend in addition to Mooncake and AttentionStore

Modifications

cache_transfer_manager.py
file_store.py

Usage or Command

python -m fastdeploy.entrypoints.openai.api_server \
  --model "PaddlePaddle/ERNIE-4.5-0.3B-Paddle" \
  --port 8300 \
  --metrics-port 8301 \
  --engine-worker-queue-port 8302 \
  --cache-queue-port 8303 \
  --max-model-len 32768 \
  --max-num-seqs 32 \
  --kvcache-storage-backend file \
  --enable-prefix-caching

Accuracy Tests

test script

cd tests/ce/accuracy_cases
python gsm8k.py

test result

base line

🎯 Evaluation Complete: Accuracy = 3.33% (23/690)
平均准确率：3.91%

file store

🎯 Evaluation Complete: Accuracy = 4.06% (28/690)
平均准确率：3.62%

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

paddle-bot · 2026-01-23T07:48:44Z

Thanks for your contribution!

Copilot

Pull request overview

该 PR 旨在为 KVCache 磁盘缓存新增一个基于文件系统持久化的存储后端（FileStore），作为 Mooncake / AttentionStore 之外的第三种可选后端，并在 cache_transfer_manager 中接入读写流程与参数支持。

Changes:

新增 FileStore KVCacheStorage 实现（本地文件落盘 / 读取 / 清理 / 查询）。
将 FileStore 接入 cache_transfer_manager（新增 backend 选项与 file_path 参数）。
扩展 Engine CLI 的 --kvcache-storage-backend 选项，并更新示例脚本。

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
fastdeploy/engine/args_utils.py	CLI 参数 `--kvcache-storage-backend` 增加 `file` 选项。
fastdeploy/cache_manager/transfer_factory/file_store/file_store.py	新增 FileStore 后端的核心实现。
fastdeploy/cache_manager/transfer_factory/file_store/init.py	暴露 FileStore / FileStoreConfig。
fastdeploy/cache_manager/transfer_factory/init.py	将 FileStore 加入 transfer_factory 导出。
fastdeploy/cache_manager/cache_transfer_manager.py	增加 file 后端初始化、读写分支与 `--kvcache_file_path` 参数。
examples/cache_storage/run.sh	示例脚本中 stop.sh 调用被注释。

Copilot · 2026-01-23T08:01:58Z

+        elif args.kvcache_storage_backend == "file":
+            logger.info("Start initialize file store...")
+            self.storage_backend = FileStore(
+                namespace=self.model_id,
+                tp_rank=self.rank,
+                tp_size=self.n_ranks,
+                file_path=args.kvcache_file_path,
+            )
+            self._init_storage_buffer(args)
+            logger.info("Initialized file store successfully")


新增 file 存储后端分支（初始化 + read/write 逻辑）目前缺少单测覆盖。仓库已有 tests/cache_manager/test_cache_transfer_manager.py 覆盖其他分支逻辑，建议补充针对 kvcache_storage_backend=="file" 的用例（可通过 mock FileStore.batch_get/batch_set 与 swap_cache_layout 调用来验证读写流程与成功块统计）。

juncaipeng

coplit的comments，也一一看下

juncaipeng

LGTM

codecov-commenter · 2026-01-30T09:52:42Z

Codecov Report

❌ Patch coverage is 20.34884% with 137 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@85db063). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
..._manager/transfer_factory/file_store/file_store.py	19.25%	130 Missing ⚠️
fastdeploy/cache_manager/cache_transfer_manager.py	12.50%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #6188   +/-   ##
==========================================
  Coverage           ?   67.13%           
==========================================
  Files              ?      387           
  Lines              ?    51390           
  Branches           ?     8009           
==========================================
  Hits               ?    34500           
  Misses             ?    14418           
  Partials           ?     2472

Flag	Coverage Δ
GPU	`67.13% <20.34%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…#6188) * fix(examples): comment out stop.sh to avoid error when script is missing * feat: add file_store support for cache manager * [fix] fix multi gpu transfer * [fix] fix global kvcache transfer * [Feature] [KVCache] support file_store kv cache backend * chore: update FileStore according to PR comments * fix: remove comments * fix: add swap_cache_layout for file store * fix: remove rank key * fix: Switch KV cache storage to pure file mode * Temporarily disable support for Tensor types * fix: remove args --kvcache_file_path & add envs FILE_BACKEND_STORAGE_DIR * fixx: Simplify cache_transfer_manager.py * fix: fix syntax bug * fix: Simplify file_store.py * fix: Use the key directly as the filename * fix: Simplify set() * fix: Simplify cache_transfer_manager.py & file_store.py * fix: Only support load to cpu buffer * feat: add FileStore backend for cache transfer * fix: guard zmq import

Moonchild1227 added 8 commits January 16, 2026 04:52

fix(examples): comment out stop.sh to avoid error when script is missing

9cb3244

feat: add file_store support for cache manager

aea58c9

Merge branch 'PaddlePaddle:develop' into my-feature-branch

f1258fa

[fix] fix multi gpu transfer

a986394

[fix] fix global kvcache transfer

9550dbf

Merge branch 'my-feature-branch' into feature/file-store

4f2f8d0

[Feature] [KVCache] support file_store kv cache backend

42bea23

Merge branch 'PaddlePaddle:develop' into feature/file-store

6afafd0

Moonchild1227 temporarily deployed to Metax_ci January 23, 2026 07:48 — with GitHub Actions Inactive

paddle-bot Bot added the contributor External developers label Jan 23, 2026

Jiang-Jia-Jun requested a review from Copilot January 23, 2026 07:49

Copilot started reviewing on behalf of Jiang-Jia-Jun January 23, 2026 07:49 View session

Copilot AI reviewed Jan 23, 2026

View reviewed changes

Moonchild1227 added 2 commits January 23, 2026 09:16

chore: update FileStore according to PR comments

32bec84

fix: remove comments

0bdc261

Moonchild1227 temporarily deployed to Metax_ci January 23, 2026 09:20 — with GitHub Actions Inactive

fix: add swap_cache_layout for file store

657f50b

Moonchild1227 temporarily deployed to Metax_ci January 23, 2026 10:08 — with GitHub Actions Inactive

juncaipeng reviewed Jan 26, 2026

View reviewed changes

Moonchild1227 added 5 commits January 27, 2026 03:15

fix: remove rank key

f60f534

fix: Switch KV cache storage to pure file mode

3211aca

Temporarily disable support for Tensor types

38c9386

fix: remove args --kvcache_file_path & add envs FILE_BACKEND_STORAGE_DIR

c74e863

Merge remote-tracking branch 'origin/develop' into feature/file-store

5a80fe4

Moonchild1227 had a problem deploying to Metax_ci January 27, 2026 07:52 — with GitHub Actions Error

Merge branch 'PaddlePaddle:develop' into feature/file-store

033f73b

Moonchild1227 temporarily deployed to Metax_ci January 27, 2026 07:53 — with GitHub Actions Inactive

juncaipeng reviewed Jan 27, 2026

View reviewed changes

Comment thread fastdeploy/cache_manager/cache_transfer_manager.py Outdated

Comment thread fastdeploy/cache_manager/cache_transfer_manager.py Outdated

Comment thread fastdeploy/cache_manager/cache_transfer_manager.py Outdated

Merge branch 'develop' into feature/file-store

9c28e19

Moonchild1227 had a problem deploying to Metax_ci January 27, 2026 09:21 — with GitHub Actions Error

Moonchild1227 added 2 commits January 27, 2026 09:24

fixx: Simplify cache_transfer_manager.py

1faf310

fix: fix syntax bug

8f28dd2

Moonchild1227 temporarily deployed to Metax_ci January 27, 2026 09:31 — with GitHub Actions Inactive

Merge branch 'develop' into feature/file-store

4f7e2cf

Moonchild1227 temporarily deployed to Metax_ci January 28, 2026 02:31 — with GitHub Actions Inactive

Moonchild1227 added 3 commits January 28, 2026 03:27

fix: Simplify file_store.py

9a875a3

fix: Use the key directly as the filename

9622cbd

fix: Simplify set()

eced210

Moonchild1227 had a problem deploying to Metax_ci January 28, 2026 03:28 — with GitHub Actions Error

Merge branch 'develop' into feature/file-store

9328905

Moonchild1227 temporarily deployed to Metax_ci January 28, 2026 03:34 — with GitHub Actions Inactive

fix: Simplify cache_transfer_manager.py & file_store.py

04ee01d

Moonchild1227 had a problem deploying to Metax_ci January 28, 2026 08:46 — with GitHub Actions Failure

fix: Only support load to cpu buffer

5ee62d7

Moonchild1227 had a problem deploying to Metax_ci January 28, 2026 11:27 — with GitHub Actions Failure

juncaipeng previously approved these changes Jan 29, 2026

View reviewed changes

Comment thread fastdeploy/cache_manager/cache_transfer_manager.py Outdated

Comment thread fastdeploy/cache_manager/cache_transfer_manager.py Outdated

feat: add FileStore backend for cache transfer

2c1bcf8

Moonchild1227 dismissed juncaipeng’s stale review via 2c1bcf8 February 2, 2026 07:14

Moonchild1227 temporarily deployed to Metax_ci February 2, 2026 07:14 — with GitHub Actions Inactive

juncaipeng previously approved these changes Feb 2, 2026

View reviewed changes

fix: guard zmq import

b39119c

Moonchild1227 dismissed juncaipeng’s stale review via b39119c February 2, 2026 13:07

Moonchild1227 had a problem deploying to Metax_ci February 2, 2026 13:07 — with GitHub Actions Failure

Jiang-Jia-Jun merged commit 39dc4b0 into PaddlePaddle:develop Feb 3, 2026
19 of 24 checks passed

Conversation

Moonchild1227 commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Usage or Command

Accuracy Tests

test script

test result

Checklist

Uh oh!

paddle-bot Bot commented Jan 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

juncaipeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juncaipeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Moonchild1227 commented Jan 23, 2026 •

edited

Loading

codecov-commenter commented Jan 30, 2026 •

edited

Loading