Skip to content

[BugFix] RDMA: retry ibv_post_send from bad_wr and drain CQ on failure #8091

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Sunny-bot1 merged 2 commits into from
Jul 2, 2026
Merged

[BugFix] RDMA: retry ibv_post_send from bad_wr and drain CQ on failure #8091

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f031238
fix rdma retry
Sunny-bot1 Jul 2, 2026
7af1cba
fix
Sunny-bot1 Jul 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion fastdeploy/cache_manager/transfer_factory/kvcache_transfer/src/kvcache_rdma.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1513,8 +1513,9 @@ bool RDMACommunicator::post_send_with_retry(struct RdmaContext* ctx,
last_wr->send_flags |= IBV_SEND_SIGNALED;
}

struct ibv_send_wr* cur_wr = wr_list;
do {
ret = ibv_post_send(ctx->qp, wr_list, &bad_wr);
ret = ibv_post_send(ctx->qp, cur_wr, &bad_wr);
if (ret == 0) {
Comment on lines +1516 to 1519
if (need_poll) {
ctx->conn.wc_count = 0;
Expand All @@ -1532,8 +1533,22 @@ bool RDMACommunicator::post_send_with_retry(struct RdmaContext* ctx,
errno,
retries + 1,
max_retries);
// Non-blocking CQ drain to free SQ slots before retrying.
// Do not use poll_cq_with_timeout here: ibv_post_send failure does not
// guarantee a CQE is available (e.g. sync errors, unsignaled WRs), so a
// blocking wait would stall the retry loop for up to
// RDMA_POLL_CQE_TIMEOUT per attempt (up to 210s total).
{
struct ibv_wc wc_array[32];
int n;
while ((n = ibv_poll_cq(ctx->cq, 32, wc_array)) > 0) {
Comment thread
Sunny-bot1 marked this conversation as resolved.
}
}
Comment on lines +1541 to +1546
usleep(1000);
retries++;
if (bad_wr)
cur_wr =
bad_wr; // resume from the failed WR instead of retrying from head
}
} while (retries < max_retries);

Expand Down
Loading