Repository navigation
GRPO grouping in multi-turn agent RL: is it valid to mix samples with different prompts in the same group? #489
Description
While debugging the calx example, we observed that as the number of training iterations increases, the agent tends to stop calling tools.
When using GRPO for multi-turn agent RL training (e.g., tool-calling scenarios), a single rollout can produce multiple training samples — one per conversation turn — where each turn has a different prompt (because earlier turns' tool calls and tool responses get appended to the context).
For example, with rollout_n=4 for one question:
| Sample | Prompt | Response | Reward |
|---|---|---|---|
| Rollout1-Turn0 | [system, user] | "Let me calculate... <tool_call>..." | 1.0 |
| Rollout1-Turn1 | [system, user, assistant, tool_response] | "The answer is 40" | 1.0 |
| Rollout2-Turn0 | [system, user] | "The answer is π square cm" (no tool) | 0.0 |
| Rollout3-Turn0 | [system, user] | "I'll use the calculator... <tool_call>..." | 1.0 |
| Rollout3-Turn1 | [system, user, assistant, tool_response] | "### ANSWER: 40" | 1.0 |
| Rollout4-Turn0 | [system, user] | "The answer is 35" (no tool) | 0.0 |
All 6 samples share the same uid (derived from the same original question), so GRPO computes advantages within this single group. Standard GRPO assumes all samples in a group are generated from the same prompt. However, in multi-turn agent scenarios, Turn0 samples have prompt [system, user], while Turn1 samples have prompt [system, user, assistant, tool_response] — these are fundamentally different inputs. Is it theoretically valid to compute group-normalized advantages across samples with different prompts?
Activity
ultmaster commentedon Feb 28, 2026
I think your understanding is correct. Xufang Luo (@XufangLuo) could you help with the theory part?
kyk666123 commentedon Apr 16, 2026
I've been thinking about the issue you mentioned. From the source code perspective, the logic AGL's GRPO algorithm uses to compute grouped advantages is this: assuming an agent has multiple roles with significantly different inputs, all different roles from the same training sample are grouped together to calculate the advantage. This approach is clearly unfair, as the reward calculation methods for different roles could be entirely different.
However, I came across a paper titled "Stronger MAS," published at ICLR 2026, which seems to have been proposed to address this exact problem. It might be worth taking a look at.