Repository navigation
[BUG] ring sum_scatter's "not supported" error never reaches the caller, the group hangs instead #4285
Description
What happens
sum_scatter is not implemented for the ring backend and says so,
ring.cpp:
throw std::runtime_error("[ring] sum_scatter not supported.");The message never arrives. A two rank probe wrapping the call in try/except:
[rank 0] calling sum_scatter
[rank 1] calling sum_scatter
and then nothing: no exception, no message, no completion. The throw
happens in ReduceScatter::eval_cpu on the stream thread and is lost on the way
back to the caller.
Four ranks behave the same way: every rank enters, none returns, 0% CPU.
This is why FSDP cannot run on the ring backend, and the first thing a user sees is
a job that stopped rather than a feature that is missing.
Not fixed by #3742
I built 1add2cfe4 and ran the same probe: still a silent indefinite hang. So the
CPU stream error propagation in that PR does not cover this path.
Two separable problems
sum_scatteris missing on ring. PR below implements it.- An exception raised from
ReduceScatter::eval_cpudoes not reach the caller.
That outlives the first fix and is worth its own look.
Activity
erwinzhang7 commentedon Aug 17, 2026
Correcting the "Not fixed by #3742" section: it cites
1add2cfe4, which nolonger exists. That branch was rewritten on 08-16 and #4174 was folded into it,
so the commit is not checkable and the claim cannot be verified as written.
Re-ran the same probe on two machines, ranks on separate hosts, and the picture
is clearer than the original report:
RuntimeError: [ReduceScatter] Not implemented yet.9ab977b56So this is a regression rather than a permanent gap. 0.32.0 throws a
different string,
[ReduceScatter] Not implemented yet., and it reaches thecaller. The throw quoted in the issue,
[ring] sum_scatter not supportedinring.cpp, is the later one and it does not. Anyone on 0.32.0 would have seen aclear error.
Please note a single host does not
reproduce this. Two ranks on one machine raise cleanly on both current main and
#3742, so
mlx.launch -n 2on one box tests nothing here, the mechanism needs apeer on the far side of a socket.
As a measure of how completely it hangs: four ranks from the original run were
still alive 23 hours 30 minutes later, at 0% CPU, paired to the second across
both machines.
The second half of the report stands unchanged: an exception raised from
ReduceScatter::eval_cpudoes not reach the caller, and that outlives any fixfor the missing op.
zcbenz commentedon Aug 21, 2026
I'm closing this as won't fix at the moment as we don't plan to work on it.