You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] Full validation fails with a single dataset #7174
On upstream c6be9199750845bd99a7d1a438862bb9ab29b630, full validation with one dataset raises TypeError: 'int' object is not subscriptable before evaluation. This does not require Engram.
Steps/Code to reproduce bug
Set full_validation=True, multiple_validation_sets=False, and eval_iters=3 on the TP source rank.
The upstream dispatch normalizes the count to a list, but broadcasts a tensor created from the original scalar. This CPU snippet isolates the type mismatch; it does not reproduce distributed evaluation:
importtorchcount=3eval_iters= [count]
eval_iters=torch.tensor(count).tolist() # 3, not [3]eval_iters[0] # TypeError
Expected behavior
Broadcast [3] and pass 3 to the evaluator. Constructing the tensor from normalized eval_iters preserves the required list shape after .tolist().
Additional context
Local tests on 8×H800 used TP=2, real NCCL broadcast and the native dispatch with a mocked evaluator. The single-dataset case failed upstream; the [2, 3] two-dataset control passed. Both passed with the one-line fix.
Environment: Python 3.12.3, PyTorch 2.12.0a0+5aff3928d8.nv26.05, CUDA 13.2. This checks dispatch, not complete dataset evaluation or official CI. The full regression tests will accompany the fix PR.
#3388 addresses a multiple-dataset deadlock; this report concerns scalar/list conversion for one dataset.
Activity
asolergi-nv commentedon Sep 10, 2026
Thanks for raising the issue. We are taking a look & will reach back shortly