Skip to content

[BUG] Full validation fails with a single dataset #7174

Open

Description

@jeyaletheia

Describe the bug

On upstream c6be9199750845bd99a7d1a438862bb9ab29b630, full validation with one dataset raises TypeError: 'int' object is not subscriptable before evaluation. This does not require Engram.

Steps/Code to reproduce bug

Set full_validation=True, multiple_validation_sets=False, and eval_iters=3 on the TP source rank.

The upstream dispatch normalizes the count to a list, but broadcasts a tensor created from the original scalar. This CPU snippet isolates the type mismatch; it does not reproduce distributed evaluation:

import torch

count = 3
eval_iters = [count]
eval_iters = torch.tensor(count).tolist()  # 3, not [3]
eval_iters[0]  # TypeError

Expected behavior

Broadcast [3] and pass 3 to the evaluator. Constructing the tensor from normalized eval_iters preserves the required list shape after .tolist().

Additional context

Local tests on 8×H800 used TP=2, real NCCL broadcast and the native dispatch with a mocked evaluator. The single-dataset case failed upstream; the [2, 3] two-dataset control passed. Both passed with the one-line fix.

Environment: Python 3.12.3, PyTorch 2.12.0a0+5aff3928d8.nv26.05, CUDA 13.2. This checks dispatch, not complete dataset evaluation or official CI. The full regression tests will accompany the fix PR.

#3388 addresses a multiple-dataset deadlock; this report concerns scalar/list conversion for one dataset.

@NVIDIA/mcore-oncall

Activity

  1. asolergi-nv commented on Sep 10, 2026

    @asolergi-nv
    Contributor

    Thanks for raising the issue. We are taking a look & will reach back shortly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions