[VikingDeng]

What does this PR do ?

First step for the SFTDataset item in #5676. With --varlen-pretokenized-dataset, VarlenDataset reads tokens and an assistant-only loss mask from two IndexedDataset streams (<prefix> and <prefix>.mask) instead of tokenizing at load time, so it works with the existing blend/split and DP-balanced scheduler code. tools/preprocess_varlen_sft.py writes both streams from SFTTokenizer targets and refuses prompt formats whose targets are all ones (like default), since those would silently train on the whole conversation.

Not in this PR: MFFD packing and thinking/tool masks from #5017. I kept this to the data format and reader so it's easy to review; happy to adjust the scope if you had something else in mind.

Tests (CPU, Linux):

pytest -q tests/unit_tests/data/test_varlen_pretokenized.py   # 14 passed
pytest -q tests/unit_tests/data/test_varlen_dataset.py        # 63 passed, 3 failed (same 3 fail on main: they need NCCL)
pytest -q tests/unit_tests/data/test_bin_reader.py            # passed

End to end, I preprocessed 5 hand-written multi-turn conversations with the Qwen2.5-0.5B tokenizer and read them back through VarlenDataset; the masked positions decode to exactly the assistant replies. One existing limitation: SFTTokenizer's identity format hits a BPE assertion across turns with the Qwen tokenizer, so that check used the nemotron-h-aligned format.

AI assistance: I used Claude Code to help write the code and tests. I chose the design, ran the validation above on my own machines and checked the results before opening this PR, and I'll handle the review.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.