Skip to content

Tensor Metrics 4: Integrate tensor observation into training - #7472

Merged
deepakn94 merged 1 commit into
NVIDIA:mainfrom
philipcmonk:pmonk/tensor-metrics-training-integration-main
Oct 27, 2026
Merged

deepakn94 merged 1 commit into
NVIDIA:mainfrom
philipcmonk:pmonk/tensor-metrics-training-integration-main

Conversation

@philipcmonk

Copy link
Copy Markdown
Contributor

Recreated version of #6650, which was automatically closed when its stacked base branch was deleted. See #6650 for the original description and review discussion.

This PR targets main and includes the two metric-definition commits from #7471, followed by the three integration commits from #6650, including the fault-tolerance timeout fix. Conflicts with current main were resolved while preserving its training callbacks and schedule-plumbing tests.

Depends on #7471, but intentionally targets main to avoid another automatic closure when a temporary base branch is deleted.

@copy-pr-bot

copy-pr-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Build scheduled tensor metric observers from the logger configuration, capture
forward observations, and collect parameter and gradient metrics before the
optimizer step. Give metric iterations their own fault-tolerance timeout section.

Preserve the current tokenizer startup option, sample accounting, training
callbacks, and schedule plumbing. Adapt the integration fixtures to current
training configuration ownership and cover timeout transitions with regression
tests.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Philip Monk <pmonk@nvidia.com>
@deepakn94
deepakn94 enabled auto-merge October 27, 2026 20:14
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the Approved All necessary approvals have been made label Oct 27, 2026
@deepakn94
deepakn94 added this pull request to the merge queue Oct 27, 2026
@nemo-automation-bot

Copy link
Copy Markdown

🔄 Merge queue validation started!

You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/40576553756

Merged via the queue into NVIDIA:main with commit 364eef7 Oct 27, 2026
106 of 108 checks passed

This branch was previously deployed

2 inactive deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved All necessary approvals have been made complexity: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants