Phase 4: Add single-card operator comparisons and drift attribution #30

Background

Split from #6 Phase 4: Single-Card Operator Integration.

This issue owns single-card train-vs-infer operator comparison and drift attribution. It should build on the Phase 3 audit/replay harness and should not introduce distributed communication attribution before the corresponding single-card operator path is understood.

Scope

• Add train/infer role pairing for the target forward and loss-side operator boundaries:
  • attention
  • lm_head / logp
  • matmul projections
  • RMSNorm
  • RoPE
  • SwiGLU
  • embedding
  • RL loss fragments such as ratio_kl, PPO/GRPO/DPO fragments when they are exposed as operator-level comparisons
• Fill missing second/reference implementations needed for single-card comparison, especially the CUDA RMSNorm comparison path.
• Expand the operator comparison harness from pass/fail tolerances to drift-attribution metrics such as max/mean/p99 error and active-token dlogp contribution when applicable.
• Extend the operator spec registry or equivalent test surface so the forward-chain operators are covered consistently.
• Add a forward-chain accumulation test that measures how small per-operator drift grows across layers.
• Treat forward-chain drift as an audit, CI, or backend-admission test, not as a production preflight before every operator call.
• Add operator-level batch-invariance tests for single-sample vs mixed-batch execution, padding/packing layouts, row position changes, and active-mask-density changes.
• Add deterministic logp bitwise repeatability tests only for backends that explicitly advertise same-backend, same-build deterministic behavior.

Non-Goals

• Do not run distributed TP/SP/CP sweeps or communication/NCCL attribution; those belong to Phase 5.
• Do not duplicate the Phase 3 replay harness, metadata validation, or result-cube ownership.
• Do not redo Phase 2 linear_logp strict fallback checks; reuse that surface as one operator case when possible.
• Do not make production Blackwell/B-card end-to-end A/B claims; [Experiment] Build Blackwell train-inference mismatch baseline for VIME post-training #24 remains the experiment entry for that.
• Do not split this into separate operator issues unless the implementation becomes too large for one PR series.

Acceptance Criteria

• Every target operator can run a single-card train-vs-infer comparison or reports a structured unsupported reason.
• A backend is not advertised as strict until same-sample outputs and, where relevant, gradients pass the covered batch-invariance tests.
• Operator-level drift attribution is visible before distributed tests run.
• The forward-chain test reports cumulative drift without hiding which operator boundary first introduced the drift.
• Deterministic/bitwise claims are separated from tolerance-based cross-backend parity claims.

Dependencies

• Roadmap: RL-Kernel integration roadmap for vime #6.
• Phase 3 audit harness: Phase 3: Build consistency audit harness and replay diagnostics #29.
• Phase 2 linear_logp production and strict-mode work: Phase 2: Productionize linear_logp backend support and telemetry #20, Phase 2: Add linear_logp strict-mode checks and fallback tests #21.