Skip to content

[¬Re] Differential Attention at Small Scale: A Paired, Cross-Stack Reproduction with No Generalization Benefit #114

Description

@guygrigsby

Original article: Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei. Differential Transformer. ICLR 2025. https://arxiv.org/abs/2410.05258

PDF URL: https://github.com/guygrigsby/diff-mlx/blob/main/docs/paper/rescience/article.pdf
Metadata URL: https://github.com/guygrigsby/diff-mlx/blob/main/docs/paper/rescience/metadata.yaml
Code URL: https://github.com/guygrigsby/diff-mlx

Scientific domain: Machine Learning
Programming language: Python
Suggested editor: No preference


Independent reimplementation of differential attention in MLX on Apple Silicon with custom Metal kernels, verified against the official PyTorch reference at 1e-7 and replicated cross-stack in PyTorch on NVIDIA CUDA. Paired byte-identical initialization throughout. A four-seed band shows the small-scale paired delta is seed noise, the 162M/2.0B run shows a train-loss win with a held-out loss and the positional profile shows no long-context advantage. Scale sits three orders of magnitude below the original, so the negative is bounded to the small-scale regime and the article says so.

Evidence dataset: https://doi.org/10.5281/zenodo.21570478. Title carries [¬Re] with the scale qualifier doing the bounding. Open to editorial guidance on the prefix given the partial, reduced-scale nature of the replication.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions