Original article: Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei. Differential Transformer. ICLR 2025. https://arxiv.org/abs/2410.05258
PDF URL: https://github.com/guygrigsby/diff-mlx/blob/main/docs/paper/rescience/article.pdf
Metadata URL: https://github.com/guygrigsby/diff-mlx/blob/main/docs/paper/rescience/metadata.yaml
Code URL: https://github.com/guygrigsby/diff-mlx
Scientific domain: Machine Learning
Programming language: Python
Suggested editor: No preference
Independent reimplementation of differential attention in MLX on Apple Silicon with custom Metal kernels, verified against the official PyTorch reference at 1e-7 and replicated cross-stack in PyTorch on NVIDIA CUDA. Paired byte-identical initialization throughout. A four-seed band shows the small-scale paired delta is seed noise, the 162M/2.0B run shows a train-loss win with a held-out loss and the positional profile shows no long-context advantage. Scale sits three orders of magnitude below the original, so the negative is bounded to the small-scale regime and the article says so.
Evidence dataset: https://doi.org/10.5281/zenodo.21570478. Title carries [¬Re] with the scale qualifier doing the bounding. Open to editorial guidance on the prefix given the partial, reduced-scale nature of the replication.
Original article: Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei. Differential Transformer. ICLR 2025. https://arxiv.org/abs/2410.05258
PDF URL: https://github.com/guygrigsby/diff-mlx/blob/main/docs/paper/rescience/article.pdf
Metadata URL: https://github.com/guygrigsby/diff-mlx/blob/main/docs/paper/rescience/metadata.yaml
Code URL: https://github.com/guygrigsby/diff-mlx
Scientific domain: Machine Learning
Programming language: Python
Suggested editor: No preference
Independent reimplementation of differential attention in MLX on Apple Silicon with custom Metal kernels, verified against the official PyTorch reference at 1e-7 and replicated cross-stack in PyTorch on NVIDIA CUDA. Paired byte-identical initialization throughout. A four-seed band shows the small-scale paired delta is seed noise, the 162M/2.0B run shows a train-loss win with a held-out loss and the positional profile shows no long-context advantage. Scale sits three orders of magnitude below the original, so the negative is bounded to the small-scale regime and the article says so.
Evidence dataset: https://doi.org/10.5281/zenodo.21570478. Title carries [¬Re] with the scale qualifier doing the bounding. Open to editorial guidance on the prefix given the partial, reduced-scale nature of the replication.