ADPO

This is the official code and data for our ICML 2026 paper, "Autoregressive Direct Preference Optimization"

💨 TL;DR

We propose Autoregressive Direct Preference Optimization (ADPO), a preference optimization method that applies the Bradley–Terry comparison through the autoregressive generation process, improving mathematical reasoning performance over standard DPO and recent variants.

📖 Introduction

Direct Preference Optimization (DPO) is a simple and effective approach for aligning language models with preference pairs, but its standard formulation treats each completion as a sequence-level comparison. We revisit this formulation and introduce Autoregressive DPO (ADPO), which explicitly incorporates the autoregressive assumption into both the reference and learnable models before applying the Bradley–Terry (BT) model.

Empirically, ADPO improves mathematical reasoning performance on benchmarks such as GSM8K and MATH500.

📝 Changelog

• [2026.06.03] Initial release of the ADPO implementation.

🚀 Quick Start

⚙️ Setup

Cloning the repository

git clone git@github.com:stjohn2007/ADPO.git
cd ADPO

Preparing the virtual environment

uv venv --python 3.10
# Modify the PyTorch and CUDA versions to match your environment if needed
uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128
uv pip install -r requirements.txt

Authorize HuggingFace

hf auth login

🧠 Training

source .venv/bin/activate
bash scripts/train_adpo.sh

📊 Evaluation

source .venv/bin/activate
bash scripts/eval_adpo.sh

⚠️ Note on Reproducibility

Training and evaluation scores may vary across runs due to stochastic sampling, GPU kernels, hardware differences, and checkpoint selection.

🙏 Acknowledgement

Our implementation is built upon the excellent work Critical-Tokens-Matter. We sincerely thank the authors for making their code publicly available.

📚 Citation

@inproceedings{
    oi2026autoregressive,
    title={Autoregressive Direct Preference Optimization},
    author={Masanari Oi and Mahiro Ukai and Masahiro Kaneko and Naoaki Okazaki and Nakamasa Inoue},
    booktitle={Forty-third International Conference on Machine Learning},
    year={2026},
}