arXivLiveBench: A Live Benchmark for Long-Context Modelling

Overview

This repository shares the code and data of our latest work arXivLiveBench: A Live Benchmark for Long-Context Modelling.

Requirements

You will need Python 3.10 or higher and install the required packages:

pip install -r requirements.txt

You will need a proper version of sglang, if you run the model locally.

Benchmark Creation

As illustrated in the paper, you need to generate the benchmark datasets for long-context modelling before evaluation. You can use the following script to generate the benchmark datasets in the paper.

python bench_create.py

The generated benchmark datasets will be located in generated_bench_v3/.

Evaluation

If you want to use a OpenAI Compatible API model for evaluation, you can setup the proper variables in start_eval_ext_model.sh and run the script to start the evaluation.

bash start_eval_ext_model.sh

If you want to use a local model for evaluation, please setup the proper model path in model_registry.py and run the following script to start the evaluation.

MODEL=<short_model_name, key of MODELS in model_registry.py> python bench_run.py

The result will be stored in outputs-v3bench/.

Citation

If you find this repository helpful, feel free to cite our paper.

coming soon