Atropos: Replication Package

This repository contains the source code and datasets for the paper "Atropos: Improving Cost-Benefit Trade-off of LLM-based Agents with Early Termination and Model Hotswap".

📂 Directory Structure

• trajectories/: Contains the raw execution logs generated by the three agents (AutoFL, AutoCodeRover, RepairAgent).
  • trajectories/hotswap/: Specifically contains the trajectories generated during the hotswap experiments.
• early_termination/: Includes the implementation for the early termination.
  • generate_dataset/: Scripts to construct Semantic Flow Graphs (SFGs) from raw trajectories.
  • results/: Stores the evaluation results of the prediction models.
  • trained_model/: Stores the checkpoints of the trained GCN models.
• hotswap/: Includes prediction result and cost calculation logic.
• experiments.ipynb: A Jupyter Notebook that analyzes the final results and reproduces the figures and tables presented in the paper.

🚀 Reproduction Guide

To reproduce the results reported in the paper, please follow the steps below in order:

1. Dataset Generation

Construct the Semantic Flow Graphs (SFGs) dataset required for training and testing the GCN model.

• Run the scripts located in early_termination/*/generate_dataset corresponding to the required setting.

2. Model Training

Train the GCN-based correctness prediction model using the SFGs generated in Step 1.

• The training scripts are located in early_termination/{agent}.

3. Hotswap Evaluation

Evaluate the hotswap technique and calculate the cost-benefit trade-offs using the early termination predictions.

• This step utilizes the raw trajectories located in trajectories/hotswap.

4. Visualization

Run experiments.ipynb to process the data and generate the final plots and tables shown in the paper.