Mirror GUI — LLM-based GUI Simulator

Mirror GUI is a GUI simulator driven by large language models (LLMs), designed to test and evaluate AI agents interacting with a desktop-like environment. It simulates an Ubuntu-style desktop with application windows, UI elements and a file system so agents can perform GUI actions and researchers can analyze behavior and safety.

Features

• Full desktop environment simulation: windows, UI elements and a simple file system.
• ReAct-based AI agents that reason and act using thought/action cycles.
• Flexible task definitions and scene generation.
• Concurrent execution for large-scale testing.
• Rich logging and interaction traces for analysis.
• Safety-oriented evaluation, including tests for deceptive popup dialogs.

Project layout

See PROJECT_STRUCTURE.md for a detailed breakdown. Key folders:

• src/ — core implementation (simulator, agent, models, task generation, alignment tools)
• data/ — example tasks, blueprints and logs
• scripts/ — helper scripts

Quick start

Requirements

• Python 3.8+
• Access to an LLM API (configure API keys or endpoints in src/utils/config.py or via environment variables)

Install

pip install -r requirements.txt

Configure

Edit src/utils/config.py and fill MODELS_CONFIG with model names, base URLs and API keys, or provide the same values via environment variables.

Generate task blueprints

python src/generation/pipeline_generator.py

Generate tasks

python src/generation/batch.py

Alternatively, you can first generate task blueprints and then produce tasks from those blueprints:

python src/generation/pipeline_generator.py
python src/generation/batch.py

Both the generation scripts and the runner support concurrent execution and can be run in parallel (multi-threaded) to speed up large-scale generation and evaluation.

Run the runner

python src/cli/main.py --task_dir data/tasks/generated_tasks --output_dir outputs/example --max_workers 1 --max_steps 15

For more options run python src/cli/main.py -h.

Contributing & License

This project is MIT-licensed — see LICENSE. Contributions are welcome via issues and pull requests.

Core Developers

• Zhang Wenqi — Lead developer
• Shen Yulin — Prompt engineering