Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,11 @@ jobs:
- name: Install dependencies
run: npm ci --silent

- name: Unit and component tests
run: npx react-scripts test --watchAll=false
env:
CI: true

- name: Lint (via build)
run: npx react-scripts build
env:
Expand Down
20 changes: 19 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

# Python
venv/
venv-*/
__pycache__/
*.py[cod]
*.so
Expand Down Expand Up @@ -31,6 +32,8 @@ build/
*.log
*.pid
*.port
logs/
.pids/

# System files
.DS_Store
Expand All @@ -52,4 +55,19 @@ credentials.json
docker-compose.override.yml

#Solution Files
docs/challenges-walkthrough.md
docs/challenges-walkthrough.md

# Develpoment Files
AGENTS.md
CLAUDE.md
agent-docs
agent-docs/*
.cursor
.cursor/*

# Promo files
media/promo
media/promo/*

# Support attachments (generated names, not public)
support_uploads/
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,8 @@ If your change involves a significant architectural decision (new dependency, sc

By submitting a contribution, you agree that it will be licensed under the same terms as the rest of the project:

- **Code contributions** (to `app/` except `app/challenges/`, `frontend/`, `guardrails/`, `scripts/`, `docker/`, `config/config.yml`) are accepted under the **Apache License 2.0**
- **Training content contributions** (to `app/challenges/`, `prompts/`, `docs/`, `media/`, `config/labs.yml`) are accepted under **Creative Commons BY-NC-SA 4.0**
- **Code contributions** (to `app/` except `app/challenges/`, `frontend/`, `guardrails/`, `scripts/`, `docker/`, `alembic/`, `tests/`, `agent-docs/`, `colab_notebooks/`, `config/config.yml`) are accepted under the **Apache License 2.0**
- **Training content contributions** (to `app/challenges/`, `prompts/`, `docs/`, `media/`, `config/labs.yml`, `config/labs/`, `config/frameworks/`) are accepted under **Creative Commons BY-NC-SA 4.0**

This keeps the project licensing consistent. See [TRAINING_LICENSE.md](TRAINING_LICENSE.md) for details on the training content license.

Expand Down
2 changes: 1 addition & 1 deletion GOVERNANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ For larger changes (new features, architectural shifts, new lab categories), ope
AI Goat uses a dual licensing model to keep the platform open while protecting training materials:

- **Platform code** (backend, frontend, infrastructure, scripts) is licensed under **Apache License 2.0**. Anyone can use, modify, and distribute the code, including for commercial purposes.
- **Training content** (challenge evaluators, prompts, labs, walkthroughs, workshop materials) is licensed under **Creative Commons BY-NC-SA 4.0**. This content is free for learning, research, and non-commercial use. Commercial training usage requires permission from the maintainers.
- **Training content** (challenge evaluators, prompts, labs, framework/risk YAML, walkthroughs, workshop materials) is licensed under **Creative Commons BY-NC-SA 4.0**. This content is free for learning, research, and non-commercial use. Commercial training usage requires permission from the maintainers. Framework identifiers and official titles are used as labels only; descriptive prose in `config/frameworks/` is AIGoat-authored.

Note: The challenge evaluation framework (`app/challenges/`) falls under CC BY-NC-SA 4.0 because the evaluators encode specific attack patterns, detection thresholds, and success criteria that represent the training methodology — not generic application logic.

Expand Down
8 changes: 8 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,13 @@ AI Goat uses two licenses to cover different parts of the project.
• guardrails/
• scripts/
• docker/
• alembic/
• tests/
• agent-docs/
• colab_notebooks/
• config/config.yml
• config/defense_profiles.yml
• config/mcp_servers.yml

You may use, modify, and distribute this code under the terms of the
Apache License 2.0. See the LICENSE file in the project root.
Expand All @@ -37,6 +43,8 @@ AI Goat uses two licenses to cover different parts of the project.
• docs/
• media/
• config/labs.yml
• config/labs/
• config/frameworks/

You may use this content for personal learning, academic research,
and non-commercial purposes. You may modify and share it under the
Expand Down
560 changes: 300 additions & 260 deletions README.md

Large diffs are not rendered by default.

96 changes: 96 additions & 0 deletions RELEASE_NOTES_v2.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# v2.0: From LLM Chatbot to Agents and MCP

v2.0 is the largest release of AIGoat so far. In v0.3, AIGoat was an LLM security lab with one target, the Cracky chatbot, and 17 labs on the OWASP LLM Top 10. v2.0 adds the attack surfaces that appear once a model can **act**: a tool-calling agent, an MCP host and client connected to real MCP servers, agent memory, and an end-to-end kill chain. Labs grow from 17 to **46** across **three OWASP frameworks**, and the platform now also runs in Google Colab.

Changes since [v0.3](RELEASE_NOTES_v0.3.md) (tag `v0.3`, March 2026).

## Highlights

| | v0.3 | v2.0 |
|---|---|---|
| **Frameworks** | OWASP LLM Top 10 (2025) | OWASP LLM Top 10 (2026), Agentic Top 10 (2026), MCP Top 10 (2025, beta) |
| **Labs** | 17 | 46 (23 LLM, 12 Agentic, 11 MCP) |
| **Attack surfaces** | Chat | Chat, RAG, agent, MCP client, MCP host |
| **Defenses** | 3 levels on the chatbot | 3 levels on every surface, built from per-surface profiles over 19 composable controls |
| **Capstone** | None | Agentic Kill Chain: The Compromised eCommerce Agent |
| **Learning** | Attack Labs and Challenges | Adds OWASP Top 10 hub, risk pages, and a Threat Modeling workbench |
| **Run options** | Local, Docker | Local, Docker (network-reachable), Google Colab |
| **Database** | `create_all` on startup | Alembic migrations |
| **Backend test files** | 12 | 67, plus frontend component tests in CI |

## 1. New attack surfaces: agents and MCP

- **Tool-calling shop agent (`agent.runner`).** The planner's output is treated as untrusted. Every tool call passes an **Intent Gate** that validates arguments and applies the active defense profile before the tool runs. Runs show the plan, each tool call, and pending approvals. Sensitive tools (refunds, customer exports) can be held for human approval.
- **Agent memory.** Per-user, per-lab notes the agent can `remember` and `recall`. At Level 0 notes are trusted as policy, which is the lesson. Level 2 adds a memory scan.
- **MCP client and host (`mcp.client`, `mcp.host`).** Built on the official `mcp` Python SDK (protocol `2026-07-28`, stateless). Four allow-listed stdio servers ship with AIGoat: an official catalog, a community support server, an untrusted lookalike, and an internal management server. Servers are spawned with a scrubbed environment and a fixed command; a request can never choose what runs.
- **Staff admin assistant.** An MCP-speaking assistant used by the host labs, with a **Switch to Admin** persona shortcut in the header.
- **Support desk.** Customers open tickets with attachments; staff manage them in a feedback inbox. Tickets are a realistic injection channel for the agent and MCP labs.

## 2. Labs: 17 to 46, on three frameworks

- **OWASP LLM Top 10 (2026), 23 labs.** All LLM labs are renumbered to the 2026 list. New labs cover indirect injection through retrieved documents, unauthorized retrieval, tool allowlist bypass, retrieval keyword stuffing, trust-tier spoofing, and stale knowledge. LLM05 is rewritten as Data and Model Poisoning.
- **OWASP Agentic Top 10 (2026), 12 labs.** One lab per risk (ASI01 to ASI10), a second memory-poisoning lab for practicing Level 2, and the kill chain capstone. Each has a learner guide under `docs/labs/agentic/`.
- **OWASP MCP Top 10 (2025, beta), 11 labs.** Tool description poisoning, rug-pull redefinition, schema drift, decoy tokens, lookalike integrations, poisoned tickets, token passthrough, session reconstruction, unverified server identity, and context over-sharing. MCP05 (command injection) is documented but deliberately has no executable sink.
- **Labs are data, not code.** Lab content moved from a hardcoded frontend object to `config/labs/*.yml` (`llm`, `rag`, `agent`, `mcp`), served by `GET /api/labs/`. Each lab declares its risks, surface, difficulty, objective, example payloads, and the expected result at each defense level.
- **Evidence-based scoring for MCP labs.** MCP labs record what actually happened (tool listings, calls, results) and score a learner's submitted finding against that evidence, with hints, replay, and per-learner rug-pull state.

## 3. Agentic Kill Chain capstone

`killchain-1` shows why memory poisoning is more than a one-shot prompt injection. A hidden instruction in a product review or a PDF invoice (white 7 pt text) is ingested into connector memory, becomes a standing note in agent memory, and fires later on a routine request: a customer data export to an attacker BCC, an internal coupon disclosure, or a $1.00 price quote. Defended mode keeps the poison but holds each sensitive operation for an administrator approval bound to the exact arguments. Guardrailed mode adds rails that an approval cannot override (an egress allowlist, card-number masking, a staff-coupon rule, an ingestion scan and answer masking), so approving the wrong call still does not send data outside the shop.

The workbench at `/challenges?killchain=1` includes a live trace, memory inspector, attacker inbox, layered cleanup, and a hard reset. Lab data is copied from the shop into its own `kc_*` tables, and nothing leaves the machine. Guide: [docs/labs/killchain/README.md](docs/labs/killchain/README.md).

## 4. Defenses rebuilt as composable controls

- **Per-surface defense profiles.** `config/defense_profiles.yml` maps each surface and level to an ordered chain of controls. Level 0 is always empty.
- **New controls** for retrieval (provenance, access control, injection scan), tools (allowlist, coupon policy, approval, result scan), memory (scan), and MCP (tool pin, schema pin, origin pin, description scan, result scan), alongside the existing input validation, intent classification, output moderation, and NeMo rails.
- **NeMo Guardrails** now also checks agent goals, tool results, and answers at Level 2. When `nemoguardrails` is not installed, the same checks run locally and the result says so.

## 5. RAG upgrades

- Retrieval provenance and trust tiers on knowledge base documents (the target of the trust-tier spoofing lab).
- Optional hybrid retrieval (BM25 plus vectors, merged with reciprocal rank fusion), off by default.
- A retrieval trace endpoint and vector sync status, surfaced on a redesigned RAG page.

## 6. Learning experience and UI

- **Navigation regrouped** into Learn (OWASP Top 10, Threat Modeling), Try (Attack Labs, Challenges), and Console (RAG, MCP, Agent).
- **OWASP Top 10 hub and risk pages** with the three frameworks side by side, and the labs and challenges mapped to each risk.
- **Threat Modeling workbench** with an architecture diagram of AIGoat itself, a framework picker (STRIDE, MITRE ATLAS, and others), and six worked scenarios.
- **One hub layout** across Attack Labs, Challenges, RAG, MCP, Agent, OWASP, and Threat Modeling. The release label (v2.0) is shown in the nav and footer.

## 7. Run anywhere

- **Google Colab notebook** (`colab_notebooks/AIGoat_Colab.ipynb`, contributed by the community) runs the full stack in a Colab runtime, with an optional tool-calling model. It prints the app link on Colab's own domain instead of a `localhost` label, and a separate link for the API docs. Guide: [colab_notebooks/AIGoat_Colab.md](colab_notebooks/AIGoat_Colab.md).
- **Docker is reachable from your network.** Published ports bind to `0.0.0.0`, and the frontend calls whichever host opened the page, so a lab machine can serve a classroom.
- **`start.sh`** pins Python 3.11 to match CI, runs Alembic migrations (and stamps an existing v0.3 database first), and syncs the support inbox and challenge metadata on every start.

## 8. Reliability fixes that change model behavior

- **System prompts now reach the model on `/api/chat`.** Ollama ignores a top-level `system` field on that endpoint, so agent, MCP, and RAG calls previously ran without their system prompt. They now send it as a leading system message. Expect live-model behavior on those labs to differ from earlier builds.
- **Agent step budget.** Agent runs reserve their last step for a written answer, so runs no longer end with a blank reply. Native tool-call history is replayed to Ollama correctly.
- **Stopping a chat** now cancels the Ollama stream on the server.

## 9. Community contributions

- Google Colab notebook and guide ([#5](https://github.com/AISecurityConsortium/AIGoat/pull/5)).
- Python 3.9 type hint compatibility for SQLAlchemy models ([#7](https://github.com/AISecurityConsortium/AIGoat/pull/7)).
- Fix for a `KeyError` when creating a coupon without optional fields ([#3](https://github.com/AISecurityConsortium/AIGoat/pull/3)).

Thank you to everyone who opened issues and pull requests.

## Upgrade notes

1. **Pull and restart.** `./scripts/stop.sh && git pull && ./scripts/start.sh`. An existing database is stamped and migrated in place. For a clean slate, use `./scripts/start.sh --fresh`. Docker users: `docker compose up --build`.
2. **New Python dependencies** (installed by `start.sh`): `alembic`, `mcp>=2.2.0,<2.3`, `pypdf`, `bm25s`, `eval-type-backport`.
3. **Lab IDs follow the 2026 numbering.** For example, the old `llm10-1` (Token Flood) is now `llm06-1`. Migration `0005` renames stored lab sessions, agent runs, and memory, and the frontend migrates saved lab completion once. Update any bookmarks, scripts, or workshop handouts that use old IDs.
4. **Pull a tool-calling model** for the agent, MCP host, and kill chain labs: `ollama pull qwen3.5:9b`, then select it in the console or set `ollama.agent_model`. Mistral remains the default and is enough for the chat and RAG labs.
5. **Docker is now reachable on your network.** Run it only on a trusted network, or publish ports on `127.0.0.1` yourself.
6. **Unchanged:** seed users and passwords, the 9 CTF challenges and their points, the flag format `AIGOAT{32 hex}` and its per-user derivation, and existing API paths. All API changes are additive.

## Known limitations

- Local models follow planted instructions probabilistically. A lab that does not trigger records that honestly instead of faking the impact; retry, or use a tool-capable model.
- Mistral does not emit native tool calls, so the agent and MCP host labs show no tool calls with it.
- The Colab notebook clones `main` and keeps no state between sessions. Colab links expire when the runtime disconnects.
- MCP05 has no lab, by design.
7 changes: 4 additions & 3 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,11 @@ The following are part of the educational design and should NOT be reported as s
| **Sensitive Info Disclosure** (LLM02) | Admin credentials, customer data, and config paths leaked through the chatbot at Level 0 |
| **Data Poisoning** (LLM04) | Injecting false information via reviews, tips, and Knowledge Base entries |
| **Insecure Output** (LLM05) | XSS via chatbot output at Defense Level 0 (HTML/JS rendering) |
| **System Prompt Leakage** (LLM07) | Extracting the chatbot's hidden system prompt and confidential config block |
| **RAG Weaknesses** (LLM08) | Poisoning the Knowledge Base, manipulating vector retrieval, context flooding |
| **Misinformation** (LLM09) | Inducing the chatbot to fabricate certifications, endorsements, or specifications |
| **System Prompt Leakage** (LLM08 Hidden Context Exposure) | Extracting the chatbot's hidden system prompt and confidential config block |
| **RAG Weaknesses** (LLM09 Vector and Embedding Weaknesses) | Poisoning the Knowledge Base, manipulating vector retrieval, context flooding |
| **Misinformation** (LLM07) | Inducing the chatbot to fabricate certifications, endorsements, or specifications |
| **Weak Credentials** | Default demo accounts (`admin/admin123`, `alice/password123`, etc.) |
| **Persona switch** | The account menu swaps Alice and Admin by reusing tokens already returned by `GET /api/auth/demo-users/`. It is training-only and demo-token-scoped. It does not bypass login for any other account. |
| **Context Override & Role Confusion** | Making the chatbot abandon its role or follow injected instructions |
| **Guardrail Bypass** | Successfully bypassing Level 1 or Level 2 defenses with creative prompts |

Expand Down
5 changes: 3 additions & 2 deletions TRAINING_LICENSE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,10 +51,11 @@ that want to use AI Goat for training — we just ask that you get in touch firs

## Platform code is separate

The platform itself (backend, frontend, infrastructure, scripts) is licensed
The platform itself (backend, frontend, infrastructure, scripts, database
migrations, tests, agent-docs, and Colab notebooks) is licensed
under **Apache License 2.0** and can be used freely, including for commercial
purposes. This training license only applies to the educational content listed
above.
above. See [NOTICE](NOTICE) for the directory list.

See the root [LICENSE](LICENSE) file for the Apache 2.0 terms.

Expand Down
44 changes: 44 additions & 0 deletions alembic.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Alembic config. The live database URL is injected in alembic/env.py from
# AIGoat's config (CONFIG_PATH / get_settings), not from this file.

[alembic]
script_location = alembic
prepend_sys_path = .
version_path_separator = os
sqlalchemy.url = sqlite:///./aigoat.db

[post_write_hooks]

[loggers]
keys = root,sqlalchemy,alembic

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = WARN
handlers = console
qualname =

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine

[logger_alembic]
level = INFO
handlers =
qualname = alembic

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S
79 changes: 79 additions & 0 deletions alembic/env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
"""Alembic environment.

Uses a *synchronous* SQLite driver even though the app engine is
``sqlite+aiosqlite``. Alembic's default online path is sync; swapping the
scheme here keeps migrations simple and is the chosen option from T050
(async template vs sync-with-driver-swap).
"""
from __future__ import annotations

import os
from logging.config import fileConfig

from alembic import context
from sqlalchemy import engine_from_config, pool

import app.models as _models
from app.core.config import get_settings
from app.core.database import Base

assert _models # registers every model on Base.metadata

config = context.config

if config.config_file_name is not None:
fileConfig(config.config_file_name)

target_metadata = Base.metadata


def _database_url() -> str:
"""Sync URL for Alembic. Prefer ALEMBIC_DATABASE_URL (tests), else app config."""
override = os.environ.get("ALEMBIC_DATABASE_URL")
raw = override if override else get_settings().database.url
if raw.startswith("sqlite+aiosqlite://"):
return "sqlite://" + raw[len("sqlite+aiosqlite://"):]
return raw


config.set_main_option("sqlalchemy.url", _database_url())


def run_migrations_offline() -> None:
url = _database_url()
context.configure(
url=url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
render_as_batch=True,
)

with context.begin_transaction():
context.run_migrations()


def run_migrations_online() -> None:
configuration = config.get_section(config.config_ini_section, {}) or {}
configuration["sqlalchemy.url"] = _database_url()
connectable = engine_from_config(
configuration,
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)

with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=True,
)

with context.begin_transaction():
context.run_migrations()


if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()
Loading
Loading