Skip to content

Add AI Trace plugin for AI-assisted debugging #5432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DenysKuchma wants to merge 2 commits into
base: 4.x
Choose a base branch
from
+964 −0
Open

Add AI Trace plugin for AI-assisted debugging #5432

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b80e931
add aiTrace plugin for AI-assisted test debugging
DenysKuchma Feb 4, 2026
bc43021
add path and little fix
DenysKuchma Feb 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
260 changes: 260 additions & 0 deletions docs/aitrace.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
---
permalink: /aitrace
title: AI Trace Plugin
---

# AI Trace Plugin

AI Trace Plugin generates AI-friendly trace files for debugging with AI agents like Claude Code.

When a test fails, you need to understand what went wrong: what the page looked like, what elements were present, what errors occurred, and what led to the failure. This plugin automatically captures all that information and organizes it in a format optimized for AI analysis.

## Quick Start

Enable the plugin in your `codecept.conf.js`:

```javascript
export const config = {
tests: './*_test.js',
output: './output',
helpers: {
Playwright: {
url: 'https://example.com',
// Optional: Enable HAR/trace for HTTP capture
recordHar: {
mode: 'minimal',
content: 'embed',
},
trace: 'on',
keepTraceForPassedTests: true,
},
},
plugins: {
aiTrace: {
enabled: true,
},
},
}
```

Run tests:

```bash
npx codeceptjs run
```

After test execution, trace files are created in `output/trace_*/trace.md`.

## Artifacts Created

For each test, a `trace_<sha256>` directory is created with the following files:

**trace.md** - AI-friendly markdown file with test execution history

**0000_screenshot.png** - Screenshot for each step

**0000_page.html** - Full HTML of the page at each step

**0000_aria.txt** - ARIA accessibility snapshot (AI-readable structure without HTML noise)

**0000_console.json** - Browser console logs

When HAR or trace recording is enabled in your helper config, links to those files are also included.

## Trace File Format

The `trace.md` file contains a structured execution log with links to all artifacts:

```markdown
file: /path/to/test.js
name: My test scenario
time: 3.45s
---

I am on page "https://example.com"
> navigated to https://example.com/
> [HTML](./0000_page.html)
> [ARIA Snapshot](./0000_aria.txt)
> [Screenshot](./0000_screenshot.png)
> [Browser Logs](0000_console.json) (7 entries)
> HTTP: see [HAR file](../har/...) for network requests

I see "Welcome"
> navigated to https://example.com/
> [HTML](./0001_page.html)
> [ARIA Snapshot](./0001_aria.txt)
> [Screenshot](./0001_screenshot.png)
> [Browser Logs](0001_console.json) (0 entries)
```

## Configuration

### Basic Configuration

```javascript
plugins: {
aiTrace: {
enabled: true,
}
}
```

### Advanced Configuration

```javascript
plugins: {
aiTrace: {
enabled: true,

// Artifact capture options
captureHTML: true, // Save HTML for each step
captureARIA: true, // Save ARIA snapshots
captureBrowserLogs: true, // Save console logs
captureHTTP: true, // Links to HAR/trace files
captureDebugOutput: true, // CodeceptJS debug messages

// Screenshot options
fullPageScreenshots: false, // Full page or viewport only

// Output options
output: './output', // Where to save traces
deleteSuccessful: false, // Delete traces for passed tests

// Step filtering
ignoreSteps: [
/^grab/, // Ignore all grab* steps
/^wait/, // Ignore all wait* steps
],
}
}
```

### Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `enabled` | boolean | `false` | Enable/disable the plugin |
| `captureHTML` | boolean | `true` | Capture HTML for each step |
| `captureARIA` | boolean | `true` | Capture ARIA snapshots |
| `captureBrowserLogs` | boolean | `true` | Capture browser console logs |
| `captureHTTP` | boolean | `true` | Capture HTTP requests (requires HAR/trace) |
| `captureDebugOutput` | boolean | `true` | Capture CodeceptJS debug output |
| `fullPageScreenshots` | boolean | `false` | Use full page screenshots |
| `output` | string | `'./output'` | Directory for trace files |
| `deleteSuccessful` | boolean | `false` | Delete traces for passed tests |
| `ignoreSteps` | array | `[]` | Steps to ignore (regex patterns) |

## Best Practices

### Optimize for Failing Tests

Save disk space by only keeping traces for failed tests:

```javascript
plugins: {
aiTrace: {
enabled: true,
deleteSuccessful: true, // Only keep failing tests
}
}
```

### Ignore Noise

Don't capture logs for `grab` and `wait` steps:

```javascript
plugins: {
aiTrace: {
enabled: true,
ignoreSteps: [/^grab/, /^wait/],
}
}
```

### Selective Artifact Capture

Capture only what you need to reduce file sizes:

```javascript
plugins: {
aiTrace: {
enabled: true,
captureHTML: false, // Skip HTML (saves ~500KB per step)
captureARIA: true, // Keep ARIA (only ~16KB)
captureBrowserLogs: false, // Skip console logs
}
}
```

### Enable HTTP Capture

For network debugging, enable HAR/trace in your helper:

```javascript
helpers: {
Playwright: {
recordHar: {
mode: 'minimal',
content: 'embed',
},
trace: 'on',
keepTraceForPassedTests: true,
},
plugins: {
aiTrace: {
enabled: true,
captureHTTP: true, // Links to HAR/trace files
},
},
}
```

## Using with AI Agents

The trace format is optimized for AI agents like Claude Code. When debugging a failing test, just point the AI agent to the `trace.md` file - it will read the file and all linked artifacts automatically to analyze the failure.

## Troubleshooting

### No trace files created

**Possible causes:**
1. Plugin not enabled
2. No steps executed
3. Tests skipped

**Solution:**
```bash
# Check if plugin is enabled
grep -A 5 "aiTrace" codecept.conf.js

# Run with verbose output
npx codeceptjs run --verbose
```

### ARIA snapshots missing

**Possible cause:** Helper doesn't support `grabAriaSnapshot`

**Solution:** Use Playwright or update to latest CodeceptJS

### HAR files missing

**Possible cause:** HAR/trace not enabled in helper config

**Solution:**
```javascript
helpers: {
Playwright: {
recordHar: { mode: 'minimal' },
trace: 'on',
},
}
```

## Related

- [AI Features](/ai) - AI-powered testing features
- [Plugins](/plugins) - All available plugins
- [Configuration](/configuration) - General configuration
- [Playwright Helper](/playwright) - Playwright-specific configuration
108 changes: 108 additions & 0 deletions docs/plugins.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,114 @@ exports.config = {

Returns **void**&#x20;

## aiTrace

Generates AI-friendly trace files for debugging with AI agents like Claude Code.

When a test fails, you need to understand what went wrong. This plugin automatically captures comprehensive information about test execution - screenshots, HTML, ARIA snapshots, console logs, and HTTP requests - and organizes it in a format optimized for AI analysis.

The generated trace files are structured markdown documents that AI agents can easily parse to understand test context and provide debugging insights.

#### Usage

Enable this plugin in your config:

```js
// in codecept.conf.js
exports.config = {
plugins: {
aiTrace: {
enabled: true
}
}
}
```

#### Configuration

* `deleteSuccessful` (boolean) - delete traces for successfully executed tests. Default: false.
* `fullPageScreenshots` (boolean) - should full page screenshots be used. Default: false.
* `output` (string) - a directory where traces should be stored. Default: `output`.
* `captureHTML` (boolean) - capture HTML for each step. Default: true.
* `captureARIA` (boolean) - capture ARIA snapshot for each step. Default: true.
* `captureBrowserLogs` (boolean) - capture browser console logs. Default: true.
* `captureHTTP` (boolean) - capture HTTP requests (requires `trace` or `recordHar` enabled in helper config). Default: true.
* `captureDebugOutput` (boolean) - capture CodeceptJS debug output. Default: true.
* `ignoreSteps` (array) - steps to ignore in trace. Array of RegExps is expected. Default: [].

#### Artifacts Created

For each test, a `trace_<sha256>` directory is created with:

* **trace.md** - AI-friendly markdown file with test execution history
* **0000_screenshot.png** - screenshot for each step
* **0000_page.html** - full HTML of the page at each step
* **0000_aria.txt** - ARIA accessibility snapshot (AI-readable structure)
* **0000_console.json** - browser console logs

When HAR or trace recording is enabled in your helper config, links to those files are also included.

#### Example Output

```markdown
file: /path/to/test.js
name: My test scenario
time: 3.45s
---

I am on page "https://example.com"
> navigated to https://example.com/
> [HTML](./0000_page.html)
> [ARIA Snapshot](./0000_aria.txt)
> [Screenshot](./0000_screenshot.png)
> [Browser Logs](0000_console.json) (7 entries)
> HTTP: see [HAR file](../har/...) for network requests

I see "Welcome"
> navigated to https://example.com/
> [HTML](./0001_page.html)
> [ARIA Snapshot](./0001_aria.txt)
> [Screenshot](./0001_screenshot.png)
> [Browser Logs](0001_console.json) (0 entries)
```

#### Best Practices

**Save disk space** - Only keep traces for failed tests:

```js
aiTrace: {
enabled: true,
deleteSuccessful: true
}
```

**Ignore noise** - Don't capture logs for `grab` and `wait` steps:

```js
aiTrace: {
enabled: true,
ignoreSteps: [/^grab/, /^wait/]
}
```

**Reduce file sizes** - Capture only what you need:

```js
aiTrace: {
enabled: true,
captureHTML: false, // Skip HTML (saves ~500KB per step)
captureARIA: true, // Keep ARIA (only ~16KB)
captureBrowserLogs: false // Skip console logs
}
```

### Parameters

* `config` **[Object][1]** Plugin configuration (optional, default `{}`)

Returns **void**&#x20;

## auth

Logs user in for the first test and reuses session for next tests.
Expand Down
8 changes: 8 additions & 0 deletions examples/codecept.config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@ export const config = {
subtitles: {
enabled: true,
},
aiTrace: {
enabled: true,
captureHTML: true,
captureARIA: true,
captureBrowserLogs: true,
captureHTTP: true,
ignoreSteps: [/^grab/, /^wait/],
},
},

tests: './*_test.js',
Expand Down
Loading