diff --git a/.github/aw/intent.md b/.github/aw/intent.md index d22f0190365..e9dc14fa8f5 100644 --- a/.github/aw/intent.md +++ b/.github/aw/intent.md @@ -77,7 +77,7 @@ Evals and operational value answer different questions. PromptPex evals test whe Select graders that test a concrete risk to achieving the intent; do not enable metrics merely because they are available. Start with the known builtin graders: use `tool-success-rate` and `tool-failure-count` when reliable collection is required; `retries`, `loops`, `execution-step-count`, and `execution-duration` when boundedness or timely escalation matters; `working-set-rebuild-factor` and `context-growth` when repeated context is an attention or cost risk; `trajectory-efficiency` when unnecessary tool churn is a concern; and `artifact-production` only when producing the intended artifacts is itself useful diagnostic evidence. -Then inspect the implemented fragments in [`shared/graders/`](../workflows/shared/graders/README.md). Use `policy-near-miss` for explicit guard or no-op requirements. For intents that require efficient investigation rather than repeated exploration, consider `state-revisit-probability-rep`, `recurrence-rate`, `recurrence-determinism`, `recurrence-laminarity`, or `recurrence-trapping-time`. For intents where varied, non-repetitive investigation is relevant, consider `event-entropy-rate` or `lempel-ziv-trajectory-complexity`. Import only the applicable implemented fragments and ensure their documented trace prerequisites are available. These trace graders diagnose execution behavior; they do not replace scenario evals or the operational-value attainment metric. +Then inspect the implemented fragments in [`shared/graders/`](../workflows/shared/graders/README.md). Use `policy-near-miss` for explicit guard or no-op requirements, and `skill-constraint-coverage` when a harness/skill declares behavioral requirements that should be exercised. Use `exploration-error` when failure may stem from insufficient search, and `exploitation-error` when the agent had enough information but failed to use it. For intents that require efficient investigation rather than repeated exploration, consider `state-revisit-probability-rep`, `recurrence-rate`, `recurrence-determinism`, `recurrence-laminarity`, or `recurrence-trapping-time`. For intents where varied, non-repetitive investigation is relevant, consider `event-entropy-rate` or `lempel-ziv-trajectory-complexity`. Use `tool-output-consumption-rate` when tool outputs going unused by later actions is a risk. Import only the applicable implemented fragments and ensure their documented trace prerequisites are available. These trace graders diagnose execution behavior; they do not replace scenario evals or the operational-value attainment metric. ## Preserve Intent on Updates diff --git a/.github/aw/mcp-clis.md b/.github/aw/mcp-clis.md index 416a985a8ce..cee55ab65ee 100644 --- a/.github/aw/mcp-clis.md +++ b/.github/aw/mcp-clis.md @@ -22,9 +22,8 @@ Each server is a standalone executable on `PATH`. Invoke from bash with `--name **Examples:** ```bash -playwright --help -playwright browser_navigate --url https://example.com -playwright browser_snapshot +github --help +github issue_read --method get --owner org --repo repo --issue_number 42 safeoutputs add_comment --item_number 42 --body "Analysis complete" diff --git a/.github/aw/syntax-core.md b/.github/aw/syntax-core.md index 7b04c7e3fb4..eae8fa227ef 100644 --- a/.github/aw/syntax-core.md +++ b/.github/aw/syntax-core.md @@ -13,7 +13,8 @@ The YAML frontmatter supports these fields: - Object: Complex trigger configuration - Special: `slash_command:` for /mention triggers - **`forks:`** - Fork allowlist for `pull_request` triggers (array or string). By default, workflows block all forks and only allow same-repo PRs. Use `["*"]` to allow all forks, or specify patterns like `["org/*", "user/repo"]` - - **`stop-after:`** - Can be included in the `on:` object to set a deadline for workflow execution. Supports absolute timestamps ("YYYY-MM-DD HH:MM:SS") or relative time deltas (+25h, +3d, +1d12h). The minimum unit for relative deltas is hours (h). Uses precise date calculations that account for varying month lengths. + - **`stop-after:`** - Can be included in the `on:` object to set a deadline for workflow execution. Supports absolute timestamps ("YYYY-MM-DD HH:MM:SS"), relative time deltas (+25h, +3d, +1d12h), or a GitHub Actions expression (e.g. `${{ inputs.stop-after }}`) resolved at runtime instead of compile time. The minimum unit for relative/literal deltas is hours (h). Literal values use precise date calculations that account for varying month lengths; recompile with `gh aw compile --refresh-stop-time` to reset a literal deadline. + - **`cooldown:`** - Can be included in the `on:` object to block the `agent` job from starting again shortly after the most recent completed run. Value must be a literal Go duration string of at least 5 minutes (e.g. `"30m"`, `"2h"`); GitHub Actions expressions are rejected. Fails open (allows activation) if run history cannot be queried. - **`reaction:`** - Add emoji reactions to triggering items - **`status-comment:`** - Post status comments when workflow starts/completes on the triggering issue, pull request, or discussion (boolean, or object with `issues`/`pull-requests`/`discussions` booleans to control trigger groups independently). Defaults to `true` for `slash_command` and `label_command` triggers; defaults to `false` for all other triggers. Must be explicitly enabled for non-command triggers with `status-comment: true`. - **`manual-approval:`** - Require manual approval using environment protection rules diff --git a/.github/aw/visual-regression.md b/.github/aw/visual-regression.md index 639f6f80af8..405062e130b 100644 --- a/.github/aw/visual-regression.md +++ b/.github/aw/visual-regression.md @@ -19,11 +19,12 @@ permissions: contents: read pull-requests: read engine: copilot +network: + allowed: + - local + - playwright tools: playwright: - allowed_domains: - - localhost - - 127.0.0.1 cache-memory: key: visual-regression-baselines-${{ github.event.pull_request.base.ref }} retention-days: 30 @@ -53,7 +54,7 @@ Otherwise compare each screenshot to its baseline. Post a comment summarizing: p - **`cache-memory` key per base branch** — scopes baselines to `main`, `develop`, etc. - **Explicit baseline source** — state whether baselines come from `cache-memory`, a generated artifact, or a branch directory; do not leave baseline origin implicit. -- **`allowed_domains: [localhost, 127.0.0.1]`** — prevents SSRF; serve app locally +- **`network.allowed: [local, playwright]`** — prevents SSRF; serve app locally, allow browser binary downloads - **`retention-days: 30`** — beyond the default 7-day cache expiry - **Filesystem-safe timestamps** — `YYYY-MM-DD-HH-MM-SS`; colons break artifact filenames - **Minimal permissions** — all PR writes go through `safe-outputs`