[trajectory-grader] Implement exploitation-error

[github-actions]

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: README.md, .github/workflows/shared/graders/README.md, .github/workflows/shared/graders/exploitation-error.md

Implements exploitation-error (Tier 2, rank 4) from the catalog in .github/workflows/shared/graders/README.md.

Why distinct from existing built-in graders: gh-aw's built-in graders (tool-success-rate, retries, loops, trajectory-efficiency, execution-step-count, execution-duration, working-set-rebuild-factor, context-growth, artifact-production) measure aggregate efficiency/reliability signals, not why a run failed to meet its objectives. exploitation-error complements the already-shipped exploration-error: it isolates runs where the agent had sufficient evidence (observations ≥ distinct states visited) but still failed one or more declared objectives, and quantifies how much of that evidence was never consumed by a later action. This is mutually exclusive with exploration-error (which covers the insufficient-search case) on the same trace.

Required IR fields: objectives[].satisfiedAtEventIndex, events[] (kind state_change, ref), states[], observations[].consumedByActionIds.

Behavior:

• All objectives satisfied → score 0.
• Objectives unmet but exploration was insufficient → not-applicable (defers to exploration-error).
• Objectives unmet and exploration sufficient → unusedObservations / observations, clamped [0,1]; lower is better.
• Missing objectives/states/observations → not-applicable with explanatory message.

See .github/workflows/shared/graders/README.md for the full catalog — 9 of 25 implemented after this PR.

Generated by 🧮 Daily Trajectory Grader Implementer · copilot · auto · 26.6 AIC · ⌖ 9.93 AIC · ⊞ 10.2K · ◷

• expires on Sep 13, 2026, 4:03 AM UTC-08:00

---

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

Create the pull request manually

# Download the artifact from the workflow run
gh run download '33310248747' -n agent -D '/tmp/agent-33310248747'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-33310248747/aw-daily-trajectory-grader-exploitation-error.bundle'
temp_ref='refs/bundles/create-pr-daily-trajectory-grader-exploitation-error-71fae74439c134fa-95e6914b'
target_ref='refs/heads/daily-trajectory-grader-exploitation-error-71fae74439c134fa'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
  bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
  echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
  exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'daily-trajectory-grader-exploitation-error-71fae74439c134fa'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'daily-trajectory-grader-exploitation-error-71fae74439c134fa'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement exploitation-error' --base 'main' --head 'daily-trajectory-grader-exploitation-error-71fae74439c134fa' --repo 'github/gh-aw'