[trajectory-grader] Implement tool-output-consumption-rate

[github-actions]

Caution

Protected files were modified in this change.
This pull request is in request_review mode and requires explicit human scrutiny before merge.

Protected files: README.md, .github/workflows/shared/graders/README.md, .github/workflows/shared/graders/tool-output-consumption-rate.md

Grader implemented

tool-output-consumption-rate — Tier 2, rank 12 (first Not started grader encountered walking the catalog tier-first: Tier 1 ranks 5-11 and Tier 2 ranks 1-4 were all already Implemented).

Why it's distinct from built-in graders

Built-in graders (tool-success-rate, retries, loops, trajectory-efficiency, execution-step-count, execution-duration, working-set-rebuild-factor, context-growth, artifact-production) measure success/failure, repetition, duration, and token growth. None of them measure whether a tool's output was ever actually used downstream. tool-output-consumption-rate fills that gap: it answers "of everything the agent read/fetched, how much of it ever fed a later decision or action?" — a distinct signal about wasted information-gathering work, separate from success rate or step count.

Required IR fields

• observations[] (canonical Trajectory IR) — specifically each observation's consumedByActionIds array, populated per the IR build procedure (step 4 in trajectory-ir.md) by checking whether later actions reference values that only appear in that observation.

Formula

consumptionRate = observationsWithNonEmptyConsumedByActionIds / totalObservations, clamped to [0, 1], direction higher_is_better. Reports applicable: false/passed: null when the trace has no recorded observations.

Catalog status

Flipped Tier 2 rank 12 from Not started to Implemented in shared/graders/README.md. See full catalog — 13 of 25 graders now implemented.

Generated by 🧮 Daily Trajectory Grader Implementer · copilot · auto · 38.3 AIC · ⌖ 9.28 AIC · ⊞ 10.2K · ◷

• expires on Sep 13, 2026, 2:04 PM UTC-08:00

---

Note

This was originally intended as a pull request, but the git push operation failed.

Original error: The process '/usr/bin/git' failed with exit code 1

Workflow Run: View run details and download bundle artifact

The bundle file is available in the agent artifact in the workflow run linked above.

Create the pull request manually

# Download the artifact from the workflow run
gh run download '33337856819' -n agent -D '/tmp/agent-33337856819'

# Resolve the bundle source ref, fetch it into a temporary ref, then create the local branch
bundle_path='/tmp/agent-33337856819/aw-daily-trajectory-grader-tool-output-consumption-rate.bundle'
temp_ref='refs/bundles/create-pr-daily-trajectory-grader-tool-output-consumption-rate-c44316731fc18231-ca4726bd'
target_ref='refs/heads/daily-trajectory-grader-tool-output-consumption-rate-c44316731fc18231'
bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 ~ /^refs\/heads\// { print $2 }')
if [ -z "$bundle_source_ref" ]; then
  bundle_source_ref=$(git bundle list-heads "$bundle_path" | awk '$2 == "HEAD" { print $2 }')
fi
if [ "$(printf '%s\n' "$bundle_source_ref" | sed '/^$/d' | wc -l | tr -d ' ')" != "1" ]; then
  echo "Expected exactly one bundle source ref, found: $bundle_source_ref" >&2
  exit 1
fi
git fetch "$bundle_path" "${bundle_source_ref}:${temp_ref}"
git update-ref "$target_ref" "$temp_ref"
git checkout 'daily-trajectory-grader-tool-output-consumption-rate-c44316731fc18231'
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d "$temp_ref"

# Push the branch to the target remote
git push 'origin' 'daily-trajectory-grader-tool-output-consumption-rate-c44316731fc18231'

# Create the pull request
gh pr create --title '[trajectory-grader] Implement tool-output-consumption-rate' --base 'main' --head 'daily-trajectory-grader-tool-output-consumption-rate-c44316731fc18231' --repo 'github/gh-aw'

[pelikhan]

@copilot fix skip-ip filter in trajectory-grader agentic workflows, it is not properly skipping when another issue is already opened.