Skip to content

Add operational-value grader for daily-caveman-optimizer - #58557

Closed
mnkiefer with Copilot wants to merge 3 commits into
mainfrom
copilot/daily-caveman-optimizer
Closed

Add operational-value grader for daily-caveman-optimizer#58557
mnkiefer with Copilot wants to merge 3 commits into
mainfrom
copilot/daily-caveman-optimizer

Conversation

Copilot AI commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

daily-caveman-optimizer had no deterministic operational-value grader. This adds one, following .github/skills/operational-value-designer/SKILL.md, measuring whether the workflow's conciseness edits actually stick in the repository rather than counting activity or trusting agent self-assessment.

Design

  • Intent: recovered from the adoption commit (9b89368509...) — trim .github/aw/.github/agents instruction files for conciseness without losing accuracy, in daily batches of 5.
  • Assignment: the round-robin batch depends on external cache state, so per-run reconstruction can't rely on git history alone. Also found that a separate workflow (pr-description-caveman) rewrites merged PR bodies on every merge, wiping out the usual gh-aw-workflow-id/run-id footer marker (confirmed against real PRs [caveman] Optimize instruction verbosity — network.md (2026-09-01) #57777, [caveman] Optimize instruction verbosity — evals.md (2026-08-28) #56727). Assignment instead searches for the run's PR by title prefix, labels, author, and a bounded time window after run start — classified bounded-ambiguity rather than fully deterministic.
  • Metric (pr-reduction-durability, direct, [0,1]): fraction of the PR's own claimed line reduction (from its files API) still present at the evidence cutoff. This avoids inventing an arbitrary target-lines threshold (no such convention exists for markdown, unlike Go source), and directly penalizes reverted or regrown trims:
reductionClaimed = sum(deletions - additions) over eligible PR files
if not merged: value = 0
else: value = clamp((reductionClaimed - growthSinceMerge) / reductionClaimed, 0, 1)
  • Maturation: fixed 7 days, derived from the workflow's own 3-day PR auto-expiry plus buffer to observe reverts.
  • Baseline: attainment-only — no prior automated/manual instruction-trimming activity existed before adoption.
  • Diagnostic: batch-size-ratio (files touched / 5), reported separately from the primary value.

Changes

  • .github/graders/daily-caveman-optimizer-operational-value.sh — new evaluator (--definition/--metric/--grade-run)
  • .github/graders/daily-caveman-optimizer-operational-value-study.json — study record (designOutcome: accepted) documenting the design decisions and uncertainties above
  • .github/workflows/daily-caveman-optimizer.md — wires in graders.operational-value
  • .github/workflows/daily-caveman-optimizer.lock.yml — recompiled
Original prompt

[operational-value-study:paper-v1] daily-caveman-optimizer

Use the repository's .github/skills/operational-value-designer/SKILL.md to design
and verify one deterministic operational-value grader for github/gh-aw
workflow daily-caveman-optimizer. This is a fresh research run; do not copy another
workflow's metric or contract.

Requirements:

  • Recover adoption-time intent and pre-adoption evidence independently.
  • Use exactly one direct primary attainment metric in [0,1]. Keep diagnostics separate.
  • Treat unavailable evidence as null, never zero. Make maturation explicit and stable.
  • Support historical assignment reconstruction when case and event are null.
  • If no direct operational metric is defensible, create only the study record below with designOutcome "blocked" and explain why. Never substitute activity, output volume, traces, or agent judgment.
  • Change only .github/workflows/daily-caveman-optimizer.md, its generated lock file,
    .github/graders/daily-caveman-optimizer-operational-value.sh, and .github/graders/daily-caveman-optimizer-operational-value-study.json.
  • Do not add generated historical reports to the pull request.

Write .github/graders/daily-caveman-optimizer-operational-value-study.json as JSON with this exact top-level shape:
{
"schemaVersion": 1,
"repository": "github/gh-aw",
"workflowId": "daily-caveman-optimizer",
"recordedAt": "ISO-8601 timestamp",
"recorder": "copilot-coding-agent",
"designOutcome": "accepted|rejected|blocked",
"rejectionReason": null,
"classification": {
"domain": "short research domain",
"subdomain": null,
"taskArchetype": "short label",
"interventionType": "short label",
"outcomeType": "short label",
"evidenceDirectness": "direct|proxy|activity",
"evidenceLocality": "same-repository|cross-repository|external",
"measurementLevel": "nominal|ordinal|interval|ratio",
"metricForm": "binary|proportion|rate|distance-to-target|index",
"assignmentDeterminism": "deterministic|bounded-ambiguity|ambiguous",
"intentExplicitness": "explicit|implicit|mixed",
"markdownSpecificity": "exact|bounded|open-ended",
"outcomeObservability": "immediate|delayed",
"causalClaim": "observational|comparative-not-causal"
},
"decisions": [{
"type": "intent|opportunity|assignment|evidence|maturation|metric|baseline|diagnostic",
"selected": "observable selected design",
"alternatives": ["observable rejected alternative"],
"rationale": "concise evidence-backed rationale, not hidden reasoning",
"evidenceReferences": ["commit, issue, PR, or file reference"],
"changedDuringReview": false
}],
"uncertainties": [{
"type": "measurement|assignment|evidence-availability|temporal|baseline|domain-fit|causal-attribution",
"level": "low|medium|high|unknown",
"description": "specific limitation",
"mitigation": null,
"evidenceReferences": []
}]
}

For blocked/rejected designs, rejectionReason must be a non-empty string. Record
observable contract choices and alternatives only; do not record chain-of-thought.

Validation for accepted designs:

  1. .github/skills/operational-value-designer/scripts/verify-operational-value-evaluator.sh .github/graders/daily-caveman-optimizer-operational-value.sh
  2. gh aw compile .github/workflows/daily-caveman-optimizer.md

Keep the pull request limited to this workflow. The local study runner will replay
history and archive reports after this task completes.

Copilot AI and others added 2 commits September 4, 2026 15:47
Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>
Copilot AI changed the title [WIP] Design operational-value grader for daily caveman optimizer Add operational-value grader for daily-caveman-optimizer Sep 4, 2026
Copilot AI requested a review from mnkiefer September 4, 2026 15:53
@mnkiefer mnkiefer closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants