Add operational value report evaluator and history functionality by mnkiefer · Pull Request #57503 · github/gh-aw

mnkiefer · 2026-08-31T20:13:58Z

Implements operational value report evaluator with loading and parsing capabilities.
Creates tests for the operational value report evaluator to ensure correct functionality.
Develops operational value report history management, including listing and grading workflow runs.
Introduces caching mechanism for operational value report observations to optimize performance.
Adds rendering functions for operational value reports in SVG and Markdown formats.
Defines necessary data structures for operational value reports, including metrics, definitions, and observations.

- Implement operational value report evaluator with loading and parsing capabilities. - Create tests for the operational value report evaluator to ensure correct functionality. - Develop operational value report history management, including listing and grading workflow runs. - Introduce caching mechanism for operational value report observations to optimize performance. - Add rendering functions for operational value reports in SVG and Markdown formats. - Define necessary data structures for operational value reports, including metrics, definitions, and observations.

Copilot

Copilot review overview

🟡 Changes recommended

History truncation, evaluator integrity, incomplete validation, and cache exposure can produce incomplete or unsafe reports.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Balanced
Findings: 3 · 4

New issues introduced by this change (7)

Severity	Finding
	`pkg/cli/graders_operational_value_report_history.go` — The `created` filter on GitHub's workflow-runs endpoint is capped at 1,000 results per search, so…
	`pkg/cli/graders_operational_value_report_evaluator.go` — The evaluator is hashed from `evaluatorContent`, but syntax checking, definition loading, and every…
	`pkg/cli/graders_operational_value_report_evaluator.go` — This accepts incomplete frozen contracts because it checks only opportunity, accepted evidence, and…
	`pkg/cli/graders_operational_value_report_cache.go` — The cache directory tree is created with world traversal/read permissions even though it contains…
	`pkg/cli/graders_operational_value_report_command.go` — Passing the evidence endpoint as `generatedAt` makes reports produced with a historical `--until`…
	`pkg/cli/graders_operational_value_report_history.go` — `workflowFile` is inserted into a URL path without escaping. Valid filenames containing…
	`.github/skills/agentic-workflows/SKILL.md` — This newly listed resource does not exist under `.github/aw`, so routed workflow requests that try…

What changed in this PR

Adds historical operational-value reporting to the graders CLI.

Changes:

Replays evaluators across workflow history with weekly caching.
Generates JSON, SVG, and Markdown reports.
Adds tests, specifications, reference documentation, and skill guidance.

File	Description
`pkg/cli/graders_operational_value_report_types.go`	Defines report data structures.
`pkg/cli/graders_operational_value_report_render.go`	Renders report artifacts.
`pkg/cli/graders_operational_value_report_history.go`	Lists and grades historical runs.
`pkg/cli/graders_operational_value_report_history_test.go`	Tests history caching behavior.
`pkg/cli/graders_operational_value_report_evaluator.go`	Loads and validates evaluators.
`pkg/cli/graders_operational_value_report_evaluator_test.go`	Tests evaluator loading and parsing.
`pkg/cli/graders_operational_value_report_command.go`	Implements the report command.
`pkg/cli/graders_operational_value_report_command_test.go`	Tests command output artifacts.
`pkg/cli/graders_operational_value_report_cache.go`	Implements weekly observation caching.
`pkg/cli/graders_operational_value_report_cache_test.go`	Tests cache behavior and safety.
`pkg/cli/graders_operational_value_report_build.go`	Builds summaries and weekly aggregates.
`pkg/cli/graders_operational_value_report_build_test.go`	Tests aggregation and rendering.
`pkg/cli/graders_integration_test.go`	Extends CLI integration coverage.
`pkg/cli/graders_command.go`	Registers the report subcommand.
`docs/src/content/docs/specs/graders-specification.md`	Specifies historical reporting requirements.
`docs/src/content/docs/reference/trace-graders.md`	Documents report usage.
`.github/skills/aw-value/SKILL.md`	Adds historical-report guidance.
`.github/skills/agentic-workflows/SKILL.md`	Adds a workflow-context reference.

Suppressed comments (1)

pkg/cli/graders_operational_value_report_cache.go:116

Cache entries contain evaluator cases, provenance, diagnostics, and messages derived from potentially private repositories, but this makes them world-readable. Use the sensitive file permission policy so choosing a shared cache root does not disclose private repository metadata to other local users.

	if err := tempFile.Chmod(constants.FilePermPublic); err != nil {

mnkiefer · 2026-08-31T20:25:58Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

When complete run report on Daily file diet and show svg.

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot · 2026-08-31T20:37:38Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the s...

Implemented the requested review-thread fixes in commit f40242d.

github-actions · 2026-08-31T20:56:27Z

Hey @mnkiefer 👋 — thanks for working on the operational value report system! This PR looks solid and ready for review. The implementation includes comprehensive test coverage across evaluator, history, caching, and rendering components, and the diff is well-focused on a single feature area. Great work!

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by ✅ Contribution Check · copilot · auto · 44.7 AIC · ⌖ 9.49 AIC · ⊞ 9.3K · ◷

…rt building logic

…or each observation

gh-aw-bot · 2026-09-01T11:47:50Z

@copilot This PR looks ready for a forward-progress pass. Please review the current status checks, refresh the branch if needed, and run the pr-finisher skill to move it toward merge readiness.

Generated by 👨🍳 PR Sous Chef
_{Comment /souschef to run again}

Generated by 👨‍🍳 PR Sous Chef · pi · gpt54 · 23.3 AIC · ⌖ 8.74 AIC · ⊞ 8.9K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

…value

github-actions · 2026-09-01T23:48:48Z

🎉 This pull request is included in a new release.

Release: v0.88.0

mnkiefer self-assigned this Aug 31, 2026

Copilot AI balanced review requested due to automatic review settings August 31, 2026 20:13

mnkiefer marked this pull request as draft August 31, 2026 20:14

Copilot started reviewing on behalf of mnkiefer August 31, 2026 20:14 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Copilot started work on behalf of mnkiefer August 31, 2026 20:26 View session

Copilot AI and others added 2 commits August 31, 2026 20:29

Plan review-thread fixes

eca7770

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Address operational-value report review feedback

f40242d

Co-authored-by: mnkiefer <8320933+mnkiefer@users.noreply.github.com>

Copilot finished work on behalf of mnkiefer August 31, 2026 20:38

This was referenced Aug 31, 2026

[Contribution Check Report] Contribution Check — 2026-08-31 #57314

Closed

[Outcome Report] Workflow Health — 2026-09-01 #57535

Open

[pr-review] Daily PR Code Quality Review — 33474746531 #57628

Closed

mnkiefer added 3 commits September 1, 2026 11:33

Add Daily File Diet operational value report and associated SVG chart

220b083

Add ID field to operational value report observations and update repo…

b62ae6f

…rt building logic

Enhance operational value report command to include stable identity f…

206fba2

…or each observation

mnkiefer marked this pull request as ready for review September 1, 2026 11:40

Merge remote-tracking branch 'origin/main' into update-op-value

69aee32

github-actions Bot mentioned this pull request Sep 1, 2026

[pr-sous-chef] pr-sous-chef run report #57526

Closed

Copilot started work on behalf of gh-aw-bot September 1, 2026 11:48 View session

Copilot AI and others added 3 commits September 1, 2026 11:51

Plan PR forward-progress pass

cfc903d

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Enhance operational value report functions

cac3f1f

Merge remote-tracking branch 'origin/update-op-value' into update-op-…

b219649

…value

mnkiefer merged commit 367f826 into main Sep 1, 2026
33 checks passed

mnkiefer deleted the update-op-value branch September 1, 2026 12:04

Copilot stopped work on behalf of gh-aw-bot due to an error September 1, 2026 12:05
The session was cancelled by the user.

Copilot AI requested a review from gh-aw-bot September 1, 2026 12:05

Navigation Menu

Uh oh!

Add operational value report evaluator and history functionality - #57503

Add operational value report evaluator and history functionality#57503
mnkiefer merged 10 commits into
mainfrom
update-op-value

mnkiefer commented Aug 31, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mnkiefer commented Aug 31, 2026

Uh oh!

Copilot AI commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

gh-aw-bot commented Sep 1, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

mnkiefer commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mnkiefer commented Aug 31, 2026

Uh oh!

Copilot AI commented Aug 31, 2026

Uh oh!

github-actions Bot commented Aug 31, 2026

Uh oh!

gh-aw-bot commented Sep 1, 2026

Uh oh!

Uh oh!

github-actions Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mnkiefer commented Aug 31, 2026 •

edited

Loading