[evals] Daily Evals Feature Report - 2026-08-02

[github-actions]

Executive Summary

40 result-producing runs across 35 workflows were analyzed from the last 7 days. 26 runs passed every eval question, but the overall YES rate was 51.6%, so the feature is DEGRADED. There were 34 non-binary answers (UNKNOWN) out of 97 total eval answers, which is a separate contract-quality signal.

Note

Status: DEGRADED — evals jobs are producing results, but the overall YES rate is only 51.6%.

Key Metrics

Metric	Value
Workflows with evals	35
Runs analyzed	40
Runs with evals results	40
Evals job success rate	100.0%
Overall YES rate	51.6%

Per-Workflow Pass Rates

Workflow	Runs	Evals Job Success	Run Pass Rate	Lowest-Scoring Question
PR Sous Chef	4	100.0%	25.0%	"Did the agent add a comment to at least one pull request?" (25% YES)
AI Moderator	3	100.0%	0.0%	"Does the agent output include a rationale explaining why the label(s) were applied or why noop was called?" (0% YES)
Artifacts Usage Report	1	100.0%	0.0%	"Was a comprehensive summary report of artifacts usage produced?" (0% YES)
Auto-Triage Issues	1	100.0%	0.0%	"Was a summary discussion created listing the issues processed and the labels applied?" (0% YES)
CLI Version Checker	1	100.0%	100.0%	"Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?" (100% YES)
Code Scanning Fixer	1	100.0%	0.0%	"Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?" (0% YES)
Code Simplifier	1	100.0%	100.0%	"Was a pull request created with simplifications, or was noop used when no improvements were needed?" (100% YES)
Contribution Check	1	100.0%	0.0%	"Was a report issue created summarizing PR compliance with the contributing guidelines?" (0% YES)
Copilot CLI Deep Research Agent	1	100.0%	0.0%	"Does the agent output identify specific missed optimization opportunities for Copilot CLI usage in this repository?" (0% YES)
Daily Assign Issue To User	1	100.0%	0.0%	"Did the agent post a comment explaining the assignment decision?" (0% YES)
Daily AstroStyleLite Markdown Spellcheck	1	100.0%	100.0%	"Did the agent run American English spellcheck on AstroStyleLite docs content?" (100% YES)
Daily Cli Tools Tester	1	100.0%	100.0%	"Did the agent run exploratory tests on the audit, logs, and compile CLI tools?" (100% YES)
Daily Community Attribution Updater	1	100.0%	100.0%	"Did the agent scan community-labeled issues and attribute contributions using the five-tier strategy?" (100% YES)
Daily Compiler Quality Check	1	100.0%	0.0%	"Was a discussion or report created with quality findings, or was noop used when all analyzed files met the quality standards?" (0% YES)
Daily Container Image Security Scan	1	100.0%	100.0%	"Did the agent report actionable image findings, or use noop when no findings required action?" (100% YES)
Daily Go Test Parallelizer	1	100.0%	100.0%	"Did the agent analyze Go tests to identify a safe candidate for t.Parallel?" (100% YES)
Daily Safe Outputs Conformance Checker	1	100.0%	0.0%	"Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?" (0% YES)
Daily Safe Outputs Git Simulator	1	100.0%	100.0%	"Did the agent report the simulator results and any systematic safe-output issues it found?" (100% YES)
Daily Semgrep Scan	1	100.0%	0.0%	"Did the agent complete a Semgrep security scan and report on the findings?" (0% YES)
Daily Skill Optimizer Improvements	1	100.0%	0.0%	"Does the agent output include at least three specific skill improvement recommendations?" (0% YES)
Daily VulnHunter Scan	1	100.0%	0.0%	"Did the agent download the prepared VulnHunter bundle artifact, load its vulnhunt skill instructions, and complete a repository scan?" (0% YES)
Daily Windows Terminal Integration Builder	1	100.0%	100.0%	"Did the agent assess the Windows CLI integration build and test workflow?" (100% YES)
Daily Workflow Updater	1	100.0%	0.0%	"Did the agent create a pull request for required updates, or report that no changes were needed?" (0% YES)
Dictation Prompt Generator	1	100.0%	0.0%	"Did the agent create a pull request containing the generated dictation prompt update?" (0% YES)
Discussion Task Miner - Code Quality Improvement Agent	1	100.0%	0.0%	"Does the agent output show that actionable tasks were identified from the analyzed discussions?" (0% YES)
Documentation Noob Tester	1	100.0%	0.0%	"Does the agent output reflect the perspective of a new user rather than an expert reviewer?" (0% YES)
ESLint Refiner	1	100.0%	0.0%	"Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities?" (0% YES)
GPL Dependency Cleaner (gpclean)	1	100.0%	0.0%	"Does the agent output show that the objective for experiment tool_verbosity was successfully completed?" (0% YES)
Go Logger Enhancement	1	100.0%	0.0%	"Does the agent output show that it ran validation commands to verify the logging changes compile correctly?" (0% YES)
Issue Arborist	1	100.0%	0.0%	"Did the agent analyze recent issues and identify related issue relationships?" (0% YES)
Multi-Device Docs Tester	1	100.0%	100.0%	"Did the agent report the multi-device test results and any responsive design or functionality findings?" (100% YES)
PR Triage Agent	1	100.0%	0.0%	"Does the agent output confirm that category, risk, and action data were determined for each processed PR?" (0% YES)
Sub-Issue Closer	1	100.0%	100.0%	"Did the agent check parent issues for the completion status of all their sub-issues?" (100% YES)
Tidy	1	100.0%	100.0%	"Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?" (100% YES)
[aw] Failure Investigator (6h)	1	100.0%	100.0%	"Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?" (100% YES)

Per-Question Breakdown per Workflow

AI Moderator

Question ID	Question	YES	NO	YES Rate
action-taken	Did the agent apply at least one label (spam, ai-generated, link-spam, or ai-inspected) or call noop?	0	0	0%
no-unsupported-action	Does the agent output show that only allowed safe-output actions (add-labels, hide-comment, noop) were used?	0	0	0%
rationale-provided	Does the agent output include a rationale explaining why the label(s) were applied or why noop was called?	0	0	0%

Artifacts Usage Report

Question ID	Question	YES	NO	YES Rate
summary_produced	Was a comprehensive summary report of artifacts usage produced?	0	0	0%
artifacts_analyzed	Did the agent analyze GitHub Actions artifacts usage across all workflows in the repository?	1	0	100%

Auto-Triage Issues

Question ID	Question	YES	NO	YES Rate
report-created	Was a summary discussion created listing the issues processed and the labels applied?	0	1	0%
labels-applied	Did the agent apply at least one label to an unlabeled issue, or correctly call noop when no unlabeled issues were found?	1	0	100%

CLI Version Checker

Question ID	Question	YES	NO	YES Rate
cli_versions_checked	Did the agent check for new versions of agentic CLI tools (Claude Code, GitHub Copilot CLI, Codex, MCP servers, etc.)?	1	0	100%
docker_images_checked	Did the agent check for new versions and digest changes of Docker images in pkg/cli/docker_images.go (actionlint, syft, grype, grant, zizmor, poutine, runner-guard, yamllint)?	1	0	100%
updates_applied_or_noop	Were version or digest updates applied and a PR created, or was noop used when all tools were already up to date?	1	0	100%

Code Scanning Fixer

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with a remediation for a code scanning alert, or was noop used when no fixable alerts existed?	0	0	0%
alerts_analyzed	Did the agent analyze code scanning alerts and identify at least one fixable alert, or correctly skip when no fixable alerts were found?	1	0	100%

Code Simplifier

Question ID	Question	YES	NO	YES Rate
code_analyzed	Did the agent analyze recently modified code for simplification opportunities?	1	0	100%
pr_created_or_noop	Was a pull request created with simplifications, or was noop used when no improvements were needed?	1	0	100%

Contribution Check

Question ID	Question	YES	NO	YES Rate
prs-evaluated	Did the agent dispatch PRs to the contribution-checker subagent and produce evaluation results for each PR?	0	0	0%
report-created	Was a report issue created summarizing PR compliance with the contributing guidelines?	0	0	0%

Copilot CLI Deep Research Agent

Question ID	Question	YES	NO	YES Rate
evidence-based	Does the agent output cite specific repository evidence (workflow files, usage patterns) to support its findings?	0	0	0%
optimization-opportunities	Does the agent output identify specific missed optimization opportunities for Copilot CLI usage in this repository?	0	0	0%
issue-created	Did the agent create a research issue with Copilot CLI findings?	1	0	100%

Daily Assign Issue To User

Question ID	Question	YES	NO	YES Rate
comment-posted	Did the agent post a comment explaining the assignment decision?	0	0	0%
contributor-selected	Does the agent output show that a relevant contributor was selected from recent merged PRs?	0	0	0%
assignment-made	Did the agent assign an issue to a user?	1	0	100%

Daily AstroStyleLite Markdown Spellcheck

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a PR created with spelling corrections, or was noop used when no misspellings were found?	1	0	100%
spellcheck_completed	Did the agent run American English spellcheck on AstroStyleLite docs content?	1	0	100%

Daily Cli Tools Tester

Question ID	Question	YES	NO	YES Rate
cli_tools_tested	Did the agent run exploratory tests on the audit, logs, and compile CLI tools?	1	0	100%
test_results_reported	Were the test results reported with any issues or anomalies identified?	1	0	100%

Daily Community Attribution Updater

Question ID	Question	YES	NO	YES Rate
community_contributions_scanned	Did the agent scan community-labeled issues and attribute contributions using the five-tier strategy?	1	0	100%
readme_or_wiki_updated	Was the README community section or the Community Contributors wiki page updated, or was noop used when no changes were needed?	1	0	100%

Daily Compiler Quality Check

Question ID	Question	YES	NO	YES Rate
discussion_created_or_noop	Was a discussion or report created with quality findings, or was noop used when all analyzed files met the quality standards?	0	0	0%
files_analyzed	Did the agent analyze compiler code files for quality standards such as readability and maintainability?	1	0	100%

Daily Container Image Security Scan

Question ID	Question	YES	NO	YES Rate
container_images_scanned	Did the agent analyze container images for vulnerabilities, updates, and rejected licenses?	1	0	100%
findings_reported_or_noop	Did the agent report actionable image findings, or use noop when no findings required action?	1	0	100%

Daily Go Test Parallelizer

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Did the agent create a pull request for a safe test change, or use noop when no safe change was available?	1	0	100%
tests_analyzed	Did the agent analyze Go tests to identify a safe candidate for t.Parallel?	1	0	100%

Daily Safe Outputs Conformance Checker

Question ID	Question	YES	NO	YES Rate
tasks_created_or_noop	Were agentic tasks created for Copilot to address critical/high/medium/low issues, or was noop used when implementation was compliant?	0	1	0%
conformance_checked	Did the agent run a conformance check against the Safe Outputs specification implementation?	1	0	100%

Daily Safe Outputs Git Simulator

Question ID	Question	YES	NO	YES Rate
git_scenarios_simulated	Did the agent simulate git configurations for create-pull-request and push-to-pull-request-branch safe outputs?	1	0	100%
simulator_findings_reported	Did the agent report the simulator results and any systematic safe-output issues it found?	1	0	100%

Daily Semgrep Scan

Question ID	Question	YES	NO	YES Rate
scan_completed	Did the agent complete a Semgrep security scan and report on the findings?	0	1	0%
alert_created_or_noop	Was a code scanning alert created for real security findings, or does the agent output confirm no vulnerabilities were found?	1	0	100%

Daily Skill Optimizer Improvements

Question ID	Question	YES	NO	YES Rate
improvements-actionable	Does the agent output describe each improvement in enough detail to be actionable rather than only describing the general area?	0	0	0%
issue-created	Did the agent create an issue with skill improvement suggestions?	0	0	0%
three-improvements	Does the agent output include at least three specific skill improvement recommendations?	0	0	0%

Daily VulnHunter Scan

Question ID	Question	YES	NO	YES Rate
scan_completed	Did the agent download the prepared VulnHunter bundle artifact, load its vulnhunt skill instructions, and complete a repository scan?	0	0	0%
issue_created_or_noop	Was a security issue created for verified exploitable findings, or was noop used when VulnHunter found nothing actionable?	1	0	100%

Daily Windows Terminal Integration Builder

Question ID	Question	YES	NO	YES Rate
failure_reported_or_noop	Did the agent create an issue for an actionable integration failure, or use noop when no action was required?	1	0	100%
windows_integration_checked	Did the agent assess the Windows CLI integration build and test workflow?	1	0	100%

Daily Workflow Updater

Question ID	Question	YES	NO	YES Rate
update_pr_created_or_noop	Did the agent create a pull request for required updates, or report that no changes were needed?	0	0	0%
action_versions_checked	Did the agent check GitHub Actions versions for available updates?	1	0	100%

Dictation Prompt Generator

Question ID	Question	YES	NO	YES Rate
dictation_prompt_generated	Did the agent generate an optimized prompt for voice dictation and speech-to-text workflows?	0	1	0%
prompt_update_created	Did the agent create a pull request containing the generated dictation prompt update?	0	0	0%

Discussion Task Miner - Code Quality Improvement Agent

Question ID	Question	YES	NO	YES Rate
labels-applied	Does the agent output confirm that the created issues include the expected labels (code-quality, automation, task-mining)?	0	0	0%
tasks-extracted	Does the agent output show that actionable tasks were identified from the analyzed discussions?	0	0	0%
output-produced	Did the agent create at least one code quality issue or add a comment?	1	0	100%

Documentation Noob Tester

Question ID	Question	YES	NO	YES Rate
confusing-steps-identified	Does the agent output identify at least one confusing, broken, or unclear step in the documentation?	0	0	0%
user-perspective-maintained	Does the agent output reflect the perspective of a new user rather than an expert reviewer?	0	0	0%
findings-reported	Did the agent report documentation testing findings (via asset upload or report)?	1	0	100%

ESLint Refiner

Question ID	Question	YES	NO	YES Rate
eslint_trends_analyzed	Did the agent analyze ESLint diagnostics trends to identify rule refinement opportunities?	0	1	0%
refinements_reported	Did the agent report actionable ESLint rule refinements or explain why no refinement was needed?	1	0	100%

GPL Dependency Cleaner (gpclean)

Question ID	Question	YES	NO	YES Rate
tool_verbosity_goal_met	Does the agent output show that the objective for experiment tool_verbosity was successfully completed?	0	0	0%
decision-explained	Does the agent output include an explanation of why a GPL issue was created or why noop was called?	1	0	100%
go-mod-analyzed	Does the agent output confirm that go.mod was analyzed for GPL-licensed transitive dependencies?	1	0	100%
issue-or-noop	Did the agent either create a GPL dependency issue or call noop?	1	0	100%

Go Logger Enhancement

Question ID	Question	YES	NO	YES Rate
validation-run	Does the agent output show that it ran validation commands to verify the logging changes compile correctly?	0	1	0%
logging-changes-made	Does the agent output confirm that logging statements were added or modified in Go source files?	1	0	100%
pr-created	Did the agent create a pull request with Go logging improvements?	1	0	100%

Issue Arborist

Question ID	Question	YES	NO	YES Rate
issues_analyzed	Did the agent analyze recent issues and identify related issue relationships?	0	0	0%
sub_issues_linked_or_noop	Were related issues linked as sub-issues, or was noop used when no linkable relationships were found?	0	0	0%

Multi-Device Docs Tester

Question ID	Question	YES	NO	YES Rate
device_tests_completed	Did the agent test the documentation site across the requested device form factors?	1	0	100%
results_reported	Did the agent report the multi-device test results and any responsive design or functionality findings?	1	0	100%

PR Sous Chef

Question ID	Question	YES	NO	YES Rate
comment-added	Did the agent add a comment to at least one pull request?	1	3	25%
nudge-targeted	Does the agent output show a specific reason why the selected PR needs a nudge toward maintainer investigation?	1	3	25%
pr-evaluated	Does the agent output confirm that it evaluated at least one open PR for nudge eligibility?	2	1	50%

PR Triage Agent

Question ID	Question	YES	NO	YES Rate
report-produced	Does the agent output include a triage report summarizing the PRs processed?	0	0	0%
triage-data-set	Does the agent output confirm that category, risk, and action data were determined for each processed PR?	0	0	0%
labels-applied	Did the agent apply triage labels to at least one pull request?	1	0	100%

Sub-Issue Closer

Question ID	Question	YES	NO	YES Rate
issues_checked	Did the agent check parent issues for the completion status of all their sub-issues?	1	0	100%
issues_closed_or_noop	Were completed parent issues closed with a comment, or does the agent output confirm no issues were ready to close?	1	0	100%

Tidy

Question ID	Question	YES	NO	YES Rate
pr_created_or_noop	Was a pull request created with formatting and tidying fixes, or was noop used when no changes were needed?	1	0	100%
tidy_completed	Did the agent run code formatting and tidying tools on the codebase?	1	0	100%

[aw] Failure Investigator (6h)

Question ID	Question	YES	NO	YES Rate
failures_investigated	Did the agent investigate agentic workflow failures from the last 6 hours and produce findings?	1	0	100%
issues_created_or_closed	Were fix sub-issues created for unresolved failures, or were resolved tracking issues closed?	1	0	100%

Quality Signals

• 30733952293 is at 100% NO across 3 run(s): PR Sous Chef
• 30735700695 is at 67% NO across 3 run(s): PR Sous Chef
• 30737490610 is at 67% NO across 3 run(s): PR Sous Chef
• pr-evaluated in PR Sous Chef is the weakest repeated question, at 25% YES over 4 runs.

Recommendations

• Triage the consistently failing prompt/rubric items first; the failures are concentrated in a small number of questions rather than spread across the fleet.
• Investigate the UNKNOWN answers. They are not binary eval outputs and should be reduced or explicitly handled in the evals contract.
• Tighten the PR Sous Chef rubric around pr-evaluated; it is the only multi-run workflow with sustained partial failure.

References

• §30737490610
• §30735696239
• §30738148113

Generated by 🧪 Daily Evals Feature Report · gpt54 · 21 AIC · ⌖ 19.6 AIC · ⊞ 11.7K · ◷

• expires on Aug 9, 2026, 12:03 AM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #49934

View newer issue

---

This action was performed automatically by the Daily Evals Feature Report workflow.