Review agent jobs for prompt anomalies #465

Activity

1. github-actions commented on Feb 28, 2026

   

   github-actionsBot

   on Feb 28, 2026 – with GitHub Actions

   Contributor

   More actions

   2 related issues

   • [agent-deep-dive] Agent deep dive: gh-aw-duplicate-issue-detector - 2026-02-21 #346 — it already audits agent-job logs for anomaly patterns and prompt-sequence adherence, which gives a concrete baseline and method for reviewing runs.
   • [agentics] Failed runs #66 — it centralizes workflow-failure sub-issues and log-review pointers, which can help gather the run set to inspect for prompt anomalies.

   ---

   What is this? | From workflow: Duplicate Issue Detector

   Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
2. github-actions commented on Feb 28, 2026

   

   github-actionsBot

   on Feb 28, 2026 – with GitHub Actions

   Contributor

   More actions

   The strongest anomaly is a verified runtime/config mismatch: workflows request GitHub MCP search toolset, but job logs show it is rejected (unrecognized toolsets ignored: search), so we should remove/replace that toolset across workflow prompts.

   Recommendation

   Update workflow source files to stop requesting unsupported search in GitHub MCP toolsets and recompile lock files; then add a CI guard that fails on MCP startup warnings containing unrecognized toolsets so this cannot regress silently.

   Findings

   • Issue context: Review agent jobs for prompt anomalies #465 has only title text (“Review agent jobs for prompt anomalies”) and no body/comments, so runtime evidence was required.
   • Verified runtime anomaly (logs):
     • Job 64344738803 (run 22241104401) contains repeated warnings:
       • Warning: unrecognized toolsets ignored: search
     • Job 64402000820 (run 22262123421) contains repeated warnings:
       • msg="Warning: unrecognized toolsets ignored" toolsets=search
   • Config causing warning (repo code):
     • .github/workflows/gh-aw-issue-triage.lock.yml:917
     • .github/workflows/gh-aw-pr-review.lock.yml:1125
     • both set "GITHUB_TOOLSETS": "repos,issues,pull_requests,search,actions"
   • Breadth:
     • search appears in many workflow definitions (e.g. .github/workflows/gh-aw-issue-triage.md:65, .github/workflows/gh-aw-pr-review.md:82, .github/workflows/agent-deep-dive.md:42) and many compiled lock files.
   • Related prior item:
     • [agent-deep-dive] Agent deep dive: gh-aw-duplicate-issue-detector - 2026-02-21 #346 reviewed one workflow’s behavior and found no hard failure, but did not identify this MCP warning class as a tracked issue.

   Verification

   Commands run in workspace:

   # Verified warnings in run 22241104401 / job 64344738803
   curl -s '(job-log-url)' | grep -nE 'unrecognized toolsets|toolsets=search'

   Output (excerpt):

   2789: ... Warning: unrecognized toolsets ignored: search
   2800: ... Warning: unrecognized toolsets ignored: search
   

   # Verified warnings in run 22262123421 / job 64402000820
   curl -s '(job-log-url)' | grep -nE 'unrecognized toolsets|toolsets=search'

   Output (excerpt):

   2794: ... level=WARN msg="Warning: unrecognized toolsets ignored" toolsets=search
   2807: ... level=WARN msg="Warning: unrecognized toolsets ignored" toolsets=search
   

   # Verified lock config includes search

   • .github/workflows/gh-aw-issue-triage.lock.yml:917
   • .github/workflows/gh-aw-pr-review.lock.yml:1125

   # Verified source workflow configs include search (many files)

   • .github/workflows/gh-aw-issue-triage.md:65
   • .github/workflows/gh-aw-pr-review.md:82
   • .github/workflows/agent-deep-dive.md:42

   Detailed Action Plan

   1. Fix source workflow configs
      • Replace search in tools.github.toolsets with supported toolsets only in source .md workflows (start with high-traffic ones such as gh-aw-issue-triage.md, gh-aw-pr-review.md, agent-deep-dive.md).
   2. Recompile lock files
      • Run repository compile flow so .lock.yml files regenerate and no longer emit GITHUB_TOOLSETS values with search.
   3. Validate with workflow lint/compile checks
      • Run existing repo checks (make compile, make lint) to ensure generated workflows stay consistent.
   4. Add regression guard
      • Add/extend CI check to fail when MCP startup logs contain unrecognized toolsets (or equivalent strict validation before merge).
   5. Roll out broadly
      • Since search is present in many source workflows, apply the same fix pattern repo-wide rather than only in one workflow.

   Related Items

   Type	Link	Relevance
   Issue	#465	Current triage request
   Issue	#346	Prior deep-dive; adjacent investigation context
   File	.github/workflows/gh-aw-issue-triage.md:65	Source toolset declaration includes search
   File	.github/workflows/gh-aw-pr-review.md:82	Source toolset declaration includes search
   File	.github/workflows/agent-deep-dive.md:42	Source toolset declaration includes search
   File	.github/workflows/gh-aw-issue-triage.lock.yml:917	Compiled GITHUB_TOOLSETS includes search
   File	.github/workflows/gh-aw-pr-review.lock.yml:1125	Compiled GITHUB_TOOLSETS includes search
   Run/Job	22241104401 / 64344738803	Runtime warning evidence
   Run/Job	22262123421 / 64402000820	Runtime warning evidence

   ---

   What is this? | From workflow: Issue Triage

   Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
3. strawgate

   closed this as not plannedon Mar 24, 2026