Repository navigation
Epic: qa-lead — close the gap between the declared persona and the served one #1643
Description
Context
qa-lead is the worked example for the Agentic Persona Standard — §7 names it as the reference other personas are copied from. It was added in #1174 and renamed to the role in #1279.
A review of its actual, measured output on 2026-09-01 found a persona whose quality is proven and whose throughput is near-zero, and a set of declared contracts that the shipped runtime does not honor. This epic closes that gap.
What was measured
.github-private, 2026-07-18 → 2026-09-01 (the only repo where a persona mention is routable):
| Metric | Value |
|---|---|
persona-mention router runs |
4,335 |
| …skipped by the pre-filter | 4,324 |
…startup_failure |
6 (2026-07-18, fixed by #1301) |
…success |
3 — one of which correctly declined to route |
…stuck in queued since 2026-08-19 |
2 |
persona-runner (agent) executions, ever |
2 |
| qa-lead advisories posted, ever | 2 (one on its own soak issue #1300) |
For scale, in the same repo: the dev-lead marker appears on 1,136 items, pr-review-agent on 715, dependency-advisory on 52, qa-lead on 3. The 4,324 skips cost 0 billable ms (job-level if:, confirmed via the timing API) — the waste is run-history noise, not spend.
The value is real — there just isn't enough of it
The single non-soak invocation landed on epic #1402 and reshaped 4 of its 9 stories:
- [Phase 1] Write docs/agentic-interaction-model.md — the event-first standard, three trigger classes, the GITHUB_TOKEN bridge rule, and the timer contract #1403 carries a section titled "Enhancements from QA Lead review (test-architect advisory, epic Formalize the agentic PR-interaction model (event-first, timers-as-backstop) — clear, repeatable, enforceable #1402)" with 2 added ACs and 2 added tasks.
- [Phase 3] Add scripts/validate-interaction-model.sh + bats, wired into lint.yml as a PR-triggered check #1406 likewise.
- [Phase 4] Add a stall canary + rollback trigger for the Class-2 timer changes, before narrowing them #1410 exists because of it. Its AC Add @claude delegation, auto-merge, and rebase handling #3 reads verbatim: "This is the acceptance bar the QA Lead advisory requires; 'budget + stop-marker respected' is explicitly not sufficient proof."
- [Phase 4] Capture the pre-rollout baseline for the three success metrics, including a new no-action-comment noise metric #1411 was "promoted from an untracked prerequisite on the epic to a real, gated story" — verbatim the advisory's fourth bullet.
One invocation, four stories improved, two of them created. The problem is not output quality. It is that the persona is pull-only, reachable in one repo, and nobody remembers to summon it.
Why throughput is zero
.github/workflows/persona-mention.ymlis adopted in exactly one repo (.github-private), pinnedpersona-mention/v1-next. The tagspersona-mention/v1-ring0,v1-ring1, andv1-stabledo not exist. [promote persona-mention 1/3] Soak through the rings (next → ring0 → ring1 → stable) .github#783 (the ring soak) was opened 2026-07-19 and has not been touched since.mentionis the only surface served.personas/qa-lead/persona.ymldeclares five more asenabled: true—pull_request,pull_request_review,issues,discussion,check_run— and nothing dispatches any of them.
Stories
| # | Story | Blocked by |
|---|---|---|
| #1644 | Provision <id>:hands-off labels across the fleet |
— |
| #1645 | Make the qa-lead eval set scorable and capture a first held-out baseline | — |
| #1646 | Serve the pull_request surface so qa-lead runs without being summoned |
#1644, #1645 |
| #1647 | Make persona.yml's declared surfaces agree with what is deployed |
#1646 |
| #1648 | Reconcile personas/qa-lead/README.md with the shipped runtime |
— |
| #1649 | Validate skills[] and fix the entries that are wrong |
— |
| #1650 | Stop posting persona advisories under a human's PAT | — |
| #1651 | Fleet-wide: 11 of 12 eval sets fail case.schema.json |
#1645 |
#1644 and #1645 gate #1646 deliberately, following this repo's own precedent from #1410/#1411: build the instrument and the escape hatch before you broaden the thing they measure and contain. Turning on an automatic PR trigger before the opt-out label exists, or before there is any quality baseline, would repeat the pattern the #860 postmortem warns against.
Out of scope / tracked elsewhere
- Ring promotion — [promote persona-mention 1/3] Soak through the rings (next → ring0 → ring1 → stable) .github#783. This epic does not duplicate it, but S3's value is capped at one repo until it moves.
prompts/bmad/scrum-master.mdstill opens "You are Bob" — the §1.6 violation Rename murat → qa-lead and make it @-addressable #1279 deliberately deferred. Still open; unrelated to qa-lead's runtime.- Two router runs stuck
queuedsince 2026-08-19 — both bot-triggered and would have been filtered, so no advisory was lost. Noted, not worth a story. - The vendored
bmad-teaSKILL.mdfrontmatter still reads "Use when the user asks to talk to Murat". Correctly not hand-edited (frameworks/is vendored); the person-name survives only in upstream context, not in our layer.
Activity
github-actions commentedon Sep 1, 2026
Thanks for this thorough write-up — the measurement table and story-level blocking analysis make the problem space very clear. This is queued for triage. One question to help with sequencing: S1 and S2 are listed as ungated, but S2 (making the eval set scorable) also blocks S8 (fleet-wide schema fix) — is there a target sprint or rough timeline for S2, or should S1 and S2 be treated as parallel tracks that start immediately?
don-petry commentedon Sep 13, 2026
Held-out baseline at parity tier — 9/9, score 1.00
Dispatched
skill-eval-report.ymlwithskill=qa-lead(run 34737158614).This is the first scoring of the holdout split since the risk-tier calibration in #1698 / PR #1773 landed.
Engine tier is the one that matters here — #1686's whole point was that the harness was scoring the
wrong model:
{ "engine_tier": "persona", "engine_model": "claude-opus-4-8", "engine_model_observed": "claude-opus-4-8" }Declared == observed, at
run_persona(Opus + Bash-only). We are testing like we run.qa-lead-hold-flaky-network-e2eqa-lead-hold-coverage-gamingqa-lead-hold-flaky-ui-timingqa-lead-hold-auth-no-contractqa-lead-hold-migration-no-rollback-testqa-lead-hold-flaky-masks-real-raceqa-lead-hold-slow-deterministic-suiteqa-lead-hold-payment-happy-onlyqa-lead-hold-well-tested-refactorscore 1.00 · 9 passed · 0 failed · gate PASS vs threshold 0.70
What this confirms, and what it does not
flaky-network-e2eis fixed on held-out data. It scored 0.22–0.25 across four separate harnessconfigurations while every other case moved — which is what identified it as a genuine persona bias
rather than a harness artifact. Root cause (feat: implement issue #1698 — qa-lead over-escalates test flakiness at production tier — the one held-out case parity did NOT fix (HIGH/escalate where MEDIUM/no is correct) #1773): the phrase "payments sandbox" appearing in a
test fixture inflated the risk tier. It now scores 1.00.
payment-happy-only— where payment risk is real and not fixturescaffolding — holds at 0.93. The persona learned to read blast radius, not to ignore the word.
Caveat, stated plainly: this measures judgment quality on curated cases. It does not measure
delivered value in the repo, because the delivery path is still broken in ways this epic is tracking —
a 403 silently destroyed a real advisory (#1734), the preservation fix is in flight (#1775), the
authorization preflight is not started (#1776), and the mention surface still ignores human holds
(#1745). A persona that scores 1.00 and cannot reliably post is worth 0 in production.
Recording here as the S2 (#1645) baseline artifact, per the standard that a number is only meaningful
with its tier attached.
don-petry commentedon Sep 21, 2026
Epic complete — all eight stories delivered and verified 2026-09-20
<id>:hands-offlabels now 9/9 in every fleet repo (was: full family in.githubonly). Root cause was that nothing checked or remediated the derived family — fixed by combsmith/.github#1140.pull_requestsurface served — qa-lead runs without being summoned.persona.ymlsurfaces match what is deployed, enforced by a fail-closed validator.skills[]validated and corrected.--schema-treeallowlist retired.Also delivered under this epic: #1734 and both its splits (#1775 advisory preservation, #1776 authorization preflight), #1778 (issue-triage's blanket hold), #1686 (engine tier), #1698 (risk-tier calibration), #1702 (weekly parity cadence), and #1795 (the non-required-check merge deadlock).
On the original question — "is qa-lead providing value?"
The held-out eval at production parity tier scored 1.00 (9 passed, 0 failed, gate threshold 0.70). State the caveat whenever citing it: that measures judgment on curated cases, not delivered value. The delivery path was the open half, and this epic closed it — the 403 that destroyed a real advisory (#1734) is fixed at both ends: a failed post now preserves the advisory (#1775), and a credential that authenticates but cannot write is caught before the engine is paid for (#1776).
Tracked elsewhere — deliberately not held open here
needs-human-review/dev-lead:needs-human. The router fix merged as feat: implement issue #1133 — persona-mention router ignores stop_markers (needs-human-review, dev-lead:needs-human) — a human hold stops dev-lead but not a mentioned persona .github#1134; the consuming-side ACs (reconcileinteraction.yml, armvalidate-interaction-model) remain, and per that issue's own sequencing the validator must not land before thepersona-mentionchannel is promoted.apply-repo-settingsstub deployment; four repos still lack the self-heal path that keeps their labels present.seed-repo-template.shdestroys files it does not own (security).prompts/bmad/scrum-master.mdstill opens "You are Bob" — the §1.6 violation Rename murat → qa-lead and make it @-addressable #1279 deferred.Closing as delivered.
don-petry commentedon Sep 21, 2026
Reopened — Phase 2: reach (2026-09-21)
Phase 1 closed the gap between qa-lead's declared and served behaviour. It did not change where qa-lead can act, and that is the remaining half of the original finding: "pull-only, reachable in one repo, and nobody remembers to summon it."
Measured reach, 2026-09-21
persona-mention.yml(the router stub).github-privateonlyqa-lead-pr-advisory.yml(the event-driven surface from #1646).github-privateonlyNeither workflow is in
DEPLOYABLE_WORKFLOWS(petry-projects/.githubscripts/deploy-standard-workflows.sh), so no sweep will ever carry them to another repo. The fleet list today is: pr-review-mention, dev-lead, agent-shield, auto-rebase, dependabot-automerge, dependabot-rebase, dependency-audit, add-to-project, initiative-driver, pr-auto-review, feature-ideation.Phase 2 stories
persona-mention/v1-stabledoes not exist today; the channel has onlyv1-next(fde249e0),v1-ring0(a781ad9d),v1-ring1(366ab877). Per pilot(mcp): scan PR diffs for secrets via run_secret_scanning #783 the soak must not skip a tier, and stable is gated on an explicit human go.persona-mention.ymltoDEPLOYABLE_WORKFLOWS.standards/workflows/persona-mention.ymlalready exists, so this is the same shape as #1045 Step D: add apply-repo-settings.yml to DEPLOYABLE_WORKFLOWS (Step C done — v1-stable exists) .github#1128 (#1045Step D for apply-repo-settings). Blocked by (1): with nov1-stabletag, a stable-tier repo computes a nonexistent pin and Initiative: Enhance the PR-review pipeline into a world-class bug hunter — semantic symbol context, issue-type specialist routing, agentic validation, and merged-PR few-shot, eval-gated and dry-run-verified #1088's assert-exists check refuses the deploy.pull_requestadvisory surface deployable — unlike the router, it has nostandards/workflows/template, so there is nothing for the sweep to carry even once it is listed.Sequencing
(1) ring soak → v1-stable exists → (2) add to DEPLOYABLE_WORKFLOWS → sweep opens one PR per fleet repoDo not reorder. This is the precondition #1128 records verbatim for the sibling agent, and the assert-exists check exists precisely to catch the inverted order.
This epic stays open until qa-lead can act outside
.github-private.feat: implement issue #1882 — Enhance AGENTS.md with the operating ru…
don-petry commentedon Sep 23, 2026
Phase 2 status, 2026-09-23 03:20 UTC
qa-lead-pr-advisory.ymlis intentionally still onmain.pr-reviewis now instandards/canary-rings.json, host-relative like dev-lead. A read-onlyevaluatehas been dispatched as execution evidence. Autocut has nothing to cut yet (no pr-review change sincepr-review/v1-next=781b63c8), so the first real cut is still pending verification.pull_request) → feat: implement issue #1165 — [qa-lead reach 3/3] Serve the pull_request surface from the persona mention router (architect decision (b), ADR-0007 consolidation) .github#1167, changes requested. Stop markers bind correctly on apull_request-sourced dispatch (the architect's hard requirement is met). But the router ignores the surface's declaredevents(it would re-advise on everysynchronize), checks only the PR author and notgithub.actor, and drops three of the local gate's five suppressors. Sent back to dev-lead with a budget guard.persona-runnerignoressurface=pull_request, so a router-served advisory loses the test-surface, budget and already-advised gates.Retirement order: combsmith/.github#1167 (revised) and #1905 land → cut
persona-mentionand advancev1-next→ verify on a realpull_request-sourced dispatch (a held PR stays silent; a no-test-surface PR stays silent) → retireqa-lead-pr-advisory.yml.don-petry commentedon Oct 10, 2026
Phase 2 status, 2026-10-10 00:25 UTC
Found: qa-lead's
pull_requestadvisory (story #1646) had posted nothing since it shipped on 2026-09-07. The gate's first read wasgh api --paginate --slurp … --jq, whichghrejects ("the--slurpoption is not supported with--jq"), so every run failed closed withchanged-file list unavailable. 25 of 25 sampled runs skipped that way; none of the last 100 reached theadvisejob. The bats suite stayed green because itsghstub accepted any flag combination.Fixed: #1909 (closes #1905), merged 2026-10-09 23:07Z as
d9a98c7c:--slurpoutput to externaljq, and the test stub rejects--slurpwith--jqthe way realghdoes (restoring the old call fails 10 tests);persona-runnerhonourssurface=pull_requestand runs qa-lead's suppressors (test surface, opt-out, human hold, budget, already-advised) before the engine, from one script shared with the local workflow;Verified live: one router-shaped dispatch (
persona=qa-lead,surface=pull_request,source_repo=petry-projects/.github,item_number=1167) → run38007523367: pre-gaterun, authorization preflight passed for a cross-repo post, advisory posted at combsmith/.github#1167 (comment). Re-running the gate on that PR now returnsskip:already-advised. Against 12 open.github-privatePRs with realgh, the merged gate returnsrunon 11 andskip:no-test-surfaceon the standards-sync PR.Not yet observed: the local workflow's own trigger (
opened/ready_for_review). No PR has been opened in.github-privatesince the merge — the org is over the 50-PR automation cap (51), so dev-lead is deferring pickups.Remaining for the epic: combsmith/.github#1167 (router serves the
pull_requestevent) →persona-mention/v1-nextcut →.github-privatestub gains thepull_requesttrigger → verify on real PRs → retireqa-lead-pr-advisory.yml.don-petry commentedon Oct 10, 2026
Phase 2 status, 2026-10-10 11:25 UTC
combsmith/.github#1167 (router serves the
pull_requestevent, closes combsmith/.github#1165) merged asfb556603.Added since the last note, both requested in review and verified before merge:
runtime.reusable(dev-lead, pr-review), so only personas without a dedicated runtime are dispatched;Verified by running the workflow's routing block against real PRs and the live manifests with the dispatch call intercepted:
openedroutes qa-lead alone;synchronizeroutes nobody; a PR carryingneeds-human-reviewis skipped at the stop marker; a failed fetch for an earlier persona still lets qa-lead route.Not live yet.
persona-mention/v1-nextis still at2e3d523b, and.github-private's stub has nopull_requesttrigger. Remaining: cutv1-next→ add the trigger to the.github-privatestub → verify on real PRs → retireqa-lead-pr-advisory.yml.don-petry commentedon Oct 10, 2026
Phase 2 status, 2026-10-10 18:10 UTC
Landed since the last note:
49770289. This repo'spersona-mention.ymlsubscribespull_request: [opened, ready_for_review, reopened], pin unchanged atpersona-mention/v1-next(nowv1.19.0= petry-projects/.githubfb556603).df5f2265. The posting login is read through one helper with three outcomes; an unreadable read is retried and reported as unreadable rather than as a wrong identity; the advisory body is written before the identity check, so it is preserved if the post fails.Observed live on PR #2188 (the first PR to carry the trigger): the router dispatched qa-lead only (dev-lead and pr-review skipped as served by their own runtime); the local workflow and the router-served run serialized in one concurrency group; the PR received exactly one advisory. The router-served run lost its advisory to the identity-read defect that #2198 has now fixed.
qa-lead advisories since the gate fix (#1909): one each on #2188, #2190, #2191, #2192, #2198. Before that, none since the workflow shipped on 2026-09-07.
Remaining: observe one PR where the router-served run posts and the local run skips as already advised (or the reverse); then retire
qa-lead-pr-advisory.ymland update the qa-lead docs; then close this epic. Fleet reach beyond this repo is tracked in combsmith/.github#781.don-petry commentedon Oct 11, 2026
Closing: the declared qa-lead is now the served one
Outcome. This epic opened on a persona that had posted 2 advisories ever and ran in one repo only when someone remembered to summon it. Between the gate fix (2026-10-09 23:07Z) and 2026-10-11 02:20Z, 35 PRs were opened in this repo: 33 received exactly one qa-lead advisory, none received two, one was correctly skipped as
no-test-surface(#2225), and one was missed for a reason recorded below (#2238).Phase 1 — the eight stories in the description (#1644–#1651) — closed in September.
Phase 2 — reach and serving path (ADR-0009, decision (b): serve the
pull_requestsurface from the shared persona-mention router):gh api --slurpwith--jq) fixed; runner honourssurface=pull_requestwith qa-lead's suppressors; personas without a registered pre-gate skippull_requestevent: declared-events filter, bot-actor check, own-runtime skip, loop finishes past a per-persona failurepull_requestpersona-mention/v1.23.0onv1-nextqa-lead-pr-advisory.ymlretired; the router is the only serving pathVerified live, not by status:
pull_requestdispatch (ADR-0009's hard requirement): the routing block run against a real PR carryingneeds-human-review→skip stop-marker needs-human-review.skip:already-advised.run, one advisory posted three minutes after the PR opened.skip:no-test-surfaceon both paths.Not done here, tracked elsewhere:
pull_requestworkflow for it, andsynchronizeis deliberately not subscribed) — qa-lead never advises a PR that was opened with merge conflicts — pull_request.opened does not run, and synchronize is deliberately not subscribed #2253. Pre-dates the router; needs a design decision.broodminder-exportcarry the stub;persona-mention/v1-stablepredates thepull_requestsurface.