Factory MVP: rebuild the agentic factory around a declared control plane, one run contract and a council of Lenses

[don-petry]

Summary

Treat the current agentic factory as a proof of concept and build the production MVP: a declared work-item lifecycle with one place that admits runs, one run contract in which the model requests effects and a separate job applies them, agents on their own identity, one release, and a council of independent Lenses that report structured findings to something that decides.

Grounding

Pull request petry-projects/.github-private#2225 — docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed) carries the three documents this idea rests on:

• PRD (docs/prd/agentic-software-factory.md): the requirements, targets with POC baselines, constraints and exit criteria.
• Architecture review (docs/architecture/factory-mvp-architecture-review.md): the evidence, root causes, options, comparison with other systems, an adversarial review, and the research on councils.
• ADR-0013, proposed (docs/architecture/adr/0013-factory-mvp-reference-architecture.md): the decision and its checkable boundaries.
• Market research (docs/research/market-agentic-software-factories-research-2026-10-10.md): what the market offers, the solo builder as a customer, a competitive teardown.

Why now

Measured on 2026-10-10:

• At least 44,968 workflow runs in seven days in this repo for 30 merged pull requests.
• Merges per week flat since August; median time to merge up from about 1 hour to 12–22 hours.
• No open pull request in the org holds an approval.
• Fixes have been moving failures instead of removing them: eight cases are listed in review §2.4.

Seven structural causes explain the eleven recurring failure classes. Each is a decision the POC never made once, so it was made many times, locally.

The hypothesis

Convergence: a change is better when a council of independent Lenses (models, tools and perspectives) has examined it and their findings have been resolved than when one agent has done its best alone. The research supports this under conditions (review §15): independent evaluation beats self-evaluation and Lenses that can see different things find different defects, but a council that converses does worse than one that reports, and a council that co-writes does worse than one author. The POC already runs a large council through free-text discussion and cannot say what any member contributes. The proposal gives every Lens, third-party reviewers included, one structured contract and measures each one.

What it does not claim

The adversarial review (review §14) found the architecture removes waste under the machine limits and does not, by itself, raise the two ceilings that bind: the maintainer's attention, and how often a change is right the first time.

Open decisions

1. Do agents move off the owner account onto a GitHub App?
2. What is the budget, defect policy and kill criterion for the migration? ADR-0013 is not in force until these exist.
3. Which outside reviewers are removed now? Review §16.6 recommends keeping three or four of nine for measurement.
4. How is it built? Two options to weigh:
   • In place, as a strangler inside this repo.
   • Greenfield in a second, new repo, with this repo and its callers switched over when the new one is proven.

Related

• petry-projects/.github-private#2134 — Research PR Fix Agents like SonarQube Gitar: review §16.4 evaluates Gitar and SonarQube as council members.
• Three cheap changes can be made before any of this and would show how much of the rest is needed (PRD OQ-11): stop updating branches that do not conflict, move agents to their own identity, and read third-party reviewers through structured surfaces only.

[don-petry]

How it is built: decided 2026-10-10

Open decision 4 above is settled. Build greenfield in a new repository; cut over as a strangler through the channel tags this repo already has. Not in place, and not a rename or one-step swap.

The analysis is in the grounding pull request, petry-projects/.github-private#2225 — docs: factory MVP — PRD, architecture review and ADR-0012 (proposed): review §18, and ADR-0012 decisions 1 and 10.

The two options as posed

	In-place strangler	Greenfield repo, then switch
Cost of each factory change	The POC's: about 56 CI jobs a push, nine reviewers, an approval gate that issues for none of 52 open pull requests	One hermetic required check and a council chosen from the first day
Risk to the running line	Shared workflows, scripts and release tags	None until cutover
"The POC takes defect fixes only"	Unenforceable in one tree	Physical: a different repo
Agent identity	Retrofitted under the live ruleset	The App authors agent changes from the first commit, so the maintainer can review them normally
Contact with real traffic	Early and continuous	Late, unless designed in
Parity	Replaced piece by piece against the same tests	At risk: about 70,000 lines of tests encode the POC's incident history
Stopping	Half a strangler is hard to unwind	Archive the repo
Cutover	Per intent, reversible	All at once

Neither is right as posed. In place, the replacement is built through the line that is failing. Greenfield with one switch meets real traffic only at the end.

Two platform facts that decide the shape

• A literal switch is not available. GitHub's documentation: "GitHub Actions does not support redirects for actions or reusable workflows." Renaming this repo, or reusing its name, breaks every consumer's reference until it is re-pinned, across twelve repos at once.
• Forwarding is available. Reusable workflows nest up to ten levels, so a reusable workflow here can become a thin caller of one in the new repo.

The hybrid

1. In this repo, now: the configuration fixes and the GitHub App identity. Both are worth having whatever happens next.
2. A new repo that starts as it means to go on: a required hermetic check, the App as author of agent changes, environment-scoped secrets and the starting council, from the first commit.
3. Real traffic in the first weeks: the decision logic runs read-only from the new repo as a poller over every repo. It needs read access only. Its disagreements with what the POC actually did are the measure of parity.
4. Cutover at the seam that already exists: a reusable workflow here is replaced by a forwarder; moving a channel tag onto that commit switches one intent for one ring; moving it back is the rollback. No consumer file changes.
5. Last, the re-pin: once every intent is served by the new repo at stable, the standards sweep points consumers at it directly.

What it costs

• Three repos for a while.
• Parity is a real risk; the read-only comparison measures decisions, not every effect.
• It makes each change cheaper, not fewer. The size of the build is the same either way and is not yet priced.
• The POC still has to be kept alive while this is built.

Conditions before the new repo is created

• A dated limit for the period in which two systems coexist.
• A defect policy for the POC, with a weekly cap.
• A kill criterion. If it is met, the configuration fixes and the identity change stay and the new repo is archived.

Still open

• The new repo's name.
• The three conditions above.
• Whether agents move to a GitHub App, and which outside reviewers are removed now (decisions 1 and 3 in the idea).

[github-actions[bot]]

Enhancement: three things that decide whether the migration finishes

Mary (BMAD analyst), focused research on this idea, 2026-10-10. The PRD, review and ADR-0012 in #2225 already cover the architecture, comparable systems, the council research and the in-place against greenfield choice. This comment does not repeat any of that. It adds work on three questions the documents leave open. Each of them decides whether the greenfield hybrid finishes, not what it builds.

1. How is parity judged when the control is known to be wrong? ADR-0012 decision 10 requires disagreements "within a bound declared beforehand" but does not say how to set that bound or how to score a disagreement.
2. What numbers go into the three preconditions? These are the dated limit, the POC defect policy and the kill criterion (open decision 2, OQ-9). ADR-0012 is not in force without them.
3. Which LLM credential does the new repo start with? Decision 1 says the repo "starts as it means to go on", and this choice is hard to change after the first commit.

---

1. Parity with a control that is known to be wrong

The read-only observer applies GitHub's Scientist pattern ("measure twice, cut once") to the factory: the POC is the control, the factory Kernel is the candidate, and only the control's result takes effect. Scientist's lesson is that raw mismatch rates mean little until every mismatch is classified. That matters more here than usual, because the POC is wrong often enough to justify the rebuild: 0 of 52 open PRs hold an approval, and 34% of merges went in without a standing approval. If the bound measures agreement with the POC, the migration is aiming to reproduce the POC's deadlocks.

Proposal: score disagreements against outcomes, with bounds that differ by direction. For most disagreements GitHub's later record already settles who was right, so the maintainer does not have to adjudicate them. That protects the binding ceiling in review §14.1, which is the maintainer's attention.

Disagreement	Settled automatically from later facts	Classified as
POC withholds approval, factory approves	Maintainer merged the same content identity with no further commits	POC false block (factory better)
POC withholds approval, factory approves	A human Finding, a revert, or a fix to the same files within 14 days	Factory false approve (unsafe)
POC approves, factory withholds	Revert or post-merge fix to the same files within 14 days	POC false approve (factory better)
Either way, no settling fact within 14 days	—	Sent to the maintainer, capped at N per week

Bound the two directions differently. A false approve is an escaped defect (SM-C1). A false block costs lead time. Suggested starting bounds: false approve upper bound ≤ 2%, false block ≤ 10%. Each is a one-sided 95% Clopper–Pearson bound.

The sample sizes set the calendar. Units are correlated within a PR, so count one unit per PR at its final content version, not per evaluation. To show a rate is ≤ 5% at 95% confidence:

Observed disagreement rate	Independent units needed	At ~30 hosting-repo PRs a week
0%	59	~2 weeks
1%	93	~3 weeks
2%	124	~4 weeks
3%	311	~10 weeks

So the approving intent family needs about four weeks of observer data at best, and longer if its true disagreement rate is near 3%. A ≤ 2% bound on false approves needs several hundred units, which means pooling consumer repos. This is a firm lower limit on the dated limit in §3. Admission and liveness decisions are higher volume and can reach their bound in days.

Also record "ignore" rules as data. Scientist lets you register known-benign mismatches. Each one should be a reviewed entry in the factory repo, never an inline exception, so the bound cannot be met by widening what is ignored. That would be the same gaming risk the PRD already guards against for tests (FR-59).

2. Use the POC defect freeze to build the parity corpus

ADR-0012 lists "incident knowledge will be lost in translation" (about 70,000 lines of bats) as a consequence. The defect policy can recover part of that. Every POC defect that is not fixed under the cap becomes a recorded-fixture replay case in the factory repo. Its GitHub timeline is the fixture, and the expected decision is the test. The freeze then grows the factory's parity suite instead of only rationing POC work, and the bats suite is replaced over time by cases from live traffic.

3. Proposed values for the three preconditions

The ranges below come from the documents' own figures: the POC needs 30–52 fix PRs a week, the line delivers 29–36, and the rewrite is about 100–150 PRs. General parallel-run guidance puts business-critical cutovers at 4–8 weeks of overlap each.

Condition	Conservative	Balanced (suggested)	Aggressive
Dated limit for two systems running together	24 weeks	16 weeks from the factory repo's first commit: about 10 weeks to build at 10–15 factory PRs a week, plus at least 4 weeks of observer data for the approving family (§1)	10 weeks; only works if OQ-11's three cheap changes already fix class A
POC defect policy	Fix everything in classes K, F and A	POC fix only for class K (corrupts/destroys), class F (credential/security), and org-wide merge outages. Cap: 10 POC fix PRs a week. Everything else gets a poc-frozen label and becomes a fixture (§2)	Classes K and F only, cap 5
Kill criterion (any one)	—	(a) Factory-repo PRs are under 30% of agent merges org-wide for 3 consecutive weeks (adapts review §14.2's signal); (b) the approving family's false-approve bound is not met 8 weeks after the observer starts; (c) maintainer non-decision time, countable once agents use the App, stays above its pre-migration baseline for 4 consecutive weeks	(a) alone
Re-scope trigger (not a kill)	—	4 weeks after the three OQ-11 changes: if ≥ 70% of agent merges hold a standing approval and p50 lead time is < 8 h, re-argue the rest only on classes B, C and D (review §14.5)	—

Criterion (c) holds the migration to the PRD's own counter-pressure: a rebuild that raises the binding ceiling, the maintainer's time, for a month has failed regardless of code quality.

4. The LLM credential: decide before the first commit

The documents treat subscription capacity (C-3) as fixed and list a technical reopen trigger (the CLI's bare-mode default). They do not cover two newer facts:

• Terms. On 2026-02-19 Anthropic stated that OAuth tokens from Free, Pro and Max plans are for Claude Code and Claude.ai only. Using them "in any other product, tool, or service — including the Agent SDK" is not permitted, and enforcement is server-side. Running the official claude CLI in Actions stays inside that line. Review §6.3's option (e), Agent SDK packages, does not on a subscription. Neither would a Python Harness that calls the model directly. If C-3 holds, the Harness must shell out to the CLI.
• Keyless metered access now exists. Anthropic Workload Identity Federation exchanges a GitHub OIDC token for an Anthropic access token that lives 600 s (configurable). A federation rule can match repo:<org>/<factory-repo>:environment:<name>.

Option	Capacity signal (FR-35/37)	Secret exposure	gh-aw (review §6.3c)	Cost shape
A. Subscription OAuth via the official CLI (status quo)	Estimate only; shared with interactive sessions	Long-lived token in an environment secret	Blocked: gh-aw's Claude engine documents that CLAUDE_CODE_OAUTH_TOKEN is not supported	Flat; cheapest at current volume
B. API key	Exact: Usage API, per workspace, 1-minute buckets, data within about 5 minutes	Long-lived key	Unblocked	Metered
C. API via WIF, pinned to a factory environment	Exact, as in B; a dedicated workspace isolates factory usage from interactive use (FR-37)	No static secret. The platform enforces decision 5's "no factory secret readable on a work-item branch". A successfully injected run job (ADR-0012's stated gap: no network containment) gets a token that expires in minutes	Unblocked	Metered; Batch API at 50% off
D. Copilot-hosted Claude via gh-aw (copilot/claude-* with github.token)	Copilot's premium-request meter	None	Native	Needs a Copilot plan, which conflicts with the free-plan non-goal

Suggested path: hybrid by Intent, starting with eval-gated promotion on C. Eval runs are new consumption. They can be batched (50% off), and they are the item ADR-0012 says "spends the scarcest resource". On C they stop competing with the subscription window entirely, and the promotion gate gets an exact cost line. Code-writing Intents stay on A until the measured cost of the eval lane sets the price of moving them. One caveat: the Usage API needs an Admin-scoped credential. It belongs in the confined reviewer-style job, never in run.

5. Competitive and platform signal since the documents were written

• gh-aw has moved from githubnext to github/gh-aw, and GitHub's own Actions documentation now covers it as public preview. It is at v0.91.7 (pre-release, published 2026-10-09; releases are near-daily). This repo pins v0.74.4. ADR-0012's trigger "a generally available orchestrator" has not fired. It is approaching, so keep the Effect vocabulary compatible with gh-aw safe outputs (review §6.3) and re-check at each gh-aw minor release.
• The Feb 2026 terms clarification (above) shows the vendor moving automation onto API identity. Read it with ADR-0012's existing bare-mode trigger: both point the same way, which makes option C the likely end state whatever is chosen now.

6. Risks and unknowns this adds

• Outcome anchoring is slow. The 14-day settling window delays the verdict on the approving family by two weeks. Mitigation: start the observer in week 1, before any cutover work.
• Survivorship in "merged the same content". A maintainer override may itself have been a mistake. Count overrides that are later reverted as factory-correct blocks.
• Pooling consumer repos for the false-approve bound mixes populations. Report per repo as well as pooled.
• Option C changes the bill. If the review's $196–280 a week is API-equivalent spend, moving every Intent to metered access costs roughly $850–1,200 a month. That is why only the eval lane moves first.
• Unverified: whether WIF rules can also match job_workflow_ref, which would pin the credential to factory-controller.yml itself and not only the environment. Settle it on the canary repo with ADR-0012 decision 10's three assumptions.

7. Phased validation

Phase	Weeks	Exit evidence
0. Preconditions	0	Dated limit, defect policy and kill criterion written (§3); LLM-credential option chosen (§4); SM-C1 baseline captured
1. Observer + adjudicator	1–4	Disagreements auto-classified ≥ 80% of the time; the maintainer adjudicates ≤ 10 a week; the poc-frozen → fixture conversion rate is 100%
2. Eval lane on WIF	2–6	Zero static Anthropic secrets in the factory repo; cost per promotion reported exactly; no promotion blocked by the subscription window
3. First cutover (approving family, one ring)	≥ 5	False-approve upper bound ≤ 2% and false-block ≤ 10% on ≥ 124 PR units; rollback by channel-tag move exercised once

Success metrics for this enhancement specifically: the bound in ADR-0012 decision 10 is a number before the observer starts; the share of disagreements needing a human is ≤ 20%; the parity suite grows every week the POC defect cap holds; and the kill criterion is checked in the weekly scorecard (UJ-9), not remembered.

Sources

• GitHub, "Scientist: Measure Twice, Cut Once": https://github.blog/2016-02-03-scientist/
• GitHub Docs, develop agentic workflows (public preview): https://docs.github.com/en/actions/tutorials/develop-agentic-workflows-in-github-actions
• gh-aw Claude engine authentication: https://github.github.com/gh-aw/engines/claude/ · releases: https://github.com/github/gh-aw/releases
• Anthropic, Workload Identity Federation for GitHub Actions: https://platform.claude.com/docs/manage-claude/wif-providers/github-actions
• Anthropic, Usage and Cost Admin API: https://platform.claude.com/docs/en/manage-claude/usage-cost-api
• Subscription OAuth terms clarification (secondary coverage): https://winbuzzer.com/2026/02/19/anthropic-bans-claude-subscription-oauth-in-third-party-apps-xcxwbn/ · https://thenewstack.io/anthropic-agent-sdk-confusion/
• Parallel-run durations (secondary): https://www.easy.bi/blog/parallel-run-legacy-migration-playbook/
• Sample sizes: one-sided 95% Clopper–Pearson upper bounds, computed for this comment.
• Internal: PR docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed) #2225 (PRD §§5, 7, 11; review §§13–18; ADR-0012 decisions 1, 5, 10, 11 and Consequences).

[don-petry]

Update: market research added, and the ADR is now ADR-0013

Two changes to the grounding pull request, petry-projects/.github-private#2225 — docs: factory MVP — PRD, architecture review, market research and ADR-0013 (proposed).

The ADR was renumbered. It was opened as ADR-0012. While the pull request was in review, that number was assigned on main to the maintainer comment gate's exemption set, so the factory MVP record is now ADR-0013 (docs/architecture/adr/0013-factory-mvp-reference-architecture.md). The earlier comment here that says "ADR-0012" means this record. The idea's body has been corrected.

Market research was added (docs/research/market-agentic-software-factories-research-2026-10-10.md), with solo founders and indie builders as the priority segment, and the PRD was revised against it. What it found:

• The workers are a commodity; the line around them is not. Turning an issue into a pull request, repairing CI on it and giving it a first review are each offered by eight or more products. Nobody offers a solo operator a governed unattended merge, a cross-repo view of why work is waiting, staged rollout across a portfolio, capacity admission, or reviewers drawn from different vendors.
• The merge decision is the step every vendor hands back. Four refuse in writing to merge to a default branch.
• The price of automation is moving from flat to metered. One platform moved its coding product to token-metered billing in June 2026; one model provider announced the same for automated use of its subscriptions and paused it on the start date.
• Nothing shows a factory beats one well-run agent for one person. That is the strongest objection in the market, and the PRD now carries a metric to answer it.
• Attention at the orchestration layer is large and does not convert to revenue.

What changed in the PRD: a statement of what is distinctive, the user described as a technical solo builder running a portfolio, a constraint that capacity terms can change on short notice, two requirements (adopt workers and reviewers through the contract; run on metered capacity under a hard cap), a single-agent baseline metric, and a build-versus-adopt table.

New open decisions: whether to adopt GitHub's Agentic Workflows for the worker half, the monthly figure at which a metered fallback stops, and whether release and operation is the next scope after the MVP.