Qualify one replayable benchmark for a Factory configuration change #50

Current triage — 16 October

Not planned now. No new replay/model runs are needed for the current delivery priority. Revisit when a specific supported configuration change could unblock Bridge, AIOS or OPD and existing evidence cannot choose it. Pin one comparable task and a small budget then; no benchmark platform, router or automatic tuning.

#8 proves that Factory can improve from observed friction. It does not establish that a different prompt, skill or model performs better on our tasks. Use one real task to qualify a repeatable comparison and connect measurement to the existing improvement process.

Acceptance

• Select one completed, bounded task with recoverable initial source and acceptance criteria; the application delivery in Prove a complete delivery on an owner-selected application #4 is a candidate if its inputs are sufficient. Pin the initial revision, task, relevant dependencies, native settings and method revision. Keep the evaluator's checks fixed before comparing candidates.
• Provide a small reusable replay procedure through the current native tools. Both configurations start from the same initial state and receive the same task and budget. Change one factor only: a method instruction or a supported model choice. Run in disposable workspaces without replaying release, production or external-write side effects.
• For a model comparison, use two independent copies of the same frozen session/thread checkpoint, not two loosely similar prompts. Pin the source session/thread ID and checkpoint, retain its pre-checkpoint transcript and task instructions privately, and give both attempts the same history, selected skills/definition, source/data snapshot, tools, access and agreed budget. Change only the model for the role under test; keep other roles and checks fixed. Neither attempt may see the other's result.
• Verify user-visible behavior independently of the writer's claims, then assess the code where relevant. For a UI task, test whether app-first verification exposes anything the current review misses; use existing review contexts rather than assuming a new permanent role is needed. Retain regressions and failures as well as successes.
• Use #49 and the coverage limits from #48 to report outcome, human assistance, rework, time and available usage. Predeclare a small repetition budget, report all attempts and treat a single trial as exploratory. Unavailable exact replay or cost data must limit the conclusion.
• Return adopt, reject, inconclusive or defer for this one candidate. An improvement claim needs the existing independent review/adoption path from Improve the method from observed session friction and outcomes #8 and an observation on a subsequent real task not used to tune the candidate. Preserve VISION.md and the agreed acceptance criteria; do not optimize the grader or expand product scope to make the result pass.

Owner: the relevant work module performs the bounded experiment; the shared evaluator assesses it and AgentOps accepts its implication for the project. This does not build a benchmark platform, automatic prompt rewriter or model router.

Sources: The software factory stack — Benchmarks and improvement, Zach's demonstration around 15:23–20:30, Cole's workflow explanation around 14:26–15:31.

Same-session replay for model comparisons

Freeze the session before either candidate produces its answer. Use a native branch/clone when it preserves the required context, or supply the same retained context package to two fresh sessions. Record both attempt IDs and their lineage to that checkpoint. Keep logs/exports in the approved private location; publish only sanitized references and results.

A transcript alone does not restore the workspace, task data, memory, permissions or external-service state. Reset or freeze the relevant state as well; retain the existing no-side-effects and no-future-answers restrictions. Record model-specific settings and any unavoidable differences in tool/system context. If the native tools cannot reproduce a material part, report that limitation and narrow the conclusion rather than claiming exact equivalence or building a custom replay engine.

“Same session” means the same input and state up to the checkpoint. The two models' answers, subsequent tool calls and repair paths can differ; these differences are what the experiment measures. Keep the repetition and whole-attempt accounting requirements above.

Video follow-up — 26 October 2026

The Factory conversation does not establish another required architecture layer. Most topics already map to #48 (retained evidence), #49 (useful outcomes and human effort) and #88 (human decisions).

One concrete replay question to retain: at 29:16, the speakers discuss agents reading future commits or completed PRs and reproducing external side effects. For a later experiment, verify what future answers and tracker state are reachable, alongside the already-required clean starting state and disposable workspace. A reset checkout alone does not settle that question.

The discussion of what “yield” means at 9:24 supports deciding the useful outcome for the selected task, rather than adopting PR count as a quality metric. These are research inputs, not a reason to change current priority or start model runs. Review-quality Lab #99 captures the separate Omarchy/Claude Code reviewer comparison.

Method and scoring clarification — 29 October 2026

The public Factory method is a shared reference for people, planning/coordinator agents and AgentOps across domains. A project selects its own definition, relevant skills and reviewed method revision; workers receive a bounded context package. This does not require every worker to fetch the whole live guide or grant it a Notion connection.

The existing factory-evaluate skill already covers the requested experiment. No additional evaluator skill or benchmark platform is needed. AgentOps commissions the evaluation; domain checks come from the work module.

For this issue's eventual trial:

• Predeclare what counts as a useful accepted result, the objective checks, any judge rubric/classifications and point mapping, pass threshold, disagreement handling, repetitions and budget. Keep the judge and other roles fixed in a model-only comparison.
• Evaluate independently of the author’s self-rating. Keep unknown/unverifiable evidence distinct from pass and fail; report denominators and coverage rather than averaging away missing results.
• Retain the frozen input/checkpoint, resulting artifacts and native run histories before disposable environments are deleted. A saved archive is not proof of supported GUI restoration or exact replay. If harness, system/tool context or state differs materially between attempts, disclose that and do not label the result a model-only comparison.
• Report each attempt and the complete comparison in the issue or linked result artifact; use a PR when adoption would change method/configuration. Preserve the earlier priority hold: this clarification does not start benchmark/model runs or adopt a router.

Verified Warp reference: Scorers use configured classifications with score values, a pass threshold and a judge model. The definition example maps test evidence to 1 and skipped tests to 0; its schema bounds label scores to 0–1. This is not a universal correctness oracle. WarpBench reconstructs historical task/configuration states and uses LLM judging; its results are evidence about its own selected task set, not a prediction for ours. Our objective domain checks remain primary, with an optional judge for criteria they do not settle.

Related experiments remain separate: ZeroShot #97, review quality #99.