From 126cd9b97a58386b5f299210edfd93729047e1cd Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 19:46:36 -0700 Subject: [PATCH 01/32] chore(porch): 1280 init spir --- .../status.yaml | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml new file mode 100644 index 000000000..77c5fec24 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -0,0 +1,20 @@ +id: '1280' +title: prompt-surface-judgment-not-ru +protocol: spir +phase: specify +plan_phases: [] +current_plan_phase: null +gates: + spec-approval: + status: pending + plan-approval: + status: pending + pr: + status: pending + verify-approval: + status: pending +iteration: 1 +build_complete: false +history: [] +started_at: '2026-08-01T02:46:36.457Z' +updated_at: '2026-08-01T02:46:36.458Z' From 81c20ce8dedfd7984cf7588584a1edb1105fbd1c Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 19:55:37 -0700 Subject: [PATCH 02/32] [Spec 1280] Initial specification draft Judgment-not-rules rewrite of the always-on prompt surface, targeting >50% reduction for a frontier-model fleet. Carries the three architect-mandated designs: the per-surface cut plan with word targets, the A/B non-inferiority eval design with a pre-registered decision rule, and the scar-rule carriage plan; plus the rollback story. Also records a measurement defect found during the spec phase and endorsed by the architect as criterion M0: scripts/measure-prompt-surface.sh derives its phase-task term from codev-skeleton/porch/prompts/, a dead Ralph-SPIR-era tree no code reads, while the live resolver loads protocols//prompts/. The metric was structurally blind to this project's largest target. Corrected always-on baseline is 33,519 served words, not 21,702. --- .../1280-prompt-surface-judgment-not-ru.md | 695 ++++++++++++++++++ codev/state/spir-1280_thread.md | 72 ++ 2 files changed, 767 insertions(+) create mode 100644 codev/specs/1280-prompt-surface-judgment-not-ru.md create mode 100644 codev/state/spir-1280_thread.md diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md new file mode 100644 index 000000000..d05f5003b --- /dev/null +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -0,0 +1,695 @@ +# Specification: Prompt surface — judgment-not-rules rewrite (>50% always-on reduction) + + + +## Metadata +- **ID**: spec-2026-07-31-prompt-surface-judgment-not-rules +- **Status**: draft +- **Created**: 2026-07-31 +- **Issue**: #1280 +- **Protocol**: SPIR + +## Clarifying Questions Asked + +Issue #1280 carries a complete charter (goal, attack order, four Baked Decisions, +required prior art). Per the specify-phase rule, no clarifying questions were put to +the architect. Three questions were resolved against the repository instead: + +1. **"What exactly is the always-on surface, as served?"** — Resolved by reading the + live composition path (`packages/codev/src/commands/porch/prompts.ts`, + `packages/codev/src/lib/skeleton.ts`) and by measuring this builder's own served + spawn artifacts (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). See + **Current State**. + +2. **"Is the committed measurement script fit to score this project?"** — No. It + measures a directory no code reads, and is blind to the project's largest target. + Finding and consequences in **Current State**; correction is criterion **M0**. + Raised with the architect via `afx send` on 2026-07-31 (non-blocking). + +3. **"Where are the eight ratified scar rules?"** — Recovered verbatim from + `builder/spir-1252:codev/resources/scar-rules.yaml` (the preserved reference + branch). Carriage plan in **Desired State**. + +## Problem Statement + +A Codev builder consumes **~33,500 served always-on words** before it reads a single +line of the code it was spawned to change (corrected measurement, below). Almost none +of that is information the builder cannot derive; it is *process narration* — recipes +for how to be an agent, written when the fleet could not be trusted to infer them. + +Spec 1252 measured the surface and proved the obvious remedy does not work: +deduplication yields **−7.0%** (21,856 → 20,324 on its own proxy), because the surface +is not duplicated, it is **over-instructed**. Its review states the conclusion plainly — +the largest remaining block is single-owned prose. + +Over-instruction is not merely a token bill. It has three compounding costs: + +1. **It crowds out judgment.** A frontier model given a 3,700-word procedure follows + the procedure. Given a 700-word contract and a goal, it reasons about the goal. + Anthropic's published account of the Claude-5-generation rewrite reports >80% of + Claude Code's system prompt deleted with no measurable performance loss, by + replacing rules with judgment, deleting worst-case padding, designing interfaces + instead of examples, and using progressive disclosure. +2. **Nobody reads it, so it rots unnoticed.** 1252 found the *served* SPIR builder + prompt had silently lost its entire `Verify Phase` section, and a detector had been + reporting the drift, unread, for months. A surface too large to read is a surface + too large to maintain. +3. **It makes its own success unmeasurable.** The committed measurement script scores + a directory the runtime never loads (below). Nothing about the prompts a builder + actually receives has been under measurement. + +The blog operation — deletion on judgment-trust grounds — was an explicit **Non-goal** +of Spec 1252 and has never been attempted in this repo. + +### The instrument is part of the deliverable + +This is the **second** measurement defect in the 1252 lineage. The first: Spec 1252 +originally shipped with no measurement plan at all — caught at a human gate, not by +CMAP. The second is the one this spec documents below: a committed, tested, reproducible +script that measures the wrong directory, so its numbers are precise and wrong. Neither +defect was caught by reading the instrument's code; both were caught by asking *what +does this claim to measure, and does it?* + +Hence an explicit project principle, adopted at the architect's direction and binding on +every criterion in this spec: + +> **The instrument is part of the deliverable, and instruments get reviewed against what +> they claim to measure — not merely against whether they run.** A measurement script, +> a check, or a baseline artifact is subject to the same adversarial review as the +> feature it scores. "Deterministic and committed" is not "correct." + +It is load-bearing here in a way it was not for 1252: this project's headline success +criterion is a number that one shell script emits. Discovering — *before* writing the +spec — that the metric was structurally blind to the project's own largest target is the +correct order of operations, and it is the reason M0 precedes M1. + +## Current State + +### How the always-on surface is composed + +| Stage | Content | Served words | +|---|---|---:| +| Session | `CLAUDE.md` (hot tier inlined) — `AGENTS.md` is the byte-identical twin, one loads per session | 5,815 | +| Spawn (once) | `protocols/spir/builder-prompt.md` (824) + inlined `protocols/spir/protocol.md` (3,703) + inlined `roles/builder.md` (1,837) | 6,364 | +| Every phase task (×I) | hot tier (`arch-critical.md` 416 + `lessons-critical.md` 320 = 736) + the resolved phase prompt **with its `{{> templates/…}}` includes expanded** (mean 1,398) | 2,134 each | + +With `I = 10` phase-task deliveries (the proxy 1252 fixed, consistent with B4's mean of +3.06 review rounds/project across ~4–6 phases): + +``` + CLAUDE.md 5,815 ++ spawn 6,364 ++ phase task ×10 21,340 + ---------------------- + ALWAYS_ON 33,519 +``` + +Per-prompt detail (SPIR, expanded): `specify` 1,402 · `plan` 1,169 · `implement` 1,065 +· `review` 1,957. Templates pulled in by those prompts: `spec.md` 632 · `plan.md` 649 · +`review.md` 641. Reviewer-side (not in the builder's total, and unmeasured today): +`consult-types/*.md`, 5 files, 2,154w. + +The same disease is present in every other protocol shipped from the skeleton +(`aspir` prompts 3,671 · `pir` prompts 4,306 · `pir/protocol.md` 2,066 · +`maintain/protocol.md` 1,765 · per-protocol `consult-types` 813–2,154). + +### The measurement defect (why M0 exists) + +`scripts/measure-prompt-surface.sh` derives `PORCH_PROMPT_MEAN` from +`codev-skeleton/porch/prompts/*.md` — 10 files, mean 400 words. The live resolver +(`porch/prompts.ts`, `loadPromptFile`) loads `protocols//prompts/.md`, +a different tree. A repo-wide grep (excluding `node_modules`, `dist`, `.git`) finds no +code reading `porch/prompts`; every hit is historical spec/plan prose. The tree is a +Ralph-SPIR-era leftover — its `specify.md` opens *"You are the **Spec Writer** hat in a +Ralph-SPIR loop."* It is dead authored surface. + +Two consequences, both disqualifying for this project: + +- The reported baseline (21,702 on today's `main`) understates the phase-task term by + ~3.5× and omits `roles/builder.md` entirely. +- **The metric cannot see this project's primary target.** Cutting the SPIR phase + prompts from 1,398 to 450 words moves `ALWAYS_ON_WORDS` by exactly zero under the + current script. A >50% claim scored on it would be phantom savings — the precise + failure mode 1252 built the script to prevent. + +### What already landed, and what is deferred + +On `main` (the 1252 harvest): the drift reconciliation, the audit, the two word +baselines, the behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded), +and the measurement tooling. Deferred to this project by the architect's pr-gate ruling: +the **scar registry and its eight ratified rule wordings**, and enforcement rebuilt +*after* the shrink rather than before it. + +Issue #1279 (dead spec/review templates) is partially overtaken by events — the SPIR +prompts now inline their templates via `{{> …}}` includes. That wiring is exactly what +makes each phase prompt ~600 words heavier than it reads, so the template question is +in scope here: an annotated 632-word template is the "examples instead of interfaces" +anti-pattern the rewrite exists to remove. + +## Desired State + +**One prompt form, written for frontier models, that states contracts and trusts +judgment.** A builder's always-on context tells it: what it owns, what artifacts it +must produce and what shape they take, where the human gates are, what is +irreversible — and then gets out of the way. Everything else is reachable on demand. + +### Rewrite principles (the standard every cut is judged against) + +1. **Contract, not recipe.** State the required outcome and its shape. Delete the + ordered procedure for reaching it. +2. **Interface, not example.** A heading skeleton with one line of intent per heading + replaces an annotated template with filler prose. +3. **No worst-case padding.** Delete instructions that exist for a failure mode a + frontier model does not exhibit (repeated all-caps prohibitions, "⚠️ BLOCKING" + banners, checklists restating the phase body). +4. **Progressive disclosure.** How-to content that a competent agent would look up + moves to skills / on-demand files, addressed by name, not inlined. +5. **Budgets are cheap words worth keeping.** A stated scope or budget line buys + bounded process; frontier models honour stated budgets precisely but never invent + them. Budget/scope lines are exempt from cuts. +6. **Scar rules are verbatim and exempt.** See carriage plan below. +7. **The instrument is part of the deliverable.** Every check, script, and baseline this + project produces is reviewed against what it claims to measure, not merely against + whether it runs. (Stated in full in **Problem Statement**; repeated here because it + binds the cuts too — a word target that is not measured on served words is not a + target.) + +### Per-surface cut plan (word targets) + +Targets are on **served, expanded** words, measured by the corrected script (M0). +Every target applies to the SPIR instance as the measured proxy **and is swept across +all protocols in both trees** (`codev/` and `codev-skeleton/`) — an unswept protocol is +a regression, not a deferral. + +| Surface | Now | Target | What survives | +|---|---:|---:|---| +| `CLAUDE.md` / `AGENTS.md` (twins) | 5,815 | **≤2,200** | repo dual nature, four-tier resolution, gates, area-label policy, the eight scar rules verbatim, the hot-tier block. Worktree recipes, CLI walkthroughs, protocol-selection prose → skills/on-demand | +| `roles/builder.md` | 1,837 | **≤600** | ownership, gates, thread contract, notification triggers, worktree path discipline | +| `protocols/*/protocol.md` (SPIR) | 3,703 | **≤800** | state machine, phase→gate map, artifact contracts, commit/branch format, consultation checkpoints | +| `protocols/*/builder-prompt.md` (SPIR) | 824 | **≤400** | mode, spec/plan/issue wiring, baked-decisions rule, PR strategy | +| `protocols/*/prompts/*.md` expanded (SPIR mean) | 1,398 | **≤450** | goal, artifact path, heading interface, signal contract | +| hot tier (`arch-critical` + `lessons-critical`) | 736 | **736 (unchanged)** | already capped, already judgment-shaped — explicitly out of scope for cuts | +| `protocols/*/consult-types/*.md` (SPIR mean) | 431 | **≤200** | rubric dimensions + verdict contract | +| `codev-skeleton/porch/prompts/**` | 4,014 authored | **0 (deleted)** | dead tree, no consumer | + +Resulting always-on total: **≤15,900 words, a ≥52% reduction from 33,519.** + +``` + CLAUDE.md 2,200 ++ spawn (400 + 800 + 600) 1,800 ++ phase task ×10 (736 + 450) 11,860 + --------------------------- + ALWAYS_ON 15,860 (−52.7%) +``` + +Note the shape of the arithmetic: the phase-task term is 71% of the post-rewrite budget +and the hot tier — which is *not* being cut — is 62% of that term. This is deliberate. +The always-on surface that survives is overwhelmingly curated judgment, not process. + +### Scar-rule carriage plan + +The eight rules ratified by the architect on 2026-07-28 (`git show +builder/spir-1252:codev/resources/scar-rules.yaml`) ship with the rewrite, **verbatim**: +`git-add-explicit`, `never-destroy-worktrees`, `no-destructive-git`, `human-gates`, +`no-hand-edit-status`, `afx-from-root`, `shellper-verified-orphan`, +`tower-restart-permission`. + +- The registry file is **rebuilt fit-for-purpose after the shrink**, not carried across + it: each rule's `must_appear_on` list is re-derived against the post-rewrite surface, + because most of the files in the 1252 lists will have been rewritten or deleted. +- Carriage is **exempt from every word target**: ~240 words of scar text per surface + that carries them is a floor, not a cut candidate. Targets above are net of this. +- Enforcement is a byte-identical-presence test over the registry — the minimum that + makes a reworded copy fail the build. Nothing larger is built until the surface it + polices has stopped moving. +- A scar rule may be **compressed only by architect ratification**, never by a builder + applying principle 1. + +### What "done" looks like operationally + +A builder spawned after this lands receives a spawn prompt it can read in full, a phase +task that fits on a screen, and no instruction it would not have followed anyway. The +rollback is one `git revert` away (see **Rollback Plan**). + +## Stakeholders + +- **Primary Users**: Codev builder agents (Claude 5 / GPT 5.6 / Gemini 3.6 class) and + the CMAP reviewer agents that consume the consult-type prompts. +- **Secondary Users**: architects (human + AI) who must be able to read and maintain + the surface; downstream adopters who receive it via `codev update`. +- **Technical Team**: this builder; the architect at both gates. +- **Business Owners**: Waleed (charter holder; ratifies scar-rule wordings, approves + the A/B verdict). + +## Success Criteria + +- [ ] **M0 — the metric measures what is served.** `scripts/measure-prompt-surface.sh` + derives the phase-task term from the prompts the live resolver loads + (`protocols//prompts/`), includes the inlined role file, and expands + `{{> …}}` includes. A test asserts the script's phase-prompt source directory is + the one `loadPromptFile` resolves, so this defect cannot silently return. +- [ ] **M0b — the corrected instrument and baseline land on `main` early**, in a small + standalone PR (precedent: #1290, the 1252 frozen-sample fix), not at the end of + this project's branch. `1252-word-baseline.md` and `1252-word-after-phase7.md` + cite figures derived from the dead tree; they are shared knowledge that other work + reads, so the record is corrected while this project builds rather than after. The + correction annotates the 1252 artifacts in place — original figures preserved, + marked superseded, with the reason — it does not rewrite their history. +- [ ] **M1 — >50% reduction.** `ALWAYS_ON_WORDS` measured by the corrected script falls + from the corrected pre-rewrite baseline (33,519 ± re-measurement) to **≤15,900**. + Before and after are measured with the *same* corrected script and both figures + are committed as generated artifacts. +- [ ] **M2 — per-surface targets met.** Every row of the cut-plan table meets its + target, in **both** `codev/` and `codev-skeleton/`. +- [ ] **M3 — sweep completeness.** Every protocol shipped in `codev-skeleton/protocols/` + is rewritten to the same standard; no protocol retains a pre-rewrite + `protocol.md`, `builder-prompt.md`, prompt set, or consult-type set. A check + enumerates protocols from disk rather than a hardcoded list. +- [ ] **M4 — scar rules intact.** All eight canonical strings present byte-identically + on every surface in the rebuilt registry; a test fails on reword or deletion and + pins the count at 8. +- [ ] **M5 — no capability lost.** Every artifact contract, gate, signal, check name, + and notification trigger present before the rewrite is present after it. Verified + by an explicit inventory diff, not by reading. +- [ ] **M6 — the dead tree is gone.** `codev-skeleton/porch/prompts/` deleted, with a + grep proving no consumer. +- [ ] **M7 — A/B non-inferiority passes.** The pre-registered decision rule in **A/B + Validation Design** returns SHIP. +- [ ] **M8 — behavioural baseline re-run.** `measure-prompt-behavior.ts` re-run and + committed; B1 compared directionally against 51.88% (n=160) with the sample + documented. +- [ ] **M9 — rollback rehearsed.** The revert path is executed once on a scratch branch + and shown to restore the pre-rewrite surface byte-for-byte. +- [ ] All existing tests pass; no reduction in coverage. New tests cover M0, M3, M4, M5. +- [ ] Documentation updated: `arch.md`/`arch-critical.md` and + `lessons-learned.md`/`lessons-critical.md` routed by tier; `CLAUDE.md`/`AGENTS.md` + byte-identical after the rewrite. + +## Constraints + +### Technical Constraints + +Copied verbatim from issue #1280's **Baked Decisions**; each is fixed and not +re-litigated by this spec, the plan, or CMAP reviewers: + +- **All prompt consumers are frontier models** (Claude 5, GPT 5.6, Gemini 3.6 class). No + weak-model tier, no fallback scaffolding variant, no tiering mechanism. One form. +- **Scar rules are exempt and verbatim** — the eight compressed canonicals developed in + Spec 1252 Phase 5 (six repo rules + shellper verified-orphan + Tower-restart + permission) ship with the rewrite; the registry/enforcement concept from 1252 is + rebuilt fit-for-purpose around the post-shrink surface, not before it. +- **Validation is A/B, not observational**: same issues executed by builders on old vs + new prompts, compared on outcomes (gate friction, review rounds, correctness). Spec + 1252's M12 established that observational baselines (n=17) can only detect large + regressions — insufficient at deletion scale. The A/B design is a first-class spec + section. +- Spec must define a rollback story (prompt surfaces are files; reverting is cheap — + say so concretely). + +Further technical constraints arising from the repository: + +- **Both trees.** `codev/` (our instance) and `codev-skeleton/` (what adopters get) + must be changed together; `CLAUDE.md` and `AGENTS.md` must stay byte-identical. +- **Four-tier resolution.** Framework files resolve at runtime; a rewrite must not + introduce a fetch-by-path instruction for a file that may not exist on disk + (deliver-don't-fetch). +- **No behaviour changes in porch.** This project rewrites content and fixes a + measurement script. Changing the state machine, gates, or check semantics is out of + scope. +- **The measured proxy is SPIR**, but the rewrite is fleet-wide (M3). + +### Business Constraints + +- Two human gates (`spec-approval`, `plan-approval`) plus the `pr` gate; the A/B verdict + is the architect's call, not the builder's. +- Adopters consume the skeleton via `codev update` — a regression ships to them, so the + rollback path must be a single revertible unit per surface. +- Scar-rule wordings are architect-ratified; a builder may not compress them. +- **The corrected instrument ships early, as its own PR** (M0b) — architect-directed on + 2026-08-01. This is an explicitly architect-requested PR under the issue's PR strategy; + the remaining phase-commits still ship as a single later PR. + +## Assumptions + +- The eight scar-rule wordings on `builder/spir-1252` remain the ratified set; if the + architect amends them, the registry is rebuilt from the amended set. +- `I = 10` phase-task deliveries remains the agreed proxy for a SPIR project's + always-on load; the metric is a *comparison* instrument, so the exact multiplier + matters less than using the same one before and after. +- Frontier-model behaviour is stable enough over the A/B window that arm differences + are attributable to the prompt surface (mitigated by pairing and by running both arms + from the same base commit). +- `builder/spir-1252` stays undeleted for the life of this project (it is the only + source of the ratified registry). +- Reviewer models are blind to the builder's prompt surface by construction — a CMAP + reviewer sees artifacts and diffs, not the prompt that produced them. + +## Solution Approaches + +### Approach 1: In-place judgment rewrite, surface by surface (RECOMMENDED) + +**Description**: Rewrite each existing file to the six principles, keeping the file +layout, the resolver, and porch untouched. Templates become heading interfaces. How-to +content relocates to existing skills. The dead `porch/prompts/` tree is deleted. The +measurement script is corrected first so every subsequent cut is scored honestly. + +**Pros**: +- Zero mechanism risk: no new code path between authoring and serving. +- Every change is a text diff — trivially reviewable, trivially revertible, per surface. +- Rollback granularity equals cut granularity (one revert per surface). +- Compatible with the deliver-don't-fetch convention already in force. + +**Cons**: +- Discipline-dependent: nothing structurally prevents re-growth (mitigated by a + budget check, below). +- Large diff across ~10 protocols × 2 trees; sweep completeness is the main risk (M3). + +**Estimated Complexity**: Medium +**Risk Level**: Low + +### Approach 2: Generate prompts from `protocol.json` + +**Description**: Treat `protocol.json` as the single source of truth and synthesize +phase prompts (phase name, artifact path, checks, gate, signal contract) at runtime, +with a small per-phase prose delta. + +**Pros**: +- Structurally prevents re-growth and drift; the state machine and the prompt can never + disagree. +- Would have made the 1252 drift bug impossible. + +**Cons**: +- Introduces a code path between authoring and serving — new failure mode, harder to + review, harder to revert, and it changes porch behaviour (an explicit constraint + above). +- Couples the shrink to a mechanism change, so an A/B regression becomes + un-attributable: was it the deletion or the generator? + +**Estimated Complexity**: High +**Risk Level**: Medium-High + +**Verdict**: right idea, wrong project. Land the shrink first; a generator over a +15,000-word surface is a plausible successor. + +### Approach 3: Shared kernel + per-protocol deltas + +**Description**: One protocol-agnostic builder kernel (gates, artifacts, thread, +notifications, scar rules) included by every protocol, plus a short per-protocol delta. + +**Pros**: +- Attacks duplication across the ten protocols, which the per-surface table does not. +- Uses the existing `{{> …}}` include mechanism — served-word-honest by construction. + +**Cons**: +- 1252 already proved dedup buys ~7% on *served* words: an include expands, so moving + text between files changes ownership, not the bill. The savings here are maintenance, + not context. +- Risks re-creating the shadow-tree class of bug (one edit silently changing ten + protocols' served prompts). + +**Verdict**: adopt selectively *inside* Approach 1 where a kernel genuinely reduces +served words for the reader (not merely authored words), and only after the shrink. + +## Open Questions + +### Critical (Blocks Progress) + +- [x] **~~Does the architect accept the corrected measurement baseline?~~** **RESOLVED + 2026-08-01** — architect verified both claims against source (script line 89; + `prompts.ts:78`) and **endorsed M0 as specced**: fix the script first, measure + before *and* after on the corrected instrument, >50% target unchanged against the + corrected 33,519-word baseline. Added: land the corrected instrument and baseline + on `main` early (M0b), and record the instrument principle explicitly (done — + **Problem Statement**, principle 7). +- [ ] **How is "gate friction" captured?** 1252 established that gate-rejection counts + are **not minable** from committed history (no `rejected` state; `requested_at` is + overwritten). Either (a) the architect scores each gate prospectively on a + three-item rubric during the A/B, or (b) a porch gate-event append-log is added. + (b) is a porch behaviour change and therefore out of scope here — the spec + assumes **(a)** unless the architect directs otherwise. + +### Important (Affects Design) + +- [ ] **A/B sample size.** The design below specifies ≥6 issue-pairs. More pairs buy + power but cost real builder runs and consult spend (~$1,478/30d at current rates). + The architect sets the ceiling. +- [ ] **Do the SPIR templates survive as interfaces, or disappear entirely?** Issue + #1279's audit is in scope. Recommendation: survive as ≤150-word heading + interfaces, since porch checks assert on headings (`spec_has_required_sections`). +- [ ] **Is the hot tier's 736 words genuinely exempt?** It is the one surface already + built to these principles (capped, judgment-shaped, displacement-enforced). The + spec exempts it; a reviewer may argue it should be re-derived post-shrink. + +### Nice-to-Know (Optimization) + +- [ ] Should a **word-budget check** run in CI (fail the build if any surface exceeds + its target by >10%)? Cheap anti-re-growth insurance; adds a maintenance surface. +- [ ] Does trimming the consult-type prompts move CMAP verdict *quality* measurably, or + only cost? B1 will show the rate; quality needs human adjudication. +- [ ] Are the ~17,000 words of `.claude/skills/` the right destination for relocated + how-tos, or does that surface need its own budget? + +## Performance Requirements + +Not a runtime-performance feature; the requirements are on the artifact and the harness. + +- **Served always-on words**: ≤15,900 per SPIR builder (from 33,519) — M1. +- **Per-surface ceilings**: as tabulated in **Desired State** — M2. +- **Measurement runtime**: `measure-prompt-surface.sh` completes in <5s and is + deterministic — same commit ⇒ byte-identical output (existing determinism property, + preserved). +- **Token/cost effect** (advisory, not a gate): a ~17,600-word always-on reduction is + ~23,000 tokens per builder-project; recorded before/after from `consult stats` and + session telemetry as context for interpreting the A/B, keying no threshold. + +## Security Considerations + +- **The scar rules are the security surface.** Every one of the eight guards an + irreversible act (destroying uncommitted work, destroying worktrees, killing live + sessions, bypassing a human gate). Deleting or weakening one is the highest-severity + failure this project can produce — hence verbatim carriage (M4), byte-identical + enforcement, and a hard rollback trigger on any observed violation in the A/B. +- **Human-gate integrity.** The rewrite must not weaken "a gate message is a + notification to the human, not authorization." Gate semantics are content, not code, + and this project edits content. +- **No secrets in prompt surfaces.** Existing property; re-verified after the rewrite + (the surfaces contain no credentials today and must not acquire any). +- **Adopter blast radius.** Skeleton changes ship to every adopter on `codev update`; + a weakened prohibition would propagate silently. This is why rollback is per-surface + and rehearsed (M9). + +## Test Scenarios + +### Functional Tests + +1. **T1 — Measurement correctness (M0).** The script's phase-prompt source directory + equals the directory `loadPromptFile` resolves for a known protocol; asserted against + the real resolver, not a hardcoded string. Regression-proofs the dead-tree defect. +2. **T2 — Include expansion (M0/M1).** A prompt with a `{{> …}}` include counts the + include's words; a fixture that moves text from prompt into template shows **zero** + change in `ALWAYS_ON_WORDS` (phantom-savings proof, preserved from 1252). +3. **T3 — Word ceilings (M1/M2).** Each surface's served word count is at or under its + target, per protocol, per tree. Failure names the surface and the overage. +4. **T4 — Scar-rule integrity (M4).** Every canonical string appears byte-identically on + every registered surface; the test pins the rule count at 8 and the eight ids; + rewording or deleting any copy fails. +5. **T5 — Capability inventory (M5).** The set of {artifact paths, gate names, signal + names, porch check names, notification triggers} extracted from the post-rewrite + surface equals the pre-rewrite set. Additions allowed; **removals fail**. +6. **T6 — Sweep completeness (M3).** Protocols are enumerated from + `codev-skeleton/protocols/` on disk; each must satisfy T3. A newly added protocol + fails the test until it is written to budget. +7. **T7 — Twin parity.** `CLAUDE.md` and `AGENTS.md` byte-identical; `codev/` and + `codev-skeleton/` copies of every rewritten framework file consistent. +8. **T8 — Dead-tree removal (M6).** `codev-skeleton/porch/prompts/` absent; no source + file references it. +9. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface + receives a spawn prompt containing every element of the artifact contract, and its + first `porch next` returns a well-formed task. ("It compiled" is not "it works" — + the real spawn path is exercised, not a unit fixture.) +10. **T10 — Rollback rehearsal (M9).** Reverting the rewrite commits on a scratch branch + restores the pre-rewrite surface byte-for-byte and `measure-prompt-surface.sh` + reproduces the pre-rewrite figure. + +### Non-Functional Tests + +1. **T11 — Determinism.** Two runs of each measurement script at the same commit emit + byte-identical output. +2. **T12 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run + post-merge with self-exclusion; B1/B2/B4 committed and compared directionally to the + 1252 baseline. +3. **T13 — A/B execution (M7).** The full pre-registered protocol below. + +## A/B Validation Design + +*(A first-class section per Baked Decision 3. This is a **non-inferiority** trial: the +claim under test is "deleting 53% of the always-on surface does not degrade outcomes", +not "it improves them".)* + +### Unit and arms + +The unit of observation is an **issue-pair**: one GitHub issue executed twice, by two +freshly-spawned builders in separate worktrees, from the same base commit. + +- **Control arm (A)**: worktree whose `codev/` + `codev-skeleton/` are at the + pre-rewrite commit. +- **Treatment arm (B)**: worktree at the post-rewrite commit. + +No code differs between arms — the prompt surface is file-resolved, so the arms are two +checkouts. This is the whole reason the design is cheap. + +### Sample + +- **≥6 pairs (12 builder runs)**, stratified: ≥3 SPIR/ASPIR (exercises spec, plan, + implement, review prompts, both gates, and the templates) and ≥3 lighter protocols + (BUGFIX/AIR — exercises the short prompts and the single consult). +- Issues drawn from the existing backlog, selected **before** either arm runs, and + frozen (no issue-body edits mid-trial). +- Arm order alternates per pair to control for time-varying factors. + +### Pre-registered outcomes + +| ID | Outcome | Instrument | Direction | +|---|---|---|---| +| **O1** | Gate friction | Architect scores each gate on a 3-item rubric at approval time: *artifact complete as specified? / required rework before approval? / did the builder need a clarifying message?* (prospective — history is not minable) | non-inferior | +| **O2** | Review rounds | Iterations to terminal state per phase, from `status.yaml` history; plus CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | non-inferior within margin | +| **O3** | Correctness | Architect's PR review findings by severity + any post-merge defect attributable to the run | non-inferior | +| **O4** | Protocol compliance | Binary per-run checklist: required artifacts present with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar-rule violation · thread committed | **zero tolerance** | +| **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory only | + +### Blinding + +CMAP reviewer models are blind by construction. The architect is not blind and cannot +be; the mitigation is that O2 and O4 are extracted mechanically from committed +artifacts, and O1/O3 are scored against a rubric written **before** any run. + +### Decision rule (pre-registered) + +**SHIP** iff all of: + +1. **O4 = zero violations in the treatment arm.** Any scar-rule violation, skipped gate, + or missing required artifact is an immediate hard stop, independent of every other + outcome. +2. **O2** treatment mean review rounds ≤ control mean **+ 0.5 rounds/phase**, and + treatment REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. +3. **O3** no treatment-arm correctness finding of severity ≥ "would block merge" that + is absent from its paired control run. +4. **O1** no pair where the treatment arm required rework at a gate that its control did + not, for the same reason. + +Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK** (below). + +### Honest power statement + +With n=6 pairs this design detects only **large** effects — roughly a doubling of review +rounds or a ≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle +regression, and this spec does not claim it can. It is nonetheless strictly stronger +than 1252's observational baseline, because each pair is matched on the issue itself — +the dominant source of variance. O4's zero-tolerance criterion is where the real +protection lives: compliance is binary, observable in every run, and is the failure mode +that deletion would plausibly cause. + +## Rollback Plan + +*(Required by Baked Decision 4. Prompt surfaces are files; reverting is cheap — here is +exactly how cheap.)* + +- **Unit.** One revertible commit per surface (CLAUDE.md/AGENTS.md · role · protocol.md · + builder-prompt · prompts+templates · consult-types · registry · measurement script), + in both trees. Rolling back one surface never requires rolling back another. +- **Mechanism.** `git revert ` restores the prior bytes. No migration, no state, + no schema, no data. Verified by T10 on a scratch branch before the PR merges. +- **Blast radius and propagation.** For this repo: effective for the next spawned + builder — in-flight builders keep the surface they were spawned with (prompts are read + at spawn/phase time, so a running builder is unaffected either way). For adopters: the + revert ships in the next release; an adopter can also pin the prior `@cluesmith/codev` + version, since framework files resolve from the installed skeleton (tier 4). +- **Triggers.** (a) any O4 violation in the A/B — immediate, no deliberation; (b) an + observed scar-rule violation in any real project post-merge; (c) O2/O3 outside the + pre-registered margins; (d) architect's judgment at the `pr` or `verify-approval` gate. +- **Partial rollback is the expected shape.** If one surface regresses (say the review + prompt lost a contract), revert that surface and keep the rest — the whole point of + per-surface commits. +- **Cost.** One revert, one release. There is no irreversible step anywhere in this + project. + +## Dependencies + +- **External Services**: none. (`gh` for issue/PR reads during the A/B; consult backends + — Gemini via `agy`, Codex, Claude — for CMAP, unchanged.) +- **Internal Systems**: the four-tier resolver (`lib/skeleton.ts`); porch prompt + composition (`commands/porch/prompts.ts`) — read, not modified; the consult CLI's + consult-type resolution; `scripts/measure-prompt-surface.sh` and + `packages/codev/scripts/measure-prompt-behavior.ts`. +- **Artifacts**: `builder/spir-1252` (ratified scar registry — must not be deleted); + `codev/resources/1252-*.md` (baselines). +- **Libraries/Frameworks**: none new. + +## References + +- Issue #1280 (this charter); Issue #1279 (dead spec/review templates); Issues #1276 + (multi-model tiering) and #1277 (controlled A/B eval), both filed by 1252 and both + superseded here or explicitly out of scope. +- PR #1278 (Spec 1252, closed unmerged) and branch `builder/spir-1252` — surface + inventory, ownership analysis, scar registry, enforcement machinery. +- `codev/reviews/1252-prompt-architecture-single-own.md`; + `codev/resources/1252-word-baseline.md`, `1252-word-after-phase7.md`, + `1252-behavior-baseline.md`, `1252-shadow-tree-audit.md`; + `codev/state/spir-1252_thread.md`. +- *The new rules of context engineering for Claude-5-generation models* — + https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models +- `codev/resources/arch.md` (four-tier resolution, repository dual nature); + `codev/resources/lessons-learned.md` (sweep-scope failures, served-surface dedup). + +## Risks and Mitigation + +| Risk | Probability | Impact | Mitigation Strategy | +|------|------------|--------|-------------------| +| A deleted instruction was load-bearing and its loss is silent | Medium | High | T5 capability-inventory diff (removals fail the build); O4 zero-tolerance in the A/B; per-surface rollback | +| A scar rule is weakened or dropped | Low | **Critical** | Verbatim carriage; byte-identical enforcement pinned at 8 rules; hard rollback trigger; architect-only rewording | +| Sweep misses a protocol or the second tree | **High** | Medium | T6 enumerates protocols from disk; T7 twin parity. 1252's top lesson was that sweep-scope failures dominated its review iterations | +| A/B underpowered; a subtle regression ships | Medium | Medium | Stated honestly in the power statement; O4 binary compliance carries the protection; T12 behavioural re-measurement post-merge as a second net | +| ~~Corrected baseline disputed at the gate~~ — **retired**, endorsed 2026-08-01 | — | — | Both figures are re-derived by one script, so any future re-scoping of the denominator recomputes mechanically | +| A *third* instrument defect ships undetected | Medium | High | Principle 7: every check and baseline is reviewed against its claim. T1/T2 assert the instrument against the live resolver; M0b puts the corrected instrument under public review early rather than at PR time | +| Surface re-grows after the project ends | **High** | Medium | Word-ceiling test (T3) run in CI is the cheap structural answer; open question on whether to gate on it | +| The A/B costs more than the shrink saves | Medium | Low | 12 runs on backlog issues that needed doing anyway; O5 tracks it; architect sets the pair ceiling | +| `builder/spir-1252` is deleted, losing the ratified registry | Low | High | Registry content is quoted in this spec's thread and will be committed to `main` as the rebuilt registry early | + +## Expert Consultation + +**Date**: pending +**Models Consulted**: Gemini (via `agy`), Codex (GPT-5.6 Sol), Claude Opus 5 — run by +porch at the specify-phase verify step. +**Sections Updated**: *(to be filled after the 3-way review; feedback is incorporated +directly into the sections above and summarized here)* + +## Approval +- [ ] Technical Lead Review +- [ ] Product Owner Review +- [ ] Stakeholder Sign-off +- [ ] Expert AI Consultation Complete + +## Notes + +**On the irony.** A specification arguing for deletion should not be padded. This one is +long because it carries three architect-mandated designs (cut plan, A/B, scar carriage) +plus a measurement correction that changes the project's headline number. The artifacts +it produces are the short ones. + +**On what this project deliberately does not do.** It does not build a prompt generator +(Approach 2), does not add tiering of any kind (Baked Decision 1), does not change porch +behaviour, and does not rebuild 1252's full enforcement machinery — only the minimum +scar-rule integrity check that the deletion itself makes necessary. Enforcement rebuilt +around a surface that is still moving is enforcement built twice. + +**On the deferred decision from 1252.** The architect's pr-gate ruling was that +structural machinery is not worth carrying for a surface about to shrink by half. That +sequencing is honoured here: shrink first, then enforce what remains. The word-ceiling +check (T3) is the successor's smallest useful enforcement primitive, and it only becomes +meaningful once the ceilings exist. diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md new file mode 100644 index 000000000..3012ae113 --- /dev/null +++ b/codev/state/spir-1280_thread.md @@ -0,0 +1,72 @@ +# spir-1280 — Prompt surface: judgment-not-rules rewrite (>50% always-on reduction) + +## Specify phase — opening survey (2026-07-31) + +Read the required prior art before drafting: `codev/reviews/1252-prompt-architecture-single-own.md`, +`1252-word-baseline.md`, `1252-word-after-phase7.md`, `1252-behavior-baseline.md`, +`scripts/measure-prompt-surface.sh`, issue #1279, and the ratified scar registry from +`builder/spir-1252:codev/resources/scar-rules.yaml` (all eight rules recovered verbatim). + +### Finding that reshapes the spec: the committed measurement script measures a dead directory + +`scripts/measure-prompt-surface.sh` computes `PORCH_PROMPT_MEAN` over +`codev-skeleton/porch/prompts/*.md` (10 files, mean 400w). The live code +(`packages/codev/src/commands/porch/prompts.ts:78`, `loadPromptFile`) resolves +`protocols//prompts/.md` — a *different* directory. Repo-wide grep +(excluding node_modules/dist/.git) finds no code reading `porch/prompts`; the only hits +are historical spec/plan prose. That tree is a Ralph-SPIR-era leftover: its `specify.md` +opens "You are the **Spec Writer** hat in a Ralph-SPIR loop." + +Consequences: +- The real SPIR phase prompts (expanded with their `{{> templates/...}}` includes) are + specify 1402, plan 1169, implement 1065, review 1957 → **mean 1398**, not 400. +- The proxy also omits `roles/builder.md` (1837w), which the spawn wrapper inlines + verbatim (verified against this worktree's own `.builder-role.md`). +- **The metric is blind to this project's single biggest target.** Cutting the phase + prompts would not move `ALWAYS_ON_WORDS` at all under the current script. + +Corrected always-on model (SPIR, I=10 task deliveries): 5,815 + 6,364 + 21,340 = **33,519** +vs the script's reported 21,702. Same methodology (served/expanded words), corrected inputs. +Spec makes fixing this M0 — before/after both measured with the corrected script, so the +>50% target is unaffected in kind, only in denominator. + +Notified the architect; not blocking on it (the fix serves the stated intent of the goal +rather than contradicting a Baked Decision). + +### Per-surface sizes captured for the cut plan + +| Surface | Words | Notes | +|---|---:|---| +| CLAUDE.md / AGENTS.md | 5,815 each | byte-identical twins; hot tier inlined | +| roles/builder.md | 1,837 | inlined into every spawn | +| spir/protocol.md | 3,703 | inlined into every spawn | +| spir/builder-prompt.md | 824 | spawn wrapper | +| spir/prompts (expanded) | 1,398 mean | ×I per project — the dominant term | +| spir/templates | 632/649/641 | pulled in by specify/plan/review prompts | +| hot tier | 736 | capped, judgment-shaped, keep | +| spir/consult-types | 2,154 (5 files) | reviewer-side always-on, unmeasured today | + +Cut plan and A/B design go in the spec. + +### Architect ruling (2026-08-01) — M0 endorsed, two additions + +Architect independently verified both claims against source (script line 89; the live +resolver at `prompts.ts:78`) and endorsed M0 as specced: fix the instrument first, +measure before AND after on it, >50% target unchanged against the corrected 33,519-word +baseline. Two additions folded into the spec: + +1. **M0b** — the corrected script + corrected baseline land on `main` in a small early + standalone PR (precedent #1290), not at the end of the branch, because the 1252 + baseline artifacts cite the wrong figure and are shared knowledge other work reads. + Recorded as an architect-requested PR under the issue's PR strategy. +2. **Principle 7, "the instrument is part of the deliverable"** — written into the spec's + Problem Statement with the lineage: this is the SECOND measurement defect in the 1252 + line (the first: 1252 originally shipped with no measurement plan at all, caught at a + human gate, not by CMAP). Neither was caught by reading the instrument's code; both by + asking what it claims to measure. + +Spec drafted (5,800w) — carries the three architect-mandated designs (per-surface cut +plan with word targets, A/B non-inferiority design with a pre-registered decision rule, +scar-rule carriage plan) plus the rollback story. Headline: 33,519 → ≤15,900 (−52.7%), +with the phase-task term (71% of the post-rewrite budget) dominated by the hot tier, +which is explicitly exempt from cuts. Signalling SPEC_DRAFTED to porch for 3-way review. From 544866e2a4acb8ed6e9bb70b4ac2986c14ca1c51 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 19:55:40 -0700 Subject: [PATCH 03/32] chore(porch): 1280 specify build-complete --- .../projects/1280-prompt-surface-judgment-not-ru/status.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 77c5fec24..8a05358be 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -14,7 +14,7 @@ gates: verify-approval: status: pending iteration: 1 -build_complete: false +build_complete: true history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T02:46:36.458Z' +updated_at: '2026-08-01T02:55:40.540Z' From e8759bc1b53a7e369abfc056d8c379b8e744cb3c Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:05:51 -0700 Subject: [PATCH 04/32] [Spec 1280] Specification with multi-agent review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CMAP round 1 (Codex + Claude, both REQUEST_CHANGES, both HIGH) fully incorporated; no rebuttal filed — every finding verified against source and accepted. Plus the architect's full-surface scope directive. Key corrections: - All per-surface ceilings restated GROSS (scar carriage counted inside them). Iteration 1 stated ceilings net-of-scar while M1 was gross; the gross landing would have been 50.2-51.4%, failing M1 while every row passed. - Baseline 33,519 -> 34,255: CLAUDE.md @imports the hot tier since #1119, so wc -w excludes 736 always-loaded words. Inherited from the measurement script's own stale comment. - Retracted iteration 1's 'no code reads porch/prompts' claim: drawn from a truncated grep. review-prompt-routing.test.ts:29 reads it (Spec 987 protection). M6's verification method rewritten. - New M10: ~25 test files pin exact prose in the surfaces being cut, incl. a pure-addition diff on builder-prompt.md incompatible with 824 -> 420. Each assertion is a prior spec's protection; retiring one is a governance act. - M0 extended: per-file four-tier resolution, role file, hot-tier transclusion, and per-audience segment reporting. - A/B: contamination controls, eligibility exclusion for issues touching surfaces under test, O1 demoted to advisory-with-tripwire. - Rollback restated as seven groups with a scar-registry dependency rule. Scope directive: full prompt surface inventoried by audience (architect / builder / phase / consultant) with resolver paths and served word counts. Segment targets: builder -53.2%, architect -61.2%, consultant -33.8%. --- .../1280-prompt-surface-judgment-not-ru.md | 1113 +++++++++-------- codev/state/spir-1280_thread.md | 51 + 2 files changed, 663 insertions(+), 501 deletions(-) diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md index d05f5003b..1d9304146 100644 --- a/codev/specs/1280-prompt-surface-judgment-not-ru.md +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -3,14 +3,14 @@ ## Metadata - **ID**: spec-2026-07-31-prompt-surface-judgment-not-rules -- **Status**: draft +- **Status**: draft (iteration 2 — CMAP round 1 incorporated) - **Created**: 2026-07-31 - **Issue**: #1280 - **Protocol**: SPIR @@ -18,656 +18,768 @@ phases, belongs in codev/plans/1280-prompt-surface-judgment-not-ru.md. ## Clarifying Questions Asked Issue #1280 carries a complete charter (goal, attack order, four Baked Decisions, -required prior art). Per the specify-phase rule, no clarifying questions were put to -the architect. Three questions were resolved against the repository instead: +required prior art), so no clarifying questions were put to the architect before +drafting. Four questions were resolved against the repository or by architect ruling: + +1. **"What exactly is the always-on surface, as served?"** — Resolved by reading the live + composition path (`commands/porch/prompts.ts`, `lib/skeleton.ts`, `lib/managed-block.ts`, + `agent-farm/commands/spawn-worktree.ts`) and by measuring this builder's own served + artifacts (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). Full enumeration + in **Current State → Inventory**. +2. **"Is the committed measurement script fit to score this project?"** — No; three + defects, one of them disqualifying. Criterion **M0**. Architect verified both original + claims against source and **endorsed M0 as specced** (2026-07-31). +3. **"How wide is the rewrite target?"** — **Architect scope directive (2026-07-31)**: the + entire prompt surface, not CLAUDE.md/AGENTS.md — architect role prompts, builder roles + and spawn wrappers, consultant/CMAP review prompts, porch phase prompts and their + template includes, and `protocol.md` texts. The instrument must **segment by audience** + so a cut concentrated in one segment while another grows is visible, not averaged away. +4. **"Where are the eight ratified scar rules?"** — Recovered verbatim from + `builder/spir-1252:codev/resources/scar-rules.yaml`. Carriage plan in **Desired State**. -1. **"What exactly is the always-on surface, as served?"** — Resolved by reading the - live composition path (`packages/codev/src/commands/porch/prompts.ts`, - `packages/codev/src/lib/skeleton.ts`) and by measuring this builder's own served - spawn artifacts (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). See - **Current State**. +## Problem Statement -2. **"Is the committed measurement script fit to score this project?"** — No. It - measures a directory no code reads, and is blind to the project's largest target. - Finding and consequences in **Current State**; correction is criterion **M0**. - Raised with the architect via `afx send` on 2026-07-31 (non-blocking). +A Codev builder consumes **~34,300 served always-on words** before it reads a line of the +code it was spawned to change. Almost none of that is information the builder could not +derive; it is *process narration* — recipes for how to be an agent, written when the fleet +could not be trusted to infer them. -3. **"Where are the eight ratified scar rules?"** — Recovered verbatim from - `builder/spir-1252:codev/resources/scar-rules.yaml` (the preserved reference - branch). Carriage plan in **Desired State**. +Spec 1252 measured the surface and proved the obvious remedy does not work: deduplication +yields **−7.0%**, because the surface is not duplicated, it is **over-instructed**. -## Problem Statement +Over-instruction has three compounding costs: -A Codev builder consumes **~33,500 served always-on words** before it reads a single -line of the code it was spawned to change (corrected measurement, below). Almost none -of that is information the builder cannot derive; it is *process narration* — recipes -for how to be an agent, written when the fleet could not be trusted to infer them. - -Spec 1252 measured the surface and proved the obvious remedy does not work: -deduplication yields **−7.0%** (21,856 → 20,324 on its own proxy), because the surface -is not duplicated, it is **over-instructed**. Its review states the conclusion plainly — -the largest remaining block is single-owned prose. - -Over-instruction is not merely a token bill. It has three compounding costs: - -1. **It crowds out judgment.** A frontier model given a 3,700-word procedure follows - the procedure. Given a 700-word contract and a goal, it reasons about the goal. - Anthropic's published account of the Claude-5-generation rewrite reports >80% of - Claude Code's system prompt deleted with no measurable performance loss, by - replacing rules with judgment, deleting worst-case padding, designing interfaces - instead of examples, and using progressive disclosure. -2. **Nobody reads it, so it rots unnoticed.** 1252 found the *served* SPIR builder - prompt had silently lost its entire `Verify Phase` section, and a detector had been - reporting the drift, unread, for months. A surface too large to read is a surface - too large to maintain. -3. **It makes its own success unmeasurable.** The committed measurement script scores - a directory the runtime never loads (below). Nothing about the prompts a builder - actually receives has been under measurement. - -The blog operation — deletion on judgment-trust grounds — was an explicit **Non-goal** -of Spec 1252 and has never been attempted in this repo. +1. **It crowds out judgment.** A frontier model given a 3,700-word procedure follows the + procedure. Given a 700-word contract and a goal, it reasons about the goal. Anthropic's + published account of the Claude-5-generation rewrite reports >80% of Claude Code's + system prompt deleted with no measurable performance loss. +2. **Nobody reads it, so it rots unnoticed.** 1252 found the *served* SPIR builder prompt + had silently lost its entire `Verify Phase` section, with a detector reporting the drift, + unread, for months. A surface too large to read is too large to maintain. +3. **It makes its own success unmeasurable.** The committed measurement script scores a + directory the runtime never loads. Nothing about the prompts a builder actually receives + has been under measurement. + +Deletion on judgment-trust grounds was an explicit **Non-goal** of Spec 1252 and has never +been attempted here. ### The instrument is part of the deliverable This is the **second** measurement defect in the 1252 lineage. The first: Spec 1252 -originally shipped with no measurement plan at all — caught at a human gate, not by -CMAP. The second is the one this spec documents below: a committed, tested, reproducible -script that measures the wrong directory, so its numbers are precise and wrong. Neither -defect was caught by reading the instrument's code; both were caught by asking *what -does this claim to measure, and does it?* +originally shipped with no measurement plan at all — caught at a human gate, not by CMAP. +The second is documented below: a committed, tested, reproducible script that measures the +wrong directory, so its numbers are precise and wrong. Neither was caught by reading the +instrument's code; both by asking *what does this claim to measure, and does it?* Hence an explicit project principle, adopted at the architect's direction and binding on -every criterion in this spec: +every criterion here: > **The instrument is part of the deliverable, and instruments get reviewed against what -> they claim to measure — not merely against whether they run.** A measurement script, -> a check, or a baseline artifact is subject to the same adversarial review as the -> feature it scores. "Deterministic and committed" is not "correct." +> they claim to measure — not merely against whether they run.** A measurement script, a +> check, or a baseline artifact is subject to the same adversarial review as the feature it +> scores. "Deterministic and committed" is not "correct." + +It is load-bearing: this project's headline criterion is a number one shell script emits. +Discovering *before* drafting that the metric was structurally blind to the project's own +largest target is the correct order of operations, and is why M0 precedes M1. -It is load-bearing here in a way it was not for 1252: this project's headline success -criterion is a number that one shell script emits. Discovering — *before* writing the -spec — that the metric was structurally blind to the project's own largest target is the -correct order of operations, and it is the reason M0 precedes M1. +**The principle applied to this spec, iteration 1 → 2.** CMAP round 1 found two further +instrument-class errors in my own Current State — a stale claim inherited from the script's +comments, and a conclusion drawn from a truncated grep. Both are corrected below and both +are recorded, not quietly fixed: a spec that argues instruments get audited must show its +own being audited. ## Current State -### How the always-on surface is composed +### Inventory — every prompt-bearing surface, by audience -| Stage | Content | Served words | -|---|---|---:| -| Session | `CLAUDE.md` (hot tier inlined) — `AGENTS.md` is the byte-identical twin, one loads per session | 5,815 | -| Spawn (once) | `protocols/spir/builder-prompt.md` (824) + inlined `protocols/spir/protocol.md` (3,703) + inlined `roles/builder.md` (1,837) | 6,364 | -| Every phase task (×I) | hot tier (`arch-critical.md` 416 + `lessons-critical.md` 320 = 736) + the resolved phase prompt **with its `{{> templates/…}}` includes expanded** (mean 1,398) | 2,134 each | +Word counts are **served and expanded** (`{{> …}}` includes resolved through the +`codev/` → `codev-skeleton/` chain), captured 2026-07-31 at `047f92f7`. -With `I = 10` phase-task deliveries (the proxy 1252 fixed, consistent with B4's mean of -3.06 review rounds/project across ~4–6 phases): +**SHARED — every agent working in this repo** -``` - CLAUDE.md 5,815 -+ spawn 6,364 -+ phase task ×10 21,340 - ---------------------- - ALWAYS_ON 33,519 -``` +| Surface | Resolver path | Words | How served | +|---|---|---:|---| +| `CLAUDE.md` | repo root | 5,815 | session, harness auto-load | +| ↳ `@codev/resources/arch-critical.md` | four-tier | 416 | **transcluded at session launch** (#1119) | +| ↳ `@codev/resources/lessons-critical.md` | four-tier | 320 | same | +| `AGENTS.md` | repo root | 5,815 | byte-identical twin; one loads per session, never both | -Per-prompt detail (SPIR, expanded): `specify` 1,402 · `plan` 1,169 · `implement` 1,065 -· `review` 1,957. Templates pulled in by those prompts: `spec.md` 632 · `plan.md` 649 · -`review.md` 641. Reviewer-side (not in the builder's total, and unmeasured today): -`consult-types/*.md`, 5 files, 2,154w. +**Session shared total: 6,551.** -The same disease is present in every other protocol shipped from the skeleton -(`aspir` prompts 3,671 · `pir` prompts 4,306 · `pir/protocol.md` 2,066 · -`maintain/protocol.md` 1,765 · per-protocol `consult-types` 813–2,154). +**ARCHITECT** -### The measurement defect (why M0 exists) +| Surface | Resolver path | Words | How served | +|---|---|---:|---| +| `roles/architect.md` | four-tier | 2,048 | read at `arch-init` (every architect session) | +| `.claude/skills/*/SKILL.md` ×10 | repo | 6,672 | on-demand (progressive disclosure — working as intended) | + +**BUILDER — spawn, once per builder** + +| Surface | Words | | Surface | Words | +|---|---:|---|---|---:| +| `roles/builder.md` (inlined) | 1,837 | | `protocols/pir/protocol.md` | 2,066 | +| `protocols/spir/protocol.md` | 3,703 | | `protocols/maintain/protocol.md` | 1,949 | +| `protocols/spir/builder-prompt.md` | 824 | | `protocols/research/protocol.md` | 1,278 | +| `protocols/pir/builder-prompt.md` | 898 | | `protocols/experiment/protocol.md` | 1,023 | +| `protocols/aspir/builder-prompt.md` | 820 | | `protocols/spike/protocol.md` | 920 | +| `protocols/research/builder-prompt.md` | 556 | | `protocols/aspir/protocol.md` | 810 | +| `protocols/air/builder-prompt.md` | 537 | | `protocols/bugfix/protocol.md` | 699 | +| `protocols/experiment/builder-prompt.md` | 472 | | `protocols/air/protocol.md` | 643 | +| `protocols/bugfix/builder-prompt.md` | 429 | | | | +| `protocols/spike/builder-prompt.md` | 400 | | `protocols/maintain/builder-prompt.md` | 374 | + +**SPIR builder spawn total: 6,364** (role 1,837 + wrapper 824 + protocol 3,703). + +**PHASE — per porch task delivery, ×I** + +Hot tier (736) rides on *every* phase prompt. Expanded phase prompts: + +| Protocol | Prompts (expanded) | Mean | +|---|---|---:| +| spir / aspir | specify 1,402 · plan 1,169 · implement 1,065 · review 1,957 | **1,398** | +| pir | review 2,414 · implement 1,151 · plan 741 | 1,435 | +| bugfix | pr 491 · fix 352 · investigate 290 | 378 | +| air | pr 471 · implement 442 | 457 | +| maintain | maintain 402 · review 310 | 356 | -`scripts/measure-prompt-surface.sh` derives `PORCH_PROMPT_MEAN` from -`codev-skeleton/porch/prompts/*.md` — 10 files, mean 400 words. The live resolver -(`porch/prompts.ts`, `loadPromptFile`) loads `protocols//prompts/.md`, -a different tree. A repo-wide grep (excluding `node_modules`, `dist`, `.git`) finds no -code reading `porch/prompts`; every hit is historical spec/plan prose. The tree is a -Ralph-SPIR-era leftover — its `specify.md` opens *"You are the **Spec Writer** hat in a -Ralph-SPIR loop."* It is dead authored surface. +**CONSULTANT — per CMAP review, ×3 models × ~10 reviews per project** -Two consequences, both disqualifying for this project: +`roles/consultant.md` 252 (system prompt) + one consult-type: spir/aspir spec-review 514 · +impl 421 · phase 421 · plan 406 · pr 392; bugfix pr 726 / impl 641; pir pr 475 / impl 507; +air pr 455 / impl 420; maintain 421 / 392. **SPIR consultant per review: 683.** -- The reported baseline (21,702 on today's `main`) understates the phase-task term by - ~3.5× and omits `roles/builder.md` entirely. -- **The metric cannot see this project's primary target.** Cutting the SPIR phase - prompts from 1,398 to 450 words moves `ALWAYS_ON_WORDS` by exactly zero under the - current script. A >50% claim scored on it would be phantom savings — the precise - failure mode 1252 built the script to prevent. +**DEAD** + +`codev-skeleton/porch/prompts/**` — 10 files, 4,009 words, no runtime consumer. + +### Always-on load, by audience + +| Audience | Composition | Words | +|---|---|---:| +| **Builder** (SPIR, I=10) | 6,551 session + 6,364 spawn + 10×(736 + 1,398) | **34,255** | +| **Architect** (per session) | 6,551 session + 2,048 role | **8,599** | +| **Consultant** (per review) | 252 role + 431 mean consult-type | **683** | + +The builder figure is the headline. `I = 10` phase-task deliveries is 1252's proxy, +consistent with B4's 3.06 review rounds/project across 4–6 phases; it is a *comparison* +constant, identical before and after. + +### The measurement defects (why M0 exists) + +**Defect 1 — the script measures a dead directory.** `measure-prompt-surface.sh:89` derives +`PORCH_PROMPT_MEAN` from `codev-skeleton/porch/prompts/*.md` (10 files, mean 400). The live +resolver (`commands/porch/prompts.ts:78`, `loadPromptFile`) loads +`protocols//prompts/.md`. Real SPIR phase prompts average **1,398**. The +dead tree is a Ralph-SPIR-era leftover — its `specify.md` opens *"You are the **Spec +Writer** hat in a Ralph-SPIR loop."* + +**Defect 2 — the script omits the inlined role.** `spawn-worktree.ts:854` writes +`roles/builder.md` to `.builder-role.md` and the harness injects it. 1,837 always-on words, +uncounted. + +**Defect 3 — the script's hot-tier accounting is stale.** Its comment (lines 44–47) asserts +CLAUDE.md "already inlines the two hot-tier files." Since #1119 (`managed-block.ts:59-67`) +CLAUDE.md carries `@import` lines, which Claude Code transcludes at session launch. So +`wc -w CLAUDE.md` = 5,815 **excludes** 736 words that are always loaded. *(Found by CMAP +round 1 — an instrument-class error I inherited from the instrument's own comments, which +is precisely the failure principle 7 names.)* + +Consequences: the reported baseline (21,702) understates the phase-task term ~3.5×, omits +the role file, and under-counts the session term by 736. **And the metric cannot see this +project's primary target** — cutting SPIR phase prompts from 1,398 to 430 moves +`ALWAYS_ON_WORDS` by exactly zero under the current script. A >50% claim scored on it would +be phantom savings, the precise failure 1252 built the script to prevent. + +**Correction to iteration 1 of this spec.** It claimed "a repo-wide grep finds no code +reading `porch/prompts`; every hit is historical spec/plan prose." That is **false**, and +the cause is mine: I piped `grep -rn` into `head -20` and drew a conclusion from truncated +output. `packages/codev/src/__tests__/review-prompt-routing.test.ts:29` pushes +`codev-skeleton/porch/prompts/review.md` onto its assertion list (a Spec 987 hot/cold +routing protection). The tree is still dead as *prompt surface* — no runtime consumer — but +it has a **test** consumer, so M6's verification method and deletion step change accordingly. +This is the sweep-scope failure class 1252 named as its dominant review-iteration cost, and +it is exactly what a truncated grep buys. ### What already landed, and what is deferred -On `main` (the 1252 harvest): the drift reconciliation, the audit, the two word -baselines, the behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded), -and the measurement tooling. Deferred to this project by the architect's pr-gate ruling: -the **scar registry and its eight ratified rule wordings**, and enforcement rebuilt -*after* the shrink rather than before it. +On `main` (the 1252 harvest): drift reconciliation, the audit, two word baselines, the +behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded), and the +measurement tooling. Deferred here by architect ruling: the **scar registry and its eight +ratified wordings**, and enforcement rebuilt *after* the shrink. -Issue #1279 (dead spec/review templates) is partially overtaken by events — the SPIR -prompts now inline their templates via `{{> …}}` includes. That wiring is exactly what -makes each phase prompt ~600 words heavier than it reads, so the template question is -in scope here: an annotated 632-word template is the "examples instead of interfaces" -anti-pattern the rewrite exists to remove. +Issue #1279 is partly overtaken — SPIR prompts now inline their templates via `{{> …}}`, +which is what makes each phase prompt ~600 words heavier than it reads. Two *separate* +constraints govern template shape, and conflating them would over-preserve surface: +`checks.ts:149-154` (`REQUIRED_SPEC_SECTIONS`) requires only **four** headings — Problem +Statement, Current State, Desired State, Success Criteria — while the 20-heading template +pressure comes from the `spec-review` consult type, advisorily. ## Desired State -**One prompt form, written for frontier models, that states contracts and trusts -judgment.** A builder's always-on context tells it: what it owns, what artifacts it -must produce and what shape they take, where the human gates are, what is -irreversible — and then gets out of the way. Everything else is reachable on demand. +**One prompt form, written for frontier models, that states contracts and trusts judgment.** +A builder's always-on context tells it what it owns, what artifacts it must produce and +what shape they take, where the human gates are, and what is irreversible — then gets out of +the way. Everything else is reachable on demand. -### Rewrite principles (the standard every cut is judged against) +### Rewrite principles -1. **Contract, not recipe.** State the required outcome and its shape. Delete the - ordered procedure for reaching it. +1. **Contract, not recipe.** State the required outcome and its shape; delete the ordered + procedure for reaching it. 2. **Interface, not example.** A heading skeleton with one line of intent per heading replaces an annotated template with filler prose. -3. **No worst-case padding.** Delete instructions that exist for a failure mode a - frontier model does not exhibit (repeated all-caps prohibitions, "⚠️ BLOCKING" - banners, checklists restating the phase body). -4. **Progressive disclosure.** How-to content that a competent agent would look up - moves to skills / on-demand files, addressed by name, not inlined. -5. **Budgets are cheap words worth keeping.** A stated scope or budget line buys - bounded process; frontier models honour stated budgets precisely but never invent - them. Budget/scope lines are exempt from cuts. -6. **Scar rules are verbatim and exempt.** See carriage plan below. -7. **The instrument is part of the deliverable.** Every check, script, and baseline this - project produces is reviewed against what it claims to measure, not merely against - whether it runs. (Stated in full in **Problem Statement**; repeated here because it - binds the cuts too — a word target that is not measured on served words is not a - target.) - -### Per-surface cut plan (word targets) - -Targets are on **served, expanded** words, measured by the corrected script (M0). -Every target applies to the SPIR instance as the measured proxy **and is swept across -all protocols in both trees** (`codev/` and `codev-skeleton/`) — an unswept protocol is -a regression, not a deferral. - -| Surface | Now | Target | What survives | -|---|---:|---:|---| -| `CLAUDE.md` / `AGENTS.md` (twins) | 5,815 | **≤2,200** | repo dual nature, four-tier resolution, gates, area-label policy, the eight scar rules verbatim, the hot-tier block. Worktree recipes, CLI walkthroughs, protocol-selection prose → skills/on-demand | -| `roles/builder.md` | 1,837 | **≤600** | ownership, gates, thread contract, notification triggers, worktree path discipline | -| `protocols/*/protocol.md` (SPIR) | 3,703 | **≤800** | state machine, phase→gate map, artifact contracts, commit/branch format, consultation checkpoints | -| `protocols/*/builder-prompt.md` (SPIR) | 824 | **≤400** | mode, spec/plan/issue wiring, baked-decisions rule, PR strategy | -| `protocols/*/prompts/*.md` expanded (SPIR mean) | 1,398 | **≤450** | goal, artifact path, heading interface, signal contract | -| hot tier (`arch-critical` + `lessons-critical`) | 736 | **736 (unchanged)** | already capped, already judgment-shaped — explicitly out of scope for cuts | -| `protocols/*/consult-types/*.md` (SPIR mean) | 431 | **≤200** | rubric dimensions + verdict contract | -| `codev-skeleton/porch/prompts/**` | 4,014 authored | **0 (deleted)** | dead tree, no consumer | - -Resulting always-on total: **≤15,900 words, a ≥52% reduction from 33,519.** +3. **No worst-case padding.** Delete instructions guarding failure modes frontier models + do not exhibit (repeated all-caps prohibitions, "⚠️ BLOCKING" banners, checklists + restating the phase body). +4. **Progressive disclosure.** How-to content a competent agent would look up moves to + skills / on-demand files, addressed by name, not inlined. +5. **Budgets are cheap words worth keeping.** Frontier models honour stated budgets + precisely but never invent them; budget/scope lines are exempt from cuts. +6. **Scar rules are verbatim and exempt from rewriting** — but **counted** in ceilings + (see carriage plan). +7. **The instrument is part of the deliverable.** Every check, script, and baseline is + reviewed against what it claims to measure. A word ceiling not measured on served words + is not a ceiling. + +### Per-surface ceilings (GROSS — scar words count inside every ceiling) + +*CMAP round 1 (both reviewers, independently) found iteration 1 stated M1 on a gross basis +while declaring ceilings "net of" scar carriage. Incompatible. Resolved: **all ceilings are +gross**, and the arithmetic below carries carriage explicitly.* + +Ceilings apply to the SPIR instance as the measured proxy **and sweep across all ten +protocols in both trees** — an unswept protocol is a regression, not a deferral. + +| Segment | Surface | Now | Ceiling (gross) | Scar carriage inside | +|---|---|---:|---:|---:| +| shared | `CLAUDE.md` / `AGENTS.md` | 5,815 | **≤1,900** | ~190 (all 8) | +| shared | hot tier (`@import`) | 736 | **736 unchanged** | — | +| architect | `roles/architect.md` | 2,048 | **≤700** | ~30 | +| builder | `roles/builder.md` | 1,837 | **≤600** | ~12 | +| builder | `protocols/*/protocol.md` (SPIR) | 3,703 | **≤700** | — | +| builder | `protocols/*/builder-prompt.md` (SPIR) | 824 | **≤420** | ~40 | +| phase | `protocols/*/prompts/*.md` expanded (SPIR mean) | 1,398 | **≤430** | ~14 | +| consultant | `protocols/*/consult-types/*.md` (SPIR mean) | 431 | **≤200** | — | +| consultant | `roles/consultant.md` | 252 | **≤252 unchanged** | — | +| dead | `codev-skeleton/porch/prompts/**` | 4,009 | **0 (deleted)** | — | + +### Post-rewrite always-on, by audience (the segmented view the directive requires) ``` - CLAUDE.md 2,200 -+ spawn (400 + 800 + 600) 1,800 -+ phase task ×10 (736 + 450) 11,860 - --------------------------- - ALWAYS_ON 15,860 (−52.7%) +BUILDER (SPIR, I=10) + session (1,900 + 736) 2,636 + + spawn (600 + 700 + 420) 1,720 + + phase ×10 (736 + 430) 11,660 + ----------------------------------------------- + 34,255 → 16,016 −53.2% + +ARCHITECT 8,599 → 3,336 (2,636 session + 700 role) −61.2% +CONSULTANT 683 → 452 (252 role + 200 consult-type) −33.8% ``` -Note the shape of the arithmetic: the phase-task term is 71% of the post-rewrite budget -and the hot tier — which is *not* being cut — is 62% of that term. This is deliberate. -The always-on surface that survives is overwhelmingly curated judgment, not process. +No segment grows. The consultant segment cuts least because `roles/consultant.md` is +already lean at 252 words — reported rather than averaged into the headline, per the +directive. + +**Margin, stated honestly.** −53.2% clears >50% by 3.2 points ≈ 1,100 words. **M2 +(per-surface ceilings) is the binding criterion; M1 is derived.** If all ceilings are met +and M1 lands in 50–52%, that is a **HOLD**, not a pass — the plan must find the remainder, +with named candidates in priority order: further `protocol.md` compression, relocating +CLAUDE.md's Runnable-Worktree recipes wholesale to a skill, and the `pir`/`spir` review +prompts (2,414 / 1,957 — the two fattest phase prompts in the fleet). + +Note the shape: the phase term is 73% of the post-rewrite builder budget and the **exempt** +hot tier is 63% of that term. What survives is overwhelmingly curated judgment, not process. ### Scar-rule carriage plan -The eight rules ratified by the architect on 2026-07-28 (`git show -builder/spir-1252:codev/resources/scar-rules.yaml`) ship with the rewrite, **verbatim**: -`git-add-explicit`, `never-destroy-worktrees`, `no-destructive-git`, `human-gates`, -`no-hand-edit-status`, `afx-from-root`, `shellper-verified-orphan`, -`tower-restart-permission`. - -- The registry file is **rebuilt fit-for-purpose after the shrink**, not carried across - it: each rule's `must_appear_on` list is re-derived against the post-rewrite surface, - because most of the files in the 1252 lists will have been rewritten or deleted. -- Carriage is **exempt from every word target**: ~240 words of scar text per surface - that carries them is a floor, not a cut candidate. Targets above are net of this. -- Enforcement is a byte-identical-presence test over the registry — the minimum that - makes a reworded copy fail the build. Nothing larger is built until the surface it - polices has stopped moving. +The eight rules ratified 2026-07-28 (`git show +builder/spir-1252:codev/resources/scar-rules.yaml`) ship **verbatim**: `git-add-explicit`, +`never-destroy-worktrees`, `no-destructive-git`, `human-gates`, `no-hand-edit-status`, +`afx-from-root`, `shellper-verified-orphan`, `tower-restart-permission` (~188 words of +canonical text total). + +- The registry is **rebuilt fit-for-purpose after the shrink**: each rule's `must_appear_on` + is re-derived against the post-rewrite surface, since most 1252-listed files will have + been rewritten or deleted. +- Carriage is **exempt from rewriting but counted in ceilings** — a ceiling a surface cannot + meet while carrying its scar rules is a wrong ceiling, and gets raised deliberately, not + met by trimming scar text. +- Enforcement is a byte-identical-presence test over the registry, pinned at 8 rules and + their ids. Nothing larger is built until the surface it polices stops moving. - A scar rule may be **compressed only by architect ratification**, never by a builder applying principle 1. -### What "done" looks like operationally - -A builder spawned after this lands receives a spawn prompt it can read in full, a phase -task that fits on a screen, and no instruction it would not have followed anyway. The -rollback is one `git revert` away (see **Rollback Plan**). - ## Stakeholders -- **Primary Users**: Codev builder agents (Claude 5 / GPT 5.6 / Gemini 3.6 class) and - the CMAP reviewer agents that consume the consult-type prompts. -- **Secondary Users**: architects (human + AI) who must be able to read and maintain - the surface; downstream adopters who receive it via `codev update`. +- **Primary Users**: builder agents; CMAP reviewer agents (consult-type prompts); architect + agents (role + skills). +- **Secondary Users**: humans who must read and maintain the surface; downstream adopters + receiving it via `codev update`. - **Technical Team**: this builder; the architect at both gates. -- **Business Owners**: Waleed (charter holder; ratifies scar-rule wordings, approves - the A/B verdict). +- **Business Owners**: Waleed — charter holder; ratifies scar wordings, rules on the A/B + verdict and on any ceiling change. ## Success Criteria -- [ ] **M0 — the metric measures what is served.** `scripts/measure-prompt-surface.sh` - derives the phase-task term from the prompts the live resolver loads - (`protocols//prompts/`), includes the inlined role file, and expands - `{{> …}}` includes. A test asserts the script's phase-prompt source directory is - the one `loadPromptFile` resolves, so this defect cannot silently return. -- [ ] **M0b — the corrected instrument and baseline land on `main` early**, in a small - standalone PR (precedent: #1290, the 1252 frozen-sample fix), not at the end of - this project's branch. `1252-word-baseline.md` and `1252-word-after-phase7.md` - cite figures derived from the dead tree; they are shared knowledge that other work - reads, so the record is corrected while this project builds rather than after. The - correction annotates the 1252 artifacts in place — original figures preserved, - marked superseded, with the reason — it does not rewrite their history. -- [ ] **M1 — >50% reduction.** `ALWAYS_ON_WORDS` measured by the corrected script falls - from the corrected pre-rewrite baseline (33,519 ± re-measurement) to **≤15,900**. - Before and after are measured with the *same* corrected script and both figures - are committed as generated artifacts. -- [ ] **M2 — per-surface targets met.** Every row of the cut-plan table meets its - target, in **both** `codev/` and `codev-skeleton/`. -- [ ] **M3 — sweep completeness.** Every protocol shipped in `codev-skeleton/protocols/` - is rewritten to the same standard; no protocol retains a pre-rewrite - `protocol.md`, `builder-prompt.md`, prompt set, or consult-type set. A check - enumerates protocols from disk rather than a hardcoded list. -- [ ] **M4 — scar rules intact.** All eight canonical strings present byte-identically - on every surface in the rebuilt registry; a test fails on reword or deletion and - pins the count at 8. -- [ ] **M5 — no capability lost.** Every artifact contract, gate, signal, check name, - and notification trigger present before the rewrite is present after it. Verified - by an explicit inventory diff, not by reading. -- [ ] **M6 — the dead tree is gone.** `codev-skeleton/porch/prompts/` deleted, with a - grep proving no consumer. -- [ ] **M7 — A/B non-inferiority passes.** The pre-registered decision rule in **A/B - Validation Design** returns SHIP. +- [ ] **M0 — the metric measures what is served, segmented by audience.** The corrected + script: (a) sources phase prompts from the directory `loadPromptFile` resolves; + (b) resolves **per-file through the full four-tier chain** (`.codev/` → `codev/` → + cache → skeleton) exactly as `resolveCodevFile` does, not two-tier directory-level + selection, so mixed per-file overrides measure correctly; (c) counts the inlined + `roles/builder.md`; (d) counts hot-tier `@import` transclusion in the session term and + **corrects the stale inlining comment**; (e) expands `{{> …}}` includes; (f) reports + **per-segment subtotals — architect / builder / phase / consultant — alongside the + total**. Tests assert (a) and (b) against the real resolver, so these defects cannot + silently return. +- [ ] **M0b — the corrected instrument and baseline land on `main` early**, as a small + standalone PR (precedent: #1290), not at the end of this branch. + `1252-word-baseline.md` and `1252-word-after-phase7.md` cite figures derived from the + dead tree and are shared knowledge other work reads; the correction annotates them in + place — original figures preserved, marked superseded, with the reason — and does not + rewrite their history. +- [ ] **M1 — >50% reduction (derived).** Builder always-on falls from the corrected baseline + (34,255) to **≤16,100**, measured before and after by the same corrected script, both + figures committed as generated artifacts. Per-segment figures reported and none + regressed. +- [ ] **M2 — per-surface ceilings met (binding).** Every row of the ceiling table, gross, in + **both** trees, for **all ten protocols**. +- [ ] **M3 — sweep completeness.** Protocols enumerated from disk, not a hardcoded list; no + protocol retains a pre-rewrite `protocol.md`, `builder-prompt.md`, prompt set, or + consult-type set. +- [ ] **M4 — scar rules intact.** Eight canonicals byte-identical on every registered + surface; test fails on reword or deletion; count pinned at 8. +- [ ] **M5 — no capability lost, deterministically checked.** A committed + `capability-inventory.json` is extracted pre-rewrite by a script with **explicit + recognition rules**: artifact paths (`codev/(specs|plans|reviews)/…` literals and + `{{artifact_name}}` forms), gate names (from `protocol.json` `gate:` fields), signal + names (`` tags), porch check names (`protocol.json` `checks:` ids), + notification triggers (`afx send architect` call sites). Normalization: lowercase, + strip backticks/punctuation, dedupe. Post-rewrite extraction must be a superset; + **any removal fails** and must be justified in the review as a deliberate retirement. +- [ ] **M6 — the dead tree is gone, with its consumer handled.** + `codev-skeleton/porch/prompts/` deleted. Verification is **not** a bare grep: an + untruncated repo-wide search (`grep -rn … | wc -l` reconciled against the full hit + list) shows zero *runtime* consumers, and the one **test** consumer — + `review-prompt-routing.test.ts:29`, a Spec 987 hot/cold-routing protection — is + updated under M10's re-baselining rule, naming Spec 987 as the originating spec. +- [ ] **M7 — A/B non-inferiority passes** per the pre-registered decision rule. - [ ] **M8 — behavioural baseline re-run.** `measure-prompt-behavior.ts` re-run and - committed; B1 compared directionally against 51.88% (n=160) with the sample - documented. -- [ ] **M9 — rollback rehearsed.** The revert path is executed once on a scratch branch - and shown to restore the pre-rewrite surface byte-for-byte. -- [ ] All existing tests pass; no reduction in coverage. New tests cover M0, M3, M4, M5. -- [ ] Documentation updated: `arch.md`/`arch-critical.md` and - `lessons-learned.md`/`lessons-critical.md` routed by tier; `CLAUDE.md`/`AGENTS.md` - byte-identical after the rewrite. + committed; B1 compared directionally to 51.88% (n=160) with the sample documented. +- [ ] **M9 — rollback rehearsed** per **Rollback Plan**, by group. +- [ ] **M10 — prose-pinned test re-baselining is deliberate and enumerated.** ~25 test files + assert exact prose in the surfaces being cut; the hardest is + `agent-farm/__tests__/baked-decisions.test.ts:143-148`, which enforces a + **pure-addition diff** against committed baselines for + `protocols/{spir,aspir,air}/builder-prompt.md` — logically incompatible with cutting + 824 → ≤420. Also: `bugfix-744-spir-pr-strategy.test.ts` (4 near-verbatim sentences), + `spec-1273-wait-discipline-docs.test.ts` (16 assertions), `bugfix-619-aspir-prompt.test.ts`, + `template-delivery.test.ts`, `framework-ref-audit.test.ts`, `governance-sweep.test.ts`, + `review-prompt-routing.test.ts`. **Each assertion is a prior spec's protection encoded + as a grep.** Therefore: every modified or retired assertion is listed in the review + with (i) the spec that created it, (ii) whether the protected behaviour survives in + the rewritten prose, (iii) the replacement assertion if the behaviour survives, or an + explicit architect-visible retirement if it does not. Re-baselining a pure-addition + baseline is permitted **only** with the originating spec named and the new baseline + committed in the same commit. Silent deletion of an assertion to make the suite green + is a project failure, not a test fix. +- [ ] All tests pass **after M10's enumerated re-baselining**; no coverage reduction. New + tests cover M0, M3, M4, M5. +- [ ] Documentation routed by tier (`arch.md`/`arch-critical.md`, + `lessons-learned.md`/`lessons-critical.md`); `CLAUDE.md`/`AGENTS.md` byte-identical. ## Constraints ### Technical Constraints -Copied verbatim from issue #1280's **Baked Decisions**; each is fixed and not -re-litigated by this spec, the plan, or CMAP reviewers: +Verbatim from issue #1280's **Baked Decisions** — fixed, not re-litigated by this spec, the +plan, or CMAP reviewers: - **All prompt consumers are frontier models** (Claude 5, GPT 5.6, Gemini 3.6 class). No weak-model tier, no fallback scaffolding variant, no tiering mechanism. One form. - **Scar rules are exempt and verbatim** — the eight compressed canonicals developed in - Spec 1252 Phase 5 (six repo rules + shellper verified-orphan + Tower-restart - permission) ship with the rewrite; the registry/enforcement concept from 1252 is - rebuilt fit-for-purpose around the post-shrink surface, not before it. -- **Validation is A/B, not observational**: same issues executed by builders on old vs - new prompts, compared on outcomes (gate friction, review rounds, correctness). Spec - 1252's M12 established that observational baselines (n=17) can only detect large - regressions — insufficient at deletion scale. The A/B design is a first-class spec - section. -- Spec must define a rollback story (prompt surfaces are files; reverting is cheap — - say so concretely). - -Further technical constraints arising from the repository: - -- **Both trees.** `codev/` (our instance) and `codev-skeleton/` (what adopters get) - must be changed together; `CLAUDE.md` and `AGENTS.md` must stay byte-identical. -- **Four-tier resolution.** Framework files resolve at runtime; a rewrite must not - introduce a fetch-by-path instruction for a file that may not exist on disk - (deliver-don't-fetch). -- **No behaviour changes in porch.** This project rewrites content and fixes a - measurement script. Changing the state machine, gates, or check semantics is out of - scope. -- **The measured proxy is SPIR**, but the rewrite is fleet-wide (M3). + Spec 1252 Phase 5 (six repo rules + shellper verified-orphan + Tower-restart permission) + ship with the rewrite; the registry/enforcement concept from 1252 is rebuilt + fit-for-purpose around the post-shrink surface, not before it. +- **Validation is A/B, not observational**: same issues executed by builders on old vs new + prompts, compared on outcomes (gate friction, review rounds, correctness). Spec 1252's + M12 established that observational baselines (n=17) can only detect large regressions — + insufficient at deletion scale. The A/B design is a first-class spec section. +- Spec must define a rollback story (prompt surfaces are files; reverting is cheap — say so + concretely). + +Arising from the repository and the architect's scope directive: + +- **Scope is the full prompt surface, segmented**: architect roles, builder roles and spawn + wrappers, consultant/CMAP prompts, phase prompts and template includes, `protocol.md` + texts. Measurement reports per-segment, never averaged away. +- **Both trees**; `CLAUDE.md` and `AGENTS.md` byte-identical. +- **Four-tier resolution, per file.** No fetch-by-path instruction for a file that may not + exist on disk (deliver-don't-fetch). +- **No porch behaviour changes.** Content rewrite plus a measurement-script fix. Changing + the state machine, gates, or check semantics is out of scope. +- **Template shape is governed by two separate constraints** (porch's 4 required headings; + the consult type's advisory 20) — they must not be conflated. ### Business Constraints -- Two human gates (`spec-approval`, `plan-approval`) plus the `pr` gate; the A/B verdict - is the architect's call, not the builder's. -- Adopters consume the skeleton via `codev update` — a regression ships to them, so the - rollback path must be a single revertible unit per surface. +- Two human gates (`spec-approval`, `plan-approval`) plus `pr`; the A/B verdict is the + architect's call. +- Adopters consume the skeleton via `codev update`, so rollback must be a revertible unit + per **group**. - Scar-rule wordings are architect-ratified; a builder may not compress them. -- **The corrected instrument ships early, as its own PR** (M0b) — architect-directed on - 2026-08-01. This is an explicitly architect-requested PR under the issue's PR strategy; - the remaining phase-commits still ship as a single later PR. +- **The corrected instrument ships early, as its own PR** (M0b) — architect-directed. The + remaining phase-commits ship as a single later PR. ## Assumptions -- The eight scar-rule wordings on `builder/spir-1252` remain the ratified set; if the - architect amends them, the registry is rebuilt from the amended set. -- `I = 10` phase-task deliveries remains the agreed proxy for a SPIR project's - always-on load; the metric is a *comparison* instrument, so the exact multiplier - matters less than using the same one before and after. -- Frontier-model behaviour is stable enough over the A/B window that arm differences - are attributable to the prompt surface (mitigated by pairing and by running both arms - from the same base commit). -- `builder/spir-1252` stays undeleted for the life of this project (it is the only - source of the ratified registry). -- Reviewer models are blind to the builder's prompt surface by construction — a CMAP - reviewer sees artifacts and diffs, not the prompt that produced them. +- The eight scar wordings on `builder/spir-1252` remain the ratified set. +- `I = 10` remains the agreed proxy; it is a comparison constant, identical both sides. +- Frontier-model behaviour is stable across the A/B window (mitigated by pairing, same base + commit, and pinned model/config versions). +- `builder/spir-1252` stays undeleted — sole source of the ratified registry. +- CMAP reviewers are blind to the builder's prompt surface by construction: a reviewer sees + artifacts and diffs, not the prompt that produced them. ## Solution Approaches ### Approach 1: In-place judgment rewrite, surface by surface (RECOMMENDED) -**Description**: Rewrite each existing file to the six principles, keeping the file -layout, the resolver, and porch untouched. Templates become heading interfaces. How-to -content relocates to existing skills. The dead `porch/prompts/` tree is deleted. The -measurement script is corrected first so every subsequent cut is scored honestly. +**Description**: Rewrite each file to the seven principles, keeping file layout, resolver +and porch untouched. Templates become heading interfaces. How-to content relocates to +existing skills. The dead tree is deleted. The instrument is corrected first, so every +subsequent cut is scored honestly. -**Pros**: -- Zero mechanism risk: no new code path between authoring and serving. -- Every change is a text diff — trivially reviewable, trivially revertible, per surface. -- Rollback granularity equals cut granularity (one revert per surface). -- Compatible with the deliver-don't-fetch convention already in force. +**Pros**: zero mechanism risk; every change is a reviewable text diff; rollback granularity +equals cut granularity; compatible with deliver-don't-fetch. -**Cons**: -- Discipline-dependent: nothing structurally prevents re-growth (mitigated by a - budget check, below). -- Large diff across ~10 protocols × 2 trees; sweep completeness is the main risk (M3). +**Cons**: discipline-dependent (nothing structurally prevents re-growth — mitigated by T3); +large diff across 10 protocols × 2 trees, so sweep completeness is the main risk; collides +with ~25 prose-pinned test files (M10). -**Estimated Complexity**: Medium -**Risk Level**: Low +**Estimated Complexity**: Medium · **Risk Level**: Low-Medium ### Approach 2: Generate prompts from `protocol.json` -**Description**: Treat `protocol.json` as the single source of truth and synthesize -phase prompts (phase name, artifact path, checks, gate, signal contract) at runtime, -with a small per-phase prose delta. +**Description**: Treat `protocol.json` as source of truth; synthesize phase prompts at +runtime with a small per-phase prose delta. -**Pros**: -- Structurally prevents re-growth and drift; the state machine and the prompt can never - disagree. -- Would have made the 1252 drift bug impossible. +**Pros**: structurally prevents re-growth and drift; would have made the 1252 drift bug +impossible. -**Cons**: -- Introduces a code path between authoring and serving — new failure mode, harder to - review, harder to revert, and it changes porch behaviour (an explicit constraint - above). -- Couples the shrink to a mechanism change, so an A/B regression becomes - un-attributable: was it the deletion or the generator? +**Cons**: introduces a code path between authoring and serving — new failure mode, harder to +review and revert, and it changes porch behaviour (an explicit constraint). Couples the +shrink to a mechanism change, so an A/B regression becomes un-attributable: deletion or +generator? -**Estimated Complexity**: High -**Risk Level**: Medium-High +**Estimated Complexity**: High · **Risk Level**: Medium-High -**Verdict**: right idea, wrong project. Land the shrink first; a generator over a -15,000-word surface is a plausible successor. +**Verdict**: right idea, wrong project. A generator over a 16,000-word surface is a +plausible successor. ### Approach 3: Shared kernel + per-protocol deltas **Description**: One protocol-agnostic builder kernel (gates, artifacts, thread, notifications, scar rules) included by every protocol, plus a short per-protocol delta. -**Pros**: -- Attacks duplication across the ten protocols, which the per-surface table does not. -- Uses the existing `{{> …}}` include mechanism — served-word-honest by construction. +**Pros**: attacks cross-protocol duplication the per-surface table does not; uses the +existing include mechanism, so it is served-word-honest. -**Cons**: -- 1252 already proved dedup buys ~7% on *served* words: an include expands, so moving - text between files changes ownership, not the bill. The savings here are maintenance, - not context. -- Risks re-creating the shadow-tree class of bug (one edit silently changing ten - protocols' served prompts). +**Cons**: 1252 proved dedup buys ~7% on *served* words — an include expands, so moving text +changes ownership, not the bill. Savings are maintenance, not context. Risks re-creating the +shadow-tree bug class (one edit silently changing ten protocols' served prompts). -**Verdict**: adopt selectively *inside* Approach 1 where a kernel genuinely reduces -served words for the reader (not merely authored words), and only after the shrink. +**Verdict**: adopt selectively *inside* Approach 1 where a kernel reduces served words for +the reader, and only after the shrink. ## Open Questions ### Critical (Blocks Progress) -- [x] **~~Does the architect accept the corrected measurement baseline?~~** **RESOLVED - 2026-08-01** — architect verified both claims against source (script line 89; - `prompts.ts:78`) and **endorsed M0 as specced**: fix the script first, measure - before *and* after on the corrected instrument, >50% target unchanged against the - corrected 33,519-word baseline. Added: land the corrected instrument and baseline - on `main` early (M0b), and record the instrument principle explicitly (done — - **Problem Statement**, principle 7). -- [ ] **How is "gate friction" captured?** 1252 established that gate-rejection counts - are **not minable** from committed history (no `rejected` state; `requested_at` is - overwritten). Either (a) the architect scores each gate prospectively on a - three-item rubric during the A/B, or (b) a porch gate-event append-log is added. - (b) is a porch behaviour change and therefore out of scope here — the spec - assumes **(a)** unless the architect directs otherwise. +*None outstanding.* Both former Critical questions are resolved: + +- [x] **~~Corrected measurement baseline accepted?~~** **RESOLVED 2026-07-31** — architect + verified both claims against source (script line 89; `prompts.ts:78`) and endorsed M0 + as specced; >50% unchanged against the corrected baseline. Added M0b and principle 7. +- [x] **~~How is "gate friction" captured?~~** **RESOLVED** — 1252 established gate + rejections are not minable (no `rejected` state; `requested_at` overwritten). A porch + gate-event log is a behaviour change and out of scope. **Decision: O1 is scored + prospectively by the architect on the rubric in the A/B section, and is + *advisory-with-a-tripwire*, not a SHIP gate** — if scoring is incomplete for any pair, + O1 is reported as incomplete and SHIP rests on O2/O3/O4. This removes the single point + of failure CMAP flagged while keeping the signal. ### Important (Affects Design) -- [ ] **A/B sample size.** The design below specifies ≥6 issue-pairs. More pairs buy - power but cost real builder runs and consult spend (~$1,478/30d at current rates). - The architect sets the ceiling. -- [ ] **Do the SPIR templates survive as interfaces, or disappear entirely?** Issue - #1279's audit is in scope. Recommendation: survive as ≤150-word heading - interfaces, since porch checks assert on headings (`spec_has_required_sections`). -- [ ] **Is the hot tier's 736 words genuinely exempt?** It is the one surface already - built to these principles (capped, judgment-shaped, displacement-enforced). The - spec exempts it; a reviewer may argue it should be re-derived post-shrink. +- [ ] **A/B sample size.** ≥6 pairs specified; more buys power at real builder and consult + cost (~$1,478/30d at current rates). The architect sets the ceiling. +- [ ] **Do SPIR templates survive as interfaces, or disappear?** Recommendation: survive as + ≤150-word heading interfaces — porch requires only 4 headings, but the interface is + what makes the artifact contract legible without narration. +- [ ] **Is the hot tier's 736 genuinely exempt?** It is the one surface already built to + these principles. The spec exempts it; a reviewer may argue it should be re-derived + post-shrink, and it is 63% of the phase term. +- [ ] **`roles/architect.md` 2,048 → ≤700** — the architect segment was outside 1252's + analysis entirely. Confirm nothing in it is load-bearing for multi-architect + coordination (Specs 755/786/823) before cutting. ### Nice-to-Know (Optimization) -- [ ] Should a **word-budget check** run in CI (fail the build if any surface exceeds - its target by >10%)? Cheap anti-re-growth insurance; adds a maintenance surface. -- [ ] Does trimming the consult-type prompts move CMAP verdict *quality* measurably, or - only cost? B1 will show the rate; quality needs human adjudication. +- [ ] Does trimming consult-type prompts move CMAP verdict *quality*, or only cost? B1 shows + the rate; quality needs human adjudication. - [ ] Are the ~17,000 words of `.claude/skills/` the right destination for relocated how-tos, or does that surface need its own budget? -## Performance Requirements +*(Iteration 1's "should a word-budget check run in CI?" is withdrawn — it contradicted T3, +which already runs the ceilings as a test. T3 stands.)* -Not a runtime-performance feature; the requirements are on the artifact and the harness. +## Performance Requirements -- **Served always-on words**: ≤15,900 per SPIR builder (from 33,519) — M1. -- **Per-surface ceilings**: as tabulated in **Desired State** — M2. -- **Measurement runtime**: `measure-prompt-surface.sh` completes in <5s and is - deterministic — same commit ⇒ byte-identical output (existing determinism property, - preserved). -- **Token/cost effect** (advisory, not a gate): a ~17,600-word always-on reduction is - ~23,000 tokens per builder-project; recorded before/after from `consult stats` and - session telemetry as context for interpreting the A/B, keying no threshold. +- **Builder always-on**: ≤16,100 words (from 34,255) — M1. +- **Per-segment**: architect ≤3,400 (from 8,599); consultant ≤460 (from 683); no segment + regresses. +- **Per-surface ceilings**: as tabulated — M2, binding. +- **Measurement runtime**: <5s, deterministic — same commit ⇒ byte-identical output. +- **Token/cost effect** (advisory): ~18,200 fewer always-on words ≈ ~24,000 tokens per + builder-project; recorded before/after as context for the A/B, keying no threshold. ## Security Considerations -- **The scar rules are the security surface.** Every one of the eight guards an - irreversible act (destroying uncommitted work, destroying worktrees, killing live - sessions, bypassing a human gate). Deleting or weakening one is the highest-severity - failure this project can produce — hence verbatim carriage (M4), byte-identical - enforcement, and a hard rollback trigger on any observed violation in the A/B. -- **Human-gate integrity.** The rewrite must not weaken "a gate message is a - notification to the human, not authorization." Gate semantics are content, not code, - and this project edits content. -- **No secrets in prompt surfaces.** Existing property; re-verified after the rewrite - (the surfaces contain no credentials today and must not acquire any). -- **Adopter blast radius.** Skeleton changes ship to every adopter on `codev update`; - a weakened prohibition would propagate silently. This is why rollback is per-surface - and rehearsed (M9). +- **The scar rules are the security surface.** All eight guard irreversible acts (destroying + uncommitted work or worktrees, killing live sessions, bypassing a human gate). Weakening + one is the highest-severity failure this project can produce — hence verbatim carriage, + byte-identical enforcement, counted-not-exempt ceilings, and a hard rollback trigger on + any observed violation. +- **Human-gate integrity.** The rewrite must not weaken "a gate message is a notification to + the human, not authorization." Gate semantics are content, and this project edits content. +- **No secrets in prompt surfaces** — existing property, re-verified after the rewrite. +- **Adopter blast radius.** Skeleton changes ship on `codev update`; a weakened prohibition + propagates silently. Hence grouped, rehearsed rollback. ## Test Scenarios ### Functional Tests -1. **T1 — Measurement correctness (M0).** The script's phase-prompt source directory - equals the directory `loadPromptFile` resolves for a known protocol; asserted against - the real resolver, not a hardcoded string. Regression-proofs the dead-tree defect. -2. **T2 — Include expansion (M0/M1).** A prompt with a `{{> …}}` include counts the - include's words; a fixture that moves text from prompt into template shows **zero** - change in `ALWAYS_ON_WORDS` (phantom-savings proof, preserved from 1252). -3. **T3 — Word ceilings (M1/M2).** Each surface's served word count is at or under its - target, per protocol, per tree. Failure names the surface and the overage. -4. **T4 — Scar-rule integrity (M4).** Every canonical string appears byte-identically on - every registered surface; the test pins the rule count at 8 and the eight ids; - rewording or deleting any copy fails. -5. **T5 — Capability inventory (M5).** The set of {artifact paths, gate names, signal - names, porch check names, notification triggers} extracted from the post-rewrite - surface equals the pre-rewrite set. Additions allowed; **removals fail**. -6. **T6 — Sweep completeness (M3).** Protocols are enumerated from - `codev-skeleton/protocols/` on disk; each must satisfy T3. A newly added protocol - fails the test until it is written to budget. -7. **T7 — Twin parity.** `CLAUDE.md` and `AGENTS.md` byte-identical; `codev/` and - `codev-skeleton/` copies of every rewritten framework file consistent. -8. **T8 — Dead-tree removal (M6).** `codev-skeleton/porch/prompts/` absent; no source - file references it. -9. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface - receives a spawn prompt containing every element of the artifact contract, and its - first `porch next` returns a well-formed task. ("It compiled" is not "it works" — - the real spawn path is exercised, not a unit fixture.) -10. **T10 — Rollback rehearsal (M9).** Reverting the rewrite commits on a scratch branch - restores the pre-rewrite surface byte-for-byte and `measure-prompt-surface.sh` - reproduces the pre-rewrite figure. +1. **T1 — Instrument sources the served directory (M0a).** The script's phase-prompt source + equals the directory `loadPromptFile` resolves, asserted against the real resolver. +2. **T1b — Instrument resolves per-file, four-tier (M0b).** A fixture with a `.codev/` + override of *one* prompt while others resolve from the skeleton measures each file at its + winning tier — kills the directory-level-selection defect class. +3. **T2 — Include expansion (phantom-savings proof).** Moving text from prompt into template + produces **zero** change in the reported total. +4. **T3 — Ceilings (M1/M2).** Each surface at or under its gross ceiling, per protocol, per + tree; failure names surface and overage. Runs in CI as the anti-re-growth guard. +5. **T4 — Scar integrity (M4).** Every canonical byte-identical on every registered surface; + count pinned at 8; reword or deletion fails. +6. **T5 — Capability inventory (M5).** Post-rewrite extraction ⊇ pre-rewrite, using M5's + recognition and normalization rules; removals fail. +7. **T6 — Sweep completeness (M3).** Protocols enumerated from disk; each satisfies T3; a + newly added protocol fails until written to budget. +8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; `codev/` and `codev-skeleton/` copies + consistent. +9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing + test updated per M10 and still protecting hot/cold routing on its remaining files. +10. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface receives + a spawn prompt containing every element of the artifact contract, and its first + `porch next` returns a well-formed task. ("It compiled" is not "it works.") +11. **T10 — Rollback rehearsal (M9).** Reverting a rollback **group** on a scratch branch + restores that group byte-for-byte and leaves the suite green. +12. **T11 — Segment reporting (M0f).** The script emits architect/builder/phase/consultant + subtotals that sum to the total; a fixture where one segment grows and another shrinks + shows both movements, not a netted zero. ### Non-Functional Tests -1. **T11 — Determinism.** Two runs of each measurement script at the same commit emit - byte-identical output. -2. **T12 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run - post-merge with self-exclusion; B1/B2/B4 committed and compared directionally to the - 1252 baseline. -3. **T13 — A/B execution (M7).** The full pre-registered protocol below. +1. **T12 — Determinism.** Two runs at the same commit emit byte-identical output. +2. **T13 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run post-merge + with self-exclusion; B1/B2/B4 committed and compared directionally. +3. **T14 — A/B execution (M7).** The full pre-registered protocol below. ## A/B Validation Design -*(A first-class section per Baked Decision 3. This is a **non-inferiority** trial: the -claim under test is "deleting 53% of the always-on surface does not degrade outcomes", -not "it improves them".)* +*(First-class section per Baked Decision 3. A **non-inferiority** trial: the claim under +test is "deleting ~53% of the always-on surface does not degrade outcomes", not "it improves +them".)* ### Unit and arms -The unit of observation is an **issue-pair**: one GitHub issue executed twice, by two -freshly-spawned builders in separate worktrees, from the same base commit. +The unit is an **issue-pair**: one GitHub issue executed twice, by two freshly-spawned +builders in separate worktrees, from the same base commit. -- **Control arm (A)**: worktree whose `codev/` + `codev-skeleton/` are at the - pre-rewrite commit. -- **Treatment arm (B)**: worktree at the post-rewrite commit. +- **Control (A)**: worktree at the pre-rewrite commit. +- **Treatment (B)**: worktree at the post-rewrite commit. No code differs between arms — the prompt surface is file-resolved, so the arms are two -checkouts. This is the whole reason the design is cheap. - -### Sample - -- **≥6 pairs (12 builder runs)**, stratified: ≥3 SPIR/ASPIR (exercises spec, plan, - implement, review prompts, both gates, and the templates) and ≥3 lighter protocols - (BUGFIX/AIR — exercises the short prompts and the single consult). -- Issues drawn from the existing backlog, selected **before** either arm runs, and - frozen (no issue-body edits mid-trial). -- Arm order alternates per pair to control for time-varying factors. +checkouts. + +### Sample and eligibility + +- **≥6 pairs (12 runs)**, stratified: ≥3 SPIR/ASPIR (exercises spec/plan/implement/review + prompts, both gates, templates) and ≥3 lighter protocols (BUGFIX/AIR). +- Issues drawn from the existing backlog, selected **before** either arm runs, then frozen + (no issue-body edits mid-trial). +- **Eligibility exclusion (CMAP round 1):** an issue is ineligible if it modifies any + surface under test. Otherwise the treatment arm's own prompt surface is simultaneously + instrument and subject. + +### Contamination controls + +- **Pin the environment**: model ids and reasoning efforts, consult backend versions, and + `.codev/config.json` frozen for the trial window and recorded with the results. +- **Arm isolation**: the second arm of a pair must not see the first arm's branch, PR, or + thread file. Arms run **sequentially with the intervening branch unpushed**, or + concurrently in isolated worktrees — either is acceptable; which was used is recorded + per pair. +- **Arm order alternates** per pair to control for time-varying factors. +- **Recording**: one committed results artifact + (`codev/resources/1280-ab-results.md`) with a row per run — pair id, arm, protocol, issue, + base commit, order, isolation mode, every outcome value, and any exclusion with its + reason. Exclusions after the fact must be justified in that file, not silently dropped. ### Pre-registered outcomes -| ID | Outcome | Instrument | Direction | +| ID | Outcome | Instrument | Role in decision | |---|---|---|---| -| **O1** | Gate friction | Architect scores each gate on a 3-item rubric at approval time: *artifact complete as specified? / required rework before approval? / did the builder need a clarifying message?* (prospective — history is not minable) | non-inferior | -| **O2** | Review rounds | Iterations to terminal state per phase, from `status.yaml` history; plus CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | non-inferior within margin | -| **O3** | Correctness | Architect's PR review findings by severity + any post-merge defect attributable to the run | non-inferior | -| **O4** | Protocol compliance | Binary per-run checklist: required artifacts present with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar-rule violation · thread committed | **zero tolerance** | -| **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory only | +| **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each scored 0 (no friction) / 1 (minor) / 2 (blocking), recorded in the results artifact at scoring time | **advisory + tripwire** | +| **O2** | Review rounds | Iterations to terminal state per phase from `status.yaml` history; CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | gate | +| **O3** | Correctness | Architect PR-review findings by severity; any post-merge defect attributable to the run | gate | +| **O4** | Protocol compliance | Binary per-run checklist: required artifacts with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar violation · thread committed | **zero tolerance** | +| **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory | ### Blinding -CMAP reviewer models are blind by construction. The architect is not blind and cannot -be; the mitigation is that O2 and O4 are extracted mechanically from committed -artifacts, and O1/O3 are scored against a rubric written **before** any run. +CMAP reviewers are blind by construction. The architect is not and cannot be; mitigations: +O2 and O4 are extracted mechanically from committed artifacts, and O1/O3 are scored against +a rubric written **before** any run. ### Decision rule (pre-registered) **SHIP** iff all of: -1. **O4 = zero violations in the treatment arm.** Any scar-rule violation, skipped gate, - or missing required artifact is an immediate hard stop, independent of every other - outcome. -2. **O2** treatment mean review rounds ≤ control mean **+ 0.5 rounds/phase**, and - treatment REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. -3. **O3** no treatment-arm correctness finding of severity ≥ "would block merge" that - is absent from its paired control run. -4. **O1** no pair where the treatment arm required rework at a gate that its control did - not, for the same reason. +1. **O4 = zero violations** in the treatment arm. Any scar violation, skipped gate, or + missing required artifact is an immediate hard stop, independent of everything else. +2. **O2**: treatment mean review rounds ≤ control **+ 0.5 rounds/phase**, and treatment + REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. +3. **O3**: no treatment-arm finding of severity ≥ "would block merge" absent from its paired + control run. +4. **O1 tripwire**: no pair where the treatment arm scored **2 (blocking)** at a gate its + control scored 0, for the same reason. If O1 scoring is incomplete for any pair, O1 is + reported incomplete and SHIP rests on 1–3. -Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK** (below). +Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK**. ### Honest power statement -With n=6 pairs this design detects only **large** effects — roughly a doubling of review -rounds or a ≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle -regression, and this spec does not claim it can. It is nonetheless strictly stronger -than 1252's observational baseline, because each pair is matched on the issue itself — -the dominant source of variance. O4's zero-tolerance criterion is where the real -protection lives: compliance is binary, observable in every run, and is the failure mode -that deletion would plausibly cause. +With n=6 pairs this detects only **large** effects — roughly a doubling of review rounds or +a ≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle regression, and +this spec does not claim it can. It is nonetheless strictly stronger than 1252's +observational baseline, because each pair is matched on the issue itself — the dominant +variance source. **O4's zero-tolerance criterion is where the real protection lives**: +compliance is binary, observable in every run, and is the failure mode deletion would +plausibly cause. ## Rollback Plan -*(Required by Baked Decision 4. Prompt surfaces are files; reverting is cheap — here is -exactly how cheap.)* - -- **Unit.** One revertible commit per surface (CLAUDE.md/AGENTS.md · role · protocol.md · - builder-prompt · prompts+templates · consult-types · registry · measurement script), - in both trees. Rolling back one surface never requires rolling back another. -- **Mechanism.** `git revert ` restores the prior bytes. No migration, no state, - no schema, no data. Verified by T10 on a scratch branch before the PR merges. -- **Blast radius and propagation.** For this repo: effective for the next spawned - builder — in-flight builders keep the surface they were spawned with (prompts are read - at spawn/phase time, so a running builder is unaffected either way). For adopters: the - revert ships in the next release; an adopter can also pin the prior `@cluesmith/codev` - version, since framework files resolve from the installed skeleton (tier 4). -- **Triggers.** (a) any O4 violation in the A/B — immediate, no deliberation; (b) an - observed scar-rule violation in any real project post-merge; (c) O2/O3 outside the - pre-registered margins; (d) architect's judgment at the `pr` or `verify-approval` gate. -- **Partial rollback is the expected shape.** If one surface regresses (say the review - prompt lost a contract), revert that surface and keep the rest — the whole point of - per-surface commits. -- **Cost.** One revert, one release. There is no irreversible step anywhere in this - project. +*(Required by Baked Decision 4.)* + +CMAP round 1 correctly flagged that iteration 1 overstated per-surface independence: prompts, +their included templates, scar-registry mappings, and integrity tests are coupled — reverting +a prompt without its template can break a required-headings check or M4. Rollback is +therefore by **group**, each group internally consistent and independently revertible: + +| Group | Contents | +|---|---| +| **G1 instrument** | measurement script + its tests + baseline artifacts | +| **G2 shared** | `CLAUDE.md` + `AGENTS.md` + hot-tier wiring | +| **G3 builder-spawn** | `roles/builder.md` + all `builder-prompt.md` + all `protocol.md` + their prose-pinned tests | +| **G4 phase** | all `prompts/*.md` + their `templates/*.md` + porch check expectations | +| **G5 consultant** | `roles/consultant.md` + all `consult-types/*.md` | +| **G6 architect** | `roles/architect.md` + relocated skill content | +| **G7 scar registry** | `scar-rules.yaml` + its enforcement test | + +- **Dependency rule**: reverting **G7** requires reverting every group whose surfaces carry + scar text (G2, G3, G4, G6) — the registry and its copies must agree. All other groups are + mutually independent. +- **Mechanism**: `git revert` restores prior bytes. No migration, state, schema, or data. + Rehearsed under T10 before the PR merges. +- **Blast radius**: for this repo, effective for the next spawned builder — in-flight + builders keep the surface they were spawned with (prompts read at spawn/phase time). For + adopters, the revert ships in the next release; an adopter can also pin the prior + `@cluesmith/codev` version, since framework files resolve from the installed skeleton. +- **Triggers**: (a) any O4 violation — immediate, no deliberation; (b) an observed scar + violation in any real project post-merge; (c) O2/O3 outside pre-registered margins; + (d) architect judgment at the `pr` or `verify-approval` gate. +- **Partial rollback is the expected shape** — revert the offending group, keep the rest. +- **Cost**: one revert, one release. No irreversible step exists anywhere in this project. ## Dependencies -- **External Services**: none. (`gh` for issue/PR reads during the A/B; consult backends - — Gemini via `agy`, Codex, Claude — for CMAP, unchanged.) -- **Internal Systems**: the four-tier resolver (`lib/skeleton.ts`); porch prompt - composition (`commands/porch/prompts.ts`) — read, not modified; the consult CLI's - consult-type resolution; `scripts/measure-prompt-surface.sh` and - `packages/codev/scripts/measure-prompt-behavior.ts`. -- **Artifacts**: `builder/spir-1252` (ratified scar registry — must not be deleted); - `codev/resources/1252-*.md` (baselines). +- **External Services**: none. (`gh` for issue/PR reads during the A/B; consult backends — + Gemini via `agy`, Codex, Claude — unchanged.) +- **Internal Systems**: four-tier resolver (`lib/skeleton.ts`); porch prompt composition + (`commands/porch/prompts.ts`) — read, not modified; managed-block hot-tier wiring + (`lib/managed-block.ts`); role injection (`agent-farm/commands/spawn-worktree.ts`); + consult-type resolution (`commands/consult/index.ts`); both measurement scripts. +- **Artifacts**: `builder/spir-1252` (ratified registry — must not be deleted); + `codev/resources/1252-*.md` baselines. +- **Test suites**: the ~25 prose-pinned files enumerated in M10 — a dependency in the real + sense that the cuts cannot land without deliberately re-baselining them. - **Libraries/Frameworks**: none new. ## References -- Issue #1280 (this charter); Issue #1279 (dead spec/review templates); Issues #1276 - (multi-model tiering) and #1277 (controlled A/B eval), both filed by 1252 and both - superseded here or explicitly out of scope. -- PR #1278 (Spec 1252, closed unmerged) and branch `builder/spir-1252` — surface - inventory, ownership analysis, scar registry, enforcement machinery. -- `codev/reviews/1252-prompt-architecture-single-own.md`; - `codev/resources/1252-word-baseline.md`, `1252-word-after-phase7.md`, - `1252-behavior-baseline.md`, `1252-shadow-tree-audit.md`; +- Issue #1280 (charter); #1279 (dead spec/review templates); #1276, #1277 (filed by 1252 — + superseded here or out of scope). +- PR #1278 (Spec 1252, closed unmerged) and `builder/spir-1252`. +- `codev/reviews/1252-prompt-architecture-single-own.md`; `codev/resources/1252-*.md`; `codev/state/spir-1252_thread.md`. - *The new rules of context engineering for Claude-5-generation models* — https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models -- `codev/resources/arch.md` (four-tier resolution, repository dual nature); +- `codev/resources/arch.md` (four-tier resolution, include directive, hot-tier injection); `codev/resources/lessons-learned.md` (sweep-scope failures, served-surface dedup). ## Risks and Mitigation | Risk | Probability | Impact | Mitigation Strategy | |------|------------|--------|-------------------| -| A deleted instruction was load-bearing and its loss is silent | Medium | High | T5 capability-inventory diff (removals fail the build); O4 zero-tolerance in the A/B; per-surface rollback | -| A scar rule is weakened or dropped | Low | **Critical** | Verbatim carriage; byte-identical enforcement pinned at 8 rules; hard rollback trigger; architect-only rewording | -| Sweep misses a protocol or the second tree | **High** | Medium | T6 enumerates protocols from disk; T7 twin parity. 1252's top lesson was that sweep-scope failures dominated its review iterations | -| A/B underpowered; a subtle regression ships | Medium | Medium | Stated honestly in the power statement; O4 binary compliance carries the protection; T12 behavioural re-measurement post-merge as a second net | -| ~~Corrected baseline disputed at the gate~~ — **retired**, endorsed 2026-08-01 | — | — | Both figures are re-derived by one script, so any future re-scoping of the denominator recomputes mechanically | -| A *third* instrument defect ships undetected | Medium | High | Principle 7: every check and baseline is reviewed against its claim. T1/T2 assert the instrument against the live resolver; M0b puts the corrected instrument under public review early rather than at PR time | -| Surface re-grows after the project ends | **High** | Medium | Word-ceiling test (T3) run in CI is the cheap structural answer; open question on whether to gate on it | -| The A/B costs more than the shrink saves | Medium | Low | 12 runs on backlog issues that needed doing anyway; O5 tracks it; architect sets the pair ceiling | -| `builder/spir-1252` is deleted, losing the ratified registry | Low | High | Registry content is quoted in this spec's thread and will be committed to `main` as the rebuilt registry early | +| A deleted instruction was load-bearing; loss is silent | Medium | High | M5 deterministic capability inventory (removals fail); O4 zero-tolerance; grouped rollback | +| A scar rule weakened or dropped | Low | **Critical** | Verbatim carriage; counted-in-ceiling so it is never trimmed to fit; byte-identical enforcement pinned at 8; hard rollback trigger; architect-only rewording | +| **Prose-pinned tests silently gutted to go green** | **High** | **High** | **M10**: every retired assertion named with its originating spec and its protected behaviour re-asserted or explicitly retired; pure-addition re-baselining only with the spec named and baseline committed together | +| Sweep misses a protocol, a tree, or a segment | High | Medium | T6 enumerates from disk; T7 twin parity; T11 segment reporting. 1252's dominant review cost was sweep-scope failure | +| >50% margin is thin (3.2 pts) | Medium | Medium | M2 ceilings are binding and M1 derived; a 50–52% landing is a HOLD with named further-cut candidates | +| A/B underpowered; subtle regression ships | Medium | Medium | Stated in the power statement; O4 binary compliance carries the protection; T13 post-merge behavioural re-measurement as a second net | +| Instrument defect #3 ships undetected | Medium | High | Principle 7; T1/T1b/T11 assert the instrument against the live resolver; M0b puts it under public review early | +| A/B costs more than the shrink saves | Medium | Low | 12 runs on backlog issues that needed doing; O5 tracks it; architect sets the pair ceiling | +| Surface re-grows after the project | High | Medium | T3 runs the ceilings in CI | +| `builder/spir-1252` deleted, losing the registry | Low | High | Registry content quoted in this project's thread; rebuilt registry committed to `main` early | ## Expert Consultation -**Date**: pending -**Models Consulted**: Gemini (via `agy`), Codex (GPT-5.6 Sol), Claude Opus 5 — run by -porch at the specify-phase verify step. -**Sections Updated**: *(to be filled after the 3-way review; feedback is incorporated -directly into the sections above and summarized here)* +**Date**: 2026-07-31 (round 1) +**Models Consulted**: Codex (GPT-5.6 Sol) · Claude Opus 5. *(Gemini/`agy` was not in +porch's model set for this consultation — the known `--type` review limitation, #1032/#1033.)* +**Verdicts**: both REQUEST_CHANGES, both HIGH confidence. + +**Sections updated in response** (all feedback verified against source before acting): + +| Finding | Raised by | Resolution | +|---|---|---| +| Ceilings stated net-of-scar while M1 is gross — incompatible | both | All ceilings restated **gross** with carriage shown per row; arithmetic rebuilt; margin disclosed | +| M0 doesn't match the resolver (per-file four-tier vs two-tier directory selection) | Codex | M0(b) + **T1b** added | +| CLAUDE.md `@import`s the hot tier (#1119); 5,815 excludes 736 always-loaded words | Claude | Current State corrected; baseline 33,519 → **34,255**; M0(d) requires fixing the script's stale comment | +| "No code reads `porch/prompts`" is false — `review-prompt-routing.test.ts:29` | Claude | **Verified and conceded**; cause (truncated grep) recorded in Current State; M6 verification method rewritten | +| ~25 prose-pinned tests block the cuts; `baked-decisions.test.ts` pure-addition diff is incompatible with 824 → ≤420 | Claude | **New criterion M10** + new risk row + Dependencies entry | +| "Gate friction" left unresolved under Critical while the A/B assumes it | both | Resolved: O1 demoted to advisory-with-tripwire, rubric scale and recording location defined, incomplete-scoring behaviour specified | +| A/B lacks contamination controls | Codex | **Contamination controls** subsection added (env pinning, arm isolation, order, recording artifact) | +| A/B should exclude issues touching surfaces under test | Claude | Added as an eligibility rule | +| Per-surface rollback independence overstated | Codex | Rewritten as **seven rollback groups** with an explicit G7 dependency rule | +| M5's inventory diff not deterministic | Codex | Recognition + normalization rules specified; committed pre-rewrite artifact | +| T3-vs-CI open question contradictory | Claude | Open question withdrawn; T3 stands | +| Template pressure conflated (porch's 4 headings vs consult type's 20) | Claude | Named separately in Current State | +| Spec dated 2026-08-01 while project date is 2026-07-31 | Codex | Corrected to **2026-07-31** throughout — the architect's messages carried UTC timestamps (`02:50Z`), local time 19:50 on 2026-07-31 | + +**Not disputed.** Every round-1 finding was accepted; no rebuttal was filed. Two were +factual errors in my own Current State, both verified against source before correction. ## Approval - [ ] Technical Lead Review @@ -677,19 +789,18 @@ directly into the sections above and summarized here)* ## Notes -**On the irony.** A specification arguing for deletion should not be padded. This one is -long because it carries three architect-mandated designs (cut plan, A/B, scar carriage) -plus a measurement correction that changes the project's headline number. The artifacts -it produces are the short ones. - -**On what this project deliberately does not do.** It does not build a prompt generator -(Approach 2), does not add tiering of any kind (Baked Decision 1), does not change porch -behaviour, and does not rebuild 1252's full enforcement machinery — only the minimum -scar-rule integrity check that the deletion itself makes necessary. Enforcement rebuilt -around a surface that is still moving is enforcement built twice. - -**On the deferred decision from 1252.** The architect's pr-gate ruling was that -structural machinery is not worth carrying for a surface about to shrink by half. That -sequencing is honoured here: shrink first, then enforce what remains. The word-ceiling -check (T3) is the successor's smallest useful enforcement primitive, and it only becomes -meaningful once the ceilings exist. +**On the irony.** A spec arguing for deletion should not be padded. This one is long because +it carries four architect-mandated designs — cut plan, A/B, scar carriage, and now a +segmented full-surface inventory — plus an instrument correction that changes the project's +headline number twice. The artifacts it produces are the short ones. + +**On what this project deliberately does not do.** No prompt generator (Approach 2), no +tiering of any kind (Baked Decision 1), no porch behaviour changes, and not 1252's full +enforcement machinery — only the minimum scar-integrity check the deletion makes necessary, +plus the ceiling test that prevents re-growth. Enforcement built around a still-moving +surface is enforcement built twice. + +**On the deferred decision from 1252.** The architect's pr-gate ruling was that structural +machinery is not worth carrying for a surface about to halve. That sequencing is honoured: +shrink first, then enforce what remains. T3 is the smallest useful enforcement primitive, +and it only becomes meaningful once the ceilings exist. diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 3012ae113..ef42be901 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -70,3 +70,54 @@ plan with word targets, A/B non-inferiority design with a pre-registered decisio scar-rule carriage plan) plus the rollback story. Headline: 33,519 → ≤15,900 (−52.7%), with the phase-task term (71% of the post-rewrite budget) dominated by the hot tier, which is explicitly exempt from cuts. Signalling SPEC_DRAFTED to porch for 3-way review. + +### Iteration 2 — CMAP round 1 + scope directive (2026-07-31) + +**Both reviewers REQUEST_CHANGES, both HIGH confidence, both worth every word.** Verified +each factual claim against source before acting (lessons-critical: summaries are evidence, +not ground truth). All findings accepted; **no rebuttal filed** — nothing was a false +positive. + +Two findings were errors in *my own* Current State, and both are the class this spec's +principle 7 exists to catch: + +1. **Stale hot-tier claim.** I wrote "CLAUDE.md (hot tier inlined)". Since #1119 + (`managed-block.ts:59-67`) it carries `@import` lines that Claude Code transcludes at + session launch. So `wc -w CLAUDE.md` = 5,815 **excludes** 736 always-loaded words. + I inherited this from the measurement script's own stale comment (lines 44-47) — i.e. I + trusted the instrument while writing a spec about auditing instruments. Baseline + 33,519 → **34,255**. +2. **Truncated grep → false claim.** I asserted "no code reads `porch/prompts`" from a + `grep -rn ... | head -20`. The 48-line full output contains + `review-prompt-routing.test.ts:29`, a Spec 987 protection that reads + `codev-skeleton/porch/prompts/review.md`. Tree is still dead as prompt surface, but M6's + verification method was wrong. **Never conclude "no consumers" from truncated output** — + this is the sweep-scope failure class 1252 named as its dominant review cost. + +Claude's biggest structural catch: **~25 test files pin exact prose in the surfaces being +cut**, worst being `baked-decisions.test.ts:143-148` — a pure-addition diff against +committed baselines for three `builder-prompt.md` files, logically incompatible with +824 → ≤420. Each assertion is a prior spec's protection encoded as a grep, so retiring one +is a governance act. Became **M10**: every modified/retired assertion named with its +originating spec, its protected behaviour re-asserted or explicitly retired. Without this a +builder under pressure quietly deletes assertions to go green — the exact silent-capability +-loss this project exists to avoid. + +Codex's sharpest: ceilings were stated net-of-scar while M1 was gross (Claude independently +found the same, computing the gross landing at 50.2–51.4% — i.e. M1 would have *failed* +while every per-surface row passed). All ceilings are now gross; M2 is binding and M1 +derived; a 50–52% landing is a HOLD with named further-cut candidates. + +**Architect scope directive** (same window): scope is the ENTIRE prompt surface — architect +roles, builder roles + spawn wrappers, consultant/CMAP prompts, phase prompts + template +includes, protocol.md texts — and the instrument must segment by audience so a cut in one +segment masking growth in another is visible, not averaged away. Built the full inventory +(every prompt-bearing surface, resolver path, served word count) and folded it into Current +State; added per-segment targets and T11 (segment reporting, with a fixture proving one +segment growing while another shrinks is not netted to zero). + +Post-rewrite, by segment: builder 34,255 → 16,016 (−53.2%) · architect 8,599 → 3,336 +(−61.2%) · consultant 683 → 452 (−33.8%). No segment grows. Consultant cuts least because +`roles/consultant.md` is already lean at 252 — reported, not averaged away. + +Spec now 6,942w. Iteration 2 ready for CMAP round 2. From 74f45fef4ca1fd3b8b9ceb55136bacdf3bf5ba8a Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:06:40 -0700 Subject: [PATCH 05/32] chore(porch): 1280 spec-approval gate-requested --- codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 8a05358be..ef8e6d0a2 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -7,6 +7,7 @@ current_plan_phase: null gates: spec-approval: status: pending + requested_at: '2026-08-01T03:06:40.784Z' plan-approval: status: pending pr: @@ -17,4 +18,4 @@ iteration: 1 build_complete: true history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T02:55:40.540Z' +updated_at: '2026-08-01T03:06:40.784Z' From c728226ef61c37b27b21c1932dd155590f44dab9 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:06:53 -0700 Subject: [PATCH 06/32] [Spec 1280] Iteration-1 review response: all CMAP findings accepted, none disputed Records the round-1 verification and resolution of all 13 findings from Codex and Claude, plus the architect's full-surface scope directive. No rebuttal filed - every finding was verified against source and accepted, including two factual errors in the spec's own Current State. --- .../1280-specify-iter1-rebuttals.md | 174 ++++++++++++++++++ 1 file changed, 174 insertions(+) create mode 100644 codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md new file mode 100644 index 000000000..403e9e580 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md @@ -0,0 +1,174 @@ +# Iteration 1 response — Spec 1280 + +**No disputes filed.** Every REQUEST_CHANGES point from both reviewers was verified against +source and accepted. This document records what changed and where, so round 2 can check the +fixes rather than re-derive the findings. + +Two of the findings were factual errors in my own Current State. Both are the exact class +this spec's principle 7 ("the instrument is part of the deliverable") exists to catch, and +both are recorded in the spec rather than quietly patched. + +--- + +## Codex findings + +### 1. Scar-rule accounting contradicts M1/M2 — ACCEPTED + +Iteration 1 declared per-surface targets as served-word ceilings *and* said "targets above +are net of this [scar carriage]." Incompatible, and Claude independently computed the +consequence: gross landing 50.2–51.4%, so **M1 would have failed while every per-surface row +passed**. + +**Changed**: all ceilings restated **gross**, with a per-row "scar carriage inside" column. +Arithmetic rebuilt (`Desired State`). Carriage is now *exempt from rewriting but counted in +ceilings* — with the corollary stated explicitly: a ceiling a surface cannot meet while +carrying its scar rules is a wrong ceiling, raised deliberately, never met by trimming scar +text. Margin disclosed (3.2 points) and M2 made binding with M1 derived; a 50–52% landing is +a HOLD with named further-cut candidates. + +### 2. M0 does not match the runtime resolver — ACCEPTED + +Correct: `loadPromptFile` resolves each file independently through the four-tier chain, while +the script does two-tier, directory-level selection. Fixing the directory alone would +reproduce the same defect class one layer down. + +**Changed**: M0 now requires per-file four-tier resolution matching `resolveCodevFile`, and +**T1b** is a new test using a fixture with a `.codex`/`.codev/` override of *one* prompt while +others resolve from the skeleton. M0 also now names the hot-tier transclusion and the +segment reporting the architect's directive requires. + +### 3. "Gate friction" unchecked under Critical while the A/B assumes it — ACCEPTED + +**Changed**: resolved and moved out of Critical. O1 is scored prospectively by the architect +on a 3-item rubric with an explicit 0/1/2 scale, recorded in a committed results artifact at +scoring time, and **demoted to advisory-with-a-tripwire** rather than a SHIP gate. If scoring +is incomplete for any pair, O1 reports incomplete and SHIP rests on O2/O3/O4. (Claude raised +the same single-point-of-failure concern independently and suggested exactly this demotion.) + +### 4. A/B contamination controls — ACCEPTED + +**Changed**: new **Contamination controls** subsection — model ids/efforts and consult +backend versions pinned and recorded; arm isolation (second arm must not see the first arm's +branch, PR, or thread; sequential-with-unpushed-branch or isolated concurrent, recorded per +pair); alternating arm order; and a committed results artifact +(`codev/resources/1280-ab-results.md`) with one row per run including base commit, isolation +mode, every outcome, and any exclusion with its reason. + +### 5. Per-surface rollback independence overstated — ACCEPTED + +Correct — prompts, included templates, registry mappings and integrity tests are coupled. + +**Changed**: rollback restated as **seven groups** (instrument / shared / builder-spawn / +phase / consultant / architect / scar-registry), each internally consistent, with an explicit +dependency rule: reverting G7 requires reverting every group carrying scar text. T10 rehearses +by group. + +### 6. M5's inventory diff not deterministic — ACCEPTED + +**Changed**: M5 now specifies a committed pre-rewrite `capability-inventory.json` with +explicit recognition rules per element type (artifact paths, gate names from `protocol.json`, +`` tags, check ids, `afx send architect` call sites) and normalization (lowercase, +strip backticks/punctuation, dedupe). Post-rewrite must be a superset; removals fail and must +be justified as deliberate retirements. + +### 7. Future-dated provenance — ACCEPTED + +The architect's instructions carried UTC timestamps (`2026-08-01T02:50Z`, `02:59Z`); local +time was 19:50/19:59 on 2026-07-31. + +**Changed**: all dates normalized to **2026-07-31**, with the UTC/local explanation recorded +in the consultation log so the provenance is auditable rather than merely corrected. + +--- + +## Claude findings + +### 1. "All existing tests pass" is unsatisfiable — ACCEPTED (highest-value finding) + +Verified: `agent-farm/__tests__/baked-decisions.test.ts:143-148` enforces a pure-addition +diff against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md`, which +the cut plan takes 824 → ≤420. Also verified the shape across +`bugfix-744-spir-pr-strategy.test.ts`, `spec-1273-wait-discipline-docs.test.ts`, +`bugfix-619-aspir-prompt.test.ts`, `template-delivery.test.ts`, `framework-ref-audit.test.ts`, +`governance-sweep.test.ts`, `review-prompt-routing.test.ts`. + +The framing is the important part and I have adopted it verbatim in intent: **each assertion +is a prior spec's protection encoded as a grep, so retiring one is a governance act, not a +test fix.** + +**Changed**: new criterion **M10** — every modified or retired assertion listed in the review +with (i) the originating spec, (ii) whether the protected behaviour survives in the rewritten +prose, (iii) the replacement assertion or an explicit architect-visible retirement. +Pure-addition re-baselining only with the originating spec named and the new baseline +committed in the same commit. Silent deletion to go green is declared a project failure. Also +added as a Risks row (High/High) and a Dependencies entry. + +### 2. M1 and M2 stated on different bases — ACCEPTED + +Same as Codex 1; see above. Claude's gross computation (50.2–51.4%) is what made the severity +concrete, and the thin-margin warning is now in the spec as a HOLD rule. + +### 3. Two factual errors in Current State — ACCEPTED, both verified + +**(a) Hot tier is `@import`ed, not inlined.** Verified `CLAUDE.md:14-15` and +`managed-block.ts:59-67`: #1119 replaced Spec 987's verbatim inlining with `@import` lines +that Claude Code transcludes at session launch. So `wc -w CLAUDE.md` = 5,815 **excludes** the +736 hot-tier words the session actually loads. Baseline corrected **33,519 → 34,255**. + +Worth stating plainly: I inherited this from the measurement script's own stale comment +(lines 44–47) while writing a spec whose headline principle is that instruments get audited +against their claims. M0(d) now requires fixing that comment, and the incident is written +into the Problem Statement as principle 7 applied to this spec itself. + +**(b) "No code reads `porch/prompts`" is false.** Verified +`review-prompt-routing.test.ts:29` pushes `codev-skeleton/porch/prompts/review.md` onto its +assertion list (Spec 987 hot/cold routing). My cause: I ran `grep -rn … | head -20` and drew +a conclusion from truncated output — the full result is 48 lines. The tree remains dead as +*prompt surface* (no runtime consumer), but M6's stated verification method was wrong. + +**Changed**: Current State carries the retraction and its cause; M6 now requires an +untruncated search reconciled against the full hit list, and routes the test consumer through +M10 naming Spec 987. Recorded in the thread as a sweep-scope failure — the class 1252 +identified as its dominant review-iteration cost. + +### 4. O1 single point of failure — ACCEPTED + +See Codex 3. Adopted Claude's suggested demotion. + +### 5. A/B arms not as clean as "two checkouts" — ACCEPTED + +Sharp catch: for issues touching Codev's own prompt surface, the treatment arm's CLAUDE.md is +simultaneously instrument and subject. + +**Changed**: eligibility rule added — an issue is ineligible for the A/B if it modifies any +surface under test. + +### 6. Template pressure conflated — ACCEPTED + +Verified `checks.ts:149-154`: `REQUIRED_SPEC_SECTIONS` is four headings (Problem Statement, +Current State, Desired State, Success Criteria), not the template's 20; the 20-heading +pressure is the `spec-review` consult type, advisorily. + +**Changed**: named as two separate constraints in Current State, so the plan does not +over-preserve template surface for a check that does not require it. + +### 7. T3-vs-CI contradiction — ACCEPTED + +**Changed**: the Nice-to-Know open question withdrawn; T3 stands and is stated as the +anti-re-growth guard that runs in CI. + +--- + +## Also incorporated this iteration (not from review) + +**Architect scope directive** (2026-07-31): scope is the entire prompt surface — architect +role prompts, builder roles and spawn wrappers, consultant/CMAP prompts, phase prompts and +their template includes, `protocol.md` texts — and the instrument must **segment by audience** +so a cut concentrated in one segment while another grows is visible rather than averaged away. + +**Changed**: Current State now carries a full inventory of every prompt-bearing surface with +its resolver path and served word count, grouped SHARED / ARCHITECT / BUILDER / PHASE / +CONSULTANT / DEAD. Per-segment ceilings and post-rewrite figures added (builder −53.2%, +architect −61.2%, consultant −33.8%, no segment growing), M0(f) requires per-segment +subtotals, and **T11** proves with a fixture that one segment growing while another shrinks +is not netted to zero. From fbfaa1ca9560e18cf49e5f55ec17b1e08aff4eba Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:07:18 -0700 Subject: [PATCH 07/32] [Spec 1280] Thread: spec-approval gate reached, two caveats flagged --- codev/state/spir-1280_thread.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index ef42be901..d8de9acec 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -121,3 +121,22 @@ Post-rewrite, by segment: builder 34,255 → 16,016 (−53.2%) · architect 8,59 `roles/consultant.md` is already lean at 252 — reported, not averaged away. Spec now 6,942w. Iteration 2 ready for CMAP round 2. + +### spec-approval gate reached (2026-07-31) + +`porch done` → `porch gate 1280`. **STOPPED, waiting for human approval.** Not running +`porch approve` — human-only gate. + +Two caveats flagged to the architect rather than silently accepted: + +1. **No CMAP round 2 ran.** Porch advanced straight from the iteration-1 rebuttal to the + gate, though `specify` has `max_iterations: 3` and only iteration 1 was consumed. The + revised spec (6,942w, substantially restructured — gross ceilings, M10, segmented + inventory, rollback groups) has therefore not been reviewed by anyone but me. Offered to + re-run round 2 if the architect wants it before the plan phase. +2. **Gemini/agy did not participate.** Porch's model set for this consult was codex+claude + only — the known `--type` review limitation (#1032/#1033). This was a 2-way, not a 3-way + review. Recorded in the spec's Expert Consultation section rather than left implicit. + +State at the gate: spec + review-response committed; worktree clean apart from the +harness-generated `.builder-*` files. From f80423eca63cf85c3248601d2c942138747e8d9c Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:08:24 -0700 Subject: [PATCH 08/32] [Spec 1280] Add rollout contract for the early instrument PR (M0b) Architect-directed. Specifies PR-1's exact contents (corrected script + its first-ever tests + corrected segmented baseline + in-place annotation of the 1252 artifacts), what is deliberately excluded (any prompt-surface edit, the scar registry, the dead-tree deletion), timing (end of first implement phase, before any cut), and the safety verification: no test asserts on either 1252 word-count artifact, so the annotation carries no re-baselining cost. --- .../1280-prompt-surface-judgment-not-ru.md | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md index 1d9304146..4273f1d08 100644 --- a/codev/specs/1280-prompt-surface-judgment-not-ru.md +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -291,6 +291,36 @@ prompts (2,414 / 1,957 — the two fattest phase prompts in the fleet). Note the shape: the phase term is 73% of the post-rewrite builder budget and the **exempt** hot tier is 63% of that term. What survives is overwhelmingly curated judgment, not process. +### Rollout: the corrected instrument lands on `main` first (M0b) + +*(Architect-directed 2026-07-31. Recorded here as a contract — the phase decomposition +belongs in the plan.)* + +**No prompt-surface word is cut before the corrected instrument is on `main`.** The +sequencing is not administrative: a cut scored by the current script is unfalsifiable, and +1252's published baselines cite figures derived from the dead tree while being shared +knowledge other work reads. The record gets corrected while this project builds, not after. + +**PR-1 (early, standalone) contains exactly:** + +| In | Out (deliberately) | +|---|---| +| Corrected `scripts/measure-prompt-surface.sh` — all six M0 fixes (served directory · per-file four-tier resolution · inlined `roles/builder.md` · hot-tier `@import` transclusion **and** the stale comment that caused defect 3 · include expansion · per-segment subtotals) | Any edit to any prompt surface | +| Its tests — T1, T1b, T2, T11, T12. The script currently has **no test at all**, which is how three defects survived in a "committed and reproducible" instrument | The scar registry (rebuilt *after* the shrink, per Baked Decision 2) | +| `codev/resources/1280-word-baseline.md` — the corrected, segmented pre-rewrite baseline (34,255) as a generated artifact | The dead-tree deletion — it has a test consumer and goes through M10 governance | +| In-place annotation of `1252-word-baseline.md` and `1252-word-after-phase7.md`: original figures **preserved**, marked superseded, with the reason and a pointer to the corrected artifact | Any re-derivation of 1252's behavioural baseline (B1 stands; M8 re-runs it post-merge) | + +**Timing**: opened at the end of the first implement phase — after the instrument is correct +and tested, before any surface is cut. **Verified safe**: no test asserts on either 1252 +word-count artifact (the frozen-sample test at `prompt-behavior-metrics.test.ts:184` pins the +*behavioural* sample, a different instrument), so the annotation carries no re-baselining +cost. + +**Why this is not pushback.** The one argument against early landing is that the corrected +baseline is only meaningful alongside the cuts it scores. That argument fails: the baseline's +value is precisely that it is *pre-cut*, and publishing it early is what makes the eventual +−53.2% claim checkable by someone who did not watch it being produced. + ### Scar-rule carriage plan The eight rules ratified 2026-07-28 (`git show @@ -337,7 +367,8 @@ canonical text total). `1252-word-baseline.md` and `1252-word-after-phase7.md` cite figures derived from the dead tree and are shared knowledge other work reads; the correction annotates them in place — original figures preserved, marked superseded, with the reason — and does not - rewrite their history. + rewrite their history. PR-1's exact contents, timing, and safety verification are + specified in **Desired State → Rollout**. - [ ] **M1 — >50% reduction (derived).** Builder always-on falls from the corrected baseline (34,255) to **≤16,100**, measured before and after by the same corrected script, both figures committed as generated artifacts. Per-segment figures reported and none From e505bcd5149ad88cb1b3ea133df4237fcb878d28 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:08:44 -0700 Subject: [PATCH 09/32] =?UTF-8?q?[Spec=201280]=20Thread:=20pre-gate=20dire?= =?UTF-8?q?ctives=20=E2=80=94=20CMAP=20round=202=20launched,=20rollout=20c?= =?UTF-8?q?ontract=20added?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- codev/state/spir-1280_thread.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index d8de9acec..2780b94c3 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -140,3 +140,34 @@ Two caveats flagged to the architect rather than silently accepted: State at the gate: spec + review-response committed; worktree clean apart from the harness-generated `.builder-*` files. + +### Pre-gate directives (2026-07-31) + +Architect directed two items before the gate goes to Waleed: + +1. **Run CMAP round 2** on the revised spec — budget allows (1 of 3 consumed), the revision + absorbed 13 findings incl. two factual corrections and a structural net→gross ceiling + change, and blast radius is every prompt every agent reads. Porch advancing without + round 2 was permissive machinery, not a judgment. Launched codex + claude in parallel + against the post-revision spec (7,306w). 2-way is correct per current lane policy; + gemini's absence needs no remedy. +2. **Answer the early-landing directive in the spec**, which I had carried only as criterion + M0b with no concrete plan. Fair catch — a criterion is not a rollout. Added + **Desired State → Rollout**: PR-1 contains the corrected script, its **first-ever tests** + (the instrument has none today — which is how three defects survived in something + "committed and reproducible"), the corrected segmented baseline artifact, and in-place + annotation of the two 1252 artifacts (originals preserved, marked superseded, reason + + pointer). Deliberately excluded: any prompt-surface edit, the scar registry, the dead-tree + deletion. Timing: end of first implement phase, before any cut. + + Verified the annotation is safe: no test asserts on either 1252 word-count artifact — the + frozen-sample test (`prompt-behavior-metrics.test.ts:184`) pins the *behavioural* sample, + a different instrument. So no re-baselining cost. + + Considered pushback and rejected it: the only argument against early landing is that the + corrected baseline is meaningful only alongside the cuts it scores, and that fails — + the baseline's value is that it is *pre-cut*, and publishing early is what makes the + eventual −53.2% claim checkable by someone who didn't watch it being produced. + +Porch remains at the spec-approval gate throughout; running consults manually does not move +state, and I am not touching the gate. From 8e79111704b1b9095768305405a5b355047e2e92 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:14:51 -0700 Subject: [PATCH 10/32] [Spec 1280] Specification with CMAP round 2: nine findings, none disputed Architect-directed re-review of the revised spec. Two arithmetic errors caught that would otherwise have shipped: - M1's HOLD-at-50-52% branch was unreachable. M1 is fully derived from the M2 ceilings (met => <=16,016 = -53.2%); the HOLD band needs 16,442-17,128, i.e. M2 already failing. Recomputed and confirmed; replaced with the reachable contingency (denominator movement). - M5 proved nothing about prompts: it extracted gate/check names from an unchanged protocol.json and notifications from unchanged call sites, so every capability would report present even if every instruction vanished from the served prompts. Now inventories the resolved, expanded prompt surface with a contract-presence assertion per capability. Also: - M0c/M0(g)/T15: the metric could not distinguish deleted words from words relocated to skills - phantom savings on an axis T2 does not cover. - Coverage restated per-surface (enumerated from disk, unioned) - 'all ten protocols in both trees' is impossible: release is project-local (10 vs 9) and four protocols intentionally lack prompts/consult-types. - Inventory gap fixed: codev/protocols/release/protocol.md (1,626w) was missing because I enumerated the skeleton, where release does not exist. - M5 removal severity resolved on M10's committed-retirements pattern. - A/B execution: M7 gates verify-approval not the PR; treatment arm merges and control closes; cost restated as ~6 of 12 runs producing merged work; architect load (~24 gates + 12 PR reviews) named as the binding constraint. - O3 split: pre-merge findings gate SHIP, post-merge defects are a 14-day rollback signal. - M2b: nothing protected CLAUDE.md's human readability at 5,815 -> 1,900. Three of this phase's four self-audit findings share one root cause: enumerating from a convenient source instead of the authoritative one. --- .../1280-specify-iter2-rebuttals.md | 105 ++++++++ .../1280-prompt-surface-judgment-not-ru.md | 255 +++++++++++++++--- codev/state/spir-1280_thread.md | 40 +++ 3 files changed, 357 insertions(+), 43 deletions(-) create mode 100644 codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md new file mode 100644 index 000000000..77c486c19 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md @@ -0,0 +1,105 @@ +# Iteration 2 response — Spec 1280 (CMAP round 2) + +**No disputes filed.** All nine findings verified and accepted. Both arithmetic claims were +independently recomputed before acceptance rather than taken on trust. + +Round 2 existed because the architect directed it — porch had advanced to the gate after the +round-1 rebuttal without re-reviewing. It found two arithmetic errors that would otherwise +have shipped, which settles the question of whether it was worth running. + +--- + +## Codex + +**C1. Segment arithmetic underspecified** — ACCEPTED. Audience loads overlap (SHARED rides in +both architect and builder) and use different multipliers, so T11's "subtotals sum to the +total" was false as written. Current State now separates **exclusive buckets** (a partition — +these sum) from **derived audience loads** (overlap by design), with the explicit +`ALWAYS_ON_WORDS` formula. M0(f) and T11 restated accordingly. + +**C2. "All ten protocols in both trees" is impossible** — ACCEPTED, verified on disk. +`codev/protocols/` has ten; `codev-skeleton/protocols/` has nine (`release` is project-local +by design). `experiment`/`research`/`spike`/`release` have no `prompts/`; +`experiment`/`research`/`spike`/`release` have no `consult-types/`. Coverage restated as +per-**surface**, enumerated from disk across both trees and unioned: absence never fails, +unmeasured presence does. M2/M3/T6 rewritten. + +**Consequence I owe you separately**: this exposed that `codev/protocols/release/protocol.md` +(1,626w) was missing from my inventory entirely, because I enumerated +`codev-skeleton/protocols/*/` and `release` lives only in `codev/`. Now in M3 and Current +State, with the cause recorded. Same root cause as round 1's truncated grep: enumerating from +a convenient source rather than the authoritative one. + +**C3. M5 does not prove prompt capability preservation** — ACCEPTED, and the sharpest finding +of the round. Gate and check names extracted from an unchanged `protocol.json`, and +notification names from unchanged call sites, remain present even if every corresponding +instruction disappears from the served prompts — the inventory would have reported success +while measuring files I am not touching. M5 now inventories the **resolved, expanded prompt +surface**, with a contract-presence assertion per capability, and is named as the primary +defence for the most aggressive row in the table (`protocol.md` −81%). + +**C4. M5 internally contradictory on removals** — ACCEPTED (Claude raised this independently). +Resolved on M10's pattern: hard fail unless the retired name is listed in a committed +`codev/resources/1280-retirements.md` in the same commit, with the reason and architect +approval. An approved listed retirement passes; anything else fails. + +**C5. O3 timing ambiguous** — ACCEPTED. "Any post-merge defect" cannot be evaluated at a +pre-merge SHIP decision. Split: pre-merge architect findings gate SHIP; post-merge defects are +a **14-day rollback signal**. Which arm merges is now answered under C5's sibling finding +below. + +--- + +## Claude + +**L1. Nothing distinguishes deleted words from relocated words** — ACCEPTED, and the most +important finding of the round. Principle 4 authorizes relocation to skills, and relocating +3,900 words scores identically to deleting them under an always-on-only metric. A −53.2% +headline is equally consistent with −30% deleted + −23% relocated, and only *deleted* content +satisfies Problem Statement claim 1 — relocated content still enters context when looked up. +This is the phantom-savings class T2 catches on the include axis, unmonitored on the +relocation axis, and my own principle 7 requires the instrument to show it. Added **M0(g)** +(report total authored surface), **M0c** (decompose the cut into deleted vs relocated), and +**T15** (fixture: moving a block to a skill must show always-on falling while total-authored +holds steady). + +**L2. A/B execution model undefined** — ACCEPTED. Three unanswered operational questions, now +answered in a new **Execution and sequencing** subsection: +- **M7 gates `verify-approval`, not the PR merge** — matching where the rollback triggers + already point, keeping a 12-run trial off the PR's critical path, and making "treatment arm + = what builders actually get" literally true. Consequence stated plainly: a SHIP failure + means rolling back a merged change, which is what the grouped rollback plan is for. +- **Arm disposition** — the treatment arm's PR is the merge candidate; the control arm's + closes unmerged after its outcomes are recorded. The cost defence is corrected: **~6 of 12 + runs produce merged work, not 12.** +- **Architect load** — ~24 gate approvals + 12 PR reviews by one person, each SPIR gate + requiring O1 rubric scoring at approval time. Named as the trial's binding scheduling + constraint and the reason the pair count is the architect's call. + +**L3. M5 severity contradiction** — same as C4. + +**L4. M1's HOLD branch is arithmetically unreachable** — ACCEPTED, verified by recomputation. +Meeting every ceiling yields ≤16,016 (−53.2%); the 50–52% band requires a total of +16,442–17,128, i.e. ceilings already exceeded and M2 already failing. The branch was dead +prose. Withdrawn and replaced with the *reachable* contingency: **denominator movement** — if +correcting the instrument surfaces always-on content not yet found (as it has twice), the +baseline and every ceiling are re-derived to preserve >50%, and that goes to the architect +rather than being absorbed silently. + +**Smaller notes, all adopted**: M2b added (nothing protected CLAUDE.md's human readability at +5,815 → 1,900 — humans are named stakeholders with no criterion behind them; the architect now +reviews usability at the gate); the fleet-wide consultant figure stated (≈20,500 words/project, +comparable to the entire builder load, so "−33.8% on 683" does not read as negligible); and the +hot-tier exemption quantified as a ceiling on the project (7,360 of 16,016 post-rewrite builder +words = 46%). + +--- + +## Net effect + +Nine findings, none disputed, two of them arithmetic errors caught only because the architect +insisted on a round the machinery had skipped. The spec's own principle 7 — instruments get +reviewed against what they claim to measure — has now caught four errors in this spec, three +sharing one root cause: enumerating from a convenient source instead of the authoritative one. +That pattern belongs in the review's lessons learned, and it is the argument for M3's +"enumerate from disk" requirement being a test rather than an instruction. diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md index 4273f1d08..a3a7df1d4 100644 --- a/codev/specs/1280-prompt-surface-judgment-not-ru.md +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -10,7 +10,7 @@ phases, belongs in codev/plans/1280-prompt-surface-judgment-not-ru.md. ## Metadata - **ID**: spec-2026-07-31-prompt-surface-judgment-not-rules -- **Status**: draft (iteration 2 — CMAP round 1 incorporated) +- **Status**: draft (iteration 3 — CMAP rounds 1 and 2 incorporated) - **Created**: 2026-07-31 - **Issue**: #1280 - **Protocol**: SPIR @@ -83,11 +83,15 @@ It is load-bearing: this project's headline criterion is a number one shell scri Discovering *before* drafting that the metric was structurally blind to the project's own largest target is the correct order of operations, and is why M0 precedes M1. -**The principle applied to this spec, iteration 1 → 2.** CMAP round 1 found two further -instrument-class errors in my own Current State — a stale claim inherited from the script's -comments, and a conclusion drawn from a truncated grep. Both are corrected below and both -are recorded, not quietly fixed: a spec that argues instruments get audited must show its -own being audited. +**The principle applied to this spec.** CMAP round 1 found two instrument-class errors in my +own Current State — a stale claim inherited from the script's comments, and a conclusion drawn +from a truncated grep. Round 2 found two more: an inventory that missed a whole protocol +(enumerated from the skeleton, where `release` does not exist) and a metric that could not +tell deletion from relocation. All four are corrected below and all four are recorded, not +quietly patched: a spec that argues instruments get audited must show its own being audited. +The pattern in three of the four is identical — **enumerating from a convenient source instead +of the authoritative one** — which is the sweep-scope failure class 1252 named as its dominant +review cost. ## Current State @@ -153,17 +157,70 @@ air pr 455 / impl 420; maintain 421 / 392. **SPIR consultant per review: 683.** `codev-skeleton/porch/prompts/**` — 10 files, 4,009 words, no runtime consumer. -### Always-on load, by audience +### Buckets and audience loads — two different things, defined separately -| Audience | Composition | Words | +*(CMAP round 2 found the iteration-2 table conflated them: audience loads overlap — SHARED +rides in both the architect's and the builder's load — and use different multipliers, so +"subtotals sum to the total" was false as written.)* + +**Surface buckets are exclusive.** Every prompt-bearing surface belongs to exactly one, and +the buckets partition the authored prompt surface with no overlap and no gap: + +| Bucket | Contents | Words | |---|---|---:| -| **Builder** (SPIR, I=10) | 6,551 session + 6,364 spawn + 10×(736 + 1,398) | **34,255** | -| **Architect** (per session) | 6,551 session + 2,048 role | **8,599** | -| **Consultant** (per review) | 252 role + 431 mean consult-type | **683** | +| `SHARED` | `CLAUDE.md` + transcluded hot tier (`AGENTS.md` excluded — twin) | 6,551 | +| `ARCHITECT` | `roles/architect.md` (skills are on-demand, tracked but not always-on) | 2,048 | +| `BUILDER_SPAWN[p]` | `roles/builder.md` + `protocols/p/builder-prompt.md` + `protocols/p/protocol.md` | per protocol | +| `PHASE[p]` | `protocols/p/prompts/*.md` expanded (hot tier counted separately as `HOT`) | per protocol | +| `CONSULTANT[p]` | `roles/consultant.md` + `protocols/p/consult-types/*.md` | per protocol | +| `DEAD` | `codev-skeleton/porch/prompts/**` | 4,009 | + +**Audience loads are derived** by explicit formula and deliberately overlap: + +``` +HOT = arch-critical + lessons-critical = 736 +ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p] + I × (HOT + mean PHASE[p]) +ALWAYS_ON(architect) = SHARED + ARCHITECT +ALWAYS_ON(consultant,p) = roles/consultant.md + mean CONSULTANT-type[p] + +ALWAYS_ON_WORDS ≡ ALWAYS_ON(builder, spir, 10) ← the single headline number + = 6,551 + 6,364 + 10 × (736 + 1,398) = 34,255 +``` -The builder figure is the headline. `I = 10` phase-task deliveries is 1252's proxy, -consistent with B4's 3.06 review rounds/project across 4–6 phases; it is a *comparison* -constant, identical before and after. +| Audience | Value | +|---|---:| +| **Builder** (SPIR, I=10) — *the headline* | **34,255** | +| Architect (per session) | 8,599 | +| Consultant (per review) | 683 | + +Fleet-wide the consultant surface is not negligible: 683 × 3 models × ~10 reviews ≈ **20,500 +words per project**, comparable to the entire builder always-on load. It sits outside the +headline because it is per-review rather than per-builder, not because it is small. + +`I = 10` is 1252's proxy, consistent with B4's 3.06 review rounds/project across 4–6 phases. +It is a *comparison* constant, identical before and after. + +### Protocol coverage is per-surface, not per-protocol × surface-type + +*(CMAP round 2: "all ten protocols in both trees" is impossible in this repository, and +demanding it would make M2/M3 unsatisfiable.)* Verified on disk: + +- `codev/protocols/` holds **ten** protocols; `codev-skeleton/protocols/` holds **nine** — + `release` is **project-local by design** and has no skeleton twin. +- `release` has *only* `protocol.md` (1,626 words): no `protocol.json`, no + `builder-prompt.md`. It is human-invoked prose an agent reads ("Let's release v1.6.0"), so + it **is** prompt surface and **is** in scope — but it is not porch-orchestrated or + spawnable. **My iteration-2 inventory missed it entirely**, because I enumerated + `codev-skeleton/protocols/*/` and `release` lives only in `codev/`. Second sweep-scope + failure of this spec phase, same root cause as the truncated grep: enumerating from a + convenient source instead of the authoritative one. +- `experiment`, `research`, `spike`, `release` have **no** `prompts/`; `experiment`, + `research`, `spike`, `release` have **no** `consult-types/`. These absences are + intentional, not gaps. + +Therefore ceilings and sweep criteria apply to **each surface that exists after resolution**, +enumerated from disk across both trees and unioned. Absence of a surface is legitimate and +must never fail a check; presence of an unmeasured surface must. ### The measurement defects (why M0 exists) @@ -359,9 +416,21 @@ canonical text total). selection, so mixed per-file overrides measure correctly; (c) counts the inlined `roles/builder.md`; (d) counts hot-tier `@import` transclusion in the session term and **corrects the stale inlining comment**; (e) expands `{{> …}}` includes; (f) reports - **per-segment subtotals — architect / builder / phase / consultant — alongside the - total**. Tests assert (a) and (b) against the real resolver, so these defects cannot - silently return. + **exclusive bucket subtotals and the derived audience loads separately**, per the + formulas in Current State, never presenting overlapping audience figures as a sum; + (g) reports **total authored prompt-surface words** (both trees + `.claude/skills/`) + alongside always-on, so relocation is visible. Tests assert (a) and (b) against the + real resolver, so these defects cannot silently return. +- [ ] **M0c — deleted words are distinguished from relocated words.** Principle 4 authorizes + moving how-to content into skills, and relocation scores identically to deletion under + an always-on-only metric — the phantom-savings class T2 catches on the *include* axis, + unmonitored on the *relocation* axis. A −53% headline is equally consistent with −30% + deleted + −23% relocated, and only deleted content satisfies Problem Statement claim 1 + ("it crowds out judgment"); relocated content still enters context when looked up. The + review therefore **decomposes the always-on reduction into deleted vs relocated**, + evidenced by the total-authored figure from M0(g). Required by this spec's own + principle 7. *(Round-2 finding; the script already computes the skeleton and skills + totals, so this is a reporting line, not new machinery.)* - [ ] **M0b — the corrected instrument and baseline land on `main` early**, as a small standalone PR (precedent: #1290), not at the end of this branch. `1252-word-baseline.md` and `1252-word-after-phase7.md` cite figures derived from the @@ -369,25 +438,59 @@ canonical text total). place — original figures preserved, marked superseded, with the reason — and does not rewrite their history. PR-1's exact contents, timing, and safety verification are specified in **Desired State → Rollout**. -- [ ] **M1 — >50% reduction (derived).** Builder always-on falls from the corrected baseline - (34,255) to **≤16,100**, measured before and after by the same corrected script, both - figures committed as generated artifacts. Per-segment figures reported and none - regressed. -- [ ] **M2 — per-surface ceilings met (binding).** Every row of the ceiling table, gross, in - **both** trees, for **all ten protocols**. -- [ ] **M3 — sweep completeness.** Protocols enumerated from disk, not a hardcoded list; no - protocol retains a pre-rewrite `protocol.md`, `builder-prompt.md`, prompt set, or - consult-type set. +- [ ] **M1 — >50% reduction (derived, arithmetically implied by M2).** `ALWAYS_ON_WORDS` + falls from 34,255 to **≤16,100**, measured before and after by the same corrected + script, both figures committed. Audience loads reported; none regressed. + **Reachability, stated honestly** *(round-2 finding)*: M1 is fully derived from the M2 + ceilings — meeting every ceiling yields ≤16,016 (−53.2%), so M1 cannot fail while M2 + passes. Iteration 2's "HOLD at 50–52%" branch was dead prose and is withdrawn. The + *reachable* contingency is **denominator movement**: if correcting the instrument + surfaces always-on content not yet found — as it already has twice — the baseline and + every ceiling are re-derived to preserve >50%, and the re-derivation goes to the + architect rather than being absorbed silently. +- [ ] **M2 — per-surface ceilings met (binding).** Every ceiling, **gross**, on **every + surface that exists after resolution**, enumerated from disk across both trees and + unioned — not a per-protocol × surface-type cross product. *(Round-2 correction: + `codev/protocols/` has ten protocols and `codev-skeleton/protocols/` nine — `release` + is project-local by design — and `experiment`/`research`/`spike`/`release` have no + `prompts/` or `consult-types/`. Demanding "all ten in both trees" was unsatisfiable.)* + Absence of a surface never fails; presence of an **unmeasured** surface does. +- [ ] **M2b — CLAUDE.md stays human-readable.** Humans are named stakeholders, and no + criterion protected them at 5,815 → ≤1,900. The rewritten file must retain a + navigable heading structure and be reviewed by the architect for human usability at + the gate — not merely pass the twin-parity byte check. *(Round-2 finding.)* +- [ ] **M3 — sweep completeness.** Surfaces enumerated from disk (both trees, unioned), not + a hardcoded list; no surface retains pre-rewrite content. Includes + `codev/protocols/release/protocol.md` (1,626w) — human-invoked prose an agent reads, + no skeleton twin, missed by iteration 2's inventory. - [ ] **M4 — scar rules intact.** Eight canonicals byte-identical on every registered surface; test fails on reword or deletion; count pinned at 8. -- [ ] **M5 — no capability lost, deterministically checked.** A committed - `capability-inventory.json` is extracted pre-rewrite by a script with **explicit - recognition rules**: artifact paths (`codev/(specs|plans|reviews)/…` literals and - `{{artifact_name}}` forms), gate names (from `protocol.json` `gate:` fields), signal - names (`` tags), porch check names (`protocol.json` `checks:` ids), - notification triggers (`afx send architect` call sites). Normalization: lowercase, - strip backticks/punctuation, dedupe. Post-rewrite extraction must be a superset; - **any removal fails** and must be justified in the review as a deliberate retirement. +- [ ] **M5 — no capability lost, proven against the *prompt text*.** A committed + `capability-inventory.json` is extracted pre-rewrite with **explicit recognition + rules** — artifact paths (`codev/(specs|plans|reviews)/…` literals and + `{{artifact_name}}` forms), gate names, signal names (`` tags), porch check + names, notification triggers — normalized (lowercase, strip backticks/punctuation, + dedupe). + + **The inventory is over the resolved, expanded prompt surface, not over + `protocol.json` or source call sites** *(round-2 correction)*. Extracting gate and + check names from an unchanged `protocol.json`, or notification names from unchanged + `afx send` call sites, would report every capability as present even if every + corresponding instruction vanished from the served prompts — proving nothing about the + thing being cut. Each inventory item must therefore be evidenced as **represented in + the served prompt text**: for every gate, check, artifact contract, signal, and + notification trigger, a contract-presence assertion that the rewritten prompts still + tell the agent about it. This is also the primary defence for the most aggressive row + in the table, `protocol.md` 3,703 → ≤700 (−81%), which is the builder's only map of + gates, artifacts, and phases. + + **Severity, unambiguous** *(both reviewers, independently)*: "any removal fails" and + "justified as a deliberate retirement" are mutually exclusive. Resolved on M10's + pattern — **a removal is a hard failure unless the retired name appears in a committed + `codev/resources/1280-retirements.md` in the same commit**, naming the capability, why + it is obsolete, and the architect approval. An approved, listed retirement passes M5; + anything else fails. Retirements *will* occur (M6 deletes a tree; `protocol.md` drops + 3,000 words), so the exception path must be explicit rather than improvised. - [ ] **M6 — the dead tree is gone, with its consumer handled.** `codev-skeleton/porch/prompts/` deleted. Verification is **not** a bare grep: an untruncated repo-wide search (`grep -rn … | wc -l` reconciled against the full hit @@ -549,7 +652,9 @@ the reader, and only after the shrink. what makes the artifact contract legible without narration. - [ ] **Is the hot tier's 736 genuinely exempt?** It is the one surface already built to these principles. The spec exempts it; a reviewer may argue it should be re-derived - post-shrink, and it is 63% of the phase term. + post-shrink. Note the ceiling this imposes: the exempt hot tier is **7,360 of the + 16,016-word post-rewrite builder budget (46%)**, so it bounds how far this project can + go without reopening the exemption. - [ ] **`roles/architect.md` 2,048 → ≤700** — the architect segment was outside 1252's analysis entirely. Confirm nothing in it is load-bearing for multi-architect coordination (Specs 755/786/823) before cutting. @@ -604,8 +709,10 @@ which already runs the ceilings as a test. T3 stands.)* count pinned at 8; reword or deletion fails. 6. **T5 — Capability inventory (M5).** Post-rewrite extraction ⊇ pre-rewrite, using M5's recognition and normalization rules; removals fail. -7. **T6 — Sweep completeness (M3).** Protocols enumerated from disk; each satisfies T3; a - newly added protocol fails until written to budget. +7. **T6 — Sweep completeness (M3).** Surfaces enumerated from disk across both trees and + unioned; each existing surface satisfies T3. Absence of `prompts/`/`consult-types/` for a + protocol that has none must **not** fail; a newly added protocol or surface fails until + written to budget. Covers `release` (project-local, `codev/` only). 8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; `codev/` and `codev-skeleton/` copies consistent. 9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing @@ -615,9 +722,14 @@ which already runs the ceilings as a test. T3 stands.)* `porch next` returns a well-formed task. ("It compiled" is not "it works.") 11. **T10 — Rollback rehearsal (M9).** Reverting a rollback **group** on a scratch branch restores that group byte-for-byte and leaves the suite green. -12. **T11 — Segment reporting (M0f).** The script emits architect/builder/phase/consultant - subtotals that sum to the total; a fixture where one segment grows and another shrinks - shows both movements, not a netted zero. +12. **T11 — Bucket and audience reporting (M0f).** The script emits the six **exclusive + bucket** subtotals, which sum to the authored prompt-surface total, *and separately* the + derived audience loads, which overlap by design and are asserted against the stated + formulas rather than against a naive sum. A fixture where one bucket grows and another + shrinks shows both movements, not a netted zero. +13. **T15 — Relocation visibility (M0c).** A fixture that moves a block from an always-on + surface into `.claude/skills/` shows always-on falling **and** total-authored holding + steady — so relocation can never be reported as deletion. ### Non-Functional Tests @@ -667,13 +779,38 @@ checkouts. base commit, order, isolation mode, every outcome value, and any exclusion with its reason. Exclusions after the fact must be justified in that file, not silently dropped. +### Execution and sequencing + +*(Round-2 finding, raised independently by both reviewers: M7 sat among the criteria with no +stated position in the sequence, and three operational questions had no answer.)* + +- **M7 gates `verify-approval`, not the PR merge.** The rewrite PR merges on the strength of + M0–M6 and M8–M10 plus architect review; the A/B then runs against merged `main` as the + treatment arm and a pinned pre-rewrite commit as control. This matches where the rollback + triggers already point, keeps a 12-run trial off the PR's critical path, and is the only + ordering under which "treatment arm = what builders actually get" is literally true. + Consequence, stated plainly: a SHIP failure means **rolling back a merged change**, which + is precisely what the grouped, rehearsed rollback plan exists for. +- **Arm disposition — one arm merges per pair.** Each pair produces two real implementations + of one backlog issue. **The treatment arm's PR is the candidate for merge; the control + arm's is closed unmerged after its outcomes are recorded** (and vice versa if the treatment + arm's work is defective on O3). The discarded arm is not wasted — it is the comparison — + but the cost defence must be stated accurately: **≈6 of 12 runs produce merged work, not + 12.** The trial's real cost is 6 duplicated implementations plus consult spend. +- **Architect load is a scheduling dependency, not a footnote.** 6 pairs with ≥3 SPIR-class + implies up to **~24 gate approvals and 12 PR reviews by one person**, each SPIR gate + requiring O1 rubric scoring *at approval time*. This is the trial's binding constraint and + the reason the pair count is the architect's call, not the builder's. If capacity forces a + smaller n, the honest consequence is stated in the power paragraph — not a quietly reduced + sample. + ### Pre-registered outcomes | ID | Outcome | Instrument | Role in decision | |---|---|---|---| | **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each scored 0 (no friction) / 1 (minor) / 2 (blocking), recorded in the results artifact at scoring time | **advisory + tripwire** | | **O2** | Review rounds | Iterations to terminal state per phase from `status.yaml` history; CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | gate | -| **O3** | Correctness | Architect PR-review findings by severity; any post-merge defect attributable to the run | gate | +| **O3** | Correctness | **At the SHIP decision**: architect PR-review findings by severity, observable pre-merge on both arms. **Post-merge defects are excluded from the SHIP gate** — they cannot be evaluated when the decision is made — and instead act as a **later rollback signal over a 14-day window** after the merged arm lands *(round-2 correction: the original wording asked one metric to be evaluated at two incompatible times)* | gate (pre-merge part) + rollback signal (post-merge part) | | **O4** | Protocol compliance | Binary per-run checklist: required artifacts with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar violation · thread committed | **zero tolerance** | | **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory | @@ -691,8 +828,9 @@ a rubric written **before** any run. missing required artifact is an immediate hard stop, independent of everything else. 2. **O2**: treatment mean review rounds ≤ control **+ 0.5 rounds/phase**, and treatment REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. -3. **O3**: no treatment-arm finding of severity ≥ "would block merge" absent from its paired - control run. +3. **O3 (pre-merge part only)**: no treatment-arm architect finding of severity ≥ "would + block merge" absent from its paired control run. Post-merge defects do not enter this + decision; they are a 14-day rollback signal. 4. **O1 tripwire**: no pair where the treatment arm scored **2 (blocking)** at a gate its control scored 0, for the same reason. If O1 scoring is incomplete for any pair, O1 is reported incomplete and SHIP rests on 1–3. @@ -812,6 +950,37 @@ porch's model set for this consultation — the known `--type` review limitation **Not disputed.** Every round-1 finding was accepted; no rebuttal was filed. Two were factual errors in my own Current State, both verified against source before correction. +### Round 2 — 2026-07-31 (architect-directed re-review of the revised spec) + +Porch had advanced to the gate after the round-1 rebuttal without re-reviewing; the architect +directed a round 2 on the grounds that a revision absorbing 13 findings — including a +structural net→gross ceiling change — needs re-reading, and that porch advancing was +permissive machinery rather than a judgment. Correct call: round 2 found nine further issues, +two of them arithmetic errors that would have shipped. + +**Verdicts**: Codex REQUEST_CHANGES (HIGH) · Claude REQUEST_CHANGES (HIGH). Both again +verified the spec's factual claims against source before judging; Claude re-confirmed every +word count and the `REQUIRED_SPEC_SECTIONS`, `spawn-worktree.ts:854`, and +no-test-on-1252-artifacts claims. + +| Finding | Raised by | Resolution | +|---|---|---| +| Segment arithmetic underspecified — audience loads overlap and use different multipliers, so "subtotals sum to the total" is false | Codex | Current State split into **exclusive buckets** (partition, summable) vs **derived audience loads** (overlap by design), with the exact `ALWAYS_ON_WORDS` formula; M0(f) and **T11** restated | +| "All ten protocols in both trees" is impossible — `release` is project-local (10 vs 9), and several protocols intentionally lack `prompts/`/`consult-types/` | Codex | **Verified on disk.** Coverage restated as per-**surface**, enumerated from disk and unioned; absence never fails, unmeasured presence does. M2/M3/T6 rewritten | +| `release/protocol.md` (1,626w) missing from the inventory entirely | *(consequence of the above)* | Added; cause recorded — I enumerated the skeleton, where `release` does not exist. Second sweep-scope failure this phase, same root cause as the truncated grep | +| M5 proves nothing about prompts — gates/checks come from unchanged `protocol.json`, notifications from unchanged call sites | Codex | M5 rewritten to inventory the **resolved, expanded prompt surface** with contract-presence assertions per capability; named as the primary defence for the −81% `protocol.md` row | +| M5 self-contradictory: "any removal fails" vs "justified retirement" | **both** | Resolved on M10's pattern — hard fail unless the name is in a committed `1280-retirements.md` in the same commit, with architect approval | +| O3 timing ambiguous — post-merge defects cannot be evaluated at a pre-merge SHIP decision | Codex | Split: pre-merge architect findings gate SHIP; post-merge defects are a **14-day rollback signal** | +| A/B execution model undefined — what M7 gates, which arm merges, architect load | **both** | New **Execution and sequencing** subsection: M7 gates `verify-approval` (not the PR); treatment arm merges, control closes; cost restated honestly as ~6 of 12 runs producing merged work; ~24 gate approvals + 12 PR reviews named as the binding scheduling constraint | +| Relocation to skills scores identically to deletion — phantom savings on an axis T2 does not cover | Claude | **New M0c + M0(g) + T15**: report total authored surface, decompose the cut into deleted vs relocated. Caught by this spec's own principle 7 | +| M1's "HOLD at 50–52%" branch is arithmetically unreachable — M1 is fully derived from M2 | Claude | **Verified** (ceilings met ⇒ ≤16,016 = −53.2%; the HOLD band needs 16,442–17,128, i.e. M2 already failing). Dead prose withdrawn; replaced with the reachable contingency, **denominator movement** | +| No criterion protects CLAUDE.md's human readability at 5,815 → 1,900 | Claude | **New M2b** — architect reviews human usability at the gate; twin-parity bytes are not a readability check | +| Consultant surface reads as negligible at "−33.8% on 683" | Claude | Fleet-wide figure added: ≈20,500 words/project, comparable to the whole builder load | +| Exempt hot tier bounds the achievable cut | Claude | Quantified in the open question: 7,360 of 16,016 post-rewrite builder words (46%) | + +**Not disputed.** Every round-2 finding was accepted; no rebuttal filed. Both arithmetic +claims (M1 reachability, ceiling landing) were independently recomputed before acceptance. + ## Approval - [ ] Technical Lead Review - [ ] Product Owner Review diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 2780b94c3..be8c8a5f6 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -171,3 +171,43 @@ Architect directed two items before the gate goes to Waleed: Porch remains at the spec-approval gate throughout; running consults manually does not move state, and I am not touching the gate. + +### CMAP round 2 (2026-07-31) — the round porch skipped, and why it mattered + +Both REQUEST_CHANGES (HIGH). **Nine findings, none disputed, two of them arithmetic errors +that would otherwise have shipped.** The architect's insistence on running a round porch had +skipped is fully vindicated. + +The two that would have shipped: + +1. **M1's HOLD branch was unreachable.** M1 is fully derived from the M2 ceilings: meeting + every ceiling yields ≤16,016 (−53.2%), while the "HOLD at 50–52%" band needs + 16,442–17,128 — i.e. ceilings already exceeded and M2 already failing. Recomputed and + confirmed before accepting. Dead prose replaced with the reachable contingency + (denominator movement). +2. **M5 proved nothing about prompts.** It extracted gate/check names from `protocol.json` + and notification names from source call sites — files this project does not touch. Every + capability would have reported present even if every corresponding instruction vanished + from the served prompts. Now inventories the resolved, expanded *prompt surface* with a + contract-presence assertion per capability. + +Claude's best structural catch: **the metric cannot tell deletion from relocation.** +Principle 4 authorizes moving how-to content to skills, and relocation scores identically to +deletion under an always-on-only metric — the phantom-savings class T2 catches on the include +axis, unmonitored on the relocation axis. −53.2% is equally consistent with −30% deleted + +−23% relocated, and only deleted content supports the "crowds out judgment" claim. Added M0(g), +M0c, T15. + +Codex's best: **"all ten protocols in both trees" is impossible** — `release` is project-local +(10 in `codev/`, 9 in the skeleton), and four protocols intentionally have no `prompts/` or +`consult-types/`. Which exposed that **my inventory missed `release/protocol.md` (1,626w) +entirely**, because I enumerated `codev-skeleton/protocols/*/`. + +**Fourth self-audit finding of this spec phase, and three share one root cause**: enumerating +from a convenient source instead of the authoritative one (truncated grep → false "no +consumers"; skeleton-only enumeration → missing protocol; script's stale comment → wrong +baseline). This is exactly the sweep-scope class 1252 named as its dominant review cost, and +it is the argument for making M3's "enumerate from disk" a *test* rather than an instruction. +Belongs in the review's lessons learned. + +Spec now 9,415w (iteration 3). Porch remains at spec-approval; I have not touched the gate. From 58c03baf01b688c16527e74d93fbae93bce289bf Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:23:15 -0700 Subject: [PATCH 11/32] [Spec 1280] Compress for signal: 9,415 -> 6,840 words MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Architect-directed pre-gate compression. Folded, not dropped: every M-criterion (14), every test (16), the full inventory, the rollout section, the A/B design, the rollback groups, and the Baked Decisions verbatim all survive unchanged. Cut: the narration of how the spec got here — round-by-round finding tables, '(round-2 finding)' asides, the superseded 33,519 figure, and the withdrawn HOLD-band arithmetic. That record lives in the iter1/iter2 rebuttal files and the thread, which is where it belongs. Self-verified: all 20 template headings present and ordered; 14/14 criteria, 16/16 tests, 5/5 outcomes, 7/7 rollback groups; every load-bearing number reproduced; porch's spec_has_required_sections passes. --- .../1280-prompt-surface-judgment-not-ru.md | 1139 +++++++---------- 1 file changed, 458 insertions(+), 681 deletions(-) diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md index a3a7df1d4..452523d54 100644 --- a/codev/specs/1280-prompt-surface-judgment-not-ru.md +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -6,117 +6,97 @@ This spec defines WHAT and WHY. The plan defines HOW and WHEN. Per-surface WORD CEILINGS are stated here because they are the measurable acceptance criteria. WHICH sentences get cut, in what order, over how many phases, belongs in codev/plans/1280-prompt-surface-judgment-not-ru.md. + +The record of how this spec evolved (13 findings in CMAP round 1, 9 in round 2, +none disputed) lives in codev/projects/1280-*/1280-specify-iter{1,2}-rebuttals.md +and codev/state/spir-1280_thread.md — deliberately not narrated here. --> ## Metadata - **ID**: spec-2026-07-31-prompt-surface-judgment-not-rules -- **Status**: draft (iteration 3 — CMAP rounds 1 and 2 incorporated) +- **Status**: draft (compressed; CMAP rounds 1–2 incorporated) - **Created**: 2026-07-31 - **Issue**: #1280 - **Protocol**: SPIR ## Clarifying Questions Asked -Issue #1280 carries a complete charter (goal, attack order, four Baked Decisions, -required prior art), so no clarifying questions were put to the architect before -drafting. Four questions were resolved against the repository or by architect ruling: - -1. **"What exactly is the always-on surface, as served?"** — Resolved by reading the live - composition path (`commands/porch/prompts.ts`, `lib/skeleton.ts`, `lib/managed-block.ts`, - `agent-farm/commands/spawn-worktree.ts`) and by measuring this builder's own served - artifacts (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). Full enumeration - in **Current State → Inventory**. -2. **"Is the committed measurement script fit to score this project?"** — No; three - defects, one of them disqualifying. Criterion **M0**. Architect verified both original - claims against source and **endorsed M0 as specced** (2026-07-31). -3. **"How wide is the rewrite target?"** — **Architect scope directive (2026-07-31)**: the - entire prompt surface, not CLAUDE.md/AGENTS.md — architect role prompts, builder roles - and spawn wrappers, consultant/CMAP review prompts, porch phase prompts and their - template includes, and `protocol.md` texts. The instrument must **segment by audience** - so a cut concentrated in one segment while another grows is visible, not averaged away. -4. **"Where are the eight ratified scar rules?"** — Recovered verbatim from - `builder/spir-1252:codev/resources/scar-rules.yaml`. Carriage plan in **Desired State**. +Issue #1280 carries a complete charter, so no clarifying questions were put to the architect +before drafting. Four were resolved against the repository or by architect ruling: + +1. **What is the always-on surface, as served?** — Read the live composition path + (`commands/porch/prompts.ts`, `lib/skeleton.ts`, `lib/managed-block.ts`, + `agent-farm/commands/spawn-worktree.ts`) and measured this builder's own served artifacts + (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). See **Inventory**. +2. **Is the committed measurement script fit to score this project?** — No; three defects, one + disqualifying. Criterion **M0**; architect verified against source and endorsed. +3. **How wide is the rewrite target?** — Architect scope directive: the **entire** prompt + surface — architect roles, builder roles and spawn wrappers, consultant/CMAP prompts, phase + prompts and their template includes, `protocol.md` texts — with the instrument **segmented + by audience** so a cut in one segment masking growth in another stays visible. +4. **Where are the eight ratified scar rules?** — Recovered verbatim from + `builder/spir-1252:codev/resources/scar-rules.yaml`. ## Problem Statement -A Codev builder consumes **~34,300 served always-on words** before it reads a line of the -code it was spawned to change. Almost none of that is information the builder could not -derive; it is *process narration* — recipes for how to be an agent, written when the fleet -could not be trusted to infer them. - -Spec 1252 measured the surface and proved the obvious remedy does not work: deduplication -yields **−7.0%**, because the surface is not duplicated, it is **over-instructed**. - -Over-instruction has three compounding costs: - -1. **It crowds out judgment.** A frontier model given a 3,700-word procedure follows the - procedure. Given a 700-word contract and a goal, it reasons about the goal. Anthropic's - published account of the Claude-5-generation rewrite reports >80% of Claude Code's - system prompt deleted with no measurable performance loss. -2. **Nobody reads it, so it rots unnoticed.** 1252 found the *served* SPIR builder prompt - had silently lost its entire `Verify Phase` section, with a detector reporting the drift, - unread, for months. A surface too large to read is too large to maintain. -3. **It makes its own success unmeasurable.** The committed measurement script scores a - directory the runtime never loads. Nothing about the prompts a builder actually receives - has been under measurement. - -Deletion on judgment-trust grounds was an explicit **Non-goal** of Spec 1252 and has never -been attempted here. - -### The instrument is part of the deliverable - -This is the **second** measurement defect in the 1252 lineage. The first: Spec 1252 -originally shipped with no measurement plan at all — caught at a human gate, not by CMAP. -The second is documented below: a committed, tested, reproducible script that measures the -wrong directory, so its numbers are precise and wrong. Neither was caught by reading the -instrument's code; both by asking *what does this claim to measure, and does it?* - -Hence an explicit project principle, adopted at the architect's direction and binding on -every criterion here: - -> **The instrument is part of the deliverable, and instruments get reviewed against what -> they claim to measure — not merely against whether they run.** A measurement script, a -> check, or a baseline artifact is subject to the same adversarial review as the feature it -> scores. "Deterministic and committed" is not "correct." - -It is load-bearing: this project's headline criterion is a number one shell script emits. -Discovering *before* drafting that the metric was structurally blind to the project's own -largest target is the correct order of operations, and is why M0 precedes M1. - -**The principle applied to this spec.** CMAP round 1 found two instrument-class errors in my -own Current State — a stale claim inherited from the script's comments, and a conclusion drawn -from a truncated grep. Round 2 found two more: an inventory that missed a whole protocol -(enumerated from the skeleton, where `release` does not exist) and a metric that could not -tell deletion from relocation. All four are corrected below and all four are recorded, not -quietly patched: a spec that argues instruments get audited must show its own being audited. -The pattern in three of the four is identical — **enumerating from a convenient source instead -of the authoritative one** — which is the sweep-scope failure class 1252 named as its dominant -review cost. +A Codev builder consumes **34,255 served always-on words** before reading a line of the code it +was spawned to change. Almost none is information it could not derive; it is *process +narration* — recipes for how to be an agent, written when the fleet could not be trusted to +infer them. + +Spec 1252 proved the obvious remedy fails: deduplication yields **−7.0%**, because the surface +is not duplicated, it is **over-instructed**. Three compounding costs: + +1. **It crowds out judgment.** A model given a 3,703-word procedure follows the procedure; + given a 700-word contract and a goal, it reasons about the goal. Anthropic's published + account of the Claude-5-generation rewrite reports >80% of Claude Code's system prompt + deleted with no measurable performance loss. +2. **Nobody reads it, so it rots.** 1252 found the *served* SPIR builder prompt had silently + lost its entire `Verify Phase` section, with a detector reporting the drift, unread, for + months. A surface too large to read is too large to maintain. +3. **It makes its own success unmeasurable.** The committed script scores a directory the + runtime never loads. + +Deletion on judgment-trust grounds was an explicit **Non-goal** of Spec 1252. + +### Principle: the instrument is part of the deliverable + +This is the second measurement defect in the 1252 lineage — the first being that 1252 +originally shipped with no measurement plan at all, caught at a human gate, not by CMAP. +Neither was found by reading the instrument's code; both by asking *what does this claim to +measure, and does it?* + +> **The instrument is part of the deliverable, and instruments get reviewed against what they +> claim to measure — not merely against whether they run.** A measurement script, check, or +> baseline artifact is subject to the same adversarial review as the feature it scores. +> "Deterministic and committed" is not "correct." + +Load-bearing here because this project's headline criterion is a number one shell script emits. +It is why **M0 precedes M1**, why **M0c** exists (an always-on metric cannot tell deletion from +relocation), and why **M5** inventories prompt text rather than the config files this project +does not touch. ## Current State ### Inventory — every prompt-bearing surface, by audience -Word counts are **served and expanded** (`{{> …}}` includes resolved through the -`codev/` → `codev-skeleton/` chain), captured 2026-07-31 at `047f92f7`. +Served and expanded words (`{{> …}}` includes resolved `codev/` → `codev-skeleton/`), captured +2026-07-31 at `047f92f7`. -**SHARED — every agent working in this repo** +**SHARED — every agent in this repo** -| Surface | Resolver path | Words | How served | -|---|---|---:|---| -| `CLAUDE.md` | repo root | 5,815 | session, harness auto-load | -| ↳ `@codev/resources/arch-critical.md` | four-tier | 416 | **transcluded at session launch** (#1119) | -| ↳ `@codev/resources/lessons-critical.md` | four-tier | 320 | same | -| `AGENTS.md` | repo root | 5,815 | byte-identical twin; one loads per session, never both | +| Surface | Words | How served | +|---|---:|---| +| `CLAUDE.md` | 5,815 | session, harness auto-load | +| ↳ `@codev/resources/arch-critical.md` | 416 | **transcluded at session launch** (#1119) | +| ↳ `@codev/resources/lessons-critical.md` | 320 | same | +| `AGENTS.md` | 5,815 | byte-identical twin; one loads per session, never both | **Session shared total: 6,551.** -**ARCHITECT** - -| Surface | Resolver path | Words | How served | -|---|---|---:|---| -| `roles/architect.md` | four-tier | 2,048 | read at `arch-init` (every architect session) | -| `.claude/skills/*/SKILL.md` ×10 | repo | 6,672 | on-demand (progressive disclosure — working as intended) | +**ARCHITECT** — `roles/architect.md` 2,048 (read at `arch-init`, every session); +`.claude/skills/*/SKILL.md` ×10 = 6,672 (on-demand — progressive disclosure working as +intended). **BUILDER — spawn, once per builder** @@ -124,20 +104,18 @@ Word counts are **served and expanded** (`{{> …}}` includes resolved through t |---|---:|---|---|---:| | `roles/builder.md` (inlined) | 1,837 | | `protocols/pir/protocol.md` | 2,066 | | `protocols/spir/protocol.md` | 3,703 | | `protocols/maintain/protocol.md` | 1,949 | -| `protocols/spir/builder-prompt.md` | 824 | | `protocols/research/protocol.md` | 1,278 | -| `protocols/pir/builder-prompt.md` | 898 | | `protocols/experiment/protocol.md` | 1,023 | -| `protocols/aspir/builder-prompt.md` | 820 | | `protocols/spike/protocol.md` | 920 | -| `protocols/research/builder-prompt.md` | 556 | | `protocols/aspir/protocol.md` | 810 | -| `protocols/air/builder-prompt.md` | 537 | | `protocols/bugfix/protocol.md` | 699 | -| `protocols/experiment/builder-prompt.md` | 472 | | `protocols/air/protocol.md` | 643 | -| `protocols/bugfix/builder-prompt.md` | 429 | | | | +| `protocols/spir/builder-prompt.md` | 824 | | `protocols/release/protocol.md` | 1,626 | +| `protocols/pir/builder-prompt.md` | 898 | | `protocols/research/protocol.md` | 1,278 | +| `protocols/aspir/builder-prompt.md` | 820 | | `protocols/experiment/protocol.md` | 1,023 | +| `protocols/research/builder-prompt.md` | 556 | | `protocols/spike/protocol.md` | 920 | +| `protocols/air/builder-prompt.md` | 537 | | `protocols/aspir/protocol.md` | 810 | +| `protocols/experiment/builder-prompt.md` | 472 | | `protocols/bugfix/protocol.md` | 699 | +| `protocols/bugfix/builder-prompt.md` | 429 | | `protocols/air/protocol.md` | 643 | | `protocols/spike/builder-prompt.md` | 400 | | `protocols/maintain/builder-prompt.md` | 374 | **SPIR builder spawn total: 6,364** (role 1,837 + wrapper 824 + protocol 3,703). -**PHASE — per porch task delivery, ×I** - -Hot tier (736) rides on *every* phase prompt. Expanded phase prompts: +**PHASE — per porch task delivery, ×I.** Hot tier (736) rides on *every* phase prompt. | Protocol | Prompts (expanded) | Mean | |---|---|---:| @@ -147,35 +125,17 @@ Hot tier (736) rides on *every* phase prompt. Expanded phase prompts: | air | pr 471 · implement 442 | 457 | | maintain | maintain 402 · review 310 | 356 | -**CONSULTANT — per CMAP review, ×3 models × ~10 reviews per project** - -`roles/consultant.md` 252 (system prompt) + one consult-type: spir/aspir spec-review 514 · -impl 421 · phase 421 · plan 406 · pr 392; bugfix pr 726 / impl 641; pir pr 475 / impl 507; -air pr 455 / impl 420; maintain 421 / 392. **SPIR consultant per review: 683.** +**CONSULTANT — per CMAP review.** `roles/consultant.md` 252 + one consult-type: spir/aspir spec +514 · impl 421 · phase 421 · plan 406 · pr 392; bugfix pr 726 / impl 641; pir pr 475 / impl +507; air pr 455 / impl 420; maintain 421 / 392. **SPIR per review: 683.** -**DEAD** +**DEAD** — `codev-skeleton/porch/prompts/**`, 10 files, 4,009 words, no runtime consumer. -`codev-skeleton/porch/prompts/**` — 10 files, 4,009 words, no runtime consumer. +### Buckets and audience loads are different things -### Buckets and audience loads — two different things, defined separately - -*(CMAP round 2 found the iteration-2 table conflated them: audience loads overlap — SHARED -rides in both the architect's and the builder's load — and use different multipliers, so -"subtotals sum to the total" was false as written.)* - -**Surface buckets are exclusive.** Every prompt-bearing surface belongs to exactly one, and -the buckets partition the authored prompt surface with no overlap and no gap: - -| Bucket | Contents | Words | -|---|---|---:| -| `SHARED` | `CLAUDE.md` + transcluded hot tier (`AGENTS.md` excluded — twin) | 6,551 | -| `ARCHITECT` | `roles/architect.md` (skills are on-demand, tracked but not always-on) | 2,048 | -| `BUILDER_SPAWN[p]` | `roles/builder.md` + `protocols/p/builder-prompt.md` + `protocols/p/protocol.md` | per protocol | -| `PHASE[p]` | `protocols/p/prompts/*.md` expanded (hot tier counted separately as `HOT`) | per protocol | -| `CONSULTANT[p]` | `roles/consultant.md` + `protocols/p/consult-types/*.md` | per protocol | -| `DEAD` | `codev-skeleton/porch/prompts/**` | 4,009 | - -**Audience loads are derived** by explicit formula and deliberately overlap: +**Surface buckets are exclusive** and partition the authored surface with no overlap or gap: +`SHARED` (6,551) · `ARCHITECT` (2,048) · `BUILDER_SPAWN[p]` · `PHASE[p]` · `CONSULTANT[p]` · +`DEAD` (4,009). **Audience loads are derived** and deliberately overlap: ``` HOT = arch-critical + lessons-critical = 736 @@ -183,129 +143,90 @@ ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p] + I × (HOT + mean PHASE[p] ALWAYS_ON(architect) = SHARED + ARCHITECT ALWAYS_ON(consultant,p) = roles/consultant.md + mean CONSULTANT-type[p] -ALWAYS_ON_WORDS ≡ ALWAYS_ON(builder, spir, 10) ← the single headline number - = 6,551 + 6,364 + 10 × (736 + 1,398) = 34,255 +ALWAYS_ON_WORDS ≡ ALWAYS_ON(builder, spir, 10) + = 6,551 + 6,364 + 10 × (736 + 1,398) = 34,255 ← the headline ``` -| Audience | Value | -|---|---:| -| **Builder** (SPIR, I=10) — *the headline* | **34,255** | -| Architect (per session) | 8,599 | -| Consultant (per review) | 683 | - -Fleet-wide the consultant surface is not negligible: 683 × 3 models × ~10 reviews ≈ **20,500 -words per project**, comparable to the entire builder always-on load. It sits outside the -headline because it is per-review rather than per-builder, not because it is small. - -`I = 10` is 1252's proxy, consistent with B4's 3.06 review rounds/project across 4–6 phases. -It is a *comparison* constant, identical before and after. - -### Protocol coverage is per-surface, not per-protocol × surface-type - -*(CMAP round 2: "all ten protocols in both trees" is impossible in this repository, and -demanding it would make M2/M3 unsatisfiable.)* Verified on disk: - -- `codev/protocols/` holds **ten** protocols; `codev-skeleton/protocols/` holds **nine** — - `release` is **project-local by design** and has no skeleton twin. -- `release` has *only* `protocol.md` (1,626 words): no `protocol.json`, no - `builder-prompt.md`. It is human-invoked prose an agent reads ("Let's release v1.6.0"), so - it **is** prompt surface and **is** in scope — but it is not porch-orchestrated or - spawnable. **My iteration-2 inventory missed it entirely**, because I enumerated - `codev-skeleton/protocols/*/` and `release` lives only in `codev/`. Second sweep-scope - failure of this spec phase, same root cause as the truncated grep: enumerating from a - convenient source instead of the authoritative one. -- `experiment`, `research`, `spike`, `release` have **no** `prompts/`; `experiment`, - `research`, `spike`, `release` have **no** `consult-types/`. These absences are - intentional, not gaps. - -Therefore ceilings and sweep criteria apply to **each surface that exists after resolution**, -enumerated from disk across both trees and unioned. Absence of a surface is legitimate and -must never fail a check; presence of an unmeasured surface must. +Architect load 8,599; consultant 683. `I = 10` is 1252's proxy, consistent with B4's 3.06 +review rounds/project across 4–6 phases — a comparison constant, identical before and after. + +Fleet-wide the consultant surface is not small: 683 × 3 models × ~10 reviews ≈ **20,500 +words/project**, comparable to the entire builder load. It sits outside the headline because it +is per-review, not because it is negligible. + +### Coverage is per-surface, not per-protocol × surface-type + +Verified on disk: `codev/protocols/` holds **ten** protocols, `codev-skeleton/protocols/` +**nine** — `release` is project-local by design, has only `protocol.md` (1,626w, no +`protocol.json`, no `builder-prompt.md`), is human-invoked prose an agent reads, and is +therefore in scope but not porch-orchestrated. `experiment`, `research`, `spike` and `release` +have no `prompts/` or `consult-types/`; those absences are intentional. + +Ceilings and sweep criteria therefore apply to **each surface that exists after resolution**, +enumerated from disk across both trees and unioned. Absence never fails a check; an +**unmeasured** present surface does. ### The measurement defects (why M0 exists) -**Defect 1 — the script measures a dead directory.** `measure-prompt-surface.sh:89` derives -`PORCH_PROMPT_MEAN` from `codev-skeleton/porch/prompts/*.md` (10 files, mean 400). The live -resolver (`commands/porch/prompts.ts:78`, `loadPromptFile`) loads -`protocols//prompts/.md`. Real SPIR phase prompts average **1,398**. The -dead tree is a Ralph-SPIR-era leftover — its `specify.md` opens *"You are the **Spec -Writer** hat in a Ralph-SPIR loop."* - -**Defect 2 — the script omits the inlined role.** `spawn-worktree.ts:854` writes -`roles/builder.md` to `.builder-role.md` and the harness injects it. 1,837 always-on words, -uncounted. - -**Defect 3 — the script's hot-tier accounting is stale.** Its comment (lines 44–47) asserts -CLAUDE.md "already inlines the two hot-tier files." Since #1119 (`managed-block.ts:59-67`) -CLAUDE.md carries `@import` lines, which Claude Code transcludes at session launch. So -`wc -w CLAUDE.md` = 5,815 **excludes** 736 words that are always loaded. *(Found by CMAP -round 1 — an instrument-class error I inherited from the instrument's own comments, which -is precisely the failure principle 7 names.)* - -Consequences: the reported baseline (21,702) understates the phase-task term ~3.5×, omits -the role file, and under-counts the session term by 736. **And the metric cannot see this -project's primary target** — cutting SPIR phase prompts from 1,398 to 430 moves -`ALWAYS_ON_WORDS` by exactly zero under the current script. A >50% claim scored on it would -be phantom savings, the precise failure 1252 built the script to prevent. - -**Correction to iteration 1 of this spec.** It claimed "a repo-wide grep finds no code -reading `porch/prompts`; every hit is historical spec/plan prose." That is **false**, and -the cause is mine: I piped `grep -rn` into `head -20` and drew a conclusion from truncated -output. `packages/codev/src/__tests__/review-prompt-routing.test.ts:29` pushes -`codev-skeleton/porch/prompts/review.md` onto its assertion list (a Spec 987 hot/cold -routing protection). The tree is still dead as *prompt surface* — no runtime consumer — but -it has a **test** consumer, so M6's verification method and deletion step change accordingly. -This is the sweep-scope failure class 1252 named as its dominant review-iteration cost, and -it is exactly what a truncated grep buys. - -### What already landed, and what is deferred - -On `main` (the 1252 harvest): drift reconciliation, the audit, two word baselines, the -behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded), and the -measurement tooling. Deferred here by architect ruling: the **scar registry and its eight -ratified wordings**, and enforcement rebuilt *after* the shrink. - -Issue #1279 is partly overtaken — SPIR prompts now inline their templates via `{{> …}}`, -which is what makes each phase prompt ~600 words heavier than it reads. Two *separate* -constraints govern template shape, and conflating them would over-preserve surface: -`checks.ts:149-154` (`REQUIRED_SPEC_SECTIONS`) requires only **four** headings — Problem -Statement, Current State, Desired State, Success Criteria — while the 20-heading template -pressure comes from the `spec-review` consult type, advisorily. +1. **Dead directory.** `measure-prompt-surface.sh:89` derives `PORCH_PROMPT_MEAN` from + `codev-skeleton/porch/prompts/*.md` (mean 400). The live resolver + (`commands/porch/prompts.ts:78`, `loadPromptFile`) loads + `protocols//prompts/.md`, mean **1,398**. The dead tree is a Ralph-SPIR-era + leftover ("You are the **Spec Writer** hat in a Ralph-SPIR loop"). +2. **Omits the inlined role.** `spawn-worktree.ts:854` writes `roles/builder.md` to + `.builder-role.md` for harness injection — 1,837 always-on words, uncounted. +3. **Stale hot-tier accounting.** The script's comment asserts CLAUDE.md "already inlines" the + hot files. Since #1119 (`managed-block.ts:59-67`) it carries `@import` lines, transcluded at + session launch, so `wc -w CLAUDE.md` **excludes** 736 always-loaded words. + +Net: the reported baseline (21,702) understates the phase term ~3.5×, omits the role file, and +under-counts the session term by 736. **And the metric cannot see this project's primary +target** — cutting SPIR phase prompts 1,398 → 430 moves `ALWAYS_ON_WORDS` by exactly zero. A +>50% claim scored on it would be phantom savings. + +The dead tree has no *runtime* consumer but does have a **test** consumer: +`review-prompt-routing.test.ts:29` asserts on `codev-skeleton/porch/prompts/review.md` (a Spec +987 hot/cold-routing protection). M6 handles it. + +### What landed, what is deferred + +On `main` from the 1252 harvest: drift reconciliation, the audit, two word baselines, the +behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded; B2 1.12 rounds/phase), +and the measurement tooling. Deferred here by architect ruling: the **scar registry and its +eight ratified wordings**, with enforcement rebuilt *after* the shrink. + +Issue #1279 is partly overtaken — SPIR prompts now inline templates via `{{> …}}`, which is what +makes each phase prompt ~600 words heavier than it reads. Two *separate* constraints govern +template shape and must not be conflated: `checks.ts:149-154` (`REQUIRED_SPEC_SECTIONS`) +requires only **four** headings; the 20-heading pressure comes from the `spec-review` consult +type, advisorily. ## Desired State -**One prompt form, written for frontier models, that states contracts and trusts judgment.** -A builder's always-on context tells it what it owns, what artifacts it must produce and -what shape they take, where the human gates are, and what is irreversible — then gets out of -the way. Everything else is reachable on demand. +**One prompt form, written for frontier models, that states contracts and trusts judgment.** A +builder's always-on context says what it owns, what artifacts it must produce and their shape, +where the human gates are, and what is irreversible — then gets out of the way. Everything else +is reachable on demand. ### Rewrite principles -1. **Contract, not recipe.** State the required outcome and its shape; delete the ordered - procedure for reaching it. -2. **Interface, not example.** A heading skeleton with one line of intent per heading - replaces an annotated template with filler prose. -3. **No worst-case padding.** Delete instructions guarding failure modes frontier models - do not exhibit (repeated all-caps prohibitions, "⚠️ BLOCKING" banners, checklists - restating the phase body). -4. **Progressive disclosure.** How-to content a competent agent would look up moves to - skills / on-demand files, addressed by name, not inlined. -5. **Budgets are cheap words worth keeping.** Frontier models honour stated budgets - precisely but never invent them; budget/scope lines are exempt from cuts. -6. **Scar rules are verbatim and exempt from rewriting** — but **counted** in ceilings - (see carriage plan). -7. **The instrument is part of the deliverable.** Every check, script, and baseline is - reviewed against what it claims to measure. A word ceiling not measured on served words - is not a ceiling. +1. **Contract, not recipe.** State the required outcome and its shape; delete the procedure. +2. **Interface, not example.** A heading skeleton with one line of intent per heading replaces + an annotated template with filler prose. +3. **No worst-case padding.** Delete instructions guarding failure modes frontier models do not + exhibit (all-caps prohibitions, "⚠️ BLOCKING" banners, checklists restating the body). +4. **Progressive disclosure.** How-to content moves to skills / on-demand files, addressed by + name. Relocation is *not* deletion and is measured separately (M0c). +5. **Budgets are cheap words worth keeping.** Frontier models honour stated budgets precisely + but never invent them; budget/scope lines are exempt from cuts. +6. **Scar rules are verbatim and exempt from rewriting** — but **counted** in ceilings. +7. **The instrument is part of the deliverable.** A ceiling not measured on served words is not + a ceiling. ### Per-surface ceilings (GROSS — scar words count inside every ceiling) -*CMAP round 1 (both reviewers, independently) found iteration 1 stated M1 on a gross basis -while declaring ceilings "net of" scar carriage. Incompatible. Resolved: **all ceilings are -gross**, and the arithmetic below carries carriage explicitly.* - -Ceilings apply to the SPIR instance as the measured proxy **and sweep across all ten -protocols in both trees** — an unswept protocol is a regression, not a deferral. +Applied to the SPIR instance as the measured proxy and swept across every existing surface in +both trees. | Segment | Surface | Now | Ceiling (gross) | Scar carriage inside | |---|---|---:|---:|---:| @@ -320,7 +241,7 @@ protocols in both trees** — an unswept protocol is a regression, not a deferra | consultant | `roles/consultant.md` | 252 | **≤252 unchanged** | — | | dead | `codev-skeleton/porch/prompts/**` | 4,009 | **0 (deleted)** | — | -### Post-rewrite always-on, by audience (the segmented view the directive requires) +### Post-rewrite always-on, by audience ``` BUILDER (SPIR, I=10) @@ -334,193 +255,147 @@ ARCHITECT 8,599 → 3,336 (2,636 session + 700 role) −61.2% CONSULTANT 683 → 452 (252 role + 200 consult-type) −33.8% ``` -No segment grows. The consultant segment cuts least because `roles/consultant.md` is -already lean at 252 words — reported rather than averaged into the headline, per the -directive. +No segment grows. Consultant cuts least because `roles/consultant.md` is already lean at 252 — +reported rather than averaged into the headline. -**Margin, stated honestly.** −53.2% clears >50% by 3.2 points ≈ 1,100 words. **M2 -(per-surface ceilings) is the binding criterion; M1 is derived.** If all ceilings are met -and M1 lands in 50–52%, that is a **HOLD**, not a pass — the plan must find the remainder, -with named candidates in priority order: further `protocol.md` compression, relocating -CLAUDE.md's Runnable-Worktree recipes wholesale to a skill, and the `pir`/`spir` review -prompts (2,414 / 1,957 — the two fattest phase prompts in the fleet). - -Note the shape: the phase term is 73% of the post-rewrite builder budget and the **exempt** -hot tier is 63% of that term. What survives is overwhelmingly curated judgment, not process. +The phase term is 73% of the post-rewrite builder budget, and the **exempt** hot tier is 7,360 +of the 16,016 total (46%). What survives is overwhelmingly curated judgment, not process — and +that exemption bounds how far this project can go without reopening it. ### Rollout: the corrected instrument lands on `main` first (M0b) -*(Architect-directed 2026-07-31. Recorded here as a contract — the phase decomposition -belongs in the plan.)* - -**No prompt-surface word is cut before the corrected instrument is on `main`.** The -sequencing is not administrative: a cut scored by the current script is unfalsifiable, and -1252's published baselines cite figures derived from the dead tree while being shared -knowledge other work reads. The record gets corrected while this project builds, not after. +**No prompt-surface word is cut before the corrected instrument is on `main`.** A cut scored by +the current script is unfalsifiable, and 1252's published baselines cite dead-tree figures while +being shared knowledge other work reads. **PR-1 (early, standalone) contains exactly:** | In | Out (deliberately) | |---|---| -| Corrected `scripts/measure-prompt-surface.sh` — all six M0 fixes (served directory · per-file four-tier resolution · inlined `roles/builder.md` · hot-tier `@import` transclusion **and** the stale comment that caused defect 3 · include expansion · per-segment subtotals) | Any edit to any prompt surface | -| Its tests — T1, T1b, T2, T11, T12. The script currently has **no test at all**, which is how three defects survived in a "committed and reproducible" instrument | The scar registry (rebuilt *after* the shrink, per Baked Decision 2) | -| `codev/resources/1280-word-baseline.md` — the corrected, segmented pre-rewrite baseline (34,255) as a generated artifact | The dead-tree deletion — it has a test consumer and goes through M10 governance | -| In-place annotation of `1252-word-baseline.md` and `1252-word-after-phase7.md`: original figures **preserved**, marked superseded, with the reason and a pointer to the corrected artifact | Any re-derivation of 1252's behavioural baseline (B1 stands; M8 re-runs it post-merge) | - -**Timing**: opened at the end of the first implement phase — after the instrument is correct -and tested, before any surface is cut. **Verified safe**: no test asserts on either 1252 -word-count artifact (the frozen-sample test at `prompt-behavior-metrics.test.ts:184` pins the -*behavioural* sample, a different instrument), so the annotation carries no re-baselining -cost. - -**Why this is not pushback.** The one argument against early landing is that the corrected -baseline is only meaningful alongside the cuts it scores. That argument fails: the baseline's -value is precisely that it is *pre-cut*, and publishing it early is what makes the eventual -−53.2% claim checkable by someone who did not watch it being produced. +| Corrected `measure-prompt-surface.sh` — all seven M0 items | Any edit to any prompt surface | +| Its tests (T1, T1b, T2, T11, T12, T15). The script has **no test at all** today, which is how three defects survived in a "committed and reproducible" instrument | The scar registry (rebuilt after the shrink, per Baked Decision 2) | +| `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline (34,255) | The dead-tree deletion (has a test consumer → M10 governance) | +| In-place annotation of `1252-word-baseline.md` and `1252-word-after-phase7.md`: originals **preserved**, marked superseded, reason + pointer | Re-derivation of 1252's behavioural baseline (B1 stands; M8 re-runs post-merge) | + +**Timing**: end of the first implement phase — instrument correct and tested, before any cut. +**Verified safe**: no test asserts on either 1252 word-count artifact (the frozen-sample test at +`prompt-behavior-metrics.test.ts:184` pins the *behavioural* sample, a different instrument), so +the annotation carries no re-baselining cost. + +The one argument against early landing — that the corrected baseline is meaningful only +alongside the cuts it scores — fails: the baseline's value is precisely that it is *pre-cut*, +and publishing it early is what makes the eventual −53.2% claim checkable by someone who did not +watch it being produced. ### Scar-rule carriage plan -The eight rules ratified 2026-07-28 (`git show -builder/spir-1252:codev/resources/scar-rules.yaml`) ship **verbatim**: `git-add-explicit`, -`never-destroy-worktrees`, `no-destructive-git`, `human-gates`, `no-hand-edit-status`, -`afx-from-root`, `shellper-verified-orphan`, `tower-restart-permission` (~188 words of -canonical text total). - -- The registry is **rebuilt fit-for-purpose after the shrink**: each rule's `must_appear_on` - is re-derived against the post-rewrite surface, since most 1252-listed files will have - been rewritten or deleted. -- Carriage is **exempt from rewriting but counted in ceilings** — a ceiling a surface cannot - meet while carrying its scar rules is a wrong ceiling, and gets raised deliberately, not - met by trimming scar text. -- Enforcement is a byte-identical-presence test over the registry, pinned at 8 rules and - their ids. Nothing larger is built until the surface it polices stops moving. -- A scar rule may be **compressed only by architect ratification**, never by a builder - applying principle 1. +The eight rules ratified 2026-07-28 ship **verbatim** (~188 words of canonical text): +`git-add-explicit`, `never-destroy-worktrees`, `no-destructive-git`, `human-gates`, +`no-hand-edit-status`, `afx-from-root`, `shellper-verified-orphan`, `tower-restart-permission`. + +- The registry is **rebuilt after the shrink** — each rule's `must_appear_on` re-derived against + the post-rewrite surface, since most 1252-listed files will be rewritten or deleted. +- Carriage is **exempt from rewriting but counted in ceilings**: a ceiling a surface cannot meet + while carrying its scar rules is a wrong ceiling, raised deliberately — never met by trimming + scar text. +- Enforcement is a byte-identical-presence test, pinned at 8 rules and their ids. +- A scar rule may be compressed **only by architect ratification**, never by a builder applying + principle 1. ## Stakeholders -- **Primary Users**: builder agents; CMAP reviewer agents (consult-type prompts); architect - agents (role + skills). -- **Secondary Users**: humans who must read and maintain the surface; downstream adopters - receiving it via `codev update`. +- **Primary Users**: builder agents; CMAP reviewer agents; architect agents. +- **Secondary Users**: humans who must read and maintain the surface (protected by M2b); + downstream adopters receiving it via `codev update`. - **Technical Team**: this builder; the architect at both gates. -- **Business Owners**: Waleed — charter holder; ratifies scar wordings, rules on the A/B - verdict and on any ceiling change. +- **Business Owners**: Waleed — ratifies scar wordings, rules on the A/B verdict and on any + ceiling change. ## Success Criteria -- [ ] **M0 — the metric measures what is served, segmented by audience.** The corrected - script: (a) sources phase prompts from the directory `loadPromptFile` resolves; - (b) resolves **per-file through the full four-tier chain** (`.codev/` → `codev/` → - cache → skeleton) exactly as `resolveCodevFile` does, not two-tier directory-level - selection, so mixed per-file overrides measure correctly; (c) counts the inlined - `roles/builder.md`; (d) counts hot-tier `@import` transclusion in the session term and - **corrects the stale inlining comment**; (e) expands `{{> …}}` includes; (f) reports - **exclusive bucket subtotals and the derived audience loads separately**, per the - formulas in Current State, never presenting overlapping audience figures as a sum; - (g) reports **total authored prompt-surface words** (both trees + `.claude/skills/`) - alongside always-on, so relocation is visible. Tests assert (a) and (b) against the - real resolver, so these defects cannot silently return. -- [ ] **M0c — deleted words are distinguished from relocated words.** Principle 4 authorizes - moving how-to content into skills, and relocation scores identically to deletion under - an always-on-only metric — the phantom-savings class T2 catches on the *include* axis, - unmonitored on the *relocation* axis. A −53% headline is equally consistent with −30% - deleted + −23% relocated, and only deleted content satisfies Problem Statement claim 1 - ("it crowds out judgment"); relocated content still enters context when looked up. The - review therefore **decomposes the always-on reduction into deleted vs relocated**, - evidenced by the total-authored figure from M0(g). Required by this spec's own - principle 7. *(Round-2 finding; the script already computes the skeleton and skills - totals, so this is a reporting line, not new machinery.)* +- [ ] **M0 — the metric measures what is served, segmented by audience.** The corrected script + (a) sources phase prompts from the directory `loadPromptFile` resolves; (b) resolves + **per-file through the full four-tier chain** as `resolveCodevFile` does, not two-tier + directory-level selection, so mixed per-file overrides measure correctly; (c) counts the + inlined `roles/builder.md`; (d) counts hot-tier `@import` transclusion in the session term + **and corrects the stale inlining comment**; (e) expands `{{> …}}` includes; (f) reports + **exclusive bucket subtotals and derived audience loads separately**, per the stated + formulas, never presenting overlapping audience figures as a sum; (g) reports **total + authored prompt-surface words** (both trees + `.claude/skills/`) alongside always-on. + Tests assert (a) and (b) against the real resolver. - [ ] **M0b — the corrected instrument and baseline land on `main` early**, as a small - standalone PR (precedent: #1290), not at the end of this branch. - `1252-word-baseline.md` and `1252-word-after-phase7.md` cite figures derived from the - dead tree and are shared knowledge other work reads; the correction annotates them in - place — original figures preserved, marked superseded, with the reason — and does not - rewrite their history. PR-1's exact contents, timing, and safety verification are - specified in **Desired State → Rollout**. -- [ ] **M1 — >50% reduction (derived, arithmetically implied by M2).** `ALWAYS_ON_WORDS` - falls from 34,255 to **≤16,100**, measured before and after by the same corrected - script, both figures committed. Audience loads reported; none regressed. - **Reachability, stated honestly** *(round-2 finding)*: M1 is fully derived from the M2 - ceilings — meeting every ceiling yields ≤16,016 (−53.2%), so M1 cannot fail while M2 - passes. Iteration 2's "HOLD at 50–52%" branch was dead prose and is withdrawn. The - *reachable* contingency is **denominator movement**: if correcting the instrument - surfaces always-on content not yet found — as it already has twice — the baseline and - every ceiling are re-derived to preserve >50%, and the re-derivation goes to the - architect rather than being absorbed silently. -- [ ] **M2 — per-surface ceilings met (binding).** Every ceiling, **gross**, on **every - surface that exists after resolution**, enumerated from disk across both trees and - unioned — not a per-protocol × surface-type cross product. *(Round-2 correction: - `codev/protocols/` has ten protocols and `codev-skeleton/protocols/` nine — `release` - is project-local by design — and `experiment`/`research`/`spike`/`release` have no - `prompts/` or `consult-types/`. Demanding "all ten in both trees" was unsatisfiable.)* - Absence of a surface never fails; presence of an **unmeasured** surface does. -- [ ] **M2b — CLAUDE.md stays human-readable.** Humans are named stakeholders, and no - criterion protected them at 5,815 → ≤1,900. The rewritten file must retain a - navigable heading structure and be reviewed by the architect for human usability at - the gate — not merely pass the twin-parity byte check. *(Round-2 finding.)* -- [ ] **M3 — sweep completeness.** Surfaces enumerated from disk (both trees, unioned), not - a hardcoded list; no surface retains pre-rewrite content. Includes - `codev/protocols/release/protocol.md` (1,626w) — human-invoked prose an agent reads, - no skeleton twin, missed by iteration 2's inventory. -- [ ] **M4 — scar rules intact.** Eight canonicals byte-identical on every registered - surface; test fails on reword or deletion; count pinned at 8. -- [ ] **M5 — no capability lost, proven against the *prompt text*.** A committed - `capability-inventory.json` is extracted pre-rewrite with **explicit recognition - rules** — artifact paths (`codev/(specs|plans|reviews)/…` literals and - `{{artifact_name}}` forms), gate names, signal names (`` tags), porch check - names, notification triggers — normalized (lowercase, strip backticks/punctuation, - dedupe). - - **The inventory is over the resolved, expanded prompt surface, not over - `protocol.json` or source call sites** *(round-2 correction)*. Extracting gate and - check names from an unchanged `protocol.json`, or notification names from unchanged - `afx send` call sites, would report every capability as present even if every - corresponding instruction vanished from the served prompts — proving nothing about the - thing being cut. Each inventory item must therefore be evidenced as **represented in - the served prompt text**: for every gate, check, artifact contract, signal, and - notification trigger, a contract-presence assertion that the rewritten prompts still - tell the agent about it. This is also the primary defence for the most aggressive row - in the table, `protocol.md` 3,703 → ≤700 (−81%), which is the builder's only map of - gates, artifacts, and phases. - - **Severity, unambiguous** *(both reviewers, independently)*: "any removal fails" and - "justified as a deliberate retirement" are mutually exclusive. Resolved on M10's - pattern — **a removal is a hard failure unless the retired name appears in a committed - `codev/resources/1280-retirements.md` in the same commit**, naming the capability, why - it is obsolete, and the architect approval. An approved, listed retirement passes M5; - anything else fails. Retirements *will* occur (M6 deletes a tree; `protocol.md` drops - 3,000 words), so the exception path must be explicit rather than improvised. + standalone PR (precedent #1290), per **Desired State → Rollout**. +- [ ] **M0c — deleted words are distinguished from relocated words.** Relocation to skills + scores identically to deletion under an always-on-only metric — the phantom-savings class + T2 catches on the *include* axis, unmonitored on the *relocation* axis. A −53% headline is + equally consistent with −30% deleted + −23% relocated, and only deleted content satisfies + Problem Statement claim 1. The review **decomposes the always-on reduction into deleted vs + relocated**, evidenced by M0(g). +- [ ] **M1 — >50% reduction (derived, arithmetically implied by M2).** `ALWAYS_ON_WORDS` falls + from 34,255 to **≤16,100**, measured before and after by the same corrected script, both + figures committed; audience loads reported, none regressed. Meeting every ceiling yields + ≤16,016 (−53.2%), so M1 cannot fail while M2 passes — but it clears >50% by only **3.2 + points ≈ 1,100 words**, so no ceiling has slack to give away. The reachable contingency is + **denominator movement**: if correcting the instrument surfaces always-on content not yet + found, the baseline and every ceiling are re-derived to preserve >50%, and that + re-derivation goes to the architect rather than being absorbed silently. +- [ ] **M2 — per-surface ceilings met (binding).** Every ceiling, **gross**, on **every surface + existing after resolution**, enumerated from disk across both trees and unioned. Absence + never fails; an unmeasured present surface does. +- [ ] **M2b — CLAUDE.md stays human-readable.** At 5,815 → ≤1,900 the rewritten file must retain + a navigable heading structure and be reviewed by the architect for human usability at the + gate — twin-parity bytes are not a readability check. +- [ ] **M3 — sweep completeness.** Surfaces enumerated from disk (both trees, unioned), never a + hardcoded list; no surface retains pre-rewrite content. Includes + `codev/protocols/release/protocol.md` (1,626w), which has no skeleton twin. +- [ ] **M4 — scar rules intact.** Eight canonicals byte-identical on every registered surface; + test fails on reword or deletion; count pinned at 8. +- [ ] **M5 — no capability lost, proven against the prompt text.** A committed + `capability-inventory.json` extracted pre-rewrite with explicit recognition rules — + artifact paths (`codev/(specs|plans|reviews)/…` literals and `{{artifact_name}}` forms), + gate names, signal names (`` tags), porch check names, notification triggers — + normalized (lowercase, strip backticks/punctuation, dedupe). + + **The inventory is over the resolved, expanded prompt surface, not over `protocol.json` or + source call sites.** Extracting gate and check names from an unchanged `protocol.json`, or + notifications from unchanged `afx send` call sites, would report every capability present + even if every corresponding instruction vanished from the served prompts. Each item must + be evidenced as **represented in served prompt text** via a contract-presence assertion. + This is the primary defence for the most aggressive row, `protocol.md` 3,703 → ≤700 + (−81%), the builder's only map of gates, artifacts and phases. + + **Severity**: a removal is a hard failure **unless** the retired name appears in a + committed `codev/resources/1280-retirements.md` in the same commit, naming the capability, + why it is obsolete, and the architect approval. Retirements will occur (M6 deletes a tree; + `protocol.md` drops ~3,000 words), so the exception path is explicit rather than + improvised. - [ ] **M6 — the dead tree is gone, with its consumer handled.** `codev-skeleton/porch/prompts/` deleted. Verification is **not** a bare grep: an - untruncated repo-wide search (`grep -rn … | wc -l` reconciled against the full hit - list) shows zero *runtime* consumers, and the one **test** consumer — - `review-prompt-routing.test.ts:29`, a Spec 987 hot/cold-routing protection — is - updated under M10's re-baselining rule, naming Spec 987 as the originating spec. -- [ ] **M7 — A/B non-inferiority passes** per the pre-registered decision rule. -- [ ] **M8 — behavioural baseline re-run.** `measure-prompt-behavior.ts` re-run and - committed; B1 compared directionally to 51.88% (n=160) with the sample documented. -- [ ] **M9 — rollback rehearsed** per **Rollback Plan**, by group. + untruncated repo-wide search reconciled against the full hit list shows zero *runtime* + consumers, and the one **test** consumer (`review-prompt-routing.test.ts:29`, a Spec 987 + protection) is updated under M10 naming Spec 987. +- [ ] **M7 — A/B non-inferiority passes** per the pre-registered decision rule; gates + `verify-approval`. +- [ ] **M8 — behavioural baseline re-run.** `measure-prompt-behavior.ts` re-run and committed; + B1 compared directionally to 51.88% (n=160) with the sample documented. +- [ ] **M9 — rollback rehearsed** by group, per **Rollback Plan**. - [ ] **M10 — prose-pinned test re-baselining is deliberate and enumerated.** ~25 test files assert exact prose in the surfaces being cut; the hardest is - `agent-farm/__tests__/baked-decisions.test.ts:143-148`, which enforces a - **pure-addition diff** against committed baselines for - `protocols/{spir,aspir,air}/builder-prompt.md` — logically incompatible with cutting - 824 → ≤420. Also: `bugfix-744-spir-pr-strategy.test.ts` (4 near-verbatim sentences), - `spec-1273-wait-discipline-docs.test.ts` (16 assertions), `bugfix-619-aspir-prompt.test.ts`, + `agent-farm/__tests__/baked-decisions.test.ts:143-148`, enforcing a **pure-addition diff** + against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md` — + incompatible with 824 → ≤420. Also `bugfix-744-spir-pr-strategy.test.ts`, + `spec-1273-wait-discipline-docs.test.ts`, `bugfix-619-aspir-prompt.test.ts`, `template-delivery.test.ts`, `framework-ref-audit.test.ts`, `governance-sweep.test.ts`, - `review-prompt-routing.test.ts`. **Each assertion is a prior spec's protection encoded - as a grep.** Therefore: every modified or retired assertion is listed in the review - with (i) the spec that created it, (ii) whether the protected behaviour survives in - the rewritten prose, (iii) the replacement assertion if the behaviour survives, or an - explicit architect-visible retirement if it does not. Re-baselining a pure-addition - baseline is permitted **only** with the originating spec named and the new baseline - committed in the same commit. Silent deletion of an assertion to make the suite green - is a project failure, not a test fix. -- [ ] All tests pass **after M10's enumerated re-baselining**; no coverage reduction. New - tests cover M0, M3, M4, M5. -- [ ] Documentation routed by tier (`arch.md`/`arch-critical.md`, - `lessons-learned.md`/`lessons-critical.md`); `CLAUDE.md`/`AGENTS.md` byte-identical. + `review-prompt-routing.test.ts`. **Each assertion is a prior spec's protection encoded as a + grep, so retiring one is a governance act.** Every modified or retired assertion is listed + in the review with (i) the originating spec, (ii) whether the protected behaviour survives + in the rewritten prose, (iii) the replacement assertion, or an explicit architect-visible + retirement. Pure-addition re-baselining only with the originating spec named and the new + baseline committed in the same commit. Silent deletion to make the suite green is a + project failure, not a test fix. +- [ ] All tests pass **after M10's enumerated re-baselining**; no coverage reduction. New tests + cover M0, M3, M4, M5. +- [ ] Documentation routed by tier; `CLAUDE.md`/`AGENTS.md` byte-identical. ## Constraints @@ -531,163 +406,131 @@ plan, or CMAP reviewers: - **All prompt consumers are frontier models** (Claude 5, GPT 5.6, Gemini 3.6 class). No weak-model tier, no fallback scaffolding variant, no tiering mechanism. One form. -- **Scar rules are exempt and verbatim** — the eight compressed canonicals developed in - Spec 1252 Phase 5 (six repo rules + shellper verified-orphan + Tower-restart permission) - ship with the rewrite; the registry/enforcement concept from 1252 is rebuilt - fit-for-purpose around the post-shrink surface, not before it. +- **Scar rules are exempt and verbatim** — the eight compressed canonicals developed in Spec + 1252 Phase 5 (six repo rules + shellper verified-orphan + Tower-restart permission) ship with + the rewrite; the registry/enforcement concept from 1252 is rebuilt fit-for-purpose around the + post-shrink surface, not before it. - **Validation is A/B, not observational**: same issues executed by builders on old vs new - prompts, compared on outcomes (gate friction, review rounds, correctness). Spec 1252's - M12 established that observational baselines (n=17) can only detect large regressions — + prompts, compared on outcomes (gate friction, review rounds, correctness). Spec 1252's M12 + established that observational baselines (n=17) can only detect large regressions — insufficient at deletion scale. The A/B design is a first-class spec section. - Spec must define a rollback story (prompt surfaces are files; reverting is cheap — say so concretely). -Arising from the repository and the architect's scope directive: +Arising from the repository and the scope directive: -- **Scope is the full prompt surface, segmented**: architect roles, builder roles and spawn - wrappers, consultant/CMAP prompts, phase prompts and template includes, `protocol.md` - texts. Measurement reports per-segment, never averaged away. -- **Both trees**; `CLAUDE.md` and `AGENTS.md` byte-identical. -- **Four-tier resolution, per file.** No fetch-by-path instruction for a file that may not - exist on disk (deliver-don't-fetch). -- **No porch behaviour changes.** Content rewrite plus a measurement-script fix. Changing - the state machine, gates, or check semantics is out of scope. -- **Template shape is governed by two separate constraints** (porch's 4 required headings; - the consult type's advisory 20) — they must not be conflated. +- **Scope is the full prompt surface, segmented**; measurement reports per-segment. +- **Both trees**; `CLAUDE.md` ≡ `AGENTS.md`. +- **Four-tier resolution, per file.** No fetch-by-path instruction for a file that may not exist + on disk (deliver-don't-fetch). +- **No porch behaviour changes** — content rewrite plus a measurement-script fix. +- **Template shape is governed by two separate constraints** (porch's 4 headings; the consult + type's advisory 20) which must not be conflated. ### Business Constraints - Two human gates (`spec-approval`, `plan-approval`) plus `pr`; the A/B verdict is the architect's call. -- Adopters consume the skeleton via `codev update`, so rollback must be a revertible unit - per **group**. -- Scar-rule wordings are architect-ratified; a builder may not compress them. -- **The corrected instrument ships early, as its own PR** (M0b) — architect-directed. The - remaining phase-commits ship as a single later PR. +- Adopters consume the skeleton via `codev update`, so rollback is a revertible unit per + **group**. +- Scar wordings are architect-ratified; a builder may not compress them. +- **The corrected instrument ships early, as its own PR** (M0b); remaining phase-commits ship as + a single later PR. ## Assumptions - The eight scar wordings on `builder/spir-1252` remain the ratified set. -- `I = 10` remains the agreed proxy; it is a comparison constant, identical both sides. +- `I = 10` remains the agreed proxy — a comparison constant, identical both sides. - Frontier-model behaviour is stable across the A/B window (mitigated by pairing, same base - commit, and pinned model/config versions). + commit, pinned model/config versions). - `builder/spir-1252` stays undeleted — sole source of the ratified registry. -- CMAP reviewers are blind to the builder's prompt surface by construction: a reviewer sees - artifacts and diffs, not the prompt that produced them. +- CMAP reviewers are blind to the builder's prompt surface by construction. ## Solution Approaches ### Approach 1: In-place judgment rewrite, surface by surface (RECOMMENDED) -**Description**: Rewrite each file to the seven principles, keeping file layout, resolver -and porch untouched. Templates become heading interfaces. How-to content relocates to -existing skills. The dead tree is deleted. The instrument is corrected first, so every -subsequent cut is scored honestly. - -**Pros**: zero mechanism risk; every change is a reviewable text diff; rollback granularity -equals cut granularity; compatible with deliver-don't-fetch. +Rewrite each file to the seven principles, keeping file layout, resolver and porch untouched. +Templates become heading interfaces; how-to content relocates to skills; the dead tree goes; the +instrument is corrected first. -**Cons**: discipline-dependent (nothing structurally prevents re-growth — mitigated by T3); -large diff across 10 protocols × 2 trees, so sweep completeness is the main risk; collides -with ~25 prose-pinned test files (M10). - -**Estimated Complexity**: Medium · **Risk Level**: Low-Medium +**Pros**: zero mechanism risk; every change a reviewable text diff; rollback granularity equals +cut granularity; compatible with deliver-don't-fetch. +**Cons**: discipline-dependent (mitigated by T3); large diff across 10 protocols × 2 trees, so +sweep completeness is the main risk; collides with ~25 prose-pinned test files (M10). +**Complexity**: Medium · **Risk**: Low-Medium ### Approach 2: Generate prompts from `protocol.json` -**Description**: Treat `protocol.json` as source of truth; synthesize phase prompts at -runtime with a small per-phase prose delta. +Synthesize phase prompts at runtime from the state machine, with a small per-phase prose delta. **Pros**: structurally prevents re-growth and drift; would have made the 1252 drift bug impossible. - **Cons**: introduces a code path between authoring and serving — new failure mode, harder to -review and revert, and it changes porch behaviour (an explicit constraint). Couples the -shrink to a mechanism change, so an A/B regression becomes un-attributable: deletion or -generator? - -**Estimated Complexity**: High · **Risk Level**: Medium-High - -**Verdict**: right idea, wrong project. A generator over a 16,000-word surface is a -plausible successor. +review and revert — and changes porch behaviour (an explicit constraint). Couples the shrink to +a mechanism change, making any A/B regression un-attributable: deletion or generator? +**Complexity**: High · **Risk**: Medium-High +**Verdict**: right idea, wrong project; a plausible successor over a 16,000-word surface. ### Approach 3: Shared kernel + per-protocol deltas -**Description**: One protocol-agnostic builder kernel (gates, artifacts, thread, -notifications, scar rules) included by every protocol, plus a short per-protocol delta. - -**Pros**: attacks cross-protocol duplication the per-surface table does not; uses the -existing include mechanism, so it is served-word-honest. +One protocol-agnostic builder kernel included by every protocol, plus a short per-protocol delta. +**Pros**: attacks cross-protocol duplication the per-surface table does not; uses the existing +include mechanism, so it is served-word-honest. **Cons**: 1252 proved dedup buys ~7% on *served* words — an include expands, so moving text -changes ownership, not the bill. Savings are maintenance, not context. Risks re-creating the -shadow-tree bug class (one edit silently changing ten protocols' served prompts). - -**Verdict**: adopt selectively *inside* Approach 1 where a kernel reduces served words for -the reader, and only after the shrink. +changes ownership, not the bill. Risks re-creating the shadow-tree bug class. +**Verdict**: adopt selectively *inside* Approach 1 where a kernel reduces served words for the +reader, and only after the shrink. ## Open Questions ### Critical (Blocks Progress) -*None outstanding.* Both former Critical questions are resolved: - -- [x] **~~Corrected measurement baseline accepted?~~** **RESOLVED 2026-07-31** — architect - verified both claims against source (script line 89; `prompts.ts:78`) and endorsed M0 - as specced; >50% unchanged against the corrected baseline. Added M0b and principle 7. -- [x] **~~How is "gate friction" captured?~~** **RESOLVED** — 1252 established gate - rejections are not minable (no `rejected` state; `requested_at` overwritten). A porch - gate-event log is a behaviour change and out of scope. **Decision: O1 is scored - prospectively by the architect on the rubric in the A/B section, and is - *advisory-with-a-tripwire*, not a SHIP gate** — if scoring is incomplete for any pair, - O1 is reported as incomplete and SHIP rests on O2/O3/O4. This removes the single point - of failure CMAP flagged while keeping the signal. +*None outstanding.* Both former Critical questions are resolved: the corrected baseline is +architect-endorsed; and gate friction (not minable from history — no `rejected` state, +`requested_at` overwritten) is scored prospectively by the architect and demoted to +**advisory-with-a-tripwire**, so incomplete scoring cannot block the decision. ### Important (Affects Design) -- [ ] **A/B sample size.** ≥6 pairs specified; more buys power at real builder and consult - cost (~$1,478/30d at current rates). The architect sets the ceiling. +- [ ] **A/B sample size.** ≥6 pairs specified; more buys power at real builder and consult cost + (~$1,478/30d at current rates). The architect sets the ceiling — see architect load under + **Execution and sequencing**. - [ ] **Do SPIR templates survive as interfaces, or disappear?** Recommendation: survive as - ≤150-word heading interfaces — porch requires only 4 headings, but the interface is - what makes the artifact contract legible without narration. -- [ ] **Is the hot tier's 736 genuinely exempt?** It is the one surface already built to - these principles. The spec exempts it; a reviewer may argue it should be re-derived - post-shrink. Note the ceiling this imposes: the exempt hot tier is **7,360 of the - 16,016-word post-rewrite builder budget (46%)**, so it bounds how far this project can - go without reopening the exemption. -- [ ] **`roles/architect.md` 2,048 → ≤700** — the architect segment was outside 1252's - analysis entirely. Confirm nothing in it is load-bearing for multi-architect - coordination (Specs 755/786/823) before cutting. + ≤150-word heading interfaces — porch requires only 4 headings, but the interface is what + makes the artifact contract legible without narration. +- [ ] **Is the hot tier's 736 genuinely exempt?** It is the one surface already built to these + principles, but it is 7,360 of the 16,016-word post-rewrite builder budget (46%), so it + bounds how far this project can go without reopening the exemption. +- [ ] **`roles/architect.md` 2,048 → ≤700** — the architect segment was outside 1252's analysis + entirely. Confirm nothing in it is load-bearing for multi-architect coordination (Specs + 755/786/823) before cutting. ### Nice-to-Know (Optimization) -- [ ] Does trimming consult-type prompts move CMAP verdict *quality*, or only cost? B1 shows - the rate; quality needs human adjudication. -- [ ] Are the ~17,000 words of `.claude/skills/` the right destination for relocated - how-tos, or does that surface need its own budget? - -*(Iteration 1's "should a word-budget check run in CI?" is withdrawn — it contradicted T3, -which already runs the ceilings as a test. T3 stands.)* +- [ ] Does trimming consult-type prompts move CMAP verdict *quality*, or only cost? +- [ ] Are the ~17,000 words of `.claude/skills/` the right destination for relocated how-tos, or + does that surface need its own budget? ## Performance Requirements - **Builder always-on**: ≤16,100 words (from 34,255) — M1. -- **Per-segment**: architect ≤3,400 (from 8,599); consultant ≤460 (from 683); no segment - regresses. +- **Per-segment**: architect ≤3,400 (from 8,599); consultant ≤460 (from 683); none regresses. - **Per-surface ceilings**: as tabulated — M2, binding. - **Measurement runtime**: <5s, deterministic — same commit ⇒ byte-identical output. - **Token/cost effect** (advisory): ~18,200 fewer always-on words ≈ ~24,000 tokens per - builder-project; recorded before/after as context for the A/B, keying no threshold. + builder-project; recorded before/after, keying no threshold. ## Security Considerations - **The scar rules are the security surface.** All eight guard irreversible acts (destroying - uncommitted work or worktrees, killing live sessions, bypassing a human gate). Weakening - one is the highest-severity failure this project can produce — hence verbatim carriage, - byte-identical enforcement, counted-not-exempt ceilings, and a hard rollback trigger on - any observed violation. -- **Human-gate integrity.** The rewrite must not weaken "a gate message is a notification to - the human, not authorization." Gate semantics are content, and this project edits content. + uncommitted work or worktrees, killing live sessions, bypassing a human gate). Weakening one is + the highest-severity failure this project can produce — hence verbatim carriage, + byte-identical enforcement, counted-not-exempt ceilings, and a hard rollback trigger on any + observed violation. +- **Human-gate integrity.** The rewrite must not weaken "a gate message is a notification to the + human, not authorization." Gate semantics are content, and this project edits content. - **No secrets in prompt surfaces** — existing property, re-verified after the rewrite. - **Adopter blast radius.** Skeleton changes ship on `codev update`; a weakened prohibition propagates silently. Hence grouped, rehearsed rollback. @@ -696,9 +539,9 @@ which already runs the ceilings as a test. T3 stands.)* ### Functional Tests -1. **T1 — Instrument sources the served directory (M0a).** The script's phase-prompt source - equals the directory `loadPromptFile` resolves, asserted against the real resolver. -2. **T1b — Instrument resolves per-file, four-tier (M0b).** A fixture with a `.codev/` +1. **T1 — Instrument sources the served directory** (M0 item a). Asserted against the real + resolver, not a hardcoded string. +2. **T1b — Instrument resolves per-file, four-tier** (M0 item b). A fixture with a `.codev/` override of *one* prompt while others resolve from the skeleton measures each file at its winning tier — kills the directory-level-selection defect class. 3. **T2 — Include expansion (phantom-savings proof).** Moving text from prompt into template @@ -707,29 +550,29 @@ which already runs the ceilings as a test. T3 stands.)* tree; failure names surface and overage. Runs in CI as the anti-re-growth guard. 5. **T4 — Scar integrity (M4).** Every canonical byte-identical on every registered surface; count pinned at 8; reword or deletion fails. -6. **T5 — Capability inventory (M5).** Post-rewrite extraction ⊇ pre-rewrite, using M5's - recognition and normalization rules; removals fail. +6. **T5 — Capability inventory (M5).** Post-rewrite extraction over served prompt text ⊇ + pre-rewrite, using M5's recognition and normalization rules; unlisted removals fail. 7. **T6 — Sweep completeness (M3).** Surfaces enumerated from disk across both trees and unioned; each existing surface satisfies T3. Absence of `prompts/`/`consult-types/` for a protocol that has none must **not** fail; a newly added protocol or surface fails until - written to budget. Covers `release` (project-local, `codev/` only). + written to budget. Covers `release` (`codev/` only). 8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; `codev/` and `codev-skeleton/` copies consistent. -9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing - test updated per M10 and still protecting hot/cold routing on its remaining files. -10. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface receives - a spawn prompt containing every element of the artifact contract, and its first - `porch next` returns a well-formed task. ("It compiled" is not "it works.") +9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing test + updated per M10 and still protecting hot/cold routing on its remaining files. +10. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface receives a + spawn prompt containing every element of the artifact contract, and its first `porch next` + returns a well-formed task. ("It compiled" is not "it works.") 11. **T10 — Rollback rehearsal (M9).** Reverting a rollback **group** on a scratch branch restores that group byte-for-byte and leaves the suite green. -12. **T11 — Bucket and audience reporting (M0f).** The script emits the six **exclusive - bucket** subtotals, which sum to the authored prompt-surface total, *and separately* the - derived audience loads, which overlap by design and are asserted against the stated - formulas rather than against a naive sum. A fixture where one bucket grows and another - shrinks shows both movements, not a netted zero. -13. **T15 — Relocation visibility (M0c).** A fixture that moves a block from an always-on - surface into `.claude/skills/` shows always-on falling **and** total-authored holding - steady — so relocation can never be reported as deletion. +12. **T11 — Bucket and audience reporting** (M0 item f). The script emits the six **exclusive + bucket** subtotals, which sum to the authored total, *and separately* the derived audience + loads, which overlap by design and are asserted against the stated formulas rather than a + naive sum. A fixture where one bucket grows and another shrinks shows both movements, not a + netted zero. +13. **T15 — Relocation visibility (M0c).** A fixture moving a block from an always-on surface + into `.claude/skills/` shows always-on falling **and** total-authored holding steady — so + relocation can never be reported as deletion. ### Non-Functional Tests @@ -740,121 +583,99 @@ which already runs the ceilings as a test. T3 stands.)* ## A/B Validation Design -*(First-class section per Baked Decision 3. A **non-inferiority** trial: the claim under -test is "deleting ~53% of the always-on surface does not degrade outcomes", not "it improves -them".)* +*(First-class section per Baked Decision 3. A **non-inferiority** trial: the claim under test is +"deleting ~53% of the always-on surface does not degrade outcomes", not "it improves them".)* ### Unit and arms -The unit is an **issue-pair**: one GitHub issue executed twice, by two freshly-spawned -builders in separate worktrees, from the same base commit. - -- **Control (A)**: worktree at the pre-rewrite commit. -- **Treatment (B)**: worktree at the post-rewrite commit. - -No code differs between arms — the prompt surface is file-resolved, so the arms are two -checkouts. +The unit is an **issue-pair**: one GitHub issue executed twice, by two freshly-spawned builders +in separate worktrees, from the same base commit. **Control (A)** = pre-rewrite commit; +**treatment (B)** = post-rewrite. No code differs — the prompt surface is file-resolved, so the +arms are two checkouts. ### Sample and eligibility - **≥6 pairs (12 runs)**, stratified: ≥3 SPIR/ASPIR (exercises spec/plan/implement/review prompts, both gates, templates) and ≥3 lighter protocols (BUGFIX/AIR). -- Issues drawn from the existing backlog, selected **before** either arm runs, then frozen - (no issue-body edits mid-trial). -- **Eligibility exclusion (CMAP round 1):** an issue is ineligible if it modifies any - surface under test. Otherwise the treatment arm's own prompt surface is simultaneously - instrument and subject. +- Issues drawn from the backlog, selected **before** either arm runs, then frozen. +- **Eligibility exclusion**: an issue is ineligible if it modifies any surface under test — + otherwise the treatment arm's prompt surface is simultaneously instrument and subject. ### Contamination controls - **Pin the environment**: model ids and reasoning efforts, consult backend versions, and `.codev/config.json` frozen for the trial window and recorded with the results. -- **Arm isolation**: the second arm of a pair must not see the first arm's branch, PR, or - thread file. Arms run **sequentially with the intervening branch unpushed**, or - concurrently in isolated worktrees — either is acceptable; which was used is recorded - per pair. -- **Arm order alternates** per pair to control for time-varying factors. -- **Recording**: one committed results artifact - (`codev/resources/1280-ab-results.md`) with a row per run — pair id, arm, protocol, issue, - base commit, order, isolation mode, every outcome value, and any exclusion with its - reason. Exclusions after the fact must be justified in that file, not silently dropped. +- **Arm isolation**: the second arm of a pair must not see the first arm's branch, PR, or thread. + Sequential with the intervening branch unpushed, or concurrent in isolated worktrees; which was + used is recorded per pair. +- **Arm order alternates** per pair. +- **Recording**: one committed artifact (`codev/resources/1280-ab-results.md`), a row per run — + pair id, arm, protocol, issue, base commit, order, isolation mode, every outcome, and any + exclusion with its reason. Post-hoc exclusions must be justified there, never silently dropped. ### Execution and sequencing -*(Round-2 finding, raised independently by both reviewers: M7 sat among the criteria with no -stated position in the sequence, and three operational questions had no answer.)* - -- **M7 gates `verify-approval`, not the PR merge.** The rewrite PR merges on the strength of - M0–M6 and M8–M10 plus architect review; the A/B then runs against merged `main` as the - treatment arm and a pinned pre-rewrite commit as control. This matches where the rollback - triggers already point, keeps a 12-run trial off the PR's critical path, and is the only - ordering under which "treatment arm = what builders actually get" is literally true. - Consequence, stated plainly: a SHIP failure means **rolling back a merged change**, which - is precisely what the grouped, rehearsed rollback plan exists for. -- **Arm disposition — one arm merges per pair.** Each pair produces two real implementations - of one backlog issue. **The treatment arm's PR is the candidate for merge; the control - arm's is closed unmerged after its outcomes are recorded** (and vice versa if the treatment - arm's work is defective on O3). The discarded arm is not wasted — it is the comparison — - but the cost defence must be stated accurately: **≈6 of 12 runs produce merged work, not - 12.** The trial's real cost is 6 duplicated implementations plus consult spend. -- **Architect load is a scheduling dependency, not a footnote.** 6 pairs with ≥3 SPIR-class - implies up to **~24 gate approvals and 12 PR reviews by one person**, each SPIR gate - requiring O1 rubric scoring *at approval time*. This is the trial's binding constraint and - the reason the pair count is the architect's call, not the builder's. If capacity forces a - smaller n, the honest consequence is stated in the power paragraph — not a quietly reduced - sample. +- **M7 gates `verify-approval`, not the PR merge.** The rewrite PR merges on M0–M6 and M8–M10 + plus architect review; the A/B then runs against merged `main` as treatment and a pinned + pre-rewrite commit as control. This matches where the rollback triggers point, keeps a 12-run + trial off the PR's critical path, and is the only ordering under which "treatment arm = what + builders actually get" is literally true. Consequence, stated plainly: a SHIP failure means + rolling back a merged change — which is what the grouped, rehearsed rollback plan is for. +- **Arm disposition**: the treatment arm's PR is the merge candidate; the control arm's closes + unmerged once its outcomes are recorded (and vice versa if the treatment arm is defective on + O3). So **~6 of 12 runs produce merged work, not 12** — the trial's real cost is 6 duplicated + implementations plus consult spend. +- **Architect load is a scheduling dependency**: 6 pairs with ≥3 SPIR-class implies up to **~24 + gate approvals and 12 PR reviews by one person**, each SPIR gate requiring O1 rubric scoring at + approval time. This is the trial's binding constraint and why the pair count is the architect's + call. If capacity forces a smaller n, the consequence is stated in the power paragraph — not a + quietly reduced sample. ### Pre-registered outcomes -| ID | Outcome | Instrument | Role in decision | +| ID | Outcome | Instrument | Role | |---|---|---|---| -| **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each scored 0 (no friction) / 1 (minor) / 2 (blocking), recorded in the results artifact at scoring time | **advisory + tripwire** | -| **O2** | Review rounds | Iterations to terminal state per phase from `status.yaml` history; CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | gate | -| **O3** | Correctness | **At the SHIP decision**: architect PR-review findings by severity, observable pre-merge on both arms. **Post-merge defects are excluded from the SHIP gate** — they cannot be evaluated when the decision is made — and instead act as a **later rollback signal over a 14-day window** after the merged arm lands *(round-2 correction: the original wording asked one metric to be evaluated at two incompatible times)* | gate (pre-merge part) + rollback signal (post-merge part) | +| **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each 0 (none) / 1 (minor) / 2 (blocking), recorded at scoring time | advisory + tripwire | +| **O2** | Review rounds | Iterations to terminal state per phase from `status.yaml`; CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | gate | +| **O3** | Correctness | **At the SHIP decision**: architect PR-review findings by severity, observable pre-merge on both arms. **Post-merge defects are excluded from the SHIP gate** — they cannot be evaluated when the decision is made — and act as a **14-day rollback signal** after the merged arm lands | gate (pre-merge) + rollback signal (post-merge) | | **O4** | Protocol compliance | Binary per-run checklist: required artifacts with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar violation · thread committed | **zero tolerance** | | **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory | ### Blinding -CMAP reviewers are blind by construction. The architect is not and cannot be; mitigations: -O2 and O4 are extracted mechanically from committed artifacts, and O1/O3 are scored against -a rubric written **before** any run. +CMAP reviewers are blind by construction. The architect is not and cannot be; mitigations: O2 and +O4 are extracted mechanically from committed artifacts, and O1/O3 are scored against a rubric +written **before** any run. ### Decision rule (pre-registered) **SHIP** iff all of: -1. **O4 = zero violations** in the treatment arm. Any scar violation, skipped gate, or - missing required artifact is an immediate hard stop, independent of everything else. +1. **O4 = zero violations** in the treatment arm. Any scar violation, skipped gate, or missing + required artifact is an immediate hard stop, independent of everything else. 2. **O2**: treatment mean review rounds ≤ control **+ 0.5 rounds/phase**, and treatment REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. -3. **O3 (pre-merge part only)**: no treatment-arm architect finding of severity ≥ "would - block merge" absent from its paired control run. Post-merge defects do not enter this - decision; they are a 14-day rollback signal. -4. **O1 tripwire**: no pair where the treatment arm scored **2 (blocking)** at a gate its - control scored 0, for the same reason. If O1 scoring is incomplete for any pair, O1 is - reported incomplete and SHIP rests on 1–3. +3. **O3 (pre-merge part only)**: no treatment-arm finding of severity ≥ "would block merge" + absent from its paired control run. +4. **O1 tripwire**: no pair where the treatment arm scored **2 (blocking)** at a gate its control + scored 0, for the same reason. If O1 scoring is incomplete for any pair, O1 reports incomplete + and SHIP rests on 1–3. Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK**. ### Honest power statement -With n=6 pairs this detects only **large** effects — roughly a doubling of review rounds or -a ≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle regression, and -this spec does not claim it can. It is nonetheless strictly stronger than 1252's -observational baseline, because each pair is matched on the issue itself — the dominant -variance source. **O4's zero-tolerance criterion is where the real protection lives**: -compliance is binary, observable in every run, and is the failure mode deletion would -plausibly cause. +With n=6 pairs this detects only **large** effects — roughly a doubling of review rounds or a +≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle regression, and this +spec does not claim it can. It is nonetheless strictly stronger than 1252's observational +baseline, because each pair is matched on the issue itself — the dominant variance source. +**O4's zero-tolerance criterion is where the real protection lives**: compliance is binary, +observable in every run, and is the failure mode deletion would plausibly cause. ## Rollback Plan -*(Required by Baked Decision 4.)* - -CMAP round 1 correctly flagged that iteration 1 overstated per-surface independence: prompts, -their included templates, scar-registry mappings, and integrity tests are coupled — reverting -a prompt without its template can break a required-headings check or M4. Rollback is -therefore by **group**, each group internally consistent and independently revertible: +Prompts, included templates, registry mappings and integrity tests are coupled, so rollback is by +**group** — each internally consistent and independently revertible: | Group | Contents | |---|---| @@ -866,40 +687,38 @@ therefore by **group**, each group internally consistent and independently rever | **G6 architect** | `roles/architect.md` + relocated skill content | | **G7 scar registry** | `scar-rules.yaml` + its enforcement test | -- **Dependency rule**: reverting **G7** requires reverting every group whose surfaces carry - scar text (G2, G3, G4, G6) — the registry and its copies must agree. All other groups are - mutually independent. +- **Dependency rule**: reverting **G7** requires reverting every group carrying scar text (G2, + G3, G4, G6) — registry and copies must agree. All other groups are mutually independent. - **Mechanism**: `git revert` restores prior bytes. No migration, state, schema, or data. Rehearsed under T10 before the PR merges. -- **Blast radius**: for this repo, effective for the next spawned builder — in-flight - builders keep the surface they were spawned with (prompts read at spawn/phase time). For - adopters, the revert ships in the next release; an adopter can also pin the prior - `@cluesmith/codev` version, since framework files resolve from the installed skeleton. -- **Triggers**: (a) any O4 violation — immediate, no deliberation; (b) an observed scar - violation in any real project post-merge; (c) O2/O3 outside pre-registered margins; - (d) architect judgment at the `pr` or `verify-approval` gate. +- **Blast radius**: effective for the next spawned builder — in-flight builders keep the surface + they were spawned with (prompts read at spawn/phase time). For adopters the revert ships in the + next release; an adopter can also pin the prior `@cluesmith/codev` version. +- **Triggers**: (a) any O4 violation — immediate, no deliberation; (b) an observed scar violation + in any real project post-merge; (c) O2/O3 outside pre-registered margins; (d) the 14-day + post-merge O3 window; (e) architect judgment at the `pr` or `verify-approval` gate. - **Partial rollback is the expected shape** — revert the offending group, keep the rest. - **Cost**: one revert, one release. No irreversible step exists anywhere in this project. ## Dependencies -- **External Services**: none. (`gh` for issue/PR reads during the A/B; consult backends — - Gemini via `agy`, Codex, Claude — unchanged.) +- **External Services**: none. (`gh` for issue/PR reads during the A/B; consult backends + unchanged.) - **Internal Systems**: four-tier resolver (`lib/skeleton.ts`); porch prompt composition (`commands/porch/prompts.ts`) — read, not modified; managed-block hot-tier wiring (`lib/managed-block.ts`); role injection (`agent-farm/commands/spawn-worktree.ts`); consult-type resolution (`commands/consult/index.ts`); both measurement scripts. - **Artifacts**: `builder/spir-1252` (ratified registry — must not be deleted); `codev/resources/1252-*.md` baselines. -- **Test suites**: the ~25 prose-pinned files enumerated in M10 — a dependency in the real - sense that the cuts cannot land without deliberately re-baselining them. +- **Test suites**: the ~25 prose-pinned files in M10 — a dependency in the real sense that the + cuts cannot land without deliberately re-baselining them. - **Libraries/Frameworks**: none new. ## References - Issue #1280 (charter); #1279 (dead spec/review templates); #1276, #1277 (filed by 1252 — - superseded here or out of scope). -- PR #1278 (Spec 1252, closed unmerged) and `builder/spir-1252`. + superseded here or out of scope); #1032/#1033 (agy `--type` review limitation). +- PR #1278 (Spec 1252, closed unmerged) and `builder/spir-1252`; PR #1290 (early-PR precedent). - `codev/reviews/1252-prompt-architecture-single-own.md`; `codev/resources/1252-*.md`; `codev/state/spir-1252_thread.md`. - *The new rules of context engineering for Claude-5-generation models* — @@ -911,75 +730,38 @@ therefore by **group**, each group internally consistent and independently rever | Risk | Probability | Impact | Mitigation Strategy | |------|------------|--------|-------------------| -| A deleted instruction was load-bearing; loss is silent | Medium | High | M5 deterministic capability inventory (removals fail); O4 zero-tolerance; grouped rollback | +| A deleted instruction was load-bearing; loss is silent | Medium | High | M5 inventory over served prompt text (unlisted removals fail); O4 zero-tolerance; grouped rollback | | A scar rule weakened or dropped | Low | **Critical** | Verbatim carriage; counted-in-ceiling so it is never trimmed to fit; byte-identical enforcement pinned at 8; hard rollback trigger; architect-only rewording | -| **Prose-pinned tests silently gutted to go green** | **High** | **High** | **M10**: every retired assertion named with its originating spec and its protected behaviour re-asserted or explicitly retired; pure-addition re-baselining only with the spec named and baseline committed together | -| Sweep misses a protocol, a tree, or a segment | High | Medium | T6 enumerates from disk; T7 twin parity; T11 segment reporting. 1252's dominant review cost was sweep-scope failure | -| >50% margin is thin (3.2 pts) | Medium | Medium | M2 ceilings are binding and M1 derived; a 50–52% landing is a HOLD with named further-cut candidates | -| A/B underpowered; subtle regression ships | Medium | Medium | Stated in the power statement; O4 binary compliance carries the protection; T13 post-merge behavioural re-measurement as a second net | -| Instrument defect #3 ships undetected | Medium | High | Principle 7; T1/T1b/T11 assert the instrument against the live resolver; M0b puts it under public review early | -| A/B costs more than the shrink saves | Medium | Low | 12 runs on backlog issues that needed doing; O5 tracks it; architect sets the pair ceiling | +| Prose-pinned tests silently gutted to go green | **High** | **High** | M10: every retired assertion named with its originating spec and its behaviour re-asserted or explicitly retired; pure-addition re-baselining only with spec named and baseline committed together | +| Sweep misses a protocol, tree, or segment | High | Medium | T6 enumerates from disk (both trees, unioned); T7 twin parity; T11 bucket reporting. Sweep-scope failure was 1252's dominant review cost | +| Relocation reported as deletion, inflating the claim | Medium | Medium | M0c + M0(g) + T15 | +| A/B underpowered; subtle regression ships | Medium | Medium | Power statement; O4 binary compliance carries the protection; T13 post-merge behavioural re-measurement as a second net | +| A further instrument defect ships undetected | Medium | High | Principle 7; T1/T1b/T11/T15 assert the instrument against the live resolver; M0b puts it under public review early; the script gets its first tests | +| A/B costs more than the shrink saves | Medium | Low | ~6 of 12 runs produce merged work; O5 tracks it; architect sets the pair ceiling | +| Architect review capacity is the trial's bottleneck | High | Medium | Load stated explicitly (~24 gates + 12 PR reviews); pair count is the architect's call; smaller n reported honestly in the power statement | | Surface re-grows after the project | High | Medium | T3 runs the ceilings in CI | | `builder/spir-1252` deleted, losing the registry | Low | High | Registry content quoted in this project's thread; rebuilt registry committed to `main` early | ## Expert Consultation -**Date**: 2026-07-31 (round 1) -**Models Consulted**: Codex (GPT-5.6 Sol) · Claude Opus 5. *(Gemini/`agy` was not in -porch's model set for this consultation — the known `--type` review limitation, #1032/#1033.)* -**Verdicts**: both REQUEST_CHANGES, both HIGH confidence. - -**Sections updated in response** (all feedback verified against source before acting): - -| Finding | Raised by | Resolution | -|---|---|---| -| Ceilings stated net-of-scar while M1 is gross — incompatible | both | All ceilings restated **gross** with carriage shown per row; arithmetic rebuilt; margin disclosed | -| M0 doesn't match the resolver (per-file four-tier vs two-tier directory selection) | Codex | M0(b) + **T1b** added | -| CLAUDE.md `@import`s the hot tier (#1119); 5,815 excludes 736 always-loaded words | Claude | Current State corrected; baseline 33,519 → **34,255**; M0(d) requires fixing the script's stale comment | -| "No code reads `porch/prompts`" is false — `review-prompt-routing.test.ts:29` | Claude | **Verified and conceded**; cause (truncated grep) recorded in Current State; M6 verification method rewritten | -| ~25 prose-pinned tests block the cuts; `baked-decisions.test.ts` pure-addition diff is incompatible with 824 → ≤420 | Claude | **New criterion M10** + new risk row + Dependencies entry | -| "Gate friction" left unresolved under Critical while the A/B assumes it | both | Resolved: O1 demoted to advisory-with-tripwire, rubric scale and recording location defined, incomplete-scoring behaviour specified | -| A/B lacks contamination controls | Codex | **Contamination controls** subsection added (env pinning, arm isolation, order, recording artifact) | -| A/B should exclude issues touching surfaces under test | Claude | Added as an eligibility rule | -| Per-surface rollback independence overstated | Codex | Rewritten as **seven rollback groups** with an explicit G7 dependency rule | -| M5's inventory diff not deterministic | Codex | Recognition + normalization rules specified; committed pre-rewrite artifact | -| T3-vs-CI open question contradictory | Claude | Open question withdrawn; T3 stands | -| Template pressure conflated (porch's 4 headings vs consult type's 20) | Claude | Named separately in Current State | -| Spec dated 2026-08-01 while project date is 2026-07-31 | Codex | Corrected to **2026-07-31** throughout — the architect's messages carried UTC timestamps (`02:50Z`), local time 19:50 on 2026-07-31 | - -**Not disputed.** Every round-1 finding was accepted; no rebuttal was filed. Two were -factual errors in my own Current State, both verified against source before correction. - -### Round 2 — 2026-07-31 (architect-directed re-review of the revised spec) - -Porch had advanced to the gate after the round-1 rebuttal without re-reviewing; the architect -directed a round 2 on the grounds that a revision absorbing 13 findings — including a -structural net→gross ceiling change — needs re-reading, and that porch advancing was -permissive machinery rather than a judgment. Correct call: round 2 found nine further issues, -two of them arithmetic errors that would have shipped. - -**Verdicts**: Codex REQUEST_CHANGES (HIGH) · Claude REQUEST_CHANGES (HIGH). Both again -verified the spec's factual claims against source before judging; Claude re-confirmed every -word count and the `REQUIRED_SPEC_SECTIONS`, `spawn-worktree.ts:854`, and -no-test-on-1252-artifacts claims. - -| Finding | Raised by | Resolution | -|---|---|---| -| Segment arithmetic underspecified — audience loads overlap and use different multipliers, so "subtotals sum to the total" is false | Codex | Current State split into **exclusive buckets** (partition, summable) vs **derived audience loads** (overlap by design), with the exact `ALWAYS_ON_WORDS` formula; M0(f) and **T11** restated | -| "All ten protocols in both trees" is impossible — `release` is project-local (10 vs 9), and several protocols intentionally lack `prompts/`/`consult-types/` | Codex | **Verified on disk.** Coverage restated as per-**surface**, enumerated from disk and unioned; absence never fails, unmeasured presence does. M2/M3/T6 rewritten | -| `release/protocol.md` (1,626w) missing from the inventory entirely | *(consequence of the above)* | Added; cause recorded — I enumerated the skeleton, where `release` does not exist. Second sweep-scope failure this phase, same root cause as the truncated grep | -| M5 proves nothing about prompts — gates/checks come from unchanged `protocol.json`, notifications from unchanged call sites | Codex | M5 rewritten to inventory the **resolved, expanded prompt surface** with contract-presence assertions per capability; named as the primary defence for the −81% `protocol.md` row | -| M5 self-contradictory: "any removal fails" vs "justified retirement" | **both** | Resolved on M10's pattern — hard fail unless the name is in a committed `1280-retirements.md` in the same commit, with architect approval | -| O3 timing ambiguous — post-merge defects cannot be evaluated at a pre-merge SHIP decision | Codex | Split: pre-merge architect findings gate SHIP; post-merge defects are a **14-day rollback signal** | -| A/B execution model undefined — what M7 gates, which arm merges, architect load | **both** | New **Execution and sequencing** subsection: M7 gates `verify-approval` (not the PR); treatment arm merges, control closes; cost restated honestly as ~6 of 12 runs producing merged work; ~24 gate approvals + 12 PR reviews named as the binding scheduling constraint | -| Relocation to skills scores identically to deletion — phantom savings on an axis T2 does not cover | Claude | **New M0c + M0(g) + T15**: report total authored surface, decompose the cut into deleted vs relocated. Caught by this spec's own principle 7 | -| M1's "HOLD at 50–52%" branch is arithmetically unreachable — M1 is fully derived from M2 | Claude | **Verified** (ceilings met ⇒ ≤16,016 = −53.2%; the HOLD band needs 16,442–17,128, i.e. M2 already failing). Dead prose withdrawn; replaced with the reachable contingency, **denominator movement** | -| No criterion protects CLAUDE.md's human readability at 5,815 → 1,900 | Claude | **New M2b** — architect reviews human usability at the gate; twin-parity bytes are not a readability check | -| Consultant surface reads as negligible at "−33.8% on 683" | Claude | Fleet-wide figure added: ≈20,500 words/project, comparable to the whole builder load | -| Exempt hot tier bounds the achievable cut | Claude | Quantified in the open question: 7,360 of 16,016 post-rewrite builder words (46%) | - -**Not disputed.** Every round-2 finding was accepted; no rebuttal filed. Both arithmetic -claims (M1 reachability, ceiling landing) were independently recomputed before acceptance. +**Round 1** — 2026-07-31 · Codex (GPT-5.6 Sol) + Claude Opus 5 · both REQUEST_CHANGES (HIGH) · +**13 findings, none disputed**. +**Round 2** — 2026-07-31, architect-directed re-review of the revision · same models · both +REQUEST_CHANGES (HIGH) · **9 findings, none disputed**. + +Every finding was verified against source (and both arithmetic claims independently recomputed) +before acceptance; all are folded into the criteria, tests, and design sections above rather than +narrated here. The finding-by-finding record is in +`codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter{1,2}-rebuttals.md`. + +Four of the corrections were errors in this spec's own analysis, three sharing a single root +cause — **enumerating from a convenient source instead of the authoritative one** (a truncated +grep; skeleton-only protocol enumeration; the measurement script's stale comment). That is the +sweep-scope class 1252 named as its dominant review cost, and it is why M3's "enumerate from +disk" is specified as a **test** rather than an instruction. + +Gemini/`agy` did not participate: the known `--type` review limitation (#1032/#1033). Per current +lane policy this 2-way review is correct and needs no remedy. ## Approval - [ ] Technical Lead Review @@ -989,18 +771,13 @@ claims (M1 reachability, ceiling landing) were independently recomputed before a ## Notes -**On the irony.** A spec arguing for deletion should not be padded. This one is long because -it carries four architect-mandated designs — cut plan, A/B, scar carriage, and now a -segmented full-surface inventory — plus an instrument correction that changes the project's -headline number twice. The artifacts it produces are the short ones. - -**On what this project deliberately does not do.** No prompt generator (Approach 2), no -tiering of any kind (Baked Decision 1), no porch behaviour changes, and not 1252's full -enforcement machinery — only the minimum scar-integrity check the deletion makes necessary, -plus the ceiling test that prevents re-growth. Enforcement built around a still-moving -surface is enforcement built twice. +**What this project deliberately does not do.** No prompt generator (Approach 2), no tiering of +any kind (Baked Decision 1), no porch behaviour changes, and not 1252's full enforcement +machinery — only the minimum scar-integrity check the deletion makes necessary, plus the ceiling +test that prevents re-growth. Enforcement built around a still-moving surface is enforcement +built twice. **On the deferred decision from 1252.** The architect's pr-gate ruling was that structural -machinery is not worth carrying for a surface about to halve. That sequencing is honoured: -shrink first, then enforce what remains. T3 is the smallest useful enforcement primitive, -and it only becomes meaningful once the ceilings exist. +machinery is not worth carrying for a surface about to halve. That sequencing is honoured: shrink +first, then enforce what remains. T3 is the smallest useful enforcement primitive, and it only +becomes meaningful once the ceilings exist. From c0529f1a5a98b4a874ca4443b9d58b63fb38916f Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:23:27 -0700 Subject: [PATCH 12/32] [Spec 1280] Thread: compression record and the one real loss it surfaced --- codev/state/spir-1280_thread.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index be8c8a5f6..158f0cee2 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -211,3 +211,29 @@ it is the argument for making M3's "enumerate from disk" a *test* rather than an Belongs in the review's lessons learned. Spec now 9,415w (iteration 3). Porch remains at spec-approval; I have not touched the gate. + +### Compression for signal (2026-07-31) — 9,415 → 6,860 words (−27%) + +Architect-directed before the gate; I had flagged the length myself. Rule applied: **fold, +don't drop.** + +**Kept whole**: all 14 M-criteria, all 16 tests, 5 outcomes, 7 rollback groups, the full +per-audience inventory, the rollout contract, the A/B design, and the four Baked Decisions +verbatim. + +**Cut**: the narration of how the spec got here — the two round-by-round finding tables +(~1,400w), "(round-2 finding)" asides threaded through the criteria, the superseded 33,519 +figure, and the withdrawn HOLD-band arithmetic. That record lives in the iter1/iter2 rebuttal +files and this thread, which is where the architect said it belongs. + +**Self-verified mechanically** (diffed old vs new): 14/14 criteria, 16/16 tests, 5/5 outcomes, +7/7 groups, all 20 template headings in order, porch's `spec_has_required_sections` passes. +Diffed every number too — eleven appeared to vanish; ten were from the deleted narration or +the withdrawn HOLD band (correctly gone). **One was a real loss**: the margin figure (−53.2% +clears >50% by 3.2 points ≈ 1,100 words), which was Claude's thin-margin warning and is real +information for an approver. Folded back into M1 as a clause rather than left dropped. + +That number-level diff is the compression analogue of principle 7 — "I kept the criteria" is +not the same claim as "I kept the content," and only the diff distinguishes them. + +Gate remains pending; still not touched. From 8b13c4ae33561450f7d39c8de4f61622cbd88748 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:31:53 -0700 Subject: [PATCH 13/32] [Spec 1280] Revise acceptance model: principle conformance, not size Gate verdict NOT APPROVED; Waleed redirected the acceptance model: 'I don't think the goal should be a particular size. That's not the right criteria. It should be to stick to the principles outlined in the blog post.' Acceptance is now principle conformance, judged per file: - Fetched the blog and quoted its principles VERBATIM as P1-P7, each restated as a per-file conformance question answerable from a diff. P5 (auto-memory) is declared N/A with reason rather than silently skipped: it is a Claude Code harness feature and Baked Decision 1's fleet includes GPT/Gemini consumers. - P7 (delete worst-case guardrails) collides with the scar-rule exemption. Named the collision and resolved it explicitly rather than papering over: the blog's guardrails protected against bad output, which judgment now handles; scar rules protect against irreversible acts, where being wrong once is unbounded. DEMOTED from acceptance to observability: the >50% target, all per-surface word ceilings, per-segment reduction goals, and the thin-margin analysis. M1/M2 are now reporting obligations that cannot fail on a number; T3 asserts report completeness, not ceilings. M0/M0b/M0c survive - they keep the project honest about what happened. NEW M11 - architect personal inspection of every changed file: per-file manifest (path, old/new wordcount, principles applied, rationale), architect reviews actual diffs file by file before each phase advances, batches <=12 files. Sized it honestly: 131 prompt-bearing files exist, but 60 codev/protocols copies are byte-identical mirrors of their skeleton twins, so inspection covers ~66 distinct content decisions with twin sync verified mechanically by T7 - otherwise half the human review is re-reading identical bytes. New T16 fails a phase if a changed file is missing from its manifest. Approach 2 (generated prompts) now also rejected as incompatible with M11: the architect cannot inspect old-vs-new diffs of files that no longer exist. UNCHANGED per directive: A/B mandatory, scar exemption, Baked Decisions, PR-1 early landing, M10 assertion-retirement-as-governance. --- .../1280-prompt-surface-judgment-not-ru.md | 733 +++++++++--------- 1 file changed, 358 insertions(+), 375 deletions(-) diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md index 452523d54..9a62065ac 100644 --- a/codev/specs/1280-prompt-surface-judgment-not-ru.md +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -1,20 +1,22 @@ -# Specification: Prompt surface — judgment-not-rules rewrite (>50% always-on reduction) +# Specification: Prompt surface — judgment-not-rules rewrite (principle conformance) ## Metadata - **ID**: spec-2026-07-31-prompt-surface-judgment-not-rules -- **Status**: draft (compressed; CMAP rounds 1–2 incorporated) +- **Status**: draft (acceptance model revised to principle conformance) - **Created**: 2026-07-31 - **Issue**: #1280 - **Protocol**: SPIR @@ -22,66 +24,68 @@ and codev/state/spir-1280_thread.md — deliberately not narrated here. ## Clarifying Questions Asked Issue #1280 carries a complete charter, so no clarifying questions were put to the architect -before drafting. Four were resolved against the repository or by architect ruling: +before drafting. Five were resolved against the repository or by architect ruling: 1. **What is the always-on surface, as served?** — Read the live composition path (`commands/porch/prompts.ts`, `lib/skeleton.ts`, `lib/managed-block.ts`, `agent-farm/commands/spawn-worktree.ts`) and measured this builder's own served artifacts (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). See **Inventory**. -2. **Is the committed measurement script fit to score this project?** — No; three defects, one +2. **Is the committed measurement script fit for this project?** — No; three defects, one disqualifying. Criterion **M0**; architect verified against source and endorsed. -3. **How wide is the rewrite target?** — Architect scope directive: the **entire** prompt - surface — architect roles, builder roles and spawn wrappers, consultant/CMAP prompts, phase - prompts and their template includes, `protocol.md` texts — with the instrument **segmented - by audience** so a cut in one segment masking growth in another stays visible. +3. **How wide is the rewrite target?** — The **entire** prompt surface: architect roles, + builder roles and spawn wrappers, consultant/CMAP prompts, phase prompts and their template + includes, `protocol.md` texts. 4. **Where are the eight ratified scar rules?** — Recovered verbatim from `builder/spir-1252:codev/resources/scar-rules.yaml`. +5. **What is the acceptance criterion?** — **Principle conformance, not size** (Waleed, above). + Measurement is retained for honesty, not for grading. ## Problem Statement -A Codev builder consumes **34,255 served always-on words** before reading a line of the code it -was spawned to change. Almost none is information it could not derive; it is *process -narration* — recipes for how to be an agent, written when the fleet could not be trusted to -infer them. +Codev's prompt surface was written for a fleet that could not be trusted to infer process. It +tells agents how to be agents: ordered procedures, all-caps prohibitions, checklists restating +the phase body, annotated templates. A builder consumes **34,255 served always-on words** +before reading a line of the code it was spawned to change. Spec 1252 proved the obvious remedy fails: deduplication yields **−7.0%**, because the surface -is not duplicated, it is **over-instructed**. Three compounding costs: +is not duplicated, it is **over-instructed**. Anthropic's published account of the +Claude-5-generation rewrite reports: + +> "We removed over 80% of Claude Code's system prompt for models like Claude Opus 5 and Claude +> Fable 5 with no measurable loss on our coding evaluations." + +Three compounding costs: 1. **It crowds out judgment.** A model given a 3,703-word procedure follows the procedure; - given a 700-word contract and a goal, it reasons about the goal. Anthropic's published - account of the Claude-5-generation rewrite reports >80% of Claude Code's system prompt - deleted with no measurable performance loss. + given a contract and a goal, it reasons about the goal. 2. **Nobody reads it, so it rots.** 1252 found the *served* SPIR builder prompt had silently lost its entire `Verify Phase` section, with a detector reporting the drift, unread, for months. A surface too large to read is too large to maintain. -3. **It makes its own success unmeasurable.** The committed script scores a directory the - runtime never loads. +3. **It makes its own success unmeasurable.** The committed measurement script scores a + directory the runtime never loads. Deletion on judgment-trust grounds was an explicit **Non-goal** of Spec 1252. ### Principle: the instrument is part of the deliverable -This is the second measurement defect in the 1252 lineage — the first being that 1252 -originally shipped with no measurement plan at all, caught at a human gate, not by CMAP. -Neither was found by reading the instrument's code; both by asking *what does this claim to -measure, and does it?* +This is the second measurement defect in the 1252 lineage — the first being that 1252 shipped +without a measurement plan at all, caught at a human gate, not by CMAP. Neither was found by +reading the instrument's code; both by asking *what does this claim to measure, and does it?* > **The instrument is part of the deliverable, and instruments get reviewed against what they -> claim to measure — not merely against whether they run.** A measurement script, check, or -> baseline artifact is subject to the same adversarial review as the feature it scores. -> "Deterministic and committed" is not "correct." +> claim to measure — not merely against whether they run.** "Deterministic and committed" is +> not "correct." -Load-bearing here because this project's headline criterion is a number one shell script emits. -It is why **M0 precedes M1**, why **M0c** exists (an always-on metric cannot tell deletion from -relocation), and why **M5** inventories prompt text rather than the config files this project -does not touch. +Under the revised acceptance model the instrument no longer *grades* the work — but it still +keeps the project honest about what actually happened, which is why **M0**, **M0b** and +**M0c** survive the demotion of every word target. ## Current State ### Inventory — every prompt-bearing surface, by audience Served and expanded words (`{{> …}}` includes resolved `codev/` → `codev-skeleton/`), captured -2026-07-31 at `047f92f7`. +2026-07-31 at `047f92f7`. **Reported for observability; no figure here is a target.** **SHARED — every agent in this repo** @@ -94,9 +98,8 @@ Served and expanded words (`{{> …}}` includes resolved `codev/` → `codev-ske **Session shared total: 6,551.** -**ARCHITECT** — `roles/architect.md` 2,048 (read at `arch-init`, every session); -`.claude/skills/*/SKILL.md` ×10 = 6,672 (on-demand — progressive disclosure working as -intended). +**ARCHITECT** — `roles/architect.md` 2,048 (read at `arch-init`); `.claude/skills/*/SKILL.md` +×10 = 6,672 (on-demand — progressive disclosure already working as intended). **BUILDER — spawn, once per builder** @@ -113,13 +116,13 @@ intended). | `protocols/bugfix/builder-prompt.md` | 429 | | `protocols/air/protocol.md` | 643 | | `protocols/spike/builder-prompt.md` | 400 | | `protocols/maintain/builder-prompt.md` | 374 | -**SPIR builder spawn total: 6,364** (role 1,837 + wrapper 824 + protocol 3,703). +**SPIR builder spawn total: 6,364.** **PHASE — per porch task delivery, ×I.** Hot tier (736) rides on *every* phase prompt. | Protocol | Prompts (expanded) | Mean | |---|---|---:| -| spir / aspir | specify 1,402 · plan 1,169 · implement 1,065 · review 1,957 | **1,398** | +| spir / aspir | specify 1,402 · plan 1,169 · implement 1,065 · review 1,957 | 1,398 | | pir | review 2,414 · implement 1,151 · plan 741 | 1,435 | | bugfix | pr 491 · fix 352 · investigate 290 | 378 | | air | pr 471 · implement 442 | 457 | @@ -127,15 +130,16 @@ intended). **CONSULTANT — per CMAP review.** `roles/consultant.md` 252 + one consult-type: spir/aspir spec 514 · impl 421 · phase 421 · plan 406 · pr 392; bugfix pr 726 / impl 641; pir pr 475 / impl -507; air pr 455 / impl 420; maintain 421 / 392. **SPIR per review: 683.** +507; air pr 455 / impl 420; maintain 421 / 392. **SPIR per review: 683.** Fleet-wide ≈ 683 × 3 +models × ~10 reviews ≈ **20,500 words/project**. **DEAD** — `codev-skeleton/porch/prompts/**`, 10 files, 4,009 words, no runtime consumer. -### Buckets and audience loads are different things +### How the reported figures are composed -**Surface buckets are exclusive** and partition the authored surface with no overlap or gap: -`SHARED` (6,551) · `ARCHITECT` (2,048) · `BUILDER_SPAWN[p]` · `PHASE[p]` · `CONSULTANT[p]` · -`DEAD` (4,009). **Audience loads are derived** and deliberately overlap: +Buckets are **exclusive** and partition the authored surface: `SHARED` (6,551) · `ARCHITECT` +(2,048) · `BUILDER_SPAWN[p]` · `PHASE[p]` · `CONSULTANT[p]` · `DEAD` (4,009). Audience loads +are **derived** and deliberately overlap — so they are reported separately, never summed: ``` HOT = arch-critical + lessons-critical = 736 @@ -144,27 +148,35 @@ ALWAYS_ON(architect) = SHARED + ARCHITECT ALWAYS_ON(consultant,p) = roles/consultant.md + mean CONSULTANT-type[p] ALWAYS_ON_WORDS ≡ ALWAYS_ON(builder, spir, 10) - = 6,551 + 6,364 + 10 × (736 + 1,398) = 34,255 ← the headline + = 6,551 + 6,364 + 10 × (736 + 1,398) = 34,255 ``` -Architect load 8,599; consultant 683. `I = 10` is 1252's proxy, consistent with B4's 3.06 -review rounds/project across 4–6 phases — a comparison constant, identical before and after. +Architect load 8,599; consultant 683. `I = 10` is 1252's proxy — a comparison constant, +identical before and after. These definitions exist so M0's report is unambiguous; **none of +them is a target.** -Fleet-wide the consultant surface is not small: 683 × 3 models × ~10 reviews ≈ **20,500 -words/project**, comparable to the entire builder load. It sits outside the headline because it -is per-review, not because it is negligible. +### File counts — what the architect will personally inspect -### Coverage is per-surface, not per-protocol × surface-type +| | Count | +|---|---:| +| Prompt-bearing `.md` files, both trees + `CLAUDE.md`/`AGENTS.md` | **131** | +| Of which `codev/protocols` copies **byte-identical** to their skeleton twin | **60** | +| `codev/protocols` copies that differ | 3 (`maintain/templates/audit-report.md`, `maintain/templates/lessons-learned.md`, `release/protocol.md` — the last has no skeleton twin) | +| `roles/*.md` — all three byte-identical across trees | 3 pairs | +| `CLAUDE.md` ≡ `AGENTS.md` | verified identical | +| **Distinct content decisions** | **~66** | -Verified on disk: `codev/protocols/` holds **ten** protocols, `codev-skeleton/protocols/` -**nine** — `release` is project-local by design, has only `protocol.md` (1,626w, no -`protocol.json`, no `builder-prompt.md`), is human-invoked prose an agent reads, and is -therefore in scope but not porch-orchestrated. `experiment`, `research`, `spike` and `release` -have no `prompts/` or `consult-types/`; those absences are intentional. +This matters for M11: reviewing all 131 diffs would mean re-reading ~65 byte-identical copies. +The inspection is over **distinct content decisions**, with twin sync verified mechanically. -Ceilings and sweep criteria therefore apply to **each surface that exists after resolution**, -enumerated from disk across both trees and unioned. Absence never fails a check; an -**unmeasured** present surface does. +### Coverage is per-surface, not per-protocol × surface-type + +`codev/protocols/` holds **ten** protocols, `codev-skeleton/protocols/` **nine** — `release` is +project-local by design, has only `protocol.md` (1,626w, no `protocol.json`, no +`builder-prompt.md`), is human-invoked prose an agent reads, and is in scope but not +porch-orchestrated. `experiment`, `research`, `spike` and `release` have no `prompts/` or +`consult-types/`; those absences are intentional. Coverage criteria therefore apply to **each +surface that exists after resolution**, enumerated from disk across both trees and unioned. ### The measurement defects (why M0 exists) @@ -180,13 +192,8 @@ enumerated from disk across both trees and unioned. Absence never fails a check; session launch, so `wc -w CLAUDE.md` **excludes** 736 always-loaded words. Net: the reported baseline (21,702) understates the phase term ~3.5×, omits the role file, and -under-counts the session term by 736. **And the metric cannot see this project's primary -target** — cutting SPIR phase prompts 1,398 → 430 moves `ALWAYS_ON_WORDS` by exactly zero. A ->50% claim scored on it would be phantom savings. - -The dead tree has no *runtime* consumer but does have a **test** consumer: -`review-prompt-routing.test.ts:29` asserts on `codev-skeleton/porch/prompts/review.md` (a Spec -987 hot/cold-routing protection). M6 handles it. +under-counts the session term by 736. The dead tree has no *runtime* consumer but does have a +**test** consumer: `review-prompt-routing.test.ts:29` (a Spec 987 hot/cold-routing protection). ### What landed, what is deferred @@ -195,204 +202,176 @@ behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded; B2 1.12 and the measurement tooling. Deferred here by architect ruling: the **scar registry and its eight ratified wordings**, with enforcement rebuilt *after* the shrink. -Issue #1279 is partly overtaken — SPIR prompts now inline templates via `{{> …}}`, which is what -makes each phase prompt ~600 words heavier than it reads. Two *separate* constraints govern -template shape and must not be conflated: `checks.ts:149-154` (`REQUIRED_SPEC_SECTIONS`) -requires only **four** headings; the 20-heading pressure comes from the `spec-review` consult -type, advisorily. +Issue #1279 is partly overtaken — SPIR prompts now inline templates via `{{> …}}`. Two separate +constraints govern template shape and must not be conflated: `checks.ts:149-154` +(`REQUIRED_SPEC_SECTIONS`) requires only **four** headings; the 20-heading pressure comes from +the `spec-review` consult type, advisorily. ## Desired State -**One prompt form, written for frontier models, that states contracts and trusts judgment.** A -builder's always-on context says what it owns, what artifacts it must produce and their shape, -where the human gates are, and what is irreversible — then gets out of the way. Everything else -is reachable on demand. - -### Rewrite principles - -1. **Contract, not recipe.** State the required outcome and its shape; delete the procedure. -2. **Interface, not example.** A heading skeleton with one line of intent per heading replaces - an annotated template with filler prose. -3. **No worst-case padding.** Delete instructions guarding failure modes frontier models do not - exhibit (all-caps prohibitions, "⚠️ BLOCKING" banners, checklists restating the body). -4. **Progressive disclosure.** How-to content moves to skills / on-demand files, addressed by - name. Relocation is *not* deletion and is measured separately (M0c). -5. **Budgets are cheap words worth keeping.** Frontier models honour stated budgets precisely - but never invent them; budget/scope lines are exempt from cuts. -6. **Scar rules are verbatim and exempt from rewriting** — but **counted** in ceilings. -7. **The instrument is part of the deliverable.** A ceiling not measured on served words is not - a ceiling. - -### Per-surface ceilings (GROSS — scar words count inside every ceiling) - -Applied to the SPIR instance as the measured proxy and swept across every existing surface in -both trees. - -| Segment | Surface | Now | Ceiling (gross) | Scar carriage inside | -|---|---|---:|---:|---:| -| shared | `CLAUDE.md` / `AGENTS.md` | 5,815 | **≤1,900** | ~190 (all 8) | -| shared | hot tier (`@import`) | 736 | **736 unchanged** | — | -| architect | `roles/architect.md` | 2,048 | **≤700** | ~30 | -| builder | `roles/builder.md` | 1,837 | **≤600** | ~12 | -| builder | `protocols/*/protocol.md` (SPIR) | 3,703 | **≤700** | — | -| builder | `protocols/*/builder-prompt.md` (SPIR) | 824 | **≤420** | ~40 | -| phase | `protocols/*/prompts/*.md` expanded (SPIR mean) | 1,398 | **≤430** | ~14 | -| consultant | `protocols/*/consult-types/*.md` (SPIR mean) | 431 | **≤200** | — | -| consultant | `roles/consultant.md` | 252 | **≤252 unchanged** | — | -| dead | `codev-skeleton/porch/prompts/**` | 4,009 | **0 (deleted)** | — | - -### Post-rewrite always-on, by audience +**Every prompt-bearing file conforms to the blog's principles.** Not "is smaller" — conformant. +A file that is principle-conformant at more words passes; a file that hits any size one might +have hoped for while still narrating procedure fails. -``` -BUILDER (SPIR, I=10) - session (1,900 + 736) 2,636 - + spawn (600 + 700 + 420) 1,720 - + phase ×10 (736 + 430) 11,660 - ----------------------------------------------- - 34,255 → 16,016 −53.2% - -ARCHITECT 8,599 → 3,336 (2,636 session + 700 role) −61.2% -CONSULTANT 683 → 452 (252 role + 200 consult-type) −33.8% -``` +### The principles, verbatim + +Quoted from *The new rules of context engineering for Claude-5-generation models*. These are +the acceptance basis; each is restated as a per-file question the architect can answer from a +diff. -No segment grows. Consultant cuts least because `roles/consultant.md` is already lean at 252 — -reported rather than averaged into the headline. +| # | Blog transition | Verbatim rationale | Per-file conformance question | +|---|---|---|---| +| **P1** | "Give Claude rules" → **"Let Claude use judgement"** | "newer models have better judgement and can handle these decisions well without explicit rules." | Does this file state a *contract* (what must be true) rather than a *procedure* (what steps to take)? Is every remaining rule one a frontier model would get wrong without it? | +| **P2** | "Give Claude examples" → **"Design interfaces"** | "giving examples actually constrains them to a certain exploration space. Instead of using examples, think more about the design of your tools, scripts and files." | Has each illustrative example been replaced by an interface — a heading skeleton, a schema, a named artifact contract? | +| **P3** | "Put it all upfront" → **"Use progressive disclosure"** | "Claude Code has gotten very competent at using progressive disclosure- loading the right context at the right times." | Is everything in this always-on file needed *every* time? Has look-it-up content moved to a skill or on-demand file, addressed by name? | +| **P4** | "Repeat yourself" → **"Simple tool descriptions"** | "We found we could delete these repeat examples and put instructions on how to use tools in the tool descriptions rather than the system prompt." | Do CLI/tool how-tos live with the tool (skills, `--help`) rather than in the prompt? Is anything repeated here that another surface already owns? | +| **P5** | "Memory in CLAUDE.md files" → **"Auto-memory"** | "Claude now automatically saves memories that are relevant to the work and to you." | **N/A for this project, with reason** — auto-memory is a Claude Code harness feature, and Baked Decision 1's fleet includes GPT 5.6 and Gemini 3.6 consumers with no equivalent. Codev's governance content stays in the hot tier rather than relying on it. Declared rather than silently skipped. | +| **P6** | "Simple specs" → **"Rich references"** | "Claude can handle increasingly more complicated references. Instead of simple markdown files, Claude can reference HTML artifacts." | Where prose restates machine-readable truth (`protocol.json` gates, checks, phases), does the file reference the structured source instead of narrating it? | +| **P7** | **"Unhobbling"** — worst-case guardrails | Old guardrails existed because "we needed to be sure that Claude avoided worst case scenarios, such as deleting files." | Has defensive padding written for weaker models been deleted — **except** the ratified scar rules (below)? | + +**P7 and the scar rules — the one deliberate exception.** The blog's worst-case example +("deleting files") is precisely the class Codev's scar rules guard. Baked Decision 2 keeps all +eight verbatim. This is a knowing departure from P7, not an oversight: the blog's guardrails +protected against *bad output*, which judgment now handles; scar rules protect against +*irreversible acts* — destroyed worktrees, killed sessions, bypassed human gates — where the +cost of being wrong once is unbounded and no amount of judgment makes the wager sensible. Every +other P7 candidate goes. + +### Conformance is judged per file, by the architect + +Acceptance is not a number and not a CMAP verdict. Each rewritten file carries a +**conformance record** — principles applied, what was cut and why, old and new word counts — +and the architect inspects the actual old-vs-new diff. See **M11**. + +### Word counts: measured, reported, never a gate + +The corrected instrument still runs before and after, still reports per-audience loads, and +still separates deletion from relocation (**M0c**) — because a project that deletes 20,000 +words should be able to say truthfully where they went. **No criterion passes or fails on any +of these numbers.** The rewrite's own projection, kept purely so the reported figures have +something to be compared against, is roughly 34,255 → ~16,000 for a SPIR builder; if principle +conformance lands somewhere else, the number moves and the spec does not. -The phase term is 73% of the post-rewrite builder budget, and the **exempt** hot tier is 7,360 -of the 16,016 total (46%). What survives is overwhelmingly curated judgment, not process — and -that exemption bounds how far this project can go without reopening it. +### Scar-rule carriage plan + +The eight rules ratified 2026-07-28 ship **verbatim** (~188 words): `git-add-explicit`, +`never-destroy-worktrees`, `no-destructive-git`, `human-gates`, `no-hand-edit-status`, +`afx-from-root`, `shellper-verified-orphan`, `tower-restart-permission`. + +- The registry is **rebuilt after the shrink** — each rule's `must_appear_on` re-derived against + the post-rewrite surface. +- Enforcement is a byte-identical-presence test, pinned at 8 rules and their ids. +- A scar rule may be compressed **only by architect ratification**, never by a builder applying + P1 or P7. ### Rollout: the corrected instrument lands on `main` first (M0b) -**No prompt-surface word is cut before the corrected instrument is on `main`.** A cut scored by -the current script is unfalsifiable, and 1252's published baselines cite dead-tree figures while -being shared knowledge other work reads. +**No prompt-surface word is rewritten before the corrected instrument is on `main`.** 1252's +published baselines cite dead-tree figures while being shared knowledge other work reads. **PR-1 (early, standalone) contains exactly:** | In | Out (deliberately) | |---|---| | Corrected `measure-prompt-surface.sh` — all seven M0 items | Any edit to any prompt surface | -| Its tests (T1, T1b, T2, T11, T12, T15). The script has **no test at all** today, which is how three defects survived in a "committed and reproducible" instrument | The scar registry (rebuilt after the shrink, per Baked Decision 2) | -| `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline (34,255) | The dead-tree deletion (has a test consumer → M10 governance) | +| Its tests (T1, T1b, T2, T11, T12, T15). The script has **no test at all** today, which is how three defects survived in a "committed and reproducible" instrument | The scar registry (rebuilt after the shrink) | +| `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline | The dead-tree deletion (has a test consumer → M10 governance) | | In-place annotation of `1252-word-baseline.md` and `1252-word-after-phase7.md`: originals **preserved**, marked superseded, reason + pointer | Re-derivation of 1252's behavioural baseline (B1 stands; M8 re-runs post-merge) | -**Timing**: end of the first implement phase — instrument correct and tested, before any cut. **Verified safe**: no test asserts on either 1252 word-count artifact (the frozen-sample test at -`prompt-behavior-metrics.test.ts:184` pins the *behavioural* sample, a different instrument), so -the annotation carries no re-baselining cost. - -The one argument against early landing — that the corrected baseline is meaningful only -alongside the cuts it scores — fails: the baseline's value is precisely that it is *pre-cut*, -and publishing it early is what makes the eventual −53.2% claim checkable by someone who did not -watch it being produced. - -### Scar-rule carriage plan - -The eight rules ratified 2026-07-28 ship **verbatim** (~188 words of canonical text): -`git-add-explicit`, `never-destroy-worktrees`, `no-destructive-git`, `human-gates`, -`no-hand-edit-status`, `afx-from-root`, `shellper-verified-orphan`, `tower-restart-permission`. - -- The registry is **rebuilt after the shrink** — each rule's `must_appear_on` re-derived against - the post-rewrite surface, since most 1252-listed files will be rewritten or deleted. -- Carriage is **exempt from rewriting but counted in ceilings**: a ceiling a surface cannot meet - while carrying its scar rules is a wrong ceiling, raised deliberately — never met by trimming - scar text. -- Enforcement is a byte-identical-presence test, pinned at 8 rules and their ids. -- A scar rule may be compressed **only by architect ratification**, never by a builder applying - principle 1. +`prompt-behavior-metrics.test.ts:184` pins the *behavioural* sample, a different instrument). ## Stakeholders - **Primary Users**: builder agents; CMAP reviewer agents; architect agents. -- **Secondary Users**: humans who must read and maintain the surface (protected by M2b); - downstream adopters receiving it via `codev update`. -- **Technical Team**: this builder; the architect at both gates. -- **Business Owners**: Waleed — ratifies scar wordings, rules on the A/B verdict and on any - ceiling change. +- **Secondary Users**: humans who must read and maintain the surface (M2b); downstream adopters + receiving it via `codev update`. +- **Technical Team**: this builder; the architect, who personally inspects every changed file. +- **Business Owners**: Waleed — sets the acceptance model, ratifies scar wordings, rules on the + A/B verdict. ## Success Criteria -- [ ] **M0 — the metric measures what is served, segmented by audience.** The corrected script - (a) sources phase prompts from the directory `loadPromptFile` resolves; (b) resolves - **per-file through the full four-tier chain** as `resolveCodevFile` does, not two-tier - directory-level selection, so mixed per-file overrides measure correctly; (c) counts the - inlined `roles/builder.md`; (d) counts hot-tier `@import` transclusion in the session term - **and corrects the stale inlining comment**; (e) expands `{{> …}}` includes; (f) reports - **exclusive bucket subtotals and derived audience loads separately**, per the stated - formulas, never presenting overlapping audience figures as a sum; (g) reports **total - authored prompt-surface words** (both trees + `.claude/skills/`) alongside always-on. - Tests assert (a) and (b) against the real resolver. +**Acceptance basis: MP1–MP7 (principle conformance) plus M11 (architect inspection). The +measurement criteria M0/M0b/M0c exist for honesty; M1 and M2 are reporting obligations that +cannot fail on a number.** + +- [ ] **MP — every prompt-bearing file conforms to P1, P2, P3, P4, P6 and P7** (P5 declared + N/A with reason), judged per file by the architect against the verbatim principle table. + A file passes on conformance regardless of its word count. Non-conformance at any size is + a failure. +- [ ] **M11 — architect personal inspection of every changed file.** Each implement phase ends + with a **per-file manifest**: path · old word count · new word count · principles applied · + one-line rationale for what was cut. The architect reviews **actual old-vs-new diffs, file + by file** — not samples, not summaries, not CMAP-mediated — before the phase advances. + Batches are humanly sized: **≤12 distinct files per review batch**. Inspection is over the + **~66 distinct content decisions**, not all 131 file-diffs — 60 `codev/protocols` copies + are byte-identical mirrors of their skeleton twins, so reviewing both would be re-reading + the same bytes; twin sync is verified mechanically by **T7** instead. Any file the + architect judges non-conformant returns to the builder before the phase advances. +- [ ] **M0 — the metric measures what is served.** The corrected script (a) sources phase + prompts from the directory `loadPromptFile` resolves; (b) resolves **per-file through the + full four-tier chain** as `resolveCodevFile` does; (c) counts the inlined + `roles/builder.md`; (d) counts hot-tier `@import` transclusion **and corrects the stale + inlining comment**; (e) expands `{{> …}}` includes; (f) reports exclusive bucket subtotals + and derived audience loads separately; (g) reports **total authored prompt-surface words** + (both trees + `.claude/skills/`). Tests assert (a) and (b) against the real resolver. - [ ] **M0b — the corrected instrument and baseline land on `main` early**, as a small - standalone PR (precedent #1290), per **Desired State → Rollout**. -- [ ] **M0c — deleted words are distinguished from relocated words.** Relocation to skills - scores identically to deletion under an always-on-only metric — the phantom-savings class - T2 catches on the *include* axis, unmonitored on the *relocation* axis. A −53% headline is - equally consistent with −30% deleted + −23% relocated, and only deleted content satisfies - Problem Statement claim 1. The review **decomposes the always-on reduction into deleted vs - relocated**, evidenced by M0(g). -- [ ] **M1 — >50% reduction (derived, arithmetically implied by M2).** `ALWAYS_ON_WORDS` falls - from 34,255 to **≤16,100**, measured before and after by the same corrected script, both - figures committed; audience loads reported, none regressed. Meeting every ceiling yields - ≤16,016 (−53.2%), so M1 cannot fail while M2 passes — but it clears >50% by only **3.2 - points ≈ 1,100 words**, so no ceiling has slack to give away. The reachable contingency is - **denominator movement**: if correcting the instrument surfaces always-on content not yet - found, the baseline and every ceiling are re-derived to preserve >50%, and that - re-derivation goes to the architect rather than being absorbed silently. -- [ ] **M2 — per-surface ceilings met (binding).** Every ceiling, **gross**, on **every surface - existing after resolution**, enumerated from disk across both trees and unioned. Absence - never fails; an unmeasured present surface does. -- [ ] **M2b — CLAUDE.md stays human-readable.** At 5,815 → ≤1,900 the rewritten file must retain - a navigable heading structure and be reviewed by the architect for human usability at the - gate — twin-parity bytes are not a readability check. + standalone PR (precedent #1290), per **Rollout**. +- [ ] **M0c — deleted words are distinguished from relocated words.** P3 authorizes moving + content to skills, and relocation scores identically to deletion under an always-on-only + metric. The review **decomposes the reduction into deleted vs relocated**, evidenced by + M0(g). This is a reporting obligation, not a threshold. +- [ ] **M1 — before/after figures measured and published.** Same corrected script both sides; + both committed as generated artifacts; per-audience loads reported. **No pass/fail + threshold attaches to any of them.** +- [ ] **M2 — per-file word counts appear in every manifest** (M11), so the architect sees the + size effect of each decision while judging conformance. **Reporting only; no ceilings.** +- [ ] **M2b — CLAUDE.md stays human-readable.** The rewritten file retains a navigable heading + structure and is reviewed by the architect for human usability — twin-parity bytes are not + a readability check. - [ ] **M3 — sweep completeness.** Surfaces enumerated from disk (both trees, unioned), never a - hardcoded list; no surface retains pre-rewrite content. Includes - `codev/protocols/release/protocol.md` (1,626w), which has no skeleton twin. + hardcoded list; every existing surface is rewritten and inspected. Includes + `codev/protocols/release/protocol.md`, which has no skeleton twin. - [ ] **M4 — scar rules intact.** Eight canonicals byte-identical on every registered surface; test fails on reword or deletion; count pinned at 8. - [ ] **M5 — no capability lost, proven against the prompt text.** A committed `capability-inventory.json` extracted pre-rewrite with explicit recognition rules — - artifact paths (`codev/(specs|plans|reviews)/…` literals and `{{artifact_name}}` forms), - gate names, signal names (`` tags), porch check names, notification triggers — - normalized (lowercase, strip backticks/punctuation, dedupe). + artifact paths, gate names, signal names (`` tags), porch check names, + notification triggers — normalized (lowercase, strip backticks/punctuation, dedupe). **The inventory is over the resolved, expanded prompt surface, not over `protocol.json` or - source call sites.** Extracting gate and check names from an unchanged `protocol.json`, or - notifications from unchanged `afx send` call sites, would report every capability present - even if every corresponding instruction vanished from the served prompts. Each item must - be evidenced as **represented in served prompt text** via a contract-presence assertion. - This is the primary defence for the most aggressive row, `protocol.md` 3,703 → ≤700 - (−81%), the builder's only map of gates, artifacts and phases. + source call sites** — extracting gate names from an unchanged `protocol.json` would report + every capability present even if every corresponding instruction vanished from the served + prompts. Each item must be evidenced as represented in served prompt text. **Severity**: a removal is a hard failure **unless** the retired name appears in a committed `codev/resources/1280-retirements.md` in the same commit, naming the capability, - why it is obsolete, and the architect approval. Retirements will occur (M6 deletes a tree; - `protocol.md` drops ~3,000 words), so the exception path is explicit rather than - improvised. + why it is obsolete, and the architect approval. - [ ] **M6 — the dead tree is gone, with its consumer handled.** `codev-skeleton/porch/prompts/` deleted. Verification is **not** a bare grep: an untruncated repo-wide search reconciled against the full hit list shows zero *runtime* consumers, and the one **test** consumer (`review-prompt-routing.test.ts:29`, a Spec 987 protection) is updated under M10 naming Spec 987. - [ ] **M7 — A/B non-inferiority passes** per the pre-registered decision rule; gates - `verify-approval`. + `verify-approval`. Mandatory per the charter: behavioural outcomes are the evidence the + principles are working. - [ ] **M8 — behavioural baseline re-run.** `measure-prompt-behavior.ts` re-run and committed; - B1 compared directionally to 51.88% (n=160) with the sample documented. + B1 compared directionally to 51.88% (n=160). - [ ] **M9 — rollback rehearsed** by group, per **Rollback Plan**. - [ ] **M10 — prose-pinned test re-baselining is deliberate and enumerated.** ~25 test files - assert exact prose in the surfaces being cut; the hardest is + assert exact prose in the surfaces being rewritten; the hardest is `agent-farm/__tests__/baked-decisions.test.ts:143-148`, enforcing a **pure-addition diff** - against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md` — - incompatible with 824 → ≤420. Also `bugfix-744-spir-pr-strategy.test.ts`, - `spec-1273-wait-discipline-docs.test.ts`, `bugfix-619-aspir-prompt.test.ts`, - `template-delivery.test.ts`, `framework-ref-audit.test.ts`, `governance-sweep.test.ts`, + against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md`. Also + `bugfix-744-spir-pr-strategy.test.ts`, `spec-1273-wait-discipline-docs.test.ts`, + `bugfix-619-aspir-prompt.test.ts`, `template-delivery.test.ts`, + `framework-ref-audit.test.ts`, `governance-sweep.test.ts`, `review-prompt-routing.test.ts`. **Each assertion is a prior spec's protection encoded as a - grep, so retiring one is a governance act.** Every modified or retired assertion is listed - in the review with (i) the originating spec, (ii) whether the protected behaviour survives - in the rewritten prose, (iii) the replacement assertion, or an explicit architect-visible - retirement. Pure-addition re-baselining only with the originating spec named and the new - baseline committed in the same commit. Silent deletion to make the suite green is a - project failure, not a test fix. + grep, so retiring one is a governance act** — a principle matter, not a size matter. Every + modified or retired assertion is listed in the review with (i) the originating spec, + (ii) whether the protected behaviour survives in the rewritten prose, (iii) the replacement + assertion, or an explicit architect-visible retirement. Silent deletion to make the suite + green is a project failure, not a test fix. - [ ] All tests pass **after M10's enumerated re-baselining**; no coverage reduction. New tests cover M0, M3, M4, M5. - [ ] Documentation routed by tier; `CLAUDE.md`/`AGENTS.md` byte-identical. @@ -417,9 +396,9 @@ plan, or CMAP reviewers: - Spec must define a rollback story (prompt surfaces are files; reverting is cheap — say so concretely). -Arising from the repository and the scope directive: +Arising from the repository and architect rulings: -- **Scope is the full prompt surface, segmented**; measurement reports per-segment. +- **Principle conformance is the acceptance criterion; size is not.** - **Both trees**; `CLAUDE.md` ≡ `AGENTS.md`. - **Four-tier resolution, per file.** No fetch-by-path instruction for a file that may not exist on disk (deliver-don't-fetch). @@ -431,172 +410,173 @@ Arising from the repository and the scope directive: - Two human gates (`spec-approval`, `plan-approval`) plus `pr`; the A/B verdict is the architect's call. -- Adopters consume the skeleton via `codev update`, so rollback is a revertible unit per - **group**. +- **The architect personally inspects every changed file** (M11) — this is the binding + throughput constraint on phase design. +- Adopters consume the skeleton via `codev update`, so rollback is a revertible unit per group. - Scar wordings are architect-ratified; a builder may not compress them. -- **The corrected instrument ships early, as its own PR** (M0b); remaining phase-commits ship as - a single later PR. +- **The corrected instrument ships early, as its own PR** (M0b). ## Assumptions - The eight scar wordings on `builder/spir-1252` remain the ratified set. -- `I = 10` remains the agreed proxy — a comparison constant, identical both sides. - Frontier-model behaviour is stable across the A/B window (mitigated by pairing, same base commit, pinned model/config versions). - `builder/spir-1252` stays undeleted — sole source of the ratified registry. - CMAP reviewers are blind to the builder's prompt surface by construction. +- The architect has capacity for ~66 file inspections across the implement phases; if not, the + phase count grows rather than the batch size. ## Solution Approaches -### Approach 1: In-place judgment rewrite, surface by surface (RECOMMENDED) +### Approach 1: In-place principle rewrite, surface by surface (RECOMMENDED) -Rewrite each file to the seven principles, keeping file layout, resolver and porch untouched. -Templates become heading interfaces; how-to content relocates to skills; the dead tree goes; the +Rewrite each file to P1–P7, keeping file layout, resolver and porch untouched. Templates become +interfaces (P2); how-to content moves to skills (P3, P4); prose restating `protocol.json` +becomes a reference to it (P6); worst-case padding goes except the scar rules (P7). The instrument is corrected first. -**Pros**: zero mechanism risk; every change a reviewable text diff; rollback granularity equals -cut granularity; compatible with deliver-don't-fetch. -**Cons**: discipline-dependent (mitigated by T3); large diff across 10 protocols × 2 trees, so -sweep completeness is the main risk; collides with ~25 prose-pinned test files (M10). +**Pros**: zero mechanism risk; every change is a reviewable text diff — which is exactly what +M11's per-file inspection requires; rollback granularity equals cut granularity. +**Cons**: discipline-dependent; ~66 distinct files to inspect; collides with ~25 prose-pinned +test files (M10). **Complexity**: Medium · **Risk**: Low-Medium ### Approach 2: Generate prompts from `protocol.json` Synthesize phase prompts at runtime from the state machine, with a small per-phase prose delta. -**Pros**: structurally prevents re-growth and drift; would have made the 1252 drift bug -impossible. -**Cons**: introduces a code path between authoring and serving — new failure mode, harder to -review and revert — and changes porch behaviour (an explicit constraint). Couples the shrink to -a mechanism change, making any A/B regression un-attributable: deletion or generator? -**Complexity**: High · **Risk**: Medium-High -**Verdict**: right idea, wrong project; a plausible successor over a 16,000-word surface. +**Pros**: the purest expression of P6; structurally prevents drift. +**Cons**: introduces a code path between authoring and serving, changes porch behaviour (an +explicit constraint), and **defeats M11** — the architect cannot inspect old-vs-new diffs of +files that no longer exist as authored artifacts. Couples the rewrite to a mechanism change, +making any A/B regression un-attributable. +**Verdict**: right idea, wrong project — and now also incompatible with the inspection mandate. ### Approach 3: Shared kernel + per-protocol deltas One protocol-agnostic builder kernel included by every protocol, plus a short per-protocol delta. -**Pros**: attacks cross-protocol duplication the per-surface table does not; uses the existing -include mechanism, so it is served-word-honest. -**Cons**: 1252 proved dedup buys ~7% on *served* words — an include expands, so moving text -changes ownership, not the bill. Risks re-creating the shadow-tree bug class. -**Verdict**: adopt selectively *inside* Approach 1 where a kernel reduces served words for the -reader, and only after the shrink. +**Pros**: directly serves P4 (stop repeating yourself across ten protocols). +**Cons**: 1252 proved dedup buys ~7% on *served* words — an include expands, so it changes +ownership, not the reader's bill. Risks re-creating the shadow-tree bug class, where one edit +silently changes ten protocols' served prompts — which also complicates per-file inspection. +**Verdict**: adopt selectively *inside* Approach 1 where a kernel genuinely serves P4, and only +where the architect can still see what each protocol serves. ## Open Questions ### Critical (Blocks Progress) -*None outstanding.* Both former Critical questions are resolved: the corrected baseline is -architect-endorsed; and gate friction (not minable from history — no `rejected` state, -`requested_at` overwritten) is scored prospectively by the architect and demoted to -**advisory-with-a-tripwire**, so incomplete scoring cannot block the decision. +*None outstanding.* ### Important (Affects Design) -- [ ] **A/B sample size.** ≥6 pairs specified; more buys power at real builder and consult cost - (~$1,478/30d at current rates). The architect sets the ceiling — see architect load under - **Execution and sequencing**. -- [ ] **Do SPIR templates survive as interfaces, or disappear?** Recommendation: survive as - ≤150-word heading interfaces — porch requires only 4 headings, but the interface is what - makes the artifact contract legible without narration. -- [ ] **Is the hot tier's 736 genuinely exempt?** It is the one surface already built to these - principles, but it is 7,360 of the 16,016-word post-rewrite builder budget (46%), so it - bounds how far this project can go without reopening the exemption. -- [ ] **`roles/architect.md` 2,048 → ≤700** — the architect segment was outside 1252's analysis - entirely. Confirm nothing in it is load-bearing for multi-architect coordination (Specs - 755/786/823) before cutting. +- [ ] **Batch size for M11.** ≤12 distinct files per review batch is this spec's proposal, + giving ~6–7 inspection batches. The architect may want smaller. +- [ ] **Do SPIR templates survive as interfaces, or disappear?** P2 says design interfaces; + porch requires only 4 headings, while the `spec-review` consult type advisorily expects 20. + Recommendation: survive as heading interfaces. +- [ ] **Is the hot tier in scope?** It is the one surface already written to these principles. + This spec leaves it unchanged; a reviewer may argue P3 applies to it too. +- [ ] **`roles/architect.md`** — outside 1252's analysis entirely. Confirm nothing in it is + load-bearing for multi-architect coordination (Specs 755/786/823) before rewriting. +- [ ] **A/B sample size.** ≥6 pairs specified; the architect sets the ceiling given the + inspection load already on them. ### Nice-to-Know (Optimization) -- [ ] Does trimming consult-type prompts move CMAP verdict *quality*, or only cost? -- [ ] Are the ~17,000 words of `.claude/skills/` the right destination for relocated how-tos, or - does that surface need its own budget? +- [ ] Does rewriting consult-type prompts move CMAP verdict *quality*, or only cost? +- [ ] Are the ~17,000 words of `.claude/skills/` the right destination for P3/P4 relocations, + or does that surface need its own conformance pass? ## Performance Requirements -- **Builder always-on**: ≤16,100 words (from 34,255) — M1. -- **Per-segment**: architect ≤3,400 (from 8,599); consultant ≤460 (from 683); none regresses. -- **Per-surface ceilings**: as tabulated — M2, binding. -- **Measurement runtime**: <5s, deterministic — same commit ⇒ byte-identical output. -- **Token/cost effect** (advisory): ~18,200 fewer always-on words ≈ ~24,000 tokens per - builder-project; recorded before/after, keying no threshold. +No runtime-performance requirement. Reporting obligations only: + +- Before/after `ALWAYS_ON_WORDS` and per-audience loads, measured by the corrected script and + committed (M1). +- Deleted-vs-relocated decomposition (M0c). +- Per-file word counts in every manifest (M2). +- Measurement runtime <5s and deterministic — same commit ⇒ byte-identical output. + +None of these is a threshold. ## Security Considerations -- **The scar rules are the security surface.** All eight guard irreversible acts (destroying - uncommitted work or worktrees, killing live sessions, bypassing a human gate). Weakening one is - the highest-severity failure this project can produce — hence verbatim carriage, - byte-identical enforcement, counted-not-exempt ceilings, and a hard rollback trigger on any - observed violation. +- **The scar rules are the security surface.** All eight guard irreversible acts. Weakening one + is the highest-severity failure this project can produce — hence verbatim carriage, the + explicit P7 exception, byte-identical enforcement, and a hard rollback trigger on any observed + violation. - **Human-gate integrity.** The rewrite must not weaken "a gate message is a notification to the human, not authorization." Gate semantics are content, and this project edits content. -- **No secrets in prompt surfaces** — existing property, re-verified after the rewrite. +- **No secrets in prompt surfaces** — existing property, re-verified. - **Adopter blast radius.** Skeleton changes ship on `codev update`; a weakened prohibition propagates silently. Hence grouped, rehearsed rollback. ## Test Scenarios +Tests verify *mechanical* properties. **Principle conformance is judged by the architect (M11), +not asserted by a test** — that is the point of the revised acceptance model. + ### Functional Tests -1. **T1 — Instrument sources the served directory** (M0 item a). Asserted against the real - resolver, not a hardcoded string. +1. **T1 — Instrument sources the served directory** (M0 item a), asserted against the real + resolver. 2. **T1b — Instrument resolves per-file, four-tier** (M0 item b). A fixture with a `.codev/` - override of *one* prompt while others resolve from the skeleton measures each file at its - winning tier — kills the directory-level-selection defect class. + override of *one* prompt while others resolve from the skeleton. 3. **T2 — Include expansion (phantom-savings proof).** Moving text from prompt into template produces **zero** change in the reported total. -4. **T3 — Ceilings (M1/M2).** Each surface at or under its gross ceiling, per protocol, per - tree; failure names surface and overage. Runs in CI as the anti-re-growth guard. +4. **T3 — Word-count reporting.** The script emits a per-file table covering every resolved + surface. **Asserts completeness of the report, not any ceiling.** 5. **T4 — Scar integrity (M4).** Every canonical byte-identical on every registered surface; count pinned at 8; reword or deletion fails. 6. **T5 — Capability inventory (M5).** Post-rewrite extraction over served prompt text ⊇ - pre-rewrite, using M5's recognition and normalization rules; unlisted removals fail. + pre-rewrite; unlisted removals fail. 7. **T6 — Sweep completeness (M3).** Surfaces enumerated from disk across both trees and - unioned; each existing surface satisfies T3. Absence of `prompts/`/`consult-types/` for a - protocol that has none must **not** fail; a newly added protocol or surface fails until - written to budget. Covers `release` (`codev/` only). -8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; `codev/` and `codev-skeleton/` copies - consistent. + unioned; absence of `prompts/`/`consult-types/` for a protocol that has none must **not** + fail; a new surface fails until rewritten and inspected. Covers `release`. +8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; every `codev/protocols` copy byte-identical + to its skeleton twin. **Load-bearing for M11**: it is what makes inspecting ~66 files instead + of 131 sound. 9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing test - updated per M10 and still protecting hot/cold routing on its remaining files. + updated per M10. 10. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface receives a spawn prompt containing every element of the artifact contract, and its first `porch next` returns a well-formed task. ("It compiled" is not "it works.") -11. **T10 — Rollback rehearsal (M9).** Reverting a rollback **group** on a scratch branch - restores that group byte-for-byte and leaves the suite green. -12. **T11 — Bucket and audience reporting** (M0 item f). The script emits the six **exclusive - bucket** subtotals, which sum to the authored total, *and separately* the derived audience - loads, which overlap by design and are asserted against the stated formulas rather than a - naive sum. A fixture where one bucket grows and another shrinks shows both movements, not a - netted zero. +11. **T10 — Rollback rehearsal (M9).** Reverting a rollback group restores it byte-for-byte and + leaves the suite green. +12. **T11 — Bucket and audience reporting** (M0 item f). Exclusive bucket subtotals sum to the + authored total; derived audience loads are asserted against the stated formulas, never a + naive sum. 13. **T15 — Relocation visibility (M0c).** A fixture moving a block from an always-on surface - into `.claude/skills/` shows always-on falling **and** total-authored holding steady — so - relocation can never be reported as deletion. + into `.claude/skills/` shows always-on falling **and** total-authored holding steady. +14. **T16 — Manifest completeness (M11).** Every file changed in a phase appears in that phase's + manifest with all four fields. A changed file missing from the manifest fails the phase — + the architect cannot inspect what is not listed. ### Non-Functional Tests 1. **T12 — Determinism.** Two runs at the same commit emit byte-identical output. -2. **T13 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run post-merge - with self-exclusion; B1/B2/B4 committed and compared directionally. +2. **T13 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run post-merge with + self-exclusion; B1/B2/B4 committed and compared directionally. 3. **T14 — A/B execution (M7).** The full pre-registered protocol below. ## A/B Validation Design -*(First-class section per Baked Decision 3. A **non-inferiority** trial: the claim under test is -"deleting ~53% of the always-on surface does not degrade outcomes", not "it improves them".)* +*(First-class section per Baked Decision 3, and unchanged by the acceptance-model revision: the +A/B is how we learn whether principle conformance actually holds up behaviourally. A +**non-inferiority** trial — the claim under test is "a principle-conformant prompt surface does +not degrade outcomes", not "it improves them".)* ### Unit and arms The unit is an **issue-pair**: one GitHub issue executed twice, by two freshly-spawned builders in separate worktrees, from the same base commit. **Control (A)** = pre-rewrite commit; -**treatment (B)** = post-rewrite. No code differs — the prompt surface is file-resolved, so the -arms are two checkouts. +**treatment (B)** = post-rewrite. No code differs — the prompt surface is file-resolved. ### Sample and eligibility -- **≥6 pairs (12 runs)**, stratified: ≥3 SPIR/ASPIR (exercises spec/plan/implement/review - prompts, both gates, templates) and ≥3 lighter protocols (BUGFIX/AIR). +- **≥6 pairs (12 runs)**, stratified: ≥3 SPIR/ASPIR and ≥3 lighter protocols (BUGFIX/AIR). - Issues drawn from the backlog, selected **before** either arm runs, then frozen. - **Eligibility exclusion**: an issue is ineligible if it modifies any surface under test — otherwise the treatment arm's prompt surface is simultaneously instrument and subject. @@ -605,61 +585,56 @@ arms are two checkouts. - **Pin the environment**: model ids and reasoning efforts, consult backend versions, and `.codev/config.json` frozen for the trial window and recorded with the results. -- **Arm isolation**: the second arm of a pair must not see the first arm's branch, PR, or thread. - Sequential with the intervening branch unpushed, or concurrent in isolated worktrees; which was - used is recorded per pair. +- **Arm isolation**: the second arm must not see the first arm's branch, PR, or thread. + Sequential with the intervening branch unpushed, or concurrent in isolated worktrees; which + was used is recorded per pair. - **Arm order alternates** per pair. - **Recording**: one committed artifact (`codev/resources/1280-ab-results.md`), a row per run — pair id, arm, protocol, issue, base commit, order, isolation mode, every outcome, and any - exclusion with its reason. Post-hoc exclusions must be justified there, never silently dropped. + exclusion with its reason. ### Execution and sequencing -- **M7 gates `verify-approval`, not the PR merge.** The rewrite PR merges on M0–M6 and M8–M10 - plus architect review; the A/B then runs against merged `main` as treatment and a pinned - pre-rewrite commit as control. This matches where the rollback triggers point, keeps a 12-run - trial off the PR's critical path, and is the only ordering under which "treatment arm = what - builders actually get" is literally true. Consequence, stated plainly: a SHIP failure means - rolling back a merged change — which is what the grouped, rehearsed rollback plan is for. +- **M7 gates `verify-approval`, not the PR merge.** The rewrite PR merges on principle + conformance (MP, M11) plus the mechanical criteria; the A/B then runs against merged `main` as + treatment and a pinned pre-rewrite commit as control. A SHIP failure means rolling back a + merged change — which is what the grouped, rehearsed rollback plan is for. - **Arm disposition**: the treatment arm's PR is the merge candidate; the control arm's closes - unmerged once its outcomes are recorded (and vice versa if the treatment arm is defective on - O3). So **~6 of 12 runs produce merged work, not 12** — the trial's real cost is 6 duplicated - implementations plus consult spend. -- **Architect load is a scheduling dependency**: 6 pairs with ≥3 SPIR-class implies up to **~24 - gate approvals and 12 PR reviews by one person**, each SPIR gate requiring O1 rubric scoring at - approval time. This is the trial's binding constraint and why the pair count is the architect's - call. If capacity forces a smaller n, the consequence is stated in the power paragraph — not a - quietly reduced sample. + unmerged once outcomes are recorded. **~6 of 12 runs produce merged work, not 12.** +- **Architect load**: 6 pairs with ≥3 SPIR-class implies up to **~24 gate approvals and 12 PR + reviews**, each SPIR gate requiring O1 rubric scoring at approval time — *on top of* M11's ~66 + file inspections. This is the project's binding constraint and why the pair count is the + architect's call. ### Pre-registered outcomes | ID | Outcome | Instrument | Role | |---|---|---|---| -| **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each 0 (none) / 1 (minor) / 2 (blocking), recorded at scoring time | advisory + tripwire | +| **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each 0 (none) / 1 (minor) / 2 (blocking) | advisory + tripwire | | **O2** | Review rounds | Iterations to terminal state per phase from `status.yaml`; CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | gate | -| **O3** | Correctness | **At the SHIP decision**: architect PR-review findings by severity, observable pre-merge on both arms. **Post-merge defects are excluded from the SHIP gate** — they cannot be evaluated when the decision is made — and act as a **14-day rollback signal** after the merged arm lands | gate (pre-merge) + rollback signal (post-merge) | +| **O3** | Correctness | **At the SHIP decision**: architect PR-review findings by severity, observable pre-merge on both arms. **Post-merge defects are excluded from the SHIP gate** and act as a **14-day rollback signal** | gate (pre-merge) + rollback signal (post-merge) | | **O4** | Protocol compliance | Binary per-run checklist: required artifacts with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar violation · thread committed | **zero tolerance** | | **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory | ### Blinding -CMAP reviewers are blind by construction. The architect is not and cannot be; mitigations: O2 and -O4 are extracted mechanically from committed artifacts, and O1/O3 are scored against a rubric -written **before** any run. +CMAP reviewers are blind by construction. The architect is not and cannot be; mitigations: O2 +and O4 are extracted mechanically from committed artifacts, and O1/O3 are scored against a +rubric written **before** any run. ### Decision rule (pre-registered) **SHIP** iff all of: 1. **O4 = zero violations** in the treatment arm. Any scar violation, skipped gate, or missing - required artifact is an immediate hard stop, independent of everything else. + required artifact is an immediate hard stop. 2. **O2**: treatment mean review rounds ≤ control **+ 0.5 rounds/phase**, and treatment REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. 3. **O3 (pre-merge part only)**: no treatment-arm finding of severity ≥ "would block merge" absent from its paired control run. 4. **O1 tripwire**: no pair where the treatment arm scored **2 (blocking)** at a gate its control - scored 0, for the same reason. If O1 scoring is incomplete for any pair, O1 reports incomplete - and SHIP rests on 1–3. + scored 0, for the same reason. If O1 scoring is incomplete, O1 reports incomplete and SHIP + rests on 1–3. Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK**. @@ -667,10 +642,10 @@ Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK**. With n=6 pairs this detects only **large** effects — roughly a doubling of review rounds or a ≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle regression, and this -spec does not claim it can. It is nonetheless strictly stronger than 1252's observational -baseline, because each pair is matched on the issue itself — the dominant variance source. -**O4's zero-tolerance criterion is where the real protection lives**: compliance is binary, -observable in every run, and is the failure mode deletion would plausibly cause. +spec does not claim it can. It is nonetheless stronger than 1252's observational baseline, +because each pair is matched on the issue itself. **O4's zero-tolerance criterion is where the +real protection lives**: compliance is binary, observable in every run, and is the failure mode +an aggressive rewrite would plausibly cause. ## Rollback Plan @@ -688,15 +663,15 @@ Prompts, included templates, registry mappings and integrity tests are coupled, | **G7 scar registry** | `scar-rules.yaml` + its enforcement test | - **Dependency rule**: reverting **G7** requires reverting every group carrying scar text (G2, - G3, G4, G6) — registry and copies must agree. All other groups are mutually independent. + G3, G4, G6). All other groups are mutually independent. - **Mechanism**: `git revert` restores prior bytes. No migration, state, schema, or data. Rehearsed under T10 before the PR merges. - **Blast radius**: effective for the next spawned builder — in-flight builders keep the surface - they were spawned with (prompts read at spawn/phase time). For adopters the revert ships in the - next release; an adopter can also pin the prior `@cluesmith/codev` version. -- **Triggers**: (a) any O4 violation — immediate, no deliberation; (b) an observed scar violation - in any real project post-merge; (c) O2/O3 outside pre-registered margins; (d) the 14-day - post-merge O3 window; (e) architect judgment at the `pr` or `verify-approval` gate. + they were spawned with. For adopters the revert ships in the next release; an adopter can also + pin the prior `@cluesmith/codev` version. +- **Triggers**: (a) any O4 violation — immediate; (b) an observed scar violation in any real + project post-merge; (c) O2/O3 outside pre-registered margins; (d) the 14-day post-merge O3 + window; (e) architect judgment at any gate. - **Partial rollback is the expected shape** — revert the offending group, keep the rest. - **Cost**: one revert, one release. No irreversible step exists anywhere in this project. @@ -710,8 +685,10 @@ Prompts, included templates, registry mappings and integrity tests are coupled, consult-type resolution (`commands/consult/index.ts`); both measurement scripts. - **Artifacts**: `builder/spir-1252` (ratified registry — must not be deleted); `codev/resources/1252-*.md` baselines. -- **Test suites**: the ~25 prose-pinned files in M10 — a dependency in the real sense that the - cuts cannot land without deliberately re-baselining them. +- **Test suites**: the ~25 prose-pinned files in M10 — the rewrite cannot land without + deliberately re-baselining them. +- **Architect availability**: M11's per-file inspection is a hard dependency on one person's + time, not a background activity. - **Libraries/Frameworks**: none new. ## References @@ -723,45 +700,45 @@ Prompts, included templates, registry mappings and integrity tests are coupled, `codev/state/spir-1252_thread.md`. - *The new rules of context engineering for Claude-5-generation models* — https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models -- `codev/resources/arch.md` (four-tier resolution, include directive, hot-tier injection); - `codev/resources/lessons-learned.md` (sweep-scope failures, served-surface dedup). + (P1–P7 quoted verbatim in **Desired State**). +- `codev/resources/arch.md`; `codev/resources/lessons-learned.md`. ## Risks and Mitigation | Risk | Probability | Impact | Mitigation Strategy | |------|------------|--------|-------------------| -| A deleted instruction was load-bearing; loss is silent | Medium | High | M5 inventory over served prompt text (unlisted removals fail); O4 zero-tolerance; grouped rollback | -| A scar rule weakened or dropped | Low | **Critical** | Verbatim carriage; counted-in-ceiling so it is never trimmed to fit; byte-identical enforcement pinned at 8; hard rollback trigger; architect-only rewording | -| Prose-pinned tests silently gutted to go green | **High** | **High** | M10: every retired assertion named with its originating spec and its behaviour re-asserted or explicitly retired; pure-addition re-baselining only with spec named and baseline committed together | -| Sweep misses a protocol, tree, or segment | High | Medium | T6 enumerates from disk (both trees, unioned); T7 twin parity; T11 bucket reporting. Sweep-scope failure was 1252's dominant review cost | -| Relocation reported as deletion, inflating the claim | Medium | Medium | M0c + M0(g) + T15 | -| A/B underpowered; subtle regression ships | Medium | Medium | Power statement; O4 binary compliance carries the protection; T13 post-merge behavioural re-measurement as a second net | -| A further instrument defect ships undetected | Medium | High | Principle 7; T1/T1b/T11/T15 assert the instrument against the live resolver; M0b puts it under public review early; the script gets its first tests | -| A/B costs more than the shrink saves | Medium | Low | ~6 of 12 runs produce merged work; O5 tracks it; architect sets the pair ceiling | -| Architect review capacity is the trial's bottleneck | High | Medium | Load stated explicitly (~24 gates + 12 PR reviews); pair count is the architect's call; smaller n reported honestly in the power statement | -| Surface re-grows after the project | High | Medium | T3 runs the ceilings in CI | +| "Principle conformance" is judged inconsistently across ~66 files | **High** | High | P1–P7 quoted verbatim with a per-file conformance question each; one architect judges all of them; the manifest records which principles were applied per file | +| A deleted instruction was load-bearing; loss is silent | Medium | High | M5 inventory over served prompt text; O4 zero-tolerance; M11 per-file inspection; grouped rollback | +| A scar rule weakened or dropped under P1/P7 | Low | **Critical** | The P7 exception is stated explicitly; verbatim carriage; byte-identical enforcement pinned at 8; architect-only rewording; hard rollback trigger | +| Prose-pinned tests silently gutted to go green | **High** | **High** | M10: every retired assertion named with its originating spec and its behaviour re-asserted or explicitly retired | +| M11 inspection load stalls the project | **High** | Medium | ≤12 files per batch; inspection scoped to ~66 distinct decisions via T7 twin parity, not 131 diffs; phase count grows rather than batch size | +| Sweep misses a protocol, tree, or surface | High | Medium | T6 enumerates from disk (both trees, unioned); T7 twin parity; T16 manifest completeness | +| Relocation reported as deletion | Medium | Low | M0c + M0(g) + T15 (now a reporting-honesty concern, not a grading one) | +| A/B underpowered; subtle regression ships | Medium | Medium | Power statement; O4 binary compliance; T13 post-merge behavioural re-measurement | +| A further instrument defect ships undetected | Medium | Medium | T1/T1b/T11/T15 assert the instrument against the live resolver; M0b puts it under public review early; the script gets its first tests | +| Surface re-grows after the project | Medium | Medium | No ceiling test exists under the revised model; re-growth is caught by the same principle review at the next MAINTAIN | | `builder/spir-1252` deleted, losing the registry | Low | High | Registry content quoted in this project's thread; rebuilt registry committed to `main` early | ## Expert Consultation **Round 1** — 2026-07-31 · Codex (GPT-5.6 Sol) + Claude Opus 5 · both REQUEST_CHANGES (HIGH) · **13 findings, none disputed**. -**Round 2** — 2026-07-31, architect-directed re-review of the revision · same models · both -REQUEST_CHANGES (HIGH) · **9 findings, none disputed**. +**Round 2** — 2026-07-31, architect-directed re-review · same models · both REQUEST_CHANGES +(HIGH) · **9 findings, none disputed**. +**Round 3** — pending: one CMAP pass on this acceptance-model revision. -Every finding was verified against source (and both arithmetic claims independently recomputed) -before acceptance; all are folded into the criteria, tests, and design sections above rather than -narrated here. The finding-by-finding record is in +Every finding was verified against source (both arithmetic claims independently recomputed) +before acceptance; all are folded into the sections above. The finding-by-finding record is in `codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter{1,2}-rebuttals.md`. -Four of the corrections were errors in this spec's own analysis, three sharing a single root -cause — **enumerating from a convenient source instead of the authoritative one** (a truncated -grep; skeleton-only protocol enumeration; the measurement script's stale comment). That is the +Four corrections were errors in this spec's own analysis, three sharing one root cause — +**enumerating from a convenient source instead of the authoritative one** (a truncated grep; +skeleton-only protocol enumeration; the measurement script's stale comment). That is the sweep-scope class 1252 named as its dominant review cost, and it is why M3's "enumerate from disk" is specified as a **test** rather than an instruction. -Gemini/`agy` did not participate: the known `--type` review limitation (#1032/#1033). Per current -lane policy this 2-way review is correct and needs no remedy. +Gemini/`agy` did not participate: the known `--type` review limitation (#1032/#1033). Per +current lane policy this 2-way review is correct. ## Approval - [ ] Technical Lead Review @@ -771,13 +748,19 @@ lane policy this 2-way review is correct and needs no remedy. ## Notes -**What this project deliberately does not do.** No prompt generator (Approach 2), no tiering of -any kind (Baked Decision 1), no porch behaviour changes, and not 1252's full enforcement -machinery — only the minimum scar-integrity check the deletion makes necessary, plus the ceiling -test that prevents re-growth. Enforcement built around a still-moving surface is enforcement -built twice. +**On the acceptance model.** An earlier draft made ">50% reduction" the headline criterion, with +per-surface word ceilings as the binding test. Waleed's redirect removed that: *"I don't think +the goal should be a particular size. That's not the right criteria. It should be to stick to +the principles outlined in the blog post."* This is a better criterion for a reason worth +recording — a word ceiling can be met by a file that still narrates procedure, and can be missed +by a file that is perfectly conformant but genuinely needs the words. Size was a proxy; +conformance is the thing. + +**What this project deliberately does not do.** No prompt generator (Approach 2 — now also +incompatible with M11), no tiering of any kind (Baked Decision 1), no porch behaviour changes, +and not 1252's full enforcement machinery — only the scar-integrity check the rewrite makes +necessary. Enforcement built around a still-moving surface is enforcement built twice. **On the deferred decision from 1252.** The architect's pr-gate ruling was that structural -machinery is not worth carrying for a surface about to halve. That sequencing is honoured: shrink -first, then enforce what remains. T3 is the smallest useful enforcement primitive, and it only -becomes meaningful once the ceilings exist. +machinery is not worth carrying for a surface about to be rewritten. That sequencing is honoured: +rewrite first, then enforce what remains. From 9872726290d001e3f57e243e1cbb45494d091d10 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:35:07 -0700 Subject: [PATCH 14/32] [Spec 1280] Thread: acceptance-model redirect + round-3 Codex findings --- codev/state/spir-1280_thread.md | 48 +++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 158f0cee2..1b0797228 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -237,3 +237,51 @@ That number-level diff is the compression analogue of principle 7 — "I kept th not the same claim as "I kept the content," and only the diff distinguishes them. Gate remains pending; still not touched. + +### Acceptance model redirected (2026-07-31) — principles, not size + +**Gate NOT APPROVED.** Waleed: *"I don't think the goal should be a particular size. That's not +the right criteria. It should be to stick to the principles outlined in the blog post."* + +Rewrote the acceptance model (commit 0821c7ab). Fetched the blog rather than paraphrasing from +the issue charter, and quoted its principles **verbatim** as P1–P7, each restated as a per-file +question answerable from a diff. + +Two honest complications I surfaced rather than smoothed over: + +- **P5 (auto-memory) does not apply.** It is a Claude Code harness feature; Baked Decision 1's + fleet includes GPT 5.6 / Gemini 3.6 consumers with no equivalent. Declared N/A with reason + rather than listing six principles and hoping nobody counted seven. +- **P7 collides with the scar-rule exemption.** The blog deletes worst-case guardrails and its + own example is "such as deleting files" — exactly what our scar rules guard. Named the + collision: the blog's guardrails protected against *bad output* (judgment now handles it); + scar rules protect against *irreversible acts*, where being wrong once is unbounded. + +Demoted to observability: >50%, all ceilings, per-segment goals, thin-margin analysis. M1/M2 +are reporting obligations that cannot fail on a number. M0/M0b/M0c survive — they keep us +honest about what happened. + +**M11** (architect per-file inspection) added with the load sized rather than assumed, and +**T16** fails a phase whose manifest omits a changed file. + +### CMAP round 3 — Codex: fifth self-audit finding, same root cause + +REQUEST_CHANGES (HIGH), five findings. The factual one lands squarely on me again: + +**My twin-file table was wrong.** I reported "3 codev/protocols copies that differ". In fact +**zero differ** — all three (`maintain/templates/audit-report.md`, +`maintain/templates/lessons-learned.md`, `release/protocol.md`) are **local-only, no skeleton +twin**. Cause: my `cmp -s` loop treated a nonzero exit as "differs", but `cmp` also exits +nonzero when a file is missing. I read an exit code without distinguishing its two causes. + +That is the **fifth** self-audit finding of this spec phase and the fourth sharing one root +cause: *trusting a convenient signal instead of checking the authoritative thing* (truncated +grep; skeleton-only enumeration; the script's stale comment; now an overloaded exit code). The +pattern is now well-evidenced enough that it belongs in lessons-learned as its own entry, not +just as this project's review note. + +Three other findings are sharp and structural: MP/M3 scope contradicts the hot-tier and skills +exclusions; the A/B cannot both use "the same base commit" and "pre-/post-rewrite commits"; +and **M5 conflicts with P6** — P6 permits replacing narrated gate/check names with a reference +to structured truth, while M5 demands those names remain in served prose, so a conformant P6 +rewrite would fail M5. Fixing after Claude's round-3 lands. From 5b42f128e69cbf10656ba6693bf7f4240f1a4fb1 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:38:04 -0700 Subject: [PATCH 15/32] [Spec 1280] CMAP round 3: 11 findings, none disputed Codex REQUEST_CHANGES (5) + Claude COMMENT (6) on the acceptance-model revision. Both caught the same error in my own inventory: I reported '3 codev/protocols copies that differ'. Zero differ - all three are local-only with no skeleton twin. My cmp -s loop treated nonzero exit as 'differs', but cmp also exits nonzero when a file is absent. T7 now runs on the intersection of twinned files. Fifth self-audit finding this phase; fourth sharing one root cause (trusting a convenient signal over the authoritative thing). Structural fixes: - M5 conflicted with P6 (P6 permits replacing narrated names with a reference to structured truth; M5 demanded the names stay in prose, so a conformant rewrite would fail). Representation now means name-in-text OR an explicit resolvable reference to the source that still defines it. M5's detection limit is also stated: it detects deletion, not inversion - gap assigned to M11/O4 plus a short set of semantic invariants asserted as behaviour. - A/B arms could not be both 'same base commit' and 'pre-/post-rewrite commits'. Rebuilt as a prompt-only overlay: both arms branch from source commit S, control applies one overlay reverting G2-G6 and nothing else, each run records source hash AND prompt-surface hash. - Added the issue-mandated per-surface cut plan as a disposition table: every category marked rewritten / inspected-but-unchanged / excluded-with-reason. Resolves the scope contradiction (hot tier, .claude/skills) too. - M12: no release between the rewrite merge and the SHIP verdict. - A/B control isolation depends on tier-2 shadowing tier-4 (verified: 0 skeleton files lack a codev/ twin). Stated as a precondition; T14 asserts it pre-flight and voids the pair on failure. - M0(g) 'total authored surface' defined as physical files, no dedup, no expansion - deliberately a different basis from the always-on buckets. Not actioned by me: issue #1280's title/Goal still state the superseded >50% target measured by the disqualified script. That is the architect's artifact; flagged rather than edited. --- .../1280-specify-iter3-rebuttals.md | 108 +++++++++++++ .../1280-prompt-surface-judgment-not-ru.md | 147 +++++++++++++++--- 2 files changed, 236 insertions(+), 19 deletions(-) create mode 100644 codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md new file mode 100644 index 000000000..c9df851fa --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md @@ -0,0 +1,108 @@ +# Iteration 3 response — Spec 1280 (CMAP round 3, acceptance-model revision) + +Codex REQUEST_CHANGES (HIGH, 5 findings) · Claude COMMENT (HIGH, 6 findings). **All accepted; +no disputes.** Both reviewers independently caught the same inventory error, and both landed on +M5's weakness from different angles. + +--- + +## Both reviewers + +**Inventory mislabel — ACCEPTED, verified, and mine.** I reported "3 `codev/protocols` copies +that differ." **Zero differ.** All three (`maintain/templates/audit-report.md`, +`maintain/templates/lessons-learned.md`, `release/protocol.md`) are **local-only with no +skeleton twin**. Cause: my `cmp -s` loop treated a nonzero exit as "differs", but `cmp` also +exits nonzero when a file is absent. I read an exit code without distinguishing its two causes. + +Fixed: table relabelled with a separate "no skeleton twin" row, and **T7 now operates on the +intersection of files that have twins**, so the three local-only files can never be spurious +failures. + +This is the **fifth** self-audit finding of this spec phase and the fourth sharing one root +cause — *trusting a convenient signal instead of checking the authoritative thing* (truncated +grep; skeleton-only enumeration; the script's stale comment; an overloaded exit code). It is +now well-evidenced enough to belong in `lessons-learned.md` as its own entry, which the review +phase will route. + +**M5 is weaker than it reads — ACCEPTED from both angles, and they compose.** + +- *Codex*: **M5 conflicts with P6.** P6 permits replacing narrated gate/check names with a + reference to structured truth; M5 as written demanded those names remain in served prose. A + *conformant* P6 rewrite would have failed M5. +- *Claude*: **M5 detects deletion, not inversion.** "A gate message is a notification to the + human, not authorization" could collapse to a bare mention of the gate name and still pass. + +Fixed together: **representation** is now defined as *either* the name appearing in served text +*or* an explicit resolvable reference to the structured source that still defines it — which +makes P6 and M5 compatible. And M5's **detection limit is stated outright**: it is a deletion +detector, not a meaning detector; the gap is assigned to M11 (architect reads the diff) and O4 +(zero-tolerance compliance), plus a short hand-curated set of **semantic invariants** asserted +as behaviour rather than name presence. + +--- + +## Codex + +**Scope contradiction — ACCEPTED.** MP/M3 said "every prompt-bearing file" while the hot tier +was left unchanged and `.claude/skills/` was measured, received relocated content, and had an +unresolved conformance status. Fixed with an authoritative **per-surface disposition table**: +every category marked **rewritten**, **inspected-but-unchanged**, or **excluded with reason**. +Scope is now exactly that table. (This also delivers the issue-mandated cut plan — see Claude 1.) + +**A/B arms cannot use both "the same base commit" and "pre-/post-rewrite commits" — ACCEPTED.** +Genuinely incompatible as written, and the naive reading also let later pairs inherit source +changes the pinned control commit lacked. Rebuilt as a **prompt-only overlay**: both arms branch +from the same source commit `S`; treatment uses `S`; control applies one overlay commit +reverting rollback groups G2–G6 and nothing else; each run records **both** the source hash and +a **prompt-surface hash** over every file in the disposition table. Source is identical within a +pair, `S` may advance between pairs, and "no code differs" becomes literally true. + +**"Total authored surface" ambiguous — ACCEPTED.** M0(g) now defines it as **physical files on +disk**, each counted once, **no twin deduplication and no transclusion expansion** — +deliberately a different basis from the always-on buckets (which dedupe and expand), because its +job is to detect relocation. Both figures are reported side by side and labelled with their +basis, so T11 and T15 have deterministic expected values. + +--- + +## Claude + +**1. The per-surface cut plan is missing — ACCEPTED, and the sharpest process catch.** Issue +#1280's Protocol section requires the spec phase to produce it. Word *targets* are withdrawn by +the architect's redirect, but the **disposition mapping survives that redirect** and was absent. +Entering implement with only "apply P1–P7" — with the architect as the throughput bottleneck — +invites rewrite→reject churn. Added as the disposition table above, with dominant +non-conformance, governing principles, and relocation destination per bucket. + +**2. Issue #1280's body now contradicts the spec — ACCEPTED, flagged to the architect.** Title +and Goal still say ">50% reduction… measured with 1252's committed measurement script" — a +script this spec disqualifies, and a goal the acceptance-model redirect replaced. The issue is +the charter CMAP reviewers load, so it will keep generating "doesn't meet the stated goal" +findings. **I have not edited the charter myself** — that is the architect's artifact. Raised +for them to update, citing the 2026-07-31 ruling. + +**4. No release-hold between merge and the SHIP verdict — ACCEPTED.** M7 gates +`verify-approval`, so the rewritten skeleton reaches `main` and is shippable to adopters before +the A/B validates it; "pin the prior version" is reactive. Added **M12**: no `@cluesmith/codev` +release between the rewrite merge and the SHIP verdict; if one must cut, it ships from a commit +predating the merge. + +**5. A/B isolation rests on an unstated resolver property — ACCEPTED, and verified.** Every +`codev-skeleton/protocols/**` and `roles/*.md` file has a `codev/` twin (**0 lack one** — +confirmed), so tier 2 shadows tier 4 for every surface under test and the control arm genuinely +serves the old surface. But **deleting a `codev/` file while keeping its skeleton twin would +silently drop the control arm through to the new skeleton** — a comparison that looks valid and +is not. Stated as a precondition; **T14 now asserts it pre-flight per pair and voids the pair on +failure**. + +**6. Minor — ACCEPTED.** Twin mislabel fixed (above). `builder/spir-1252` confirmed present on +`origin` (`ee310a64`), so the "sole source" risk is downgraded Low/High → Low/Medium. + +--- + +## Net + +Eleven findings across two reviewers, all accepted, none disputed. The two structural ones — +scope contradiction and the A/B's impossible arm construction — would both have surfaced during +implementation as confusion rather than as a clean defect, which is the case for having run a +third round on a spec that had already passed two. diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md index 9a62065ac..c7691ed1a 100644 --- a/codev/specs/1280-prompt-surface-judgment-not-ru.md +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -160,14 +160,24 @@ them is a target.** | | Count | |---|---:| | Prompt-bearing `.md` files, both trees + `CLAUDE.md`/`AGENTS.md` | **131** | -| Of which `codev/protocols` copies **byte-identical** to their skeleton twin | **60** | -| `codev/protocols` copies that differ | 3 (`maintain/templates/audit-report.md`, `maintain/templates/lessons-learned.md`, `release/protocol.md` — the last has no skeleton twin) | +| `codev/protocols` copies **byte-identical** to their skeleton twin | **60** | +| `codev/protocols` copies that **differ** from a twin | **0** | +| `codev/protocols` files with **no skeleton twin** (local-only) | 3 — `maintain/templates/audit-report.md`, `maintain/templates/lessons-learned.md`, `release/protocol.md` | | `roles/*.md` — all three byte-identical across trees | 3 pairs | | `CLAUDE.md` ≡ `AGENTS.md` | verified identical | | **Distinct content decisions** | **~66** | This matters for M11: reviewing all 131 diffs would mean re-reading ~65 byte-identical copies. -The inspection is over **distinct content decisions**, with twin sync verified mechanically. +The inspection is over **distinct content decisions**, with twin sync verified mechanically +(**T7**), which must therefore operate on the **intersection of files that have twins** — the +three local-only files are inspected once and are not twin-parity candidates. + +**Resolver precondition (load-bearing for the A/B).** Every `codev-skeleton/protocols/**` and +`roles/*.md` file currently has a `codev/` twin — verified: **0 skeleton files lack one** — so +tier 2 shadows the installed-package skeleton (tier 4) for every surface under test. The +control arm only genuinely serves the old surface while this holds. **Deleting a `codev/` file +while keeping its skeleton twin would silently drop the control arm through to the new +skeleton**, invalidating the comparison without any error. Asserted as a pre-flight in T14. ### Coverage is per-surface, not per-protocol × surface-type @@ -237,6 +247,37 @@ protected against *bad output*, which judgment now handles; scar rules protect a cost of being wrong once is unbounded and no amount of judgment makes the wager sensible. Every other P7 candidate goes. +### Per-surface disposition and cut plan + +Issue #1280's Protocol section requires the spec phase to produce "the per-surface cut plan…, +the A/B eval design, and the scar-rule carriage plan." The *word targets* are withdrawn by the +acceptance-model redirect; the **disposition mapping survives it**, and is the authoritative +answer to "which surfaces are in scope, and what is expected to change in each." + +Every category is marked **rewritten**, **inspected-but-unchanged**, or **excluded with +reason** — no category is left implicit. + +| Bucket | Disposition | Dominant non-conformance today | Governing principles | Relocation destination | +|---|---|---|---|---| +| `CLAUDE.md` / `AGENTS.md` | **rewritten** | CLI walkthroughs, worktree recipes, protocol-selection prose — all needed rarely, loaded always | P3, P4, P1 | `.claude/skills/` (afx, codev, porch, consult), `--help` | +| hot tier (`arch-critical`, `lessons-critical`) | **inspected-but-unchanged** | none — already capped, judgment-shaped, displacement-enforced (Spec 987) | — | — | +| `roles/architect.md` | **rewritten** | procedure narration for coordination already covered by skills | P1, P3 | `arch-init` / `afx` skills | +| `roles/builder.md` | **rewritten** | ordered procedure + repeated prohibitions | P1, P7 | — | +| `roles/consultant.md` | **inspected-but-unchanged (expected)** | already lean at 252w; rewritten only if inspection finds non-conformance | P1 | — | +| `protocols/*/protocol.md` | **rewritten** | narrates the state machine that `protocol.json` already defines; checklists restate phase bodies | **P6**, P1, P7 | reference `protocol.json` | +| `protocols/*/builder-prompt.md` | **rewritten** | worst-case padding, all-caps prohibitions | P1, P7 | — | +| `protocols/*/prompts/*.md` | **rewritten** | step-by-step process; annotated templates inlined via `{{> …}}` | **P2**, P1 | heading interfaces | +| `protocols/*/templates/*.md` | **rewritten** | annotated examples with filler prose | **P2** | heading interfaces | +| `protocols/*/consult-types/*.md` | **rewritten** | process prose around a rubric + verdict contract | P1, P2 | — | +| `.claude/skills/**` | **excluded from rewrite; in scope for measurement** | on-demand already — P3 working as intended. Receives relocated content, so it is measured (M0(g)) and grows by design | — | — | +| `codev-skeleton/porch/prompts/**` | **deleted** (M6) | dead — no runtime consumer | — | — | +| `codev/protocols/release/protocol.md` | **rewritten** | human-invoked prose, no skeleton twin, missed by earlier inventories | P1, P3 | — | + +**Scope is exactly this table.** MP and M3 apply to every row marked *rewritten*; rows marked +*inspected-but-unchanged* are still inspected under M11 (the architect confirms conformance +rather than approving a diff); the one *excluded* row is excluded for a stated reason and is +still measured. + ### Conformance is judged per file, by the architect Acceptance is not a number and not a CMAP verdict. Each rewritten file carries a @@ -314,8 +355,15 @@ cannot fail on a number.** full four-tier chain** as `resolveCodevFile` does; (c) counts the inlined `roles/builder.md`; (d) counts hot-tier `@import` transclusion **and corrects the stale inlining comment**; (e) expands `{{> …}}` includes; (f) reports exclusive bucket subtotals - and derived audience loads separately; (g) reports **total authored prompt-surface words** - (both trees + `.claude/skills/`). Tests assert (a) and (b) against the real resolver. + and derived audience loads separately; (g) reports **total authored prompt-surface words**, defined + unambiguously as **physical files on disk** — every `.md` under `codev/protocols`, + `codev-skeleton/protocols`, `codev/roles`, `codev-skeleton/roles`, `.claude/skills`, plus + `CLAUDE.md` and `AGENTS.md`, each counted once, **no deduplication of twins and no + transclusion expansion**. This is deliberately a *different* basis from the always-on + buckets (which dedupe twins and expand `@import`/`{{> …}}`), because its job is to detect + relocation — content moved out of an always-on file must still show up somewhere. Both + figures are reported side by side and labelled with their basis, so T11 and T15 have + deterministic expected values. Tests assert (a) and (b) against the real resolver. - [ ] **M0b — the corrected instrument and baseline land on `main` early**, as a small standalone PR (precedent #1290), per **Rollout**. - [ ] **M0c — deleted words are distinguished from relocated words.** P3 authorizes moving @@ -343,7 +391,26 @@ cannot fail on a number.** **The inventory is over the resolved, expanded prompt surface, not over `protocol.json` or source call sites** — extracting gate names from an unchanged `protocol.json` would report every capability present even if every corresponding instruction vanished from the served - prompts. Each item must be evidenced as represented in served prompt text. + prompts. + + **Representation, defined so M5 does not contradict P6.** P6 explicitly permits replacing + narrated gate/check/phase names with a reference to the structured source. A naive + "every extracted name must still appear in prose" rule would make a *conformant* P6 + rewrite fail. A capability is therefore **represented** if either (a) it is named in + served prompt text, **or** (b) the served text carries an explicit, resolvable reference + to the structured source that defines it (e.g. "gates, checks and phase order are defined + in `protocol.json`; read it") **and** that source still defines it. (b) satisfies M5. + + **Detection limit, stated rather than implied.** Set-inclusion over names catches the + *deletion* of a capability; it does **not** catch *inversion or gutting* of the + instruction attached to one. "A gate message is a notification to the human, not + authorization" could collapse to a bare mention of the gate name and still pass M5 and + T6. That gap is covered by **M11** (the architect reads the actual diff) and **O4** (the + A/B's zero-tolerance compliance checklist), not by M5. To narrow it further, a small + hand-curated set of **semantic invariants** — human-gate semantics, artifact-contract + obligations, the scar prohibitions — is asserted as *behaviour present in the served + text*, not as name presence. This is a short list by design; the honest claim is that M5 + is a deletion detector, not a meaning detector. **Severity**: a removal is a hard failure **unless** the retired name appears in a committed `codev/resources/1280-retirements.md` in the same commit, naming the capability, @@ -372,6 +439,11 @@ cannot fail on a number.** (ii) whether the protected behaviour survives in the rewritten prose, (iii) the replacement assertion, or an explicit architect-visible retirement. Silent deletion to make the suite green is a project failure, not a test fix. +- [ ] **M12 — no release between the rewrite merge and the SHIP verdict.** M7 gates + `verify-approval`, not merge, so the rewritten skeleton is on `main` — and therefore + shippable to adopters via `codev update` — before the A/B has validated it. "Pin the prior + version" is a reactive remedy for a problem this criterion prevents. If a release must cut + inside the window, it ships from a commit predating the rewrite merge. - [ ] All tests pass **after M10's enumerated re-baselining**; no coverage reduction. New tests cover M0, M3, M4, M5. - [ ] Documentation routed by tier; `CLAUDE.md`/`AGENTS.md` byte-identical. @@ -535,9 +607,11 @@ not asserted by a test** — that is the point of the revised acceptance model. 7. **T6 — Sweep completeness (M3).** Surfaces enumerated from disk across both trees and unioned; absence of `prompts/`/`consult-types/` for a protocol that has none must **not** fail; a new surface fails until rewritten and inspected. Covers `release`. -8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; every `codev/protocols` copy byte-identical - to its skeleton twin. **Load-bearing for M11**: it is what makes inspecting ~66 files instead - of 131 sound. +8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; every `codev/protocols` file **that has a + skeleton twin** is byte-identical to it — the assertion runs on the intersection, so the + three local-only files (`maintain/templates/audit-report.md`, + `maintain/templates/lessons-learned.md`, `release/protocol.md`) are not failures. + **Load-bearing for M11**: it is what makes inspecting ~66 files instead of 131 sound. 9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing test updated per M10. 10. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface receives a @@ -559,7 +633,11 @@ not asserted by a test** — that is the point of the revised acceptance model. 1. **T12 — Determinism.** Two runs at the same commit emit byte-identical output. 2. **T13 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run post-merge with self-exclusion; B1/B2/B4 committed and compared directionally. -3. **T14 — A/B execution (M7).** The full pre-registered protocol below. +3. **T14 — A/B execution (M7).** The full pre-registered protocol below, including a + **pre-flight assertion per pair**: every surface under test resolves from tier 2 (`codev/`), + i.e. no skeleton file lacks a `codev/` twin. If that fails, the control arm would silently + serve the *new* skeleton and the pair is void — so the pre-flight aborts the pair rather than + producing a comparison that looks valid and is not. ## A/B Validation Design @@ -571,8 +649,28 @@ not degrade outcomes", not "it improves them".)* ### Unit and arms The unit is an **issue-pair**: one GitHub issue executed twice, by two freshly-spawned builders -in separate worktrees, from the same base commit. **Control (A)** = pre-rewrite commit; -**treatment (B)** = post-rewrite. No code differs — the prompt surface is file-resolved. +in separate worktrees. + +**Arms are a prompt-only overlay on one source snapshot — not two different commits.** An +earlier draft said both "from the same base commit" *and* "control = pre-rewrite commit, +treatment = post-rewrite commit." Those are incompatible, and the naive reading also lets later +pairs inherit source changes the pinned control commit does not have. The construction is: + +1. Both arms branch from the **same source commit** `S` (current `main` at pair start). +2. The **treatment** arm uses `S` unmodified — the rewritten prompt surface. +3. The **control** arm applies one **prompt-only overlay commit** on top of `S`: the rollback + groups (G2–G6) reverted, restoring the pre-rewrite prompt surface and touching nothing else. +4. Each run records **both hashes** — source commit `S` and a `prompt-surface hash` (a digest + over every file in the disposition table) — so any later audit can prove the arms differed + in prompts and only in prompts. + +This keeps source identical within a pair, lets `S` advance between pairs without contaminating +comparisons, and makes "no code differs" literally true rather than approximately true. + +**Precondition**: control-arm isolation depends on tier 2 (`codev/`) shadowing the installed +skeleton (tier 4) for every surface under test — true today (0 skeleton files lack a `codev/` +twin) but silently breakable if the rewrite *deletes* a `codev/` file while keeping its +skeleton twin. Asserted pre-flight in T14. ### Sample and eligibility @@ -605,6 +703,11 @@ in separate worktrees, from the same base commit. **Control (A)** = pre-rewrite reviews**, each SPIR gate requiring O1 rubric scoring at approval time — *on top of* M11's ~66 file inspections. This is the project's binding constraint and why the pair count is the architect's call. +- **Release hold (M12).** Because M7 gates `verify-approval` rather than merge, the rewritten + skeleton reaches `main` before the A/B runs, and adopters would consume it on the next npm + release. "Pin the prior version" is purely reactive. Therefore: **no `@cluesmith/codev` + release between the rewrite merge and the SHIP verdict.** If a release must cut inside that + window, it ships from a commit predating the rewrite merge. ### Pre-registered outcomes @@ -717,7 +820,7 @@ Prompts, included templates, registry mappings and integrity tests are coupled, | A/B underpowered; subtle regression ships | Medium | Medium | Power statement; O4 binary compliance; T13 post-merge behavioural re-measurement | | A further instrument defect ships undetected | Medium | Medium | T1/T1b/T11/T15 assert the instrument against the live resolver; M0b puts it under public review early; the script gets its first tests | | Surface re-grows after the project | Medium | Medium | No ceiling test exists under the revised model; re-growth is caught by the same principle review at the next MAINTAIN | -| `builder/spir-1252` deleted, losing the registry | Low | High | Registry content quoted in this project's thread; rebuilt registry committed to `main` early | +| `builder/spir-1252` deleted, losing the registry | Low | Medium | The branch exists on `origin` (verified), not only locally; registry content is also quoted in this project's thread; rebuilt registry committed to `main` early | ## Expert Consultation @@ -725,17 +828,23 @@ Prompts, included templates, registry mappings and integrity tests are coupled, **13 findings, none disputed**. **Round 2** — 2026-07-31, architect-directed re-review · same models · both REQUEST_CHANGES (HIGH) · **9 findings, none disputed**. -**Round 3** — pending: one CMAP pass on this acceptance-model revision. +**Round 3** — 2026-07-31, on the acceptance-model revision · Codex REQUEST_CHANGES (HIGH, 5) · +Claude COMMENT (HIGH, 6) · **11 findings, none disputed**. Both independently caught a +twin-file mislabel in this spec's own inventory (0 files differ; 3 are twinless — my `cmp -s` +loop conflated "differs" with "absent"), and both landed on M5's weakness from different angles +(P6 conflict / inversion-not-detected), which composed into one fix. Every finding was verified against source (both arithmetic claims independently recomputed) before acceptance; all are folded into the sections above. The finding-by-finding record is in `codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter{1,2}-rebuttals.md`. -Four corrections were errors in this spec's own analysis, three sharing one root cause — -**enumerating from a convenient source instead of the authoritative one** (a truncated grep; -skeleton-only protocol enumeration; the measurement script's stale comment). That is the -sweep-scope class 1252 named as its dominant review cost, and it is why M3's "enumerate from -disk" is specified as a **test** rather than an instruction. +Five corrections were errors in this spec's own analysis, four sharing one root cause — +**trusting a convenient signal instead of checking the authoritative thing**: a truncated grep; +skeleton-only protocol enumeration; the measurement script's stale comment; and an overloaded +`cmp` exit code read as "differs" when it also means "absent". That is the sweep-scope class +1252 named as its dominant review cost, it is why M3's "enumerate from disk" is a **test** +rather than an instruction, and at five instances it is a `lessons-learned.md` entry the review +phase will route. Gemini/`agy` did not participate: the known `--type` review limitation (#1032/#1033). Per current lane policy this 2-way review is correct. From 514d96b940866b79fda3a3ffe3ca3bfb7352208e Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:38:19 -0700 Subject: [PATCH 16/32] [Spec 1280] Thread: round-3 complete --- codev/state/spir-1280_thread.md | 36 +++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 1b0797228..f7382cd5e 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -285,3 +285,39 @@ exclusions; the A/B cannot both use "the same base commit" and "pre-/post-rewrit and **M5 conflicts with P6** — P6 permits replacing narrated gate/check names with a reference to structured truth, while M5 demands those names remain in served prose, so a conformant P6 rewrite would fail M5. Fixing after Claude's round-3 lands. + +### CMAP round 3 complete — 11 findings, none disputed (2026-07-31) + +Codex REQUEST_CHANGES (HIGH, 5) · Claude COMMENT (HIGH, 6). Commit be95ef40. + +**The two structural ones would have surfaced in implement as confusion, not as clean defects:** + +1. **The A/B was impossible as written.** It said both "both arms from the same base commit" + AND "control = pre-rewrite commit, treatment = post-rewrite commit." Rebuilt as a + **prompt-only overlay**: both arms branch from source commit `S`; control applies one overlay + reverting G2–G6 and nothing else; each run records source hash AND prompt-surface hash. Also + fixes the latent bug Codex spotted — later pairs would otherwise inherit source changes the + pinned control commit lacked. +2. **M5 conflicted with P6.** P6 permits replacing narrated gate/check names with a reference to + structured truth; M5 demanded the names stay in prose. A *conformant* rewrite would have + failed the capability check. Representation now means name-in-text OR an explicit resolvable + reference to a source that still defines it. Claude hit the same criterion from the other + side — it detects deletion, not inversion — so both fixes composed: the limit is now stated + outright and the gap assigned to M11/O4 plus a short set of semantic invariants. + +**Claude's best process catch**: the issue-mandated **per-surface cut plan** was missing. Word +targets died with the redirect; the *disposition mapping* survived it and I had dropped both. +Added as a disposition table marking every category rewritten / inspected-but-unchanged / +excluded-with-reason — which also resolved Codex's scope contradiction (hot tier, .claude/skills). + +**Also**: M12 (no release between merge and SHIP verdict — adopters would otherwise consume an +unvalidated skeleton); the A/B's unstated resolver precondition (tier-2 shadows tier-4; verified +0 skeleton files lack a codev/ twin) now asserted pre-flight in T14, voiding a pair on failure +rather than producing a comparison that looks valid and isn't. + +**Not actioned by me**: issue #1280's title/Goal still state ">50% reduction… measured with +1252's committed measurement script" — superseded on both counts. That is the architect's +artifact; flagged, not edited. Left unchanged it will keep drawing "doesn't meet the stated +goal" findings from every future CMAP round. + +Spec 8,212w. Gate remains pending and untouched. From b0e1daacb2b826bccd7ec503ffbee780a77be107 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:39:54 -0700 Subject: [PATCH 17/32] [Spec 1280] Thread: charter amendment verified, Baked Decisions byte-intact --- codev/state/spir-1280_thread.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index f7382cd5e..5c9c70732 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -321,3 +321,29 @@ artifact; flagged, not edited. Left unchanged it will keep drawing "doesn't meet goal" findings from every future CMAP round. Spec 8,212w. Gate remains pending and untouched. + +### Charter amended by the architect (2026-08-01 UTC / 2026-07-31 local) + +Issue #1280 retitled "Prompt-surface judgment rewrite: acceptance = blog-principles conformance +per file (size is reporting-only)" with a dated AMENDMENT block superseding the original Goal. +Original Goal preserved above it and marked superseded — history stays honest. + +**Verified rather than assumed** (the spec quotes Baked Decisions verbatim, so an amendment +touching them would silently invalidate a Constraints section): + +- All **four Baked Decisions are byte-intact** — diffed the issue's bullets against the spec's + Constraints, all four match. No re-quoting needed. +- The amendment cites the spec's P7-vs-scar-rules resolution and states architect inspection as + charter-level, matching what the spec specifies. + +One residual, not worth raising: the Protocol section still reads "the per-surface cut plan with +word targets." Amendment point 2 (size reporting-only) supersedes the "word targets" clause, and +the spec already handles it explicitly — targets withdrawn, disposition mapping retained. No +action. + +Fifth self-audit instance confirmed for the lessons ledger by the architect: **"trusting a +convenient signal over the authoritative thing"** — `cmp -s` exit codes joining truncated greps +and skeleton-only enumeration. Named pattern; the review phase routes it to lessons-learned. + +**Gate presentation to Waleed is out with the architect's recommendation. Nothing is pending on +me. Not touching the gate; waiting.** From 10ce855b12be1e292d56fdc040cd9a76385c183d Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:48:22 -0700 Subject: [PATCH 18/32] chore(porch): 1280 spec-approval gate-approved --- .../projects/1280-prompt-surface-judgment-not-ru/status.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index ef8e6d0a2..261ca4470 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -6,8 +6,9 @@ plan_phases: [] current_plan_phase: null gates: spec-approval: - status: pending + status: approved requested_at: '2026-08-01T03:06:40.784Z' + approved_at: '2026-08-01T03:48:22.599Z' plan-approval: status: pending pr: @@ -18,4 +19,4 @@ iteration: 1 build_complete: true history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T03:06:40.784Z' +updated_at: '2026-08-01T03:48:22.600Z' From f84bc9c998a151b3a1e9431c44bc210ad4bd1183 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:48:30 -0700 Subject: [PATCH 19/32] chore(porch): 1280 plan phase-transition --- .../1280-prompt-surface-judgment-not-ru/status.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 261ca4470..348e10585 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -1,7 +1,7 @@ id: '1280' title: prompt-surface-judgment-not-ru protocol: spir -phase: specify +phase: plan plan_phases: [] current_plan_phase: null gates: @@ -16,7 +16,7 @@ gates: verify-approval: status: pending iteration: 1 -build_complete: true +build_complete: false history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T03:48:22.600Z' +updated_at: '2026-08-01T03:48:30.711Z' From ddde48cde0a84b59153ebbe47e3b44b96dd9d506 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:51:42 -0700 Subject: [PATCH 20/32] [Spec 1280] Initial plan draft: 10 phases, 67 decisions, batches <=12 Phase boundaries drawn by M11 inspection load rather than subsystem grouping, since the architect's per-file review is the throughput constraint. Enumerated the 67 decisions against disk rather than trusting the spec's '~66': protocol.md 10 (9 skeleton + release codev-local) + builder-prompt 9 + prompts 18 + templates 8 (6 skeleton + 2 codev-local maintain) + consult-types 18 + roles 3 + CLAUDE/AGENTS 1. Batches: 4,10,9,11,10,10,9,4 = 67, max 11 against the cap of 12. Sequencing rationale: - P0 (corrected instrument) ships as PR-1 before any prompt word changes - rewriting first would make every later measurement unfalsifiable. - P1 kept small (4) - highest blast radius, and it calibrates the conformance standard for the phases that follow. - P3 is only 9 decisions but carries the entire M10 test-retirement burden (baked-decisions.test.ts pure-addition diff vs rewriting builder-prompt.md), which is why it is not merged with P1. - P8 rebuilds the scar registry last, against the settled surface. Named two capability risks that P2 (interfaces-not-examples) could plausibly destroy: the plan template's machine-readable phases JSON (porch requires it) and the consult verdict format (consult parses it). Both get live integration checks rather than fixtures. --- .../1280-prompt-surface-judgment-not-ru.md | 532 ++++++++++++++++++ codev/state/spir-1280_thread.md | 37 ++ 2 files changed, 569 insertions(+) create mode 100644 codev/plans/1280-prompt-surface-judgment-not-ru.md diff --git a/codev/plans/1280-prompt-surface-judgment-not-ru.md b/codev/plans/1280-prompt-surface-judgment-not-ru.md new file mode 100644 index 000000000..a03492be3 --- /dev/null +++ b/codev/plans/1280-prompt-surface-judgment-not-ru.md @@ -0,0 +1,532 @@ +# Plan: Prompt surface — judgment-not-rules rewrite (principle conformance) + +## Metadata +- **ID**: plan-2026-07-31-prompt-surface-judgment-not-rules +- **Status**: draft +- **Specification**: [codev/specs/1280-prompt-surface-judgment-not-ru.md](../specs/1280-prompt-surface-judgment-not-ru.md) +- **Created**: 2026-07-31 +- **Issue**: #1280 (charter amended 2026-08-01: acceptance = principle conformance, size reporting-only) + +## Executive Summary + +Approach 1 from the spec: **in-place principle rewrite, surface by surface**, keeping file +layout, the four-tier resolver, and porch untouched. Approach 2 (generate prompts from +`protocol.json`) is rejected twice over — it changes porch behaviour and it defeats M11, since +the architect cannot inspect old-vs-new diffs of files that no longer exist as authored +artifacts. + +**The binding constraint is not the writing — it is M11.** The architect personally inspects +the old-vs-new diff of every changed file. There are **67 distinct content decisions** (enumerated +below), and the spec caps review batches at **≤12**. Phase boundaries are therefore drawn by +*inspection load*, not by subsystem elegance: every phase is a reviewable batch that ends at the +architect's per-file review and does not advance until it passes. + +Phase 0 ships separately and early as **PR-1** (M0b): the corrected measurement instrument and +its first-ever tests, before a single prompt word is rewritten. Everything after it is one PR. + +### The 67 decisions + +| Category | Count | Notes | +|---|---:|---| +| `protocol.md` | 10 | 9 skeleton + `release` (codev-local, no twin) | +| `builder-prompt.md` | 9 | `release` has none | +| `prompts/*.md` | 18 | spir 4 · aspir 4 · pir 3 · bugfix 3 · air 2 · maintain 2 | +| `templates/*.md` | 8 | spir 3 · experiment 1 · maintain 1 · spike 1 + 2 maintain codev-local | +| `consult-types/*.md` | 18 | spir 5 · aspir 5 · pir 2 · bugfix 2 · air 2 · maintain 2 | +| `roles/*.md` | 3 | architect · builder · consultant | +| `CLAUDE.md` + `AGENTS.md` | 1 | one decision, two byte-identical files | +| **Total** | **67** | | + +Each decision is applied to **both trees** where a twin exists; T7 asserts twin parity, which is +what makes the architect's review of 67 decisions sound rather than 131 diffs. + +## Success Metrics + +From the specification (acceptance = principle conformance; size is reporting-only): + +- [ ] **MP** — every file marked *rewritten* in the spec's disposition table conforms to P1, P2, + P3, P4, P6, P7 (P5 N/A with reason), judged per file by the architect +- [ ] **M11** — architect inspected the old-vs-new diff of every changed file, in ≤12-file + batches, with a complete manifest per phase +- [ ] **M0 / M0b / M0c** — corrected instrument, landed early as PR-1, reporting deleted vs relocated +- [ ] **M1 / M2** — before/after figures and per-file counts published (no threshold) +- [ ] **M2b** — CLAUDE.md human-readable, architect-confirmed +- [ ] **M3** — every surface enumerated from disk is rewritten and inspected +- [ ] **M4** — eight scar canonicals byte-identical; count pinned at 8 +- [ ] **M5** — capability inventory over served prompt text; unlisted removals fail +- [ ] **M6** — dead tree deleted, its Spec 987 test consumer handled +- [ ] **M7** — A/B non-inferiority SHIP verdict (gates `verify-approval`) +- [ ] **M8** — behavioural baseline re-run +- [ ] **M9** — rollback rehearsed by group +- [ ] **M10** — every retired prose-pinned assertion named with its originating spec +- [ ] **M12** — no release between the rewrite merge and the SHIP verdict +- [ ] All tests pass after M10 re-baselining; no coverage reduction + +## Phases (Machine Readable) + + + +```json +{ + "phases": [ + {"id": "phase_0_instrument", "title": "Corrected measurement instrument (PR-1, ships early)"}, + {"id": "phase_1_shared_roles", "title": "CLAUDE.md/AGENTS.md + three role files (4 decisions)"}, + {"id": "phase_2_protocol_md", "title": "protocol.md across ten protocols (10 decisions)"}, + {"id": "phase_3_builder_prompts", "title": "builder-prompt.md across nine protocols (9 decisions)"}, + {"id": "phase_4_prompts_heavy", "title": "Phase prompts: spir, aspir, pir (11 decisions)"}, + {"id": "phase_5_prompts_light_spir_templates", "title": "Phase prompts: bugfix, air, maintain + spir templates (10 decisions)"}, + {"id": "phase_6_templates_consult_spir", "title": "Remaining templates + spir consult-types (10 decisions)"}, + {"id": "phase_7_consult_types_a", "title": "Consult-types: aspir, bugfix, air (9 decisions)"}, + {"id": "phase_8_consult_registry_deadtree", "title": "Consult-types: pir, maintain + scar registry + dead-tree deletion (4 decisions + M4/M6)"}, + {"id": "phase_9_integration", "title": "Capability inventory, governance docs, measurement report, PR"} + ] +} +``` + +## Phase Breakdown + +### Phase 0: Corrected measurement instrument (PR-1, ships early) +**Dependencies**: None. **Ships as its own PR before any prompt rewriting** (M0b). + +#### Objectives +- Make the instrument measure what is actually served, before anything is scored by it +- Correct the public record: 1252's baselines cite dead-tree figures and are read by other work + +#### Deliverables +- [ ] `scripts/measure-prompt-surface.sh` corrected — all seven M0 items: served directory · + per-file four-tier resolution · inlined `roles/builder.md` · hot-tier `@import` + transclusion **and the stale comment that caused defect 3** · include expansion · + bucket/audience reporting · total-authored-surface reporting +- [ ] First tests for the script: T1, T1b, T2, T11, T12, T15 +- [ ] `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline +- [ ] In-place annotation of `1252-word-baseline.md` / `1252-word-after-phase7.md`: originals + preserved, marked superseded, reason + pointer +- [ ] PR opened, reviewed, merged + +#### Implementation Details +- The script's `PORCH_DIR` (line 89) is replaced by resolution through the same path + `loadPromptFile` uses (`protocols/

/prompts/`), asserted against the real resolver in T1. +- Per-file four-tier resolution mirrors `resolveCodevFile` rather than two-tier directory + selection (T1b fixture: one prompt overridden in `.codev/`, others from the skeleton). +- Two reporting bases, labelled: **always-on** (dedupes twins, expands `@import`/`{{> …}}`) and + **total authored** (physical files, no dedup, no expansion) — the second is what makes + relocation visible. +- **No prompt-surface file is touched in this phase.** + +#### Acceptance Criteria +- [ ] T1, T1b, T2, T11, T12, T15 pass +- [ ] Script output deterministic at a fixed commit +- [ ] Architect reviews the script diff and both baseline artifacts (M11 applies — small batch) +- [ ] PR-1 merged + +#### Test Plan +- **Unit**: the six tests above, including the phantom-savings and relocation fixtures +- **Manual**: run against `main`, confirm the corrected baseline reproduces + +#### Rollback Strategy +Group **G1**. Revert the PR; no prompt surface has changed, so nothing else is affected. + +#### Risks +A fourth instrument defect. Mitigated by the tests being written *with* the fix and by the +architect reviewing the script before any cut is scored by it. + +--- + +### Phase 1: CLAUDE.md/AGENTS.md + three role files (4 decisions) +**Dependencies**: Phase 0 merged. + +#### Objectives +- Establish the conformance pattern on the highest-blast-radius surfaces first, so the + architect's review calibrates the standard for the seven phases that follow + +#### Deliverables +- [ ] `CLAUDE.md` + `AGENTS.md` rewritten (1 decision, 2 byte-identical files) +- [ ] `roles/architect.md`, `roles/builder.md` rewritten; `roles/consultant.md` inspected + (rewritten only if inspection finds non-conformance — spec disposition table) +- [ ] All eight scar canonicals present byte-identically in CLAUDE.md/AGENTS.md +- [ ] Relocated how-to content landed in the named skills +- [ ] Per-file manifest + architect review + +#### Implementation Details +- Governing principles: **P3** (worktree recipes, CLI walkthroughs, protocol-selection prose are + needed rarely and loaded always) and **P4** (tool how-tos belong with the tool), plus **P1**. +- Relocation destinations are the existing skills: `afx`, `codev`, `porch`, `consult`. Content + moves *by name*, never by "go read this path" — deliver-don't-fetch still applies. +- `roles/architect.md`: confirm nothing is load-bearing for multi-architect coordination + (Specs 755/786/823) **before** cutting — spec Open Question. +- M2b: the architect confirms CLAUDE.md is still navigable by a human, not just parity-clean. + +#### Acceptance Criteria +- [ ] T4 (scar integrity), T7 (twin parity) pass +- [ ] Architect judges all 4 decisions conformant against P1/P3/P4 +- [ ] M2b confirmed +- [ ] M10: any prose-pinned assertion touched is re-baselined with its originating spec named + +#### Test Plan +- **Unit**: T4, T7; existing governance-sweep and framework-ref-audit suites +- **Integration**: `codev doctor` clean +- **Manual**: architect reads the rewritten CLAUDE.md end to end + +#### Rollback Strategy +Groups **G2** (shared) and **G6** (architect role). Independent of later phases. + +#### Risks +Relocating content that turns out to be needed every time. Mitigated by M11 inspection and by +M0c making relocation visible rather than scoring it as deletion. + +--- + +### Phase 2: protocol.md across ten protocols (10 decisions) +**Dependencies**: Phase 1 reviewed. + +#### Objectives +- Apply **P6** where it bites hardest: `protocol.md` narrates a state machine that + `protocol.json` already defines + +#### Deliverables +- [ ] Ten `protocol.md` files rewritten (spir, aspir, pir, maintain, research, experiment, + spike, bugfix, air, **release** — the last is codev-local with no skeleton twin) +- [ ] Per-file manifest + architect review + +#### Implementation Details +- **P6 is the lever**: replace narrated gate/check/phase enumerations with an explicit, + resolvable reference to `protocol.json`. Per M5's representation rule, that reference + *satisfies* the capability inventory — this is the case M5 was amended to allow. +- **P7**: delete worst-case padding (all-caps prohibition blocks, "⚠️ BLOCKING" banners, + checklists restating the phase body) — **except** scar rules. +- `release/protocol.md` has no `protocol.json`; it is human-invoked prose, so P6 does not apply + and it is rewritten on P1/P3 alone. +- Largest single cut in the project (`spir/protocol.md`); M5's contract-presence assertions are + the primary defence. + +#### Acceptance Criteria +- [ ] T5 (capability inventory over served prompt text) passes — every gate, check, signal and + artifact contract still represented, by name or by resolvable reference +- [ ] T6, T7 pass +- [ ] Architect judges all 10 decisions conformant + +#### Test Plan +- **Unit**: T5, T6, T7 +- **Integration**: `porch next` on a scratch project returns a well-formed task per phase +- **Manual**: architect diff review, 10 files + +#### Rollback Strategy +Group **G3**. + +#### Risks +P6 over-applied — a prompt that references `protocol.json` for something an agent needs +*inline*. Mitigated by T9's live spawn probe in Phase 9 and by M11. + +--- + +### Phase 3: builder-prompt.md across nine protocols (9 decisions) +**Dependencies**: Phase 2 reviewed. + +#### Objectives +- Remove worst-case padding from the spawn wrappers while preserving every artifact contract + +#### Deliverables +- [ ] Nine `builder-prompt.md` files rewritten +- [ ] **M10 re-baselining executed and enumerated** — this is the phase that collides with + `agent-farm/__tests__/baked-decisions.test.ts:143-148` +- [ ] Per-file manifest + architect review + +#### Implementation Details +- Governing principles: **P1**, **P7**. +- **M10 is the load-bearing work here, not the rewriting.** `baked-decisions.test.ts` enforces a + *pure-addition diff* against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md` + — structurally incompatible with rewriting them. Also colliding: + `bugfix-744-spir-pr-strategy.test.ts` (4 near-verbatim sentences), + `spec-1273-wait-discipline-docs.test.ts` (16 assertions), `bugfix-619-aspir-prompt.test.ts`. +- For each: name the originating spec, state whether the protected behaviour survives in the + rewritten prose, and either write the replacement assertion or record an + architect-visible retirement in `codev/resources/1280-retirements.md`. +- Re-baselining a pure-addition baseline requires the originating spec named and the new + baseline committed **in the same commit**. + +#### Acceptance Criteria +- [ ] Every touched assertion enumerated with its originating spec; none silently deleted +- [ ] Baked-decisions, PR-strategy, wait-discipline and aspir-prompt behaviours either + re-asserted or explicitly retired with approval +- [ ] T4, T7 pass; full suite green +- [ ] Architect judges all 9 decisions conformant **and** approves each assertion retirement + +#### Test Plan +- **Unit**: the four named suites, re-baselined; T4, T7 +- **Integration**: `afx spawn --help` path and a live spawn smoke check +- **Manual**: architect reviews 9 diffs + the retirements file + +#### Rollback Strategy +Group **G3** (shares the group with Phase 2 — they touch the same spawn-time surface and their +tests are coupled). + +#### Risks +The highest-risk phase: silently gutting a prior spec's protection to make the suite green. +Mitigated by M10 being an explicit deliverable with architect sign-off per assertion. + +--- + +### Phase 4: Phase prompts — spir, aspir, pir (11 decisions) +**Dependencies**: Phase 3 reviewed. + +#### Objectives +- Apply **P2** (interfaces, not examples) to the heaviest phase prompts in the fleet + +#### Deliverables +- [ ] spir: specify, plan, implement, review (4) +- [ ] aspir: specify, plan, implement, review (4) +- [ ] pir: plan, implement, review (3) +- [ ] Per-file manifest + architect review + +#### Implementation Details +- These carry the fleet's two fattest prompts (`pir/review` 2,414w, `spir/review` 1,957w). +- **P2**: the `{{> …}}` template includes are what make each prompt ~600 words heavier than it + reads. Templates themselves are rewritten in Phases 5–6; this phase rewrites the prompt bodies + and keeps the include mechanism. +- Two separate constraints on template shape, not to be conflated: porch's + `REQUIRED_SPEC_SECTIONS` needs **4** headings (`checks.ts:149-154`); the `spec-review` consult + type advisorily expects 20. +- Signal contracts (`` tags) are capability-inventory items — preserved or retired + explicitly. + +#### Acceptance Criteria +- [ ] T5 passes — every signal, artifact path and check name still represented +- [ ] Porch checks still pass on a scratch project (`spec_has_required_sections`, `has_phases_json`) +- [ ] Architect judges all 11 decisions conformant + +#### Test Plan +- **Unit**: T5, T6, T7; `template-delivery.test.ts` re-baselined per M10 if touched +- **Integration**: drive a scratch project through specify→plan with the rewritten prompts +- **Manual**: architect reviews 11 diffs + +#### Rollback Strategy +Group **G4**. + +#### Risks +Dropping a porch-required heading and breaking a gate check. Mitigated by the integration test +driving real porch checks, not just unit fixtures. + +--- + +### Phase 5: Phase prompts — bugfix, air, maintain + spir templates (10 decisions) +**Dependencies**: Phase 4 reviewed. + +#### Deliverables +- [ ] bugfix: investigate, fix, pr (3); air: implement, pr (2); maintain: maintain, review (2) +- [ ] spir templates: `spec.md`, `plan.md`, `review.md` (3) → heading interfaces +- [ ] Per-file manifest + architect review + +#### Implementation Details +- The lighter protocols are already closer to conformant (means 356–457w); expect confirmation + rather than large rewrites — and per the acceptance model, **a file that is already conformant + passes unchanged**. +- spir templates are the clearest **P2** case in the project: annotated examples with filler + prose become heading skeletons with one line of intent per heading. +- `plan.md`'s machine-readable phases JSON block is a **capability**, not an example — it is + required by porch's `has_phases_json` check and must survive. + +#### Acceptance Criteria +- [ ] `has_phases_json` and `min_two_phases` still pass against a plan produced from the + rewritten template +- [ ] T5, T6, T7 pass +- [ ] Architect judges all 10 decisions conformant + +#### Test Plan +- **Unit**: T5, T6, T7 +- **Integration**: generate a plan from the rewritten template, run porch's plan checks against it +- **Manual**: architect reviews 10 diffs + +#### Rollback Strategy +Group **G4**. + +#### Risks +Trimming the plan template's JSON block as "an example". Called out explicitly above. + +--- + +### Phase 6: Remaining templates + spir consult-types (10 decisions) +**Dependencies**: Phase 5 reviewed. + +#### Deliverables +- [ ] Templates: experiment (1), maintain (1), spike (1), **maintain codev-local ×2** + (`audit-report.md`, `lessons-learned.md` — no skeleton twin) (5) +- [ ] spir consult-types: spec, plan, impl, phase, pr (5) +- [ ] Per-file manifest + architect review + +#### Implementation Details +- The two codev-local maintain templates have **no skeleton twin** — inspected once, excluded + from T7's twin-parity intersection. +- Consult-types: keep the rubric dimensions and the verdict contract; delete the process prose + around them (**P1**, **P2**). +- The verdict format (`VERDICT: APPROVE | REQUEST_CHANGES | COMMENT`) is a **capability** — + `consult` parses it. Preserved exactly. + +#### Acceptance Criteria +- [ ] A live `consult -m claude --type spec-review` returns a parseable verdict +- [ ] T5, T6, T7 pass +- [ ] Architect judges all 10 decisions conformant + +#### Test Plan +- **Unit**: T5, T6, T7 +- **Integration**: one live consult per rewritten type, verdict parsed successfully +- **Manual**: architect reviews 10 diffs + +#### Rollback Strategy +Groups **G4** (templates) and **G5** (consult-types). + +#### Risks +Breaking verdict parsing, which would silently degrade every future CMAP round. Mitigated by the +live-consult integration check rather than a fixture. + +--- + +### Phase 7: Consult-types — aspir, bugfix, air (9 decisions) +**Dependencies**: Phase 6 reviewed. + +#### Deliverables +- [ ] aspir ×5, bugfix ×2, air ×2 +- [ ] Per-file manifest + architect review + +#### Implementation Details +- Same treatment as Phase 6. bugfix's two are the fleet's largest consult-types (pr 726, impl + 641) and carry the most process prose. +- aspir's five mirror spir's; if the Phase 6 rewrites apply cleanly, these are largely mechanical + — but each is still a separate decision and a separate diff. + +#### Acceptance Criteria +- [ ] Live consult per rewritten type returns a parseable verdict +- [ ] T5, T6, T7 pass +- [ ] Architect judges all 9 decisions conformant + +#### Test Plan +As Phase 6. + +#### Rollback Strategy +Group **G5**. + +#### Risks +Mechanical application without judgment — "same as spir" is a rules-not-judgment failure in a +project about exactly that. Each file is judged on its own diff. + +--- + +### Phase 8: Consult-types pir + maintain, scar registry, dead-tree deletion (4 decisions + M4/M6) +**Dependencies**: Phase 7 reviewed. + +#### Objectives +- Finish the per-file rewrite, then rebuild the scar registry **against the settled surface** + +#### Deliverables +- [ ] pir ×2, maintain ×2 consult-types (4 decisions) +- [ ] `codev/resources/scar-rules.yaml` rebuilt — eight canonicals verbatim, `must_appear_on` + re-derived against the **post-rewrite** surface +- [ ] Scar enforcement test (byte-identical presence, count pinned at 8) +- [ ] `codev-skeleton/porch/prompts/` deleted (M6), with `review-prompt-routing.test.ts:29` + updated under M10 naming **Spec 987** +- [ ] Per-file manifest + architect review + +#### Implementation Details +- The registry is rebuilt **now, not earlier** — Baked Decision 2 defers enforcement until the + surface stops moving, and `must_appear_on` lists derived before the rewrite would be stale. +- M6 verification is **not** a bare grep: an untruncated repo-wide search reconciled against the + full hit list. (An earlier truncated grep in this project's spec phase produced a false + "no consumers" claim — the failure this step is written to avoid.) + +#### Acceptance Criteria +- [ ] T4 passes with the rebuilt registry; count pinned at 8; reword/deletion fails +- [ ] T8 passes — tree absent, no runtime reference, Spec 987 protection preserved on remaining files +- [ ] Architect judges the 4 decisions conformant **and** ratifies the registry's `must_appear_on` + +#### Test Plan +- **Unit**: T4, T8; the updated Spec 987 routing test +- **Manual**: architect reviews 4 diffs + registry + the routing-test change + +#### Rollback Strategy +Group **G7** (registry) — note the dependency rule: reverting G7 requires reverting every group +carrying scar text (G2, G3, G4, G6). Group **G5** for the consult-types. + +#### Risks +Registry `must_appear_on` drifting from where scar text actually landed. Mitigated by deriving +it from the post-rewrite surface and by T4 failing loudly. + +--- + +### Phase 9: Capability inventory, governance docs, measurement report, PR +**Dependencies**: Phase 8 reviewed. + +#### Objectives +- Prove nothing was lost, report honestly what changed, and open the PR + +#### Deliverables +- [ ] Post-rewrite capability inventory extracted and compared (M5); any removal listed in + `codev/resources/1280-retirements.md` with architect approval +- [ ] Measurement re-run: before/after, per-audience, **deleted vs relocated** (M0c, M1, M2) +- [ ] T9 live spawn probe; T10 rollback rehearsal by group (M9) +- [ ] Governance docs routed by tier (`arch.md`/`arch-critical.md`, + `lessons-learned.md`/`lessons-critical.md`) — including the + **"trust the authoritative source, not the convenient signal"** lesson (five instances in + this project's own spec phase) +- [ ] Review document; PR opened +- [ ] **M12 recorded**: no release between merge and the SHIP verdict + +#### Implementation Details +- The measurement report is the project's honesty artifact: it must state where relocated words + went, not merely that always-on fell. +- T10 rehearses a **group** revert on a scratch branch and confirms the suite stays green. +- The A/B (M7) runs **after** merge and gates `verify-approval` — its design, including the + prompt-only overlay construction and the T14 pre-flight, is in the spec. + +#### Acceptance Criteria +- [ ] M5 passes; retirements file complete and approved +- [ ] Measurement artifacts committed +- [ ] T9, T10 pass; full suite green +- [ ] Review document complete; PR opened + +#### Test Plan +- **Unit**: full suite +- **Integration**: T9 live spawn probe end-to-end +- **Manual**: T10 rollback rehearsal; architect final review + +#### Rollback Strategy +The PR itself is revertible by group; T10 has rehearsed it. + +#### Risks +Discovering at the end that a capability was lost several phases ago. Mitigated by T5 running in +every phase, not only here. + +## Timeline & Dependencies + +Strictly sequential — each phase ends at the architect's per-file review and does not advance +until it passes. Phase 0 ships as PR-1 and merges before Phase 1 begins; Phases 1–9 accumulate as +commits on one branch and ship as a single PR. + +``` +P0 (PR-1, merged) → P1 → P2 → P3 → P4 → P5 → P6 → P7 → P8 → P9 (PR) + 4 10 9 11 10 10 9 4 — = 67 decisions +``` + +## Rollback Strategy (whole project) + +Per the spec's seven groups: G1 instrument · G2 shared · G3 builder-spawn · G4 phase · +G5 consultant · G6 architect · G7 scar registry. Reverting **G7** requires reverting every group +carrying scar text (G2, G3, G4, G6); all others are mutually independent. `git revert` restores +prior bytes — no migration, state, or schema. Rehearsed under T10 before the PR merges. + +## Open Questions Carried Into Implementation + +- **Batch size** — ≤12 is the spec's cap; the architect may prefer smaller, which grows the phase + count rather than the batches. +- **`roles/consultant.md`** — marked *inspected-but-unchanged (expected)*; rewritten only if + Phase 1 inspection finds non-conformance. +- **Hot tier** — out of scope by the spec's disposition table; a reviewer may argue P3 applies. + +## Notes + +**Why phases are drawn by inspection load.** A conventional plan would group by subsystem. Here +the architect's per-file review is the throughput constraint, so the phase boundary that matters +is "a batch a human can review in one sitting." Phase 3 is the exception that proves it: it is +only 9 decisions but carries the entire M10 test-retirement burden, which is why it is not merged +with the 4-decision Phase 1. + +**Phase 0 is not optional sequencing.** Rewriting before the instrument is corrected would make +every subsequent measurement unfalsifiable — the spec's principle 7. diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 5c9c70732..6bb4be1dd 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -347,3 +347,40 @@ and skeleton-only enumeration. Named pattern; the review phase routes it to less **Gate presentation to Waleed is out with the architect's recommendation. Nothing is pending on me. Not touching the gate; waiting.** + +### spec-approval APPROVED → plan phase (2026-07-31) + +Waleed approved; architect relayed. Ran `porch approve 1280 spec-approval` myself per the flow +(porch required the `--a-human-explicitly-approved-this` flag — correct guard). Advanced to plan. + +**Plan drafted: 10 phases, 67 decisions, max batch 11 (cap 12).** + +Phase boundaries are drawn by **inspection load, not subsystem elegance** — M11 makes the +architect's per-file review the throughput constraint, so the boundary that matters is "a batch +a human can review in one sitting." + +``` +P0 (PR-1) → P1(4) → P2(10) → P3(9) → P4(11) → P5(10) → P6(10) → P7(9) → P8(4) → P9 + sum = 67 +``` + +Verified the enumeration against disk rather than trusting the spec's "~66": protocol.md 10 +(9 skeleton + release local) · builder-prompt 9 · prompts 18 · templates 8 (6 + 2 codev-local) · +consult-types 18 · roles 3 · CLAUDE/AGENTS 1 = **67**. + +Design decisions worth recording: + +- **P0 ships as PR-1 before any prompt word changes.** Not administrative sequencing — rewriting + first would make every later measurement unfalsifiable (principle 7). +- **P1 is deliberately small (4).** Highest blast radius (CLAUDE.md carries all 8 scar rules), + and it calibrates the architect's conformance standard for the seven phases after it. +- **P3 is only 9 decisions but is the riskiest phase**, because it carries the whole M10 burden: + `baked-decisions.test.ts` enforces a pure-addition diff on three `builder-prompt.md` files, + structurally incompatible with rewriting them. Kept separate from P1 for that reason alone. +- **P8 rebuilds the scar registry last**, against the settled surface — Baked Decision 2 defers + enforcement until the surface stops moving, and `must_appear_on` derived earlier would be stale. +- **Capability risks named per phase** rather than deferred: the plan template's phases-JSON block + (P5) and the consult verdict format (P6/P7) are *capabilities*, not examples, and would be + plausible casualties of P2 applied carelessly. Both get live integration checks, not fixtures. + +Porch checks pass: plan_exists, has_phases_json, min_two_phases (10). From 7ccb5b168dd99b304716a1c303d583e1b8bf1bc8 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 20:51:45 -0700 Subject: [PATCH 21/32] chore(porch): 1280 plan build-complete --- .../projects/1280-prompt-surface-judgment-not-ru/status.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 348e10585..d2e1fcdcc 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -16,7 +16,7 @@ gates: verify-approval: status: pending iteration: 1 -build_complete: false +build_complete: true history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T03:48:30.711Z' +updated_at: '2026-08-01T03:51:45.083Z' From e1fae8e5f5b4444eab50f2efedde584201066c78 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:00:09 -0700 Subject: [PATCH 22/32] [Spec 1280] Plan with multi-agent review: 15 findings, none disputed Both reviewers reproduced the 67-decision count independently - the accounting held. What failed was the supporting work riding alongside the decisions, which a plan drawn purely by inspection load under-specifies. Two blocking mechanism gaps, both verified against source: - P6 had no adopter-resolvable mechanism. protocol.md is inlined at spawn via {{protocol_reference}}; protocol.json is inlined NOWHERE and doesn't exist on disk in fresh adopter projects, so 'reference protocol.json' was the fetch-by-path CLAUDE.md forbids. Resolved by verifying resolveCodevIncludes is extension-agnostic: a fenced json block with {{> protocols/

/protocol.json}} resolves through all four tiers, no porch change. New T18 covers strict AND soft mode. - Skill relocation is a FOUR-tree sync (.claude, .codex, and both skeleton copies, with existing drift). M0(g) counted one, so relocated content would have left Codex agents and adopters without it and been reported as DELETED - inverting the honesty artifact. Phase 0 widens the basis; new T17 asserts parity. Also: pre-rewrite capability inventory frozen in Phase 0 (was extracted last while Phase 2 asserted against it); verify phase given an explicit home for M7/M8/M12/T13/T14; T3/T16 into Phase 0; 'review batch' defined to include tests and registry, with Phases 4 and 9 declaring two batches each; scar canonicals verified against the ratified YAML in Phase 1 with T4 deferred to Phase 9; commit-level group purity replacing the broken phase-level mapping; per-phase green suite; M10 reassigned to where each collision actually lands. Phase count 10 -> 11; decisions 67; max batch 11. --- .../1280-prompt-surface-judgment-not-ru.md | 652 ++++++++---------- .../1280-plan-iter1-rebuttals.md | 113 +++ codev/state/spir-1280_thread.md | 39 ++ 3 files changed, 435 insertions(+), 369 deletions(-) create mode 100644 codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md diff --git a/codev/plans/1280-prompt-surface-judgment-not-ru.md b/codev/plans/1280-prompt-surface-judgment-not-ru.md index a03492be3..071d2a343 100644 --- a/codev/plans/1280-prompt-surface-judgment-not-ru.md +++ b/codev/plans/1280-prompt-surface-judgment-not-ru.md @@ -2,7 +2,7 @@ ## Metadata - **ID**: plan-2026-07-31-prompt-surface-judgment-not-rules -- **Status**: draft +- **Status**: draft (CMAP round 1 incorporated) - **Specification**: [codev/specs/1280-prompt-surface-judgment-not-ru.md](../specs/1280-prompt-surface-judgment-not-ru.md) - **Created**: 2026-07-31 - **Issue**: #1280 (charter amended 2026-08-01: acceptance = principle conformance, size reporting-only) @@ -10,21 +10,15 @@ ## Executive Summary Approach 1 from the spec: **in-place principle rewrite, surface by surface**, keeping file -layout, the four-tier resolver, and porch untouched. Approach 2 (generate prompts from -`protocol.json`) is rejected twice over — it changes porch behaviour and it defeats M11, since -the architect cannot inspect old-vs-new diffs of files that no longer exist as authored -artifacts. +layout, the four-tier resolver, and porch untouched. Approach 2 is rejected twice over — it +changes porch behaviour and defeats M11, since the architect cannot inspect old-vs-new diffs of +files that no longer exist as authored artifacts. -**The binding constraint is not the writing — it is M11.** The architect personally inspects -the old-vs-new diff of every changed file. There are **67 distinct content decisions** (enumerated -below), and the spec caps review batches at **≤12**. Phase boundaries are therefore drawn by -*inspection load*, not by subsystem elegance: every phase is a reviewable batch that ends at the -architect's per-file review and does not advance until it passes. +**The binding constraint is M11, not the writing.** The architect inspects the old-vs-new diff of +every changed file, in batches of ≤12. Phase boundaries are drawn by *inspection load*. -Phase 0 ships separately and early as **PR-1** (M0b): the corrected measurement instrument and -its first-ever tests, before a single prompt word is rewritten. Everything after it is one PR. - -### The 67 decisions +**Decision count: 67** (the spec says "~66"; **67 is the correct figure** and both CMAP reviewers +independently reproduced it — readers should not have to re-derive it): | Category | Count | Notes | |---|---:|---| @@ -37,30 +31,72 @@ its first-ever tests, before a single prompt word is rewritten. Everything after | `CLAUDE.md` + `AGENTS.md` | 1 | one decision, two byte-identical files | | **Total** | **67** | | -Each decision is applied to **both trees** where a twin exists; T7 asserts twin parity, which is -what makes the architect's review of 67 decisions sound rather than 131 diffs. +### Review batch — defined, because "one phase = one batch" was false + +A **review batch** is *every distinct file the architect reads in one sitting* — prompt +decisions **plus** the supporting test files, registry, retirements entries and manifest that +ride along. A phase may contain **more than one batch**; each batch is ≤12 and is reviewed +before the next begins. Phases below declare their batches explicitly. + +### Rollback groups are commit-pure, phases may span groups + +Phases are drawn by inspection load, so a phase can touch several rollback groups. The +invariant is at the **commit** level: **every commit belongs to exactly one group**, so any group +reverts cleanly regardless of which phase produced it. Each phase declares its groups; T10 +rehearses **every group the project touched**, not a sample. (CMAP round 1 correctly found the +first draft's group mapping was wrong — Phase 1 claimed G2/G6 while also rewriting +`roles/builder.md` (G3) and `roles/consultant.md` (G5).) + +**M6's dead-tree deletion is assigned to G4** (phase surfaces), which owns the `prompts/` axis. + +### Two mechanisms that must be settled before the work they gate + +**1. P6 delivery of `protocol.json` (gates Phase 3).** Verified: `protocol.md` is inlined at +spawn via `{{protocol_reference}}` (`spawn-roles.ts:112-124`), but **`protocol.json` is inlined +nowhere** — `spawn-roles.ts:267` only reads it for validation. In a fresh adopter project +`codev/protocols/

/protocol.json` does not exist on disk; it resolves from tier 4. So a prose +instruction "read `protocol.json`" is exactly the fetch-by-path of a framework file that +CLAUDE.md forbids and the spec's own constraint restates. + +**Decision: deliver it through the existing include resolver.** `resolveCodevIncludes` +(`skeleton.ts:108-119`) is **extension-agnostic** — verified — so `protocol.md` carries a fenced +```` ```json ```` block containing `{{> protocols/

/protocol.json}}`. This resolves through all +four tiers, works in fresh installs, requires **no porch change**, and is the literal expression +of P6 ("rich references"). Cost, stated honestly: it adds the JSON back as served words (spir +570, aspir 568, pir 375, others 77–282) — acceptable because size is reporting-only under the +amended charter, and it replaces narration with authoritative structured truth. +**Tested in both strict mode (porch-driven) and soft mode (builder reads `protocol.md` with no +porch)** — the asymmetry matters: strict-mode builders receive checks and gates as porch tasks, +soft-mode builders have only the prompt. + +**2. Skill relocation is a FOUR-tree sync (gates Phase 1).** Verified: skills exist in +`.claude/skills` (10), `.codex/skills` (10, **byte-identical** to `.claude`), +`codev-skeleton/.claude/skills` (7) and `codev-skeleton/.codex/skills` (7) — with **existing +drift** (`afx`, `porch` differ repo-vs-skeleton; `forge`, `skill-creator`, `team` are absent from +the skeleton). Consequences if unaddressed: content relocated out of CLAUDE.md into one copy +leaves **Codex agents without it**, leaves **adopters without it** after `codev update`, and is +reported as **deleted** by M0c/T15 — inverting the project's honesty artifact. Phase 0 widens +M0(g)'s basis to all four trees; Phase 1 adds **T17** (skills parity) and treats every relocation +as a four-copy write. ## Success Metrics -From the specification (acceptance = principle conformance; size is reporting-only): - -- [ ] **MP** — every file marked *rewritten* in the spec's disposition table conforms to P1, P2, - P3, P4, P6, P7 (P5 N/A with reason), judged per file by the architect -- [ ] **M11** — architect inspected the old-vs-new diff of every changed file, in ≤12-file - batches, with a complete manifest per phase -- [ ] **M0 / M0b / M0c** — corrected instrument, landed early as PR-1, reporting deleted vs relocated +- [ ] **MP** — every file marked *rewritten* in the spec's disposition table conforms to + P1, P2, P3, P4, P6, P7 (P5 N/A with reason), judged per file by the architect +- [ ] **M11** — architect inspected every changed file, in ≤12-file batches, with a complete manifest +- [ ] **M0 / M0b / M0c** — corrected instrument landed early as PR-1; deleted vs relocated reported - [ ] **M1 / M2** — before/after figures and per-file counts published (no threshold) - [ ] **M2b** — CLAUDE.md human-readable, architect-confirmed -- [ ] **M3** — every surface enumerated from disk is rewritten and inspected +- [ ] **M3** — every surface enumerated from disk rewritten and inspected - [ ] **M4** — eight scar canonicals byte-identical; count pinned at 8 - [ ] **M5** — capability inventory over served prompt text; unlisted removals fail -- [ ] **M6** — dead tree deleted, its Spec 987 test consumer handled -- [ ] **M7** — A/B non-inferiority SHIP verdict (gates `verify-approval`) -- [ ] **M8** — behavioural baseline re-run -- [ ] **M9** — rollback rehearsed by group +- [ ] **M6** — dead tree deleted, Spec 987 test consumer handled +- [ ] **M7** — A/B SHIP verdict (verify phase) +- [ ] **M8** — behavioural baseline re-run (verify phase) +- [ ] **M9** — rollback rehearsed for every group touched - [ ] **M10** — every retired prose-pinned assertion named with its originating spec -- [ ] **M12** — no release between the rewrite merge and the SHIP verdict -- [ ] All tests pass after M10 re-baselining; no coverage reduction +- [ ] **M12** — no release between merge and SHIP verdict (spans the verify phase) +- [ ] Suite green **at the end of every phase**, not only at the end ## Phases (Machine Readable) @@ -69,464 +105,342 @@ From the specification (acceptance = principle conformance; size is reporting-on ```json { "phases": [ - {"id": "phase_0_instrument", "title": "Corrected measurement instrument (PR-1, ships early)"}, - {"id": "phase_1_shared_roles", "title": "CLAUDE.md/AGENTS.md + three role files (4 decisions)"}, - {"id": "phase_2_protocol_md", "title": "protocol.md across ten protocols (10 decisions)"}, - {"id": "phase_3_builder_prompts", "title": "builder-prompt.md across nine protocols (9 decisions)"}, - {"id": "phase_4_prompts_heavy", "title": "Phase prompts: spir, aspir, pir (11 decisions)"}, - {"id": "phase_5_prompts_light_spir_templates", "title": "Phase prompts: bugfix, air, maintain + spir templates (10 decisions)"}, - {"id": "phase_6_templates_consult_spir", "title": "Remaining templates + spir consult-types (10 decisions)"}, - {"id": "phase_7_consult_types_a", "title": "Consult-types: aspir, bugfix, air (9 decisions)"}, - {"id": "phase_8_consult_registry_deadtree", "title": "Consult-types: pir, maintain + scar registry + dead-tree deletion (4 decisions + M4/M6)"}, - {"id": "phase_9_integration", "title": "Capability inventory, governance docs, measurement report, PR"} + {"id": "phase_0_instrument", "title": "Corrected instrument + frozen capability inventory (PR-1, ships early)"}, + {"id": "phase_1_shared_skills", "title": "CLAUDE.md/AGENTS.md + four-tree skill relocation (G2)"}, + {"id": "phase_2_roles", "title": "Three role files (G6, G3, G5)"}, + {"id": "phase_3_protocol_md", "title": "protocol.md x10 with the P6 include mechanism (G3)"}, + {"id": "phase_4_builder_prompts", "title": "builder-prompt.md x9 + M10 test-retirement burden (G3)"}, + {"id": "phase_5_prompts_heavy", "title": "Phase prompts: spir, aspir, pir (G4)"}, + {"id": "phase_6_prompts_light_spir_templates", "title": "Phase prompts: bugfix, air, maintain + spir templates (G4)"}, + {"id": "phase_7_templates_consult_spir", "title": "Remaining templates + spir consult-types (G4, G5)"}, + {"id": "phase_8_consult_types_a", "title": "Consult-types: aspir, bugfix, air (G5)"}, + {"id": "phase_9_consult_registry_deadtree", "title": "Consult-types pir/maintain + scar registry + dead-tree deletion (G5, G7, G4)"}, + {"id": "phase_10_integration", "title": "Capability verification, measurement report, rollback rehearsal, governance docs"} ] } ``` ## Phase Breakdown -### Phase 0: Corrected measurement instrument (PR-1, ships early) +### Phase 0: Corrected instrument + frozen capability inventory (PR-1, ships early) +**Groups**: G1 · **Batches**: 1 (script + ~6 test files + 3 artifacts ≈ 10) **Dependencies**: None. **Ships as its own PR before any prompt rewriting** (M0b). -#### Objectives -- Make the instrument measure what is actually served, before anything is scored by it -- Correct the public record: 1252's baselines cite dead-tree figures and are read by other work - #### Deliverables -- [ ] `scripts/measure-prompt-surface.sh` corrected — all seven M0 items: served directory · - per-file four-tier resolution · inlined `roles/builder.md` · hot-tier `@import` - transclusion **and the stale comment that caused defect 3** · include expansion · - bucket/audience reporting · total-authored-surface reporting -- [ ] First tests for the script: T1, T1b, T2, T11, T12, T15 +- [ ] `scripts/measure-prompt-surface.sh` corrected — all seven M0 items +- [ ] **M0(g) basis widened to all four skill trees** (`.claude/skills`, `.codex/skills`, and + both `codev-skeleton/` copies) — otherwise relocation reports as deletion +- [ ] First tests for the script: **T1, T1b, T2, T3, T11, T12, T15** +- [ ] **T16** (manifest completeness) implemented **now** — it is the mechanical guard on M11, + the project's binding constraint, and must exist before Phase 1 produces the first manifest +- [ ] **`codev/resources/1280-capability-inventory.json` extracted from the PRE-rewrite surface + and committed** — M5 requires a frozen pre-rewrite baseline, and Phase 3 onward asserts + against it - [ ] `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline -- [ ] In-place annotation of `1252-word-baseline.md` / `1252-word-after-phase7.md`: originals - preserved, marked superseded, reason + pointer -- [ ] PR opened, reviewed, merged - -#### Implementation Details -- The script's `PORCH_DIR` (line 89) is replaced by resolution through the same path - `loadPromptFile` uses (`protocols/

/prompts/`), asserted against the real resolver in T1. -- Per-file four-tier resolution mirrors `resolveCodevFile` rather than two-tier directory - selection (T1b fixture: one prompt overridden in `.codev/`, others from the skeleton). -- Two reporting bases, labelled: **always-on** (dedupes twins, expands `@import`/`{{> …}}`) and - **total authored** (physical files, no dedup, no expansion) — the second is what makes - relocation visible. -- **No prompt-surface file is touched in this phase.** +- [ ] In-place annotation of `1252-word-baseline.md` / `1252-word-after-phase7.md` +- [ ] **Manifest format defined** at `codev/projects/1280-*/manifests/phase-N.md`: one row per + changed file — path · old words · new words · principles applied · one-line rationale + +#### PR-1 operational mechanics +1. Cut `builder/1280-instrument` from the current branch point +2. Open PR-1, architect review, merge +3. `git fetch origin main && git checkout -b builder/1280-rewrite origin/main` — **never** + `git checkout main` (a worktree cannot check out a branch checked out elsewhere) +4. Record: `porch done 1280 --pr --branch builder/1280-instrument`, then + `porch done 1280 --merged ` +5. Verify the rewrite branch contains **no duplicate Phase-0 commits** (`git log origin/main..HEAD`) #### Acceptance Criteria -- [ ] T1, T1b, T2, T11, T12, T15 pass -- [ ] Script output deterministic at a fixed commit -- [ ] Architect reviews the script diff and both baseline artifacts (M11 applies — small batch) -- [ ] PR-1 merged - -#### Test Plan -- **Unit**: the six tests above, including the phantom-savings and relocation fixtures -- **Manual**: run against `main`, confirm the corrected baseline reproduces +- [ ] T1, T1b, T2, T3, T11, T12, T15, T16 pass; output deterministic +- [ ] Pre-rewrite capability inventory committed and non-empty +- [ ] Architect reviews the batch; PR-1 merged; suite green #### Rollback Strategy -Group **G1**. Revert the PR; no prompt surface has changed, so nothing else is affected. - -#### Risks -A fourth instrument defect. Mitigated by the tests being written *with* the fix and by the -architect reviewing the script before any cut is scored by it. +Group **G1**. No prompt surface has changed. --- -### Phase 1: CLAUDE.md/AGENTS.md + three role files (4 decisions) +### Phase 1: CLAUDE.md/AGENTS.md + four-tree skill relocation (G2) +**Groups**: G2 · **Batches**: 1 (CLAUDE+AGENTS, ≤6 skill files across 4 trees, 2 test files ≈ 10) **Dependencies**: Phase 0 merged. -#### Objectives -- Establish the conformance pattern on the highest-blast-radius surfaces first, so the - architect's review calibrates the standard for the seven phases that follow - #### Deliverables - [ ] `CLAUDE.md` + `AGENTS.md` rewritten (1 decision, 2 byte-identical files) -- [ ] `roles/architect.md`, `roles/builder.md` rewritten; `roles/consultant.md` inspected - (rewritten only if inspection finds non-conformance — spec disposition table) -- [ ] All eight scar canonicals present byte-identically in CLAUDE.md/AGENTS.md -- [ ] Relocated how-to content landed in the named skills -- [ ] Per-file manifest + architect review +- [ ] Relocated how-to content written to **all four skill trees**, not one +- [ ] **T17 — skills parity**: `.claude/skills` ≡ `.codex/skills`; every skill present in the + repo that the skeleton ships is in sync. Pre-existing drift (`afx`, `porch`) and + skeleton-absent skills (`forge`, `skill-creator`, `team`) are recorded as known state, not + silently "fixed" — but any skill this project *touches* must be four-way consistent +- [ ] All eight scar canonicals present byte-identically +- [ ] **M10 here, not Phase 4**: `spec-1273-wait-discipline-docs.test.ts:31` asserts on + `.claude/skills/afx/SKILL.md` + `.codex/skills/afx/SKILL.md` — relocation into `afx` breaks + it in **this** phase +- [ ] Manifest + architect review; suite green #### Implementation Details -- Governing principles: **P3** (worktree recipes, CLI walkthroughs, protocol-selection prose are - needed rarely and loaded always) and **P4** (tool how-tos belong with the tool), plus **P1**. -- Relocation destinations are the existing skills: `afx`, `codev`, `porch`, `consult`. Content - moves *by name*, never by "go read this path" — deliver-don't-fetch still applies. -- `roles/architect.md`: confirm nothing is load-bearing for multi-architect coordination - (Specs 755/786/823) **before** cutting — spec Open Question. -- M2b: the architect confirms CLAUDE.md is still navigable by a human, not just parity-clean. +- Governing principles: **P3**, **P4**, **P1**. +- **Scar verification here is against the ratified source, not T4.** T4 and the registry are + built in Phase 9, after the surface settles (Baked Decision 2). This phase verifies the eight + canonicals byte-for-byte against `builder/spir-1252:codev/resources/scar-rules.yaml` directly. + (CMAP round 1 caught the first draft asserting T4 in Phase 1 while creating it in Phase 8.) +- M2b: architect confirms CLAUDE.md is still navigable by a human. #### Acceptance Criteria -- [ ] T4 (scar integrity), T7 (twin parity) pass -- [ ] Architect judges all 4 decisions conformant against P1/P3/P4 -- [ ] M2b confirmed -- [ ] M10: any prose-pinned assertion touched is re-baselined with its originating spec named - -#### Test Plan -- **Unit**: T4, T7; existing governance-sweep and framework-ref-audit suites -- **Integration**: `codev doctor` clean -- **Manual**: architect reads the rewritten CLAUDE.md end to end +- [ ] Eight canonicals verified against the ratified YAML; T7, T17 pass +- [ ] Architect judges the decision conformant; M2b confirmed +- [ ] Every touched assertion re-baselined with its originating spec named +- [ ] Suite green #### Rollback Strategy -Groups **G2** (shared) and **G6** (architect role). Independent of later phases. - -#### Risks -Relocating content that turns out to be needed every time. Mitigated by M11 inspection and by -M0c making relocation visible rather than scoring it as deletion. +Group **G2**, commit-pure. --- -### Phase 2: protocol.md across ten protocols (10 decisions) -**Dependencies**: Phase 1 reviewed. - -#### Objectives -- Apply **P6** where it bites hardest: `protocol.md` narrates a state machine that - `protocol.json` already defines +### Phase 2: Three role files (G6, G3, G5) +**Groups**: G6 (`architect`), G3 (`builder`), G5 (`consultant`) — **three commits, one per group** +**Batches**: 1 (3 decisions × 2 trees + 1 test ≈ 7) #### Deliverables -- [ ] Ten `protocol.md` files rewritten (spir, aspir, pir, maintain, research, experiment, - spike, bugfix, air, **release** — the last is codev-local with no skeleton twin) -- [ ] Per-file manifest + architect review - -#### Implementation Details -- **P6 is the lever**: replace narrated gate/check/phase enumerations with an explicit, - resolvable reference to `protocol.json`. Per M5's representation rule, that reference - *satisfies* the capability inventory — this is the case M5 was amended to allow. -- **P7**: delete worst-case padding (all-caps prohibition blocks, "⚠️ BLOCKING" banners, - checklists restating the phase body) — **except** scar rules. -- `release/protocol.md` has no `protocol.json`; it is human-invoked prose, so P6 does not apply - and it is rewritten on P1/P3 alone. -- Largest single cut in the project (`spir/protocol.md`); M5's contract-presence assertions are - the primary defence. - -#### Acceptance Criteria -- [ ] T5 (capability inventory over served prompt text) passes — every gate, check, signal and - artifact contract still represented, by name or by resolvable reference -- [ ] T6, T7 pass -- [ ] Architect judges all 10 decisions conformant - -#### Test Plan -- **Unit**: T5, T6, T7 -- **Integration**: `porch next` on a scratch project returns a well-formed task per phase -- **Manual**: architect diff review, 10 files +- [ ] `roles/architect.md` rewritten (G6) — confirm nothing is load-bearing for multi-architect + coordination (Specs 755/786/823) **before** cutting +- [ ] `roles/builder.md` rewritten (G3) +- [ ] `roles/consultant.md` — *inspected-but-unchanged (expected)*; rewritten only if inspection + finds non-conformance (G5) +- [ ] **M10**: `spec-1273-wait-discipline-docs.test.ts:26` (`ROLE_DOCS` = `codev/roles/builder.md` + + skeleton twin) breaks here +- [ ] Manifest + architect review; suite green #### Rollback Strategy -Group **G3**. - -#### Risks -P6 over-applied — a prompt that references `protocol.json` for something an agent needs -*inline*. Mitigated by T9's live spawn probe in Phase 9 and by M11. +Three group-pure commits: G6, G3, G5. --- -### Phase 3: builder-prompt.md across nine protocols (9 decisions) -**Dependencies**: Phase 2 reviewed. - -#### Objectives -- Remove worst-case padding from the spawn wrappers while preserving every artifact contract +### Phase 3: protocol.md ×10 with the P6 include mechanism (G3) +**Groups**: G3 · **Batches**: 1 (10 decisions + include-mechanism test ≈ 11) +**Dependencies**: the P6 mechanism decision above. #### Deliverables -- [ ] Nine `builder-prompt.md` files rewritten -- [ ] **M10 re-baselining executed and enumerated** — this is the phase that collides with - `agent-farm/__tests__/baked-decisions.test.ts:143-148` -- [ ] Per-file manifest + architect review +- [ ] Ten `protocol.md` files rewritten (incl. `release`, codev-local, no twin) +- [ ] **P6 include mechanism implemented**: fenced ```` ```json ```` block containing + `{{> protocols/

/protocol.json}}` +- [ ] **T18 — P6 delivery, both modes**: strict (porch-driven spawn resolves the include) **and** + soft (a builder reading `protocol.md` with no porch still receives the structured source); + asserted against a simulated fresh-install resolution where `codev/protocols/` is absent +- [ ] Manifest + architect review; suite green #### Implementation Details -- Governing principles: **P1**, **P7**. -- **M10 is the load-bearing work here, not the rewriting.** `baked-decisions.test.ts` enforces a - *pure-addition diff* against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md` - — structurally incompatible with rewriting them. Also colliding: - `bugfix-744-spir-pr-strategy.test.ts` (4 near-verbatim sentences), - `spec-1273-wait-discipline-docs.test.ts` (16 assertions), `bugfix-619-aspir-prompt.test.ts`. -- For each: name the originating spec, state whether the protected behaviour survives in the - rewritten prose, and either write the replacement assertion or record an - architect-visible retirement in `codev/resources/1280-retirements.md`. -- Re-baselining a pure-addition baseline requires the originating spec named and the new - baseline committed **in the same commit**. +- `release/protocol.md` has no `protocol.json`; P6 does not apply — rewritten on P1/P3 alone. +- **P7**: delete worst-case padding except scar rules. +- Largest single cut in the project (`spir/protocol.md`, 3,703w); M5's contract-presence + assertions against the Phase-0 frozen inventory are the primary defence. #### Acceptance Criteria -- [ ] Every touched assertion enumerated with its originating spec; none silently deleted -- [ ] Baked-decisions, PR-strategy, wait-discipline and aspir-prompt behaviours either - re-asserted or explicitly retired with approval -- [ ] T4, T7 pass; full suite green -- [ ] Architect judges all 9 decisions conformant **and** approves each assertion retirement - -#### Test Plan -- **Unit**: the four named suites, re-baselined; T4, T7 -- **Integration**: `afx spawn --help` path and a live spawn smoke check -- **Manual**: architect reviews 9 diffs + the retirements file - -#### Rollback Strategy -Group **G3** (shares the group with Phase 2 — they touch the same spawn-time surface and their -tests are coupled). - -#### Risks -The highest-risk phase: silently gutting a prior spec's protection to make the suite green. -Mitigated by M10 being an explicit deliverable with architect sign-off per assertion. +- [ ] T5 passes against the **frozen pre-rewrite inventory** — every gate, check, signal and + artifact contract still represented by name or resolvable reference +- [ ] T18 passes in both modes +- [ ] Architect judges all 10 conformant; suite green --- -### Phase 4: Phase prompts — spir, aspir, pir (11 decisions) +### Phase 4: builder-prompt.md ×9 + M10 test-retirement burden (G3) +**Groups**: G3 · **Batches**: **2** — (A) 9 prompt decisions; (B) 4 test suites + retirements file **Dependencies**: Phase 3 reviewed. -#### Objectives -- Apply **P2** (interfaces, not examples) to the heaviest phase prompts in the fleet - #### Deliverables -- [ ] spir: specify, plan, implement, review (4) -- [ ] aspir: specify, plan, implement, review (4) -- [ ] pir: plan, implement, review (3) -- [ ] Per-file manifest + architect review +- [ ] Nine `builder-prompt.md` files rewritten *(batch A)* +- [ ] **M10 executed and enumerated** *(batch B)*: `baked-decisions.test.ts:139-148` + (pure-addition diff on `protocols/{spir,aspir,air}/builder-prompt.md` — structurally + incompatible with rewriting them), `bugfix-744-spir-pr-strategy.test.ts`, + `bugfix-619-aspir-prompt.test.ts`, plus any `governance-sweep` / `framework-ref-audit` + assertions touched +- [ ] `codev/resources/1280-retirements.md` entries with architect approval *(batch B)* +- [ ] Manifest + architect review of **both** batches; suite green #### Implementation Details -- These carry the fleet's two fattest prompts (`pir/review` 2,414w, `spir/review` 1,957w). -- **P2**: the `{{> …}}` template includes are what make each prompt ~600 words heavier than it - reads. Templates themselves are rewritten in Phases 5–6; this phase rewrites the prompt bodies - and keeps the include mechanism. -- Two separate constraints on template shape, not to be conflated: porch's - `REQUIRED_SPEC_SECTIONS` needs **4** headings (`checks.ts:149-154`); the `spec-review` consult - type advisorily expects 20. -- Signal contracts (`` tags) are capability-inventory items — preserved or retired - explicitly. - -#### Acceptance Criteria -- [ ] T5 passes — every signal, artifact path and check name still represented -- [ ] Porch checks still pass on a scratch project (`spec_has_required_sections`, `has_phases_json`) -- [ ] Architect judges all 11 decisions conformant - -#### Test Plan -- **Unit**: T5, T6, T7; `template-delivery.test.ts` re-baselined per M10 if touched -- **Integration**: drive a scratch project through specify→plan with the rewritten prompts -- **Manual**: architect reviews 11 diffs - -#### Rollback Strategy -Group **G4**. +For each assertion: name the originating spec, state whether the protected behaviour survives in +the rewritten prose, and either write the replacement assertion or record an architect-visible +retirement. Re-baselining a pure-addition baseline requires the originating spec named and the +new baseline committed **in the same commit**. #### Risks -Dropping a porch-required heading and breaking a gate check. Mitigated by the integration test -driving real porch checks, not just unit fixtures. +The highest-risk phase: silently gutting a prior spec's protection to make the suite green. +Mitigated by M10 being an explicit deliverable with architect sign-off per assertion, reviewed as +its own batch rather than buried among prompt diffs. --- -### Phase 5: Phase prompts — bugfix, air, maintain + spir templates (10 decisions) -**Dependencies**: Phase 4 reviewed. +### Phase 5: Phase prompts — spir, aspir, pir (G4) +**Groups**: G4 · **Batches**: 1 (11 decisions) #### Deliverables -- [ ] bugfix: investigate, fix, pr (3); air: implement, pr (2); maintain: maintain, review (2) -- [ ] spir templates: `spec.md`, `plan.md`, `review.md` (3) → heading interfaces -- [ ] Per-file manifest + architect review +- [ ] spir ×4, aspir ×4, pir ×3 +- [ ] **M10**: `template-delivery.test.ts` if the include wiring is touched +- [ ] Manifest + architect review; suite green #### Implementation Details -- The lighter protocols are already closer to conformant (means 356–457w); expect confirmation - rather than large rewrites — and per the acceptance model, **a file that is already conformant - passes unchanged**. -- spir templates are the clearest **P2** case in the project: annotated examples with filler - prose become heading skeletons with one line of intent per heading. -- `plan.md`'s machine-readable phases JSON block is a **capability**, not an example — it is - required by porch's `has_phases_json` check and must survive. +- **P2** is the lever; the `{{> …}}` template includes make each prompt ~600 words heavier than + it reads. Templates themselves are rewritten in Phases 6–7. +- Two separate constraints, not to be conflated: porch's `REQUIRED_SPEC_SECTIONS` needs **4** + headings (`checks.ts:149-154`); the `spec-review` consult type advisorily expects 20. +- `` tags are capability-inventory items — preserved or retired explicitly. #### Acceptance Criteria -- [ ] `has_phases_json` and `min_two_phases` still pass against a plan produced from the - rewritten template -- [ ] T5, T6, T7 pass -- [ ] Architect judges all 10 decisions conformant - -#### Test Plan -- **Unit**: T5, T6, T7 -- **Integration**: generate a plan from the rewritten template, run porch's plan checks against it -- **Manual**: architect reviews 10 diffs - -#### Rollback Strategy -Group **G4**. - -#### Risks -Trimming the plan template's JSON block as "an example". Called out explicitly above. +- [ ] T5 passes; porch checks pass on a scratch project driven specify→plan +- [ ] Architect judges all 11 conformant; suite green --- -### Phase 6: Remaining templates + spir consult-types (10 decisions) -**Dependencies**: Phase 5 reviewed. +### Phase 6: Phase prompts — bugfix, air, maintain + spir templates (G4) +**Groups**: G4 · **Batches**: 1 (10 decisions) #### Deliverables -- [ ] Templates: experiment (1), maintain (1), spike (1), **maintain codev-local ×2** - (`audit-report.md`, `lessons-learned.md` — no skeleton twin) (5) -- [ ] spir consult-types: spec, plan, impl, phase, pr (5) -- [ ] Per-file manifest + architect review +- [ ] bugfix ×3, air ×2, maintain ×2; spir templates `spec.md`/`plan.md`/`review.md` ×3 +- [ ] Manifest + architect review; suite green #### Implementation Details -- The two codev-local maintain templates have **no skeleton twin** — inspected once, excluded - from T7's twin-parity intersection. -- Consult-types: keep the rubric dimensions and the verdict contract; delete the process prose - around them (**P1**, **P2**). -- The verdict format (`VERDICT: APPROVE | REQUEST_CHANGES | COMMENT`) is a **capability** — - `consult` parses it. Preserved exactly. +- Lighter protocols are already closer to conformant (means 356–457w); **a file already + conformant passes unchanged** under the acceptance model. +- **`plan.md`'s machine-readable phases JSON block is a CAPABILITY, not an example** — porch's + `has_phases_json` and `min_two_phases` checks require it. It survives P2 untouched. #### Acceptance Criteria -- [ ] A live `consult -m claude --type spec-review` returns a parseable verdict -- [ ] T5, T6, T7 pass -- [ ] Architect judges all 10 decisions conformant - -#### Test Plan -- **Unit**: T5, T6, T7 -- **Integration**: one live consult per rewritten type, verdict parsed successfully -- **Manual**: architect reviews 10 diffs - -#### Rollback Strategy -Groups **G4** (templates) and **G5** (consult-types). - -#### Risks -Breaking verdict parsing, which would silently degrade every future CMAP round. Mitigated by the -live-consult integration check rather than a fixture. +- [ ] A plan generated from the rewritten template passes `has_phases_json` + `min_two_phases` +- [ ] T5, T6, T7 pass; architect judges all 10 conformant; suite green --- -### Phase 7: Consult-types — aspir, bugfix, air (9 decisions) -**Dependencies**: Phase 6 reviewed. +### Phase 7: Remaining templates + spir consult-types (G4, G5) +**Groups**: G4 (templates), G5 (consult-types) — **two commits, one per group** +**Batches**: 1 (10 decisions + `bugfix-742` test ≈ 11) #### Deliverables -- [ ] aspir ×5, bugfix ×2, air ×2 -- [ ] Per-file manifest + architect review +- [ ] Templates: experiment 1, maintain 1, spike 1, **maintain codev-local ×2** (no skeleton twin) = 5 +- [ ] spir consult-types ×5 +- [ ] **M10**: `bugfix-742-consult-templates.test.ts:27-28` pins prose in + `spir/consult-types/{pr,impl}-review.md` — breaks here +- [ ] Manifest + architect review; suite green #### Implementation Details -- Same treatment as Phase 6. bugfix's two are the fleet's largest consult-types (pr 726, impl - 641) and carry the most process prose. -- aspir's five mirror spir's; if the Phase 6 rewrites apply cleanly, these are largely mechanical - — but each is still a separate decision and a separate diff. +- The two codev-local maintain templates have no skeleton twin — inspected once, excluded from + T7's twin-parity intersection. +- **The verdict format (`VERDICT: APPROVE | REQUEST_CHANGES | COMMENT`) is a CAPABILITY** — + `consult` parses it. Preserved exactly. #### Acceptance Criteria -- [ ] Live consult per rewritten type returns a parseable verdict -- [ ] T5, T6, T7 pass -- [ ] Architect judges all 9 decisions conformant +- [ ] A live `consult --type spec-review` returns a parseable verdict +- [ ] T5, T6, T7 pass; architect judges all 10 conformant; suite green -#### Test Plan -As Phase 6. +--- -#### Rollback Strategy -Group **G5**. +### Phase 8: Consult-types — aspir, bugfix, air (G5) +**Groups**: G5 · **Batches**: 1 (9 decisions + `bugfix-742` bugfix-side assertions ≈ 10) + +#### Deliverables +- [ ] aspir ×5, bugfix ×2, air ×2 +- [ ] **M10**: `bugfix-742-consult-templates.test.ts:25-26` pins `bugfix/consult-types/{pr,impl}-review.md` +- [ ] Manifest + architect review; suite green #### Risks -Mechanical application without judgment — "same as spir" is a rules-not-judgment failure in a -project about exactly that. Each file is judged on its own diff. +Mechanical application without judgment — "same as spir" is itself a rules-not-judgment failure +in a project about exactly that. Each file is judged on its own diff. --- -### Phase 8: Consult-types pir + maintain, scar registry, dead-tree deletion (4 decisions + M4/M6) -**Dependencies**: Phase 7 reviewed. - -#### Objectives -- Finish the per-file rewrite, then rebuild the scar registry **against the settled surface** +### Phase 9: Consult-types pir/maintain + scar registry + dead-tree deletion (G5, G7, G4) +**Groups**: G5 (consult-types), G7 (registry), G4 (dead tree) — **three commits, one per group** +**Batches**: **2** — (A) 4 consult-type decisions; (B) registry + T4 + dead-tree deletion + T8 + routing test #### Deliverables -- [ ] pir ×2, maintain ×2 consult-types (4 decisions) +- [ ] pir ×2, maintain ×2 consult-types *(batch A)* - [ ] `codev/resources/scar-rules.yaml` rebuilt — eight canonicals verbatim, `must_appear_on` - re-derived against the **post-rewrite** surface -- [ ] Scar enforcement test (byte-identical presence, count pinned at 8) -- [ ] `codev-skeleton/porch/prompts/` deleted (M6), with `review-prompt-routing.test.ts:29` - updated under M10 naming **Spec 987** -- [ ] Per-file manifest + architect review + re-derived against the **post-rewrite** surface *(batch B)* +- [ ] **T4** scar enforcement test created here — first phase where it can be meaningful *(batch B)* +- [ ] `codev-skeleton/porch/prompts/` deleted (M6, group G4), with + `review-prompt-routing.test.ts:29` updated under M10 naming **Spec 987** *(batch B)* +- [ ] Manifest + architect review of both batches; suite green #### Implementation Details -- The registry is rebuilt **now, not earlier** — Baked Decision 2 defers enforcement until the - surface stops moving, and `must_appear_on` lists derived before the rewrite would be stale. +- The registry is rebuilt **now** — Baked Decision 2 defers enforcement until the surface stops + moving; `must_appear_on` derived earlier would be stale. - M6 verification is **not** a bare grep: an untruncated repo-wide search reconciled against the - full hit list. (An earlier truncated grep in this project's spec phase produced a false - "no consumers" claim — the failure this step is written to avoid.) + full hit list. (A truncated grep in this project's own spec phase produced a false + "no consumers" claim — the failure this step exists to avoid.) #### Acceptance Criteria - [ ] T4 passes with the rebuilt registry; count pinned at 8; reword/deletion fails -- [ ] T8 passes — tree absent, no runtime reference, Spec 987 protection preserved on remaining files -- [ ] Architect judges the 4 decisions conformant **and** ratifies the registry's `must_appear_on` - -#### Test Plan -- **Unit**: T4, T8; the updated Spec 987 routing test -- **Manual**: architect reviews 4 diffs + registry + the routing-test change - -#### Rollback Strategy -Group **G7** (registry) — note the dependency rule: reverting G7 requires reverting every group -carrying scar text (G2, G3, G4, G6). Group **G5** for the consult-types. - -#### Risks -Registry `must_appear_on` drifting from where scar text actually landed. Mitigated by deriving -it from the post-rewrite surface and by T4 failing loudly. +- [ ] T8 passes; Spec 987 protection preserved on remaining files +- [ ] Architect judges the 4 decisions conformant **and** ratifies `must_appear_on`; suite green --- -### Phase 9: Capability inventory, governance docs, measurement report, PR -**Dependencies**: Phase 8 reviewed. - -#### Objectives -- Prove nothing was lost, report honestly what changed, and open the PR +### Phase 10: Capability verification, measurement report, rollback rehearsal, governance docs +**Groups**: all (verification only) · **Batches**: 1 (artifacts) #### Deliverables -- [ ] Post-rewrite capability inventory extracted and compared (M5); any removal listed in - `codev/resources/1280-retirements.md` with architect approval -- [ ] Measurement re-run: before/after, per-audience, **deleted vs relocated** (M0c, M1, M2) -- [ ] T9 live spawn probe; T10 rollback rehearsal by group (M9) -- [ ] Governance docs routed by tier (`arch.md`/`arch-critical.md`, - `lessons-learned.md`/`lessons-critical.md`) — including the - **"trust the authoritative source, not the convenient signal"** lesson (five instances in - this project's own spec phase) -- [ ] Review document; PR opened -- [ ] **M12 recorded**: no release between merge and the SHIP verdict - -#### Implementation Details -- The measurement report is the project's honesty artifact: it must state where relocated words - went, not merely that always-on fell. -- T10 rehearses a **group** revert on a scratch branch and confirms the suite stays green. -- The A/B (M7) runs **after** merge and gates `verify-approval` — its design, including the - prompt-only overlay construction and the T14 pre-flight, is in the spec. +- [ ] Post-rewrite capability inventory extracted and compared against the Phase-0 frozen + baseline (M5); any removal listed in `1280-retirements.md` with architect approval +- [ ] Measurement re-run: before/after, per-audience, **deleted vs relocated** across all four + skill trees (M0c, M1, M2) +- [ ] **T9** live spawn probe; **T10** rollback rehearsal for **every group touched** + (G1–G7), not a sample (M9) +- [ ] Governance docs routed by tier — including the **"trust the authoritative source, not the + convenient signal"** lesson (five instances in this project's spec phase alone) +- [ ] Suite green + +#### Note on scope +The **review document and PR belong to porch's `review` phase**, not here — the first draft +folded them into an implement sub-phase where porch's review phase would re-run over them. -#### Acceptance Criteria -- [ ] M5 passes; retirements file complete and approved -- [ ] Measurement artifacts committed -- [ ] T9, T10 pass; full suite green -- [ ] Review document complete; PR opened +--- -#### Test Plan -- **Unit**: full suite -- **Integration**: T9 live spawn probe end-to-end -- **Manual**: T10 rollback rehearsal; architect final review +## Post-merge: the verify phase (M7, M8, M12, T13, T14) -#### Rollback Strategy -The PR itself is revertible by group; T10 has rehearsed it. +SPIR's `verify` phase is where the A/B lives; it is **not** an implement phase and is listed here +so it has an explicit home rather than being assumed. -#### Risks -Discovering at the end that a capability was lost several phases ago. Mitigated by T5 running in -every phase, not only here. +- [ ] **M12 release hold** in force from the moment the rewrite PR merges until the SHIP verdict +- [ ] **T14 pre-flight per pair**: every surface under test resolves from tier 2 — no skeleton + file lacks a `codev/` twin. Failure **voids the pair** rather than producing a comparison + that looks valid and is not +- [ ] **M7**: ≥6 issue-pairs, prompt-only overlay construction (both arms from source commit `S`; + control applies one overlay reverting G2–G6), both hashes recorded per run +- [ ] `codev/resources/1280-ab-results.md` — one row per run +- [ ] **M8 / T13**: `measure-prompt-behavior.ts` re-run, B1 compared directionally to 51.88% +- [ ] SHIP / HOLD / ROLLBACK verdict → `verify-approval` ## Timeline & Dependencies -Strictly sequential — each phase ends at the architect's per-file review and does not advance -until it passes. Phase 0 ships as PR-1 and merges before Phase 1 begins; Phases 1–9 accumulate as -commits on one branch and ship as a single PR. +Strictly sequential; each phase ends at the architect's per-file review and a green suite. ``` -P0 (PR-1, merged) → P1 → P2 → P3 → P4 → P5 → P6 → P7 → P8 → P9 (PR) - 4 10 9 11 10 10 9 4 — = 67 decisions +P0 (PR-1, merged) → P1 → P2 → P3 → P4 → P5 → P6 → P7 → P8 → P9 → P10 → [PR] → verify + 1 3 10 9 11 10 10 9 4 — + decisions = 67 ``` ## Rollback Strategy (whole project) -Per the spec's seven groups: G1 instrument · G2 shared · G3 builder-spawn · G4 phase · -G5 consultant · G6 architect · G7 scar registry. Reverting **G7** requires reverting every group -carrying scar text (G2, G3, G4, G6); all others are mutually independent. `git revert` restores -prior bytes — no migration, state, or schema. Rehearsed under T10 before the PR merges. +Seven groups per the spec: G1 instrument · G2 shared · G3 builder-spawn · G4 phase · +G5 consultant · G6 architect · G7 scar registry. **Every commit is group-pure**, so any group +reverts cleanly. Reverting **G7** requires reverting every group carrying scar text (G2, G3, G4, +G6); all others are mutually independent. T10 rehearses **every** group touched. ## Open Questions Carried Into Implementation -- **Batch size** — ≤12 is the spec's cap; the architect may prefer smaller, which grows the phase - count rather than the batches. -- **`roles/consultant.md`** — marked *inspected-but-unchanged (expected)*; rewritten only if - Phase 1 inspection finds non-conformance. +- **Batch size** — ≤12 is the spec's cap; smaller grows the phase count, not the batches. +- **`roles/consultant.md`** — *inspected-but-unchanged (expected)*; rewritten only on finding. - **Hot tier** — out of scope by the spec's disposition table; a reviewer may argue P3 applies. +- **Pre-existing skills drift** (`afx`, `porch` repo-vs-skeleton; `forge`/`skill-creator`/`team` + skeleton-absent) — recorded as known state. Fixing it is arguably in scope for P3/P4 + relocation and arguably a separate concern; **architect's call at the plan gate.** ## Notes -**Why phases are drawn by inspection load.** A conventional plan would group by subsystem. Here -the architect's per-file review is the throughput constraint, so the phase boundary that matters -is "a batch a human can review in one sitting." Phase 3 is the exception that proves it: it is -only 9 decisions but carries the entire M10 test-retirement burden, which is why it is not merged -with the 4-decision Phase 1. +**Why phases are drawn by inspection load.** The architect's per-file review is the throughput +constraint, so the boundary that matters is "a batch a human can review in one sitting." Phase 4 +is the exception that proves it: only 9 decisions, but it carries the entire M10 burden, so it is +split into two explicit batches. -**Phase 0 is not optional sequencing.** Rewriting before the instrument is corrected would make -every subsequent measurement unfalsifiable — the spec's principle 7. +**Phase 0 is not optional sequencing.** Rewriting before the instrument is corrected — and before +the capability inventory is frozen — would make every subsequent measurement and every M5 +assertion unfalsifiable. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md new file mode 100644 index 000000000..66c5238e4 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md @@ -0,0 +1,113 @@ +# Plan iteration 1 response — Spec 1280 + +Codex REQUEST_CHANGES (HIGH, 8) · Claude REQUEST_CHANGES (HIGH, 7). **All accepted; no +disputes.** Both reviewers verified the plan's decision arithmetic independently and both +reproduced 67 — the accounting held; what failed was the *supporting* work riding alongside the +decisions, which the first draft under-specified because it drew boundaries purely by inspection +load. + +Two findings were **blocking mechanism gaps** that would have surfaced mid-implementation as +adopter breakage rather than as clean failures. + +--- + +## Blocking mechanism gaps (both reviewers, verified) + +### 1. P6's `protocol.json` reference had no adopter-resolvable mechanism + +Verified against source: `protocol.md` is inlined at spawn via `{{protocol_reference}}` +(`spawn-roles.ts:112-124`); **`protocol.json` is inlined nowhere** — `spawn-roles.ts:267` reads it +only for validation. In a fresh adopter project `codev/protocols/

/protocol.json` does not +exist on disk. So "read `protocol.json`" is exactly the fetch-by-path of a framework file that +CLAUDE.md forbids and the spec's own constraint restates — and it was carrying the largest single +cut in the project. + +**Resolved by verifying the resolver rather than guessing**: `resolveCodevIncludes` +(`skeleton.ts:108-119`) is **extension-agnostic**, so `protocol.md` carries a fenced ```json +block containing `{{> protocols/

/protocol.json}}`. Resolves through all four tiers, works in +fresh installs, needs **no porch change**, and is the literal expression of P6. Cost stated +honestly (spir +570 words back) — acceptable because size is reporting-only under the amended +charter. New **T18** tests both **strict** and **soft** mode, because the asymmetry is real: +strict-mode builders get checks/gates as porch tasks, soft-mode builders have only the prompt. + +### 2. Skill relocation is a FOUR-tree sync the instrument could only see one quarter of + +Claude's finding, verified: skills exist in `.claude/skills` (10), `.codex/skills` (10, +**byte-identical**), `codev-skeleton/.claude/skills` (7), `codev-skeleton/.codex/skills` (7) — +with **existing drift** (`afx`, `porch` differ repo-vs-skeleton; `forge`, `skill-creator`, `team` +skeleton-absent). M0(g) counted only `.claude/skills`, so relocating content there would have +left Codex agents and adopters without it **and reported it as deleted** by M0c/T15 — inverting +the project's honesty artifact. + +Fixed: Phase 0 widens M0(g) to all four trees; Phase 1 adds **T17** (skills parity) and treats +every relocation as a four-copy write. Pre-existing drift is recorded as known state rather than +silently "fixed" — with the scope question raised for the architect at the gate. + +--- + +## Codex + +**Pre-rewrite capability inventory never created — ACCEPTED.** Phase 2 asserted T5 while Phase 9 +first extracted the inventory. M5 requires a *committed pre-rewrite* baseline. Moved into Phase 0 +(PR-1) and frozen there; every later phase asserts against it. + +**Post-merge work has no executable home — ACCEPTED.** Added an explicit **verify phase** +section covering M7, M8, M12, T13, T14, `1280-ab-results.md`, and the SHIP/HOLD/ROLLBACK verdict. + +**T3, T13, T14, T16 unhomed — ACCEPTED.** T3 and **T16** into Phase 0 — T16 especially, since it +is the mechanical guard on M11 and must exist *before* Phase 1 produces the first manifest. T13 +and T14 into verify. Manifest format and location now specified. + +**"≤12 batches" didn't count supporting changes — ACCEPTED, and it would have broken the +architect's own mandate.** Added an explicit definition: a **review batch** is every distinct file +the architect reads, including tests, registry and retirements. Phases 4 and 9 now declare **two +batches each**. + +**Scar-test sequencing contradictory — ACCEPTED.** Phase 1 asserted T4 while Phase 8 created it. +Phase 1 now verifies the eight canonicals byte-for-byte against the ratified +`builder/spir-1252:scar-rules.yaml` directly; T4 is created in Phase 9 and applies from there. + +**Grouped rollback not achievable from the proposed commits — ACCEPTED** (Claude found the same +from the mapping side). Resolved with a commit-level invariant: **every commit is group-pure**; +phases may span groups and now declare them. M6's dead-tree deletion assigned to **G4**. T10 +rehearses **every** group touched, not a sample. + +**PR-1 mechanics unstated — ACCEPTED.** Added: branch cut, merge wait, re-branch via +`git fetch origin main && git checkout -b … origin/main` (never `git checkout main` from a +worktree), `porch done --pr` / `--merged` recording, and a duplicate-commit check. + +--- + +## Claude + +**Rollback mapping contradicted the spec — ACCEPTED.** Phase 1 claimed G2/G6 while rewriting +`roles/builder.md` (spec G3) and `roles/consultant.md` (spec G5); a G3 revert would have pulled +Phase 1 work out and T10 would have rehearsed the wrong map. Roles now live in their own phase +with **three group-pure commits** (G6/G3/G5). + +**M10 concentrated in one phase but collisions are spread — one misassigned — ACCEPTED, +verified.** `spec-1273-wait-discipline-docs.test.ts:26` targets `codev/roles/builder.md` + its +skeleton twin → breaks in the **roles** phase, not the builder-prompt phase; line 31 targets +`.claude`/`.codex` `afx/SKILL.md` → breaks in **Phase 1** on relocation. +`bugfix-742-consult-templates.test.ts:25-28` pins spir *and* bugfix consult-types → Phases 7 and +8. M10 is now a per-phase deliverable, assigned where each collision actually lands. + +**No per-phase green-suite requirement — ACCEPTED.** The branch could have sat red across eight +review batches while the architect inspected diffs on a broken tree. Every phase now ends green, +and it is in Success Metrics. + +**Reconcile "~66" vs 67 — ACCEPTED.** The plan states 67 is correct and notes both reviewers +reproduced it. + +**Phase 9 folded review-phase deliverables into an implement phase — ACCEPTED.** Review document +and PR moved out to porch's `review` phase, which would otherwise re-run over them. + +--- + +## Net + +Fifteen findings, none disputed. The plan's decision accounting survived both reviews intact; +everything that failed was work that *accompanies* the decisions — tests, inventories, group +purity, sync obligations — which is exactly what a plan drawn by inspection load will +under-specify if nobody checks. Phase count 10 → 11, with two phases now carrying explicit +double batches. diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 6bb4be1dd..c124b5d2b 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -384,3 +384,42 @@ Design decisions worth recording: plausible casualties of P2 applied carelessly. Both get live integration checks, not fixtures. Porch checks pass: plan_exists, has_phases_json, min_two_phases (10). + +### Plan CMAP round 1 — 15 findings, none disputed (2026-07-31) + +Codex REQUEST_CHANGES (8) · Claude REQUEST_CHANGES (7). Both independently reproduced the 67 +decision count — **the accounting held; what failed was everything riding alongside it.** A plan +drawn purely by inspection load under-specifies the supporting work, and that is precisely what +both reviewers found. + +**Two blocking mechanism gaps, both verified against source before accepting:** + +1. **P6 had no adopter-resolvable mechanism.** `protocol.md` is inlined at spawn via + `{{protocol_reference}}` (spawn-roles.ts:112-124); **protocol.json is inlined nowhere** (:267 + reads it only for validation). In a fresh adopter project the file isn't on disk. So my + "reference protocol.json" was the fetch-by-path CLAUDE.md forbids — carrying the largest cut + in the project. Resolved by *checking the resolver rather than guessing*: + `resolveCodevIncludes` is extension-agnostic, so a fenced ```json block with + `{{> protocols/

/protocol.json}}` resolves through all four tiers with no porch change. + New T18 tests strict AND soft mode — soft-mode builders have only the prompt, no porch tasks. + +2. **Skill relocation is a FOUR-tree sync** (Claude). `.claude/skills` (10), `.codex/skills` (10, + byte-identical), and both skeleton copies (7 each) — with existing drift (afx, porch) and + three skeleton-absent skills. M0(g) counted only `.claude/skills`, so relocated content would + have left Codex agents and adopters without it **and been reported as deleted** — inverting + the project's own honesty artifact. Phase 0 widens the basis; new T17 asserts parity. + +**Other structural fixes**: pre-rewrite capability inventory frozen in Phase 0 (was first +extracted in the final phase, while Phase 2 already asserted against it); verify phase given an +explicit home for M7/M8/M12/T13/T14; T3/T16 into Phase 0 (T16 guards M11 and had to exist before +the first manifest); "review batch" *defined* to include tests/registry/retirements, with Phases +4 and 9 declaring two batches each; scar canonicals verified against the ratified YAML in Phase 1 +with T4 deferred to Phase 9; commit-level group purity replacing the broken phase-level mapping; +per-phase green suite; M10 reassigned to where each collision actually lands +(spec-1273 → roles phase and Phase 1, not Phase 4; bugfix-742 → Phases 7/8). + +Phase count 10 → 11. Decisions still 67, max batch 11. + +**Notable**: Claude caught that my rollback mapping contradicted the spec I wrote — Phase 1 +claimed G2/G6 while rewriting roles/builder.md (G3) and roles/consultant.md (G5). A G3 revert +would have silently pulled Phase 1 work out and T10 would have rehearsed the wrong map. From 90742b3add8d6b272c1e57b4610d71c6556d8de8 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:00:12 -0700 Subject: [PATCH 23/32] chore(porch): 1280 plan-approval gate-requested --- codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index d2e1fcdcc..898a1f20c 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -11,6 +11,7 @@ gates: approved_at: '2026-08-01T03:48:22.599Z' plan-approval: status: pending + requested_at: '2026-08-01T04:00:12.126Z' pr: status: pending verify-approval: @@ -19,4 +20,4 @@ iteration: 1 build_complete: true history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T03:51:45.083Z' +updated_at: '2026-08-01T04:00:12.126Z' From 558fb4f4a4e22e2480c177faeb8ab40922bb1871 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:26:10 -0700 Subject: [PATCH 24/32] chore(porch): 1280 plan-approval gate-approved --- .../projects/1280-prompt-surface-judgment-not-ru/status.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 898a1f20c..4a02d2a0e 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -10,8 +10,9 @@ gates: requested_at: '2026-08-01T03:06:40.784Z' approved_at: '2026-08-01T03:48:22.599Z' plan-approval: - status: pending + status: approved requested_at: '2026-08-01T04:00:12.126Z' + approved_at: '2026-08-01T04:26:10.219Z' pr: status: pending verify-approval: @@ -20,4 +21,4 @@ iteration: 1 build_complete: true history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T04:00:12.126Z' +updated_at: '2026-08-01T04:26:10.220Z' From 0e7cf666cd3b047405bf31939f84f828f769f809 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:26:12 -0700 Subject: [PATCH 25/32] chore(porch): 1280 implement phase-transition --- .../status.yaml | 43 ++++++++++++++++--- 1 file changed, 38 insertions(+), 5 deletions(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 4a02d2a0e..5d59a25cb 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -1,9 +1,42 @@ id: '1280' title: prompt-surface-judgment-not-ru protocol: spir -phase: plan -plan_phases: [] -current_plan_phase: null +phase: implement +plan_phases: + - id: phase_0_instrument + title: Corrected instrument + frozen capability inventory (PR-1, ships early) + status: in_progress + - id: phase_1_shared_skills + title: CLAUDE.md/AGENTS.md + four-tree skill relocation (G2) + status: pending + - id: phase_2_roles + title: Three role files (G6, G3, G5) + status: pending + - id: phase_3_protocol_md + title: protocol.md x10 with the P6 include mechanism (G3) + status: pending + - id: phase_4_builder_prompts + title: builder-prompt.md x9 + M10 test-retirement burden (G3) + status: pending + - id: phase_5_prompts_heavy + title: 'Phase prompts: spir, aspir, pir (G4)' + status: pending + - id: phase_6_prompts_light_spir_templates + title: 'Phase prompts: bugfix, air, maintain + spir templates (G4)' + status: pending + - id: phase_7_templates_consult_spir + title: Remaining templates + spir consult-types (G4, G5) + status: pending + - id: phase_8_consult_types_a + title: 'Consult-types: aspir, bugfix, air (G5)' + status: pending + - id: phase_9_consult_registry_deadtree + title: Consult-types pir/maintain + scar registry + dead-tree deletion (G5, G7, G4) + status: pending + - id: phase_10_integration + title: Capability verification, measurement report, rollback rehearsal, governance docs + status: pending +current_plan_phase: phase_0_instrument gates: spec-approval: status: approved @@ -18,7 +51,7 @@ gates: verify-approval: status: pending iteration: 1 -build_complete: true +build_complete: false history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T04:26:10.220Z' +updated_at: '2026-08-01T04:26:12.253Z' From 47a9ab579db514bbc07600e7571894b98c7f2582 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:26:36 -0700 Subject: [PATCH 26/32] [Spec 1280] Plan: record the plan-gate skills-drift ruling Any skill this project touches gets four-tree parity (T17 enforces, scoped to the touched set). Pre-existing drift on untouched skills and the skeleton-absence product question go to a separate architect-filed issue and are recorded-known-state here - they must not fail T17. --- .../1280-prompt-surface-judgment-not-ru.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/codev/plans/1280-prompt-surface-judgment-not-ru.md b/codev/plans/1280-prompt-surface-judgment-not-ru.md index 071d2a343..29403dbbf 100644 --- a/codev/plans/1280-prompt-surface-judgment-not-ru.md +++ b/codev/plans/1280-prompt-surface-judgment-not-ru.md @@ -167,10 +167,12 @@ Group **G1**. No prompt surface has changed. #### Deliverables - [ ] `CLAUDE.md` + `AGENTS.md` rewritten (1 decision, 2 byte-identical files) - [ ] Relocated how-to content written to **all four skill trees**, not one -- [ ] **T17 — skills parity**: `.claude/skills` ≡ `.codex/skills`; every skill present in the - repo that the skeleton ships is in sync. Pre-existing drift (`afx`, `porch`) and - skeleton-absent skills (`forge`, `skill-creator`, `team`) are recorded as known state, not - silently "fixed" — but any skill this project *touches* must be four-way consistent +- [ ] **T17 — skills parity, scoped by the plan-gate ruling (2026-08-01)**: every skill this + project **touches** must be four-tree consistent (`.claude/skills`, `.codex/skills`, and + both `codev-skeleton/` copies). Untouched skills are **exempt** — T17 asserts over the + touched set only, so pre-existing drift (`afx`, `porch`) and skeleton-absent skills + (`forge`, `skill-creator`, `team`) cannot fail it. Those go to the architect's separate + issue and are recorded here as known state - [ ] All eight scar canonicals present byte-identically - [ ] **M10 here, not Phase 4**: `spec-1273-wait-discipline-docs.test.ts:31` asserts on `.claude/skills/afx/SKILL.md` + `.codex/skills/afx/SKILL.md` — relocation into `afx` breaks @@ -430,9 +432,12 @@ G6); all others are mutually independent. T10 rehearses **every** group touched. - **Batch size** — ≤12 is the spec's cap; smaller grows the phase count, not the batches. - **`roles/consultant.md`** — *inspected-but-unchanged (expected)*; rewritten only on finding. - **Hot tier** — out of scope by the spec's disposition table; a reviewer may argue P3 applies. -- **Pre-existing skills drift** (`afx`, `porch` repo-vs-skeleton; `forge`/`skill-creator`/`team` - skeleton-absent) — recorded as known state. Fixing it is arguably in scope for P3/P4 - relocation and arguably a separate concern; **architect's call at the plan gate.** +- ~~**Pre-existing skills drift**~~ — **RULED at the plan gate (2026-08-01)**: any skill this + project **touches** gets full four-tree parity as part of touching it (**T17** enforces). + Pre-existing drift on **untouched** skills (`afx`, `porch` repo-vs-skeleton) and the + skeleton-absence product question (`forge`, `skill-creator`, `team`) go to a **separate + issue**, filed by the architect. This plan treats them as **recorded known state, nothing + more** — they are not repaired here and their presence must not fail T17. ## Notes From 2b0b68ae891e4eaf4f44a4c63ad9745a024b884b Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:35:20 -0700 Subject: [PATCH 27/32] [Spec 1280][Phase: phase_0_instrument] fix: correct the prompt-surface measurement instrument Spec 1280 M0/M0b/M0c. No prompt-surface file is touched by this commit - the instrument must be correct and public before anything is scored by it. THREE DEFECTS CORRECTED (the 1252 instrument was committed, deterministic, and wrong): 1. Phase-task term was derived from codev-skeleton/porch/prompts/, a dead Ralph-SPIR-era tree with no runtime consumer, while loadPromptFile loads protocols/

/prompts/. Real SPIR phase prompts average 1,396 words; the dead tree averaged 400. The metric was blind to the largest always-on surface. 2. roles/builder.md (1,837w) was omitted despite spawn-worktree.ts:854 injecting it into every builder spawn. 3. The script's own comment claimed CLAUDE.md 'already inlines' the hot tier. Since #1119 it @imports, which Claude Code transcludes - so wc -w excluded 736 always-loaded words. Reported baseline: 21,702 -> 34,235. ALSO FIXED, found while testing (a fourth inaccuracy, mine): include expansion was ADDITIVE - it counted a {{> path}} directive's own tokens plus the content substituted for them, over-reporting ~2 words per include. expand_text now does real substitution mirroring resolveCodevIncludes (regex replace in place, recursive, depth-guarded, unresolved -> empty), which also makes the T2 phantom-savings property exact. This is why the figure is 34,235 rather than the 34,255 quoted in the spec; size is reporting-only, so no criterion moves. New in this instrument: - per-file four-tier resolution (.codev > codev > skeleton), replacing two-tier directory-level selection that mis-measured single-file overrides - exclusive buckets (which sum) reported separately from derived audience loads (which overlap) - never presented as a sum - TOTAL_AUTHORED on a deliberately different basis (physical files, no dedup, no expansion) across ALL FOUR skill trees, so relocation cannot be reported as deletion (M0c) Frozen pre-rewrite capability inventory (M5): 57 capabilities, 47 present in served prompts, 10 absent pre-existing (porch delivers those via task JSON, not authored text). Extraction is over served prompt text, not protocol.json - extracting from unchanged config would report every capability present even if every instruction vanished. Representation accepts a resolvable {{> }} include, so a conformant P6 rewrite cannot fail M5. A pipefail/grep -q bug in the extractor initially reported all 57 as absent: grep -q exits on first match, printf takes SIGPIPE, and the pipeline reports failure BECAUSE the match succeeded. Fixed with a here-string. Same failure class this project keeps recording: an exit code with two causes, read as one. Tests: T1, T1b, T2, T3, T11, T12, T15 (instrument) and T16 (manifest completeness, written before the first manifest exists because the guard must predate what it guards). 22 passing. 1252-word-baseline.md and 1252-word-after-phase7.md annotated in place - originals preserved, marked superseded, with the reason and a pointer. --- .../manifests/README.md | 37 +++ codev/resources/1252-word-after-phase7.md | 27 ++ codev/resources/1252-word-baseline.md | 27 ++ .../resources/1280-capability-inventory.json | 69 ++++ codev/resources/1280-word-baseline.md | 68 ++++ .../spec-1280-measurement-instrument.test.ts | 283 ++++++++++++++++ .../spec-1280-phase-manifest.test.ts | 119 +++++++ scripts/extract-capability-inventory.sh | 156 +++++++++ scripts/measure-prompt-surface.sh | 307 ++++++++++++------ 9 files changed, 994 insertions(+), 99 deletions(-) create mode 100644 codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md create mode 100644 codev/resources/1280-capability-inventory.json create mode 100644 codev/resources/1280-word-baseline.md create mode 100644 packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts create mode 100644 packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts create mode 100755 scripts/extract-capability-inventory.sh diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md b/codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md new file mode 100644 index 000000000..438c01072 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md @@ -0,0 +1,37 @@ +# Per-phase inspection manifests (Spec 1280, M11) + +The architect inspects the **old-vs-new diff of every changed file**, per phase, before porch +advances. A manifest is what makes that inspection possible: **the architect cannot inspect what +is not listed**, so a changed prompt-bearing file absent from its phase's manifest fails the +phase (**T16**, `spec-1280-phase-manifest.test.ts`). + +## File + +`phase--.md`, one per implement phase, committed with that phase's work. + +## Required shape + +A manifest has one row per changed file with **all four fields**. Phases with more than 12 +files declare explicit `## Batch N` sections, each ≤12 — the cap is on *what the architect reads +in one sitting*, which includes tests, registry and retirements entries, not only prompt files. + +```markdown +# Phase 3 — protocol.md x10 (G3) + +## Batch 1 + +| File | Old | New | Principles | Rationale | +|---|---:|---:|---|---| +| `codev-skeleton/protocols/spir/protocol.md` | 3703 | 690 | P6, P1, P7 | State machine now references protocol.json via `{{> }}`; deleted phase-body checklists | +| `codev-skeleton/protocols/air/protocol.md` | 643 | 380 | P1, P7 | Deleted worst-case padding; kept artifact contract | +``` + +- **Old / New** — served word counts from `scripts/measure-prompt-surface.sh`. +- **Principles** — which of P1–P7 were applied (or `none` for an inspected-but-unchanged file). + P5 is N/A project-wide, with reason, per the spec. +- **Rationale** — one line. What was cut, and why it was safe to cut. + +## What a manifest is not + +It is not a substitute for reading the diff. It is the index that makes the diff review +complete and bounded. diff --git a/codev/resources/1252-word-after-phase7.md b/codev/resources/1252-word-after-phase7.md index f36be3fe7..8339dae81 100644 --- a/codev/resources/1252-word-after-phase7.md +++ b/codev/resources/1252-word-after-phase7.md @@ -56,3 +56,30 @@ largest block (protocol.md, 3,703 words) is protocol semantics — excluded by the spec's Non-goals. The 20% arithmetic assumed more of the surface was duplicated rules than measurement bore out. Whether even −7% moved behaviour is the verify phase's question (M12b). + +--- + +## SUPERSEDED by Spec 1280 (2026-08-01) + +**The figures above were produced by an instrument with three defects and should not be +cited.** They are preserved unaltered because the record of what was believed, and when, +is part of the history — not because they are correct. + +The instrument (`scripts/measure-prompt-surface.sh` as of Spec 1252): + +1. derived its phase-task term from `codev-skeleton/porch/prompts/`, a dead Ralph-SPIR-era + tree with **no runtime consumer**, while the live resolver + (`commands/porch/prompts.ts`, `loadPromptFile`) loads `protocols/

/prompts/`. Real SPIR + phase prompts average ~1,396 words; the dead tree averaged 400; +2. omitted `roles/builder.md` (1,837 words), which `spawn-worktree.ts:854` injects into + **every** builder spawn; +3. asserted in its own comments that `CLAUDE.md` "already inlines" the hot tier. Since #1119 + it carries `@import` lines, which are **transcluded** at session launch — so `wc -w + CLAUDE.md` excludes 736 always-loaded words. + +**Corrected pre-rewrite baseline: `codev/resources/1280-word-baseline.md` +(ALWAYS_ON_WORDS = 34,235 for a SPIR builder at I=10, versus the 21,702 implied here).** + +The behavioural baseline in `1252-behavior-baseline.md` is **unaffected** — it uses a +different instrument (`measure-prompt-behavior.ts`) and remains the valid "before" for +Spec 1280's M8. diff --git a/codev/resources/1252-word-baseline.md b/codev/resources/1252-word-baseline.md index 99aa8abe5..08dd93707 100644 --- a/codev/resources/1252-word-baseline.md +++ b/codev/resources/1252-word-baseline.md @@ -37,3 +37,30 @@ session, not both), and the hot-tier files appear inside CLAUDE.md's count (they ARE counted once per phase task, where porch injects them separately). ALWAYS_ON_WORDS=21856 + +--- + +## SUPERSEDED by Spec 1280 (2026-08-01) + +**The figures above were produced by an instrument with three defects and should not be +cited.** They are preserved unaltered because the record of what was believed, and when, +is part of the history — not because they are correct. + +The instrument (`scripts/measure-prompt-surface.sh` as of Spec 1252): + +1. derived its phase-task term from `codev-skeleton/porch/prompts/`, a dead Ralph-SPIR-era + tree with **no runtime consumer**, while the live resolver + (`commands/porch/prompts.ts`, `loadPromptFile`) loads `protocols/

/prompts/`. Real SPIR + phase prompts average ~1,396 words; the dead tree averaged 400; +2. omitted `roles/builder.md` (1,837 words), which `spawn-worktree.ts:854` injects into + **every** builder spawn; +3. asserted in its own comments that `CLAUDE.md` "already inlines" the hot tier. Since #1119 + it carries `@import` lines, which are **transcluded** at session launch — so `wc -w + CLAUDE.md` excludes 736 always-loaded words. + +**Corrected pre-rewrite baseline: `codev/resources/1280-word-baseline.md` +(ALWAYS_ON_WORDS = 34,235 for a SPIR builder at I=10, versus the 21,702 implied here).** + +The behavioural baseline in `1252-behavior-baseline.md` is **unaffected** — it uses a +different instrument (`measure-prompt-behavior.ts`) and remains the valid "before" for +Spec 1280's M8. diff --git a/codev/resources/1280-capability-inventory.json b/codev/resources/1280-capability-inventory.json new file mode 100644 index 000000000..c9a791271 --- /dev/null +++ b/codev/resources/1280-capability-inventory.json @@ -0,0 +1,69 @@ +{ + "_spec": "1280", + "_criterion": "M5", + "_basis": "served prompt text (includes expanded); protocol.json supplies the expected set only", + "_detection_limit": "detects deletion, not inversion or gutting — see M11 and O4", + "_commit": "1056834d", + "gates": [ + {"name": "dev-approval", "present_in_served_prompts": true}, + {"name": "experiment-complete", "present_in_served_prompts": false}, + {"name": "maintain-complete", "present_in_served_prompts": false}, + {"name": "plan-approval", "present_in_served_prompts": true}, + {"name": "pr", "present_in_served_prompts": true}, + {"name": "research-complete", "present_in_served_prompts": false}, + {"name": "scope-approval", "present_in_served_prompts": true}, + {"name": "spec-approval", "present_in_served_prompts": true}, + {"name": "verify-approval", "present_in_served_prompts": true} ], + "checks": [ + {"name": "build", "present_in_served_prompts": true}, + {"name": "e2e_tests", "present_in_served_prompts": false}, + {"name": "has_phases_json", "present_in_served_prompts": false}, + {"name": "min_two_phases", "present_in_served_prompts": false}, + {"name": "plan_exists", "present_in_served_prompts": false}, + {"name": "pr_exists", "present_in_served_prompts": true}, + {"name": "regression_test", "present_in_served_prompts": false}, + {"name": "review_has_arch_updates", "present_in_served_prompts": true}, + {"name": "review_has_lessons_updates", "present_in_served_prompts": true}, + {"name": "spec_exists", "present_in_served_prompts": false}, + {"name": "spec_has_required_sections", "present_in_served_prompts": false}, + {"name": "tests", "present_in_served_prompts": true} ], + "signals": [ + {"name": "AWAITING_INPUT", "present_in_served_prompts": true}, + {"name": "BLOCKED", "present_in_served_prompts": true}, + {"name": "PHASE_COMPLETE", "present_in_served_prompts": true}, + {"name": "PLAN_DRAFTED", "present_in_served_prompts": true}, + {"name": "PR_READY", "present_in_served_prompts": true}, + {"name": "REVIEW_COMPLETE", "present_in_served_prompts": true}, + {"name": "SPEC_DRAFTED", "present_in_served_prompts": true}, + {"name": "TOO_COMPLEX", "present_in_served_prompts": true} ], + "artifact_paths": [ + {"name": "codev/plans/{{artifact_name}}.md", "present_in_served_prompts": true}, + {"name": "codev/plans/0042-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/plans/42-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/plans/XXXX-*.md", "present_in_served_prompts": true}, + {"name": "codev/plans/XXXX-name.md", "present_in_served_prompts": true}, + {"name": "codev/reviews/{{artifact_name}}.md", "present_in_served_prompts": true}, + {"name": "codev/reviews/42-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/reviews/XXXX-name.md", "present_in_served_prompts": true}, + {"name": "codev/specs/...", "present_in_served_prompts": true}, + {"name": "codev/specs/{{artifact_name}}.md", "present_in_served_prompts": true}, + {"name": "codev/specs/{{project_id}}-*.md", "present_in_served_prompts": true}, + {"name": "codev/specs/0042-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/specs/1-test-infrastructure.md", "present_in_served_prompts": true}, + {"name": "codev/specs/42-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/specs/42-feature.md", "present_in_served_prompts": true}, + {"name": "codev/specs/spec-file.md", "present_in_served_prompts": true}, + {"name": "codev/specs/XXXX-*.md", "present_in_served_prompts": true}, + {"name": "codev/specs/XXXX-name.md", "present_in_served_prompts": true}, + {"name": "codev/state/*.md", "present_in_served_prompts": true}, + {"name": "codev/state/spir-823_thread.md", "present_in_served_prompts": true} ], + "notification_triggers": [ + {"name": "afx send ", "present_in_served_prompts": true}, + {"name": "afx send ", "present_in_served_prompts": true}, + {"name": "afx send :architect", "present_in_served_prompts": true}, + {"name": "afx send ", "present_in_served_prompts": true}, + {"name": "afx send architect", "present_in_served_prompts": true}, + {"name": "afx send architect:", "present_in_served_prompts": true}, + {"name": "afx send architect:ob-refine", "present_in_served_prompts": true}, + {"name": "afx send marketmaker:architect", "present_in_served_prompts": true} ] +} diff --git a/codev/resources/1280-word-baseline.md b/codev/resources/1280-word-baseline.md new file mode 100644 index 000000000..46d321a5a --- /dev/null +++ b/codev/resources/1280-word-baseline.md @@ -0,0 +1,68 @@ + +# Prompt-surface measurement + +Commit: `1056834d` +Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. + +## Exclusive buckets (partition the authored surface; these SUM) + +| Bucket | Words | +|---|---:| +| SHARED (CLAUDE.md + transcluded hot tier) | 6551 | +| ARCHITECT (roles/architect.md) | 2048 | +| DEAD (codev-skeleton/porch/prompts, 10 files) | 4009 | + +## Per-protocol (resolved per file, four-tier) + +| Protocol | BUILDER_SPAWN | PHASE mean | CONSULT mean | +|---|---:|---:|---:| +| air | 3017 | 456 | 437 | +| aspir | 3467 | 1396 | 430 | +| bugfix | 2965 | 377 | 683 | +| experiment | 3330 | 0 | 0 | +| maintain | 4158 | 356 | 406 | +| pir | 4801 | 1435 | 491 | +| release | 3463 | 0 | 0 | +| research | 3671 | 0 | 0 | +| spike | 3155 | 0 | 0 | +| spir | 6364 | 1396 | 430 | + +## Derived audience loads (these OVERLAP by design — never sum them) + + HOT = arch-critical(416) + lessons-critical(320) = 736 + ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p] + I x (HOT + mean PHASE[p]) + ALWAYS_ON(architect) = SHARED + ARCHITECT + ALWAYS_ON(consultant,p) = roles/consultant.md + mean CONSULT-type[p] + +| Audience | Words | +|---|---:| +| **Builder (spir, I=10)** — the headline | **34235** | +| Architect (per session) | 8599 | +| Consultant (per review, spir) | 682 | + +## Total authored surface (relocation detector — different basis, see header) + +| Component | Words | +|---|---:| +| CLAUDE.md + AGENTS.md (no twin dedup here) | 11630 | +| codev/protocols + codev-skeleton/protocols | 88475 | +| codev/roles + codev-skeleton/roles | 8274 | +| skills, ALL FOUR trees (42 files) | 44840 | +| **TOTAL_AUTHORED** | **153219** | + +Reference (on-demand, not always-on): arch.md 20367, lessons-learned.md 21270. + +ALWAYS_ON_WORDS=34235 +TOTAL_AUTHORED_WORDS=153219 diff --git a/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts b/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts new file mode 100644 index 000000000..d1998a820 --- /dev/null +++ b/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts @@ -0,0 +1,283 @@ +/** + * Spec 1280 — Phase 0: the corrected measurement instrument. + * + * The Spec 1252 instrument was committed, deterministic, and wrong: it derived + * its phase-task term from `codev-skeleton/porch/prompts/`, a dead tree with no + * runtime consumer, while the live resolver loads `protocols/

/prompts/`. It + * also omitted the spawn-inlined `roles/builder.md` and mis-stated how the hot + * tier reaches CLAUDE.md. + * + * These tests exist because "deterministic and committed" is not "correct". + * They assert the instrument against the REAL resolver and the REAL runtime + * loader, so a future edit cannot quietly reintroduce any of the three defects. + * + * Covers T1, T1b, T2, T3, T11, T12, T15. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +const repoRoot = path.resolve(import.meta.dirname, '../../../..'); +const script = path.join(repoRoot, 'scripts/measure-prompt-surface.sh'); + +function run(root: string = repoRoot, env: Record = {}): string { + return execFileSync('bash', [script, root], { + encoding: 'utf-8', + env: { ...process.env, ...env }, + maxBuffer: 20 * 1024 * 1024, + }); +} + +function num(output: string, key: string): number { + const m = output.match(new RegExp(`^${key}=(\\d+)$`, 'm')); + if (!m) throw new Error(`${key} not found in measurement output`); + return Number(m[1]); +} + +/** Minimal fixture repo: only what the script reads. */ +function makeFixture(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'spec1280-')); + const mk = (rel: string, body: string) => { + fs.mkdirSync(path.join(dir, path.dirname(rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); + }; + mk('CLAUDE.md', 'alpha bravo charlie\n'); + mk('AGENTS.md', 'alpha bravo charlie\n'); + mk('codev/resources/arch-critical.md', 'one two\n'); + mk('codev/resources/lessons-critical.md', 'three four\n'); + mk('codev/roles/builder.md', 'role word here\n'); + mk('codev/roles/architect.md', 'architect role words\n'); + mk('codev/roles/consultant.md', 'consultant words\n'); + mk('codev-skeleton/protocols/spir/builder-prompt.md', 'wrapper words here\n'); + mk('codev-skeleton/protocols/spir/protocol.md', 'protocol words here now\n'); + mk('codev-skeleton/protocols/spir/prompts/specify.md', 'aa bb cc dd ee\n'); + mk('codev-skeleton/protocols/spir/consult-types/spec-review.md', 'rubric words\n'); + return dir; +} + +describe('T1 — the instrument sources the directory the runtime actually loads', () => { + it('reads protocols/

/prompts/, not the dead porch/prompts tree', () => { + const src = fs.readFileSync(script, 'utf-8'); + // Assert on EXECUTABLE lines only. The header comment legitimately names the + // dead tree while explaining the defect, and a blanket string ban would + // forbid documenting the very bug this test guards. + const code = src + .split('\n') + .filter((l) => !/^\s*#/.test(l) && l.trim() !== '') + .join('\n'); + expect(code).toMatch(/protocols\/\$p\/prompts\//); + expect(code).not.toMatch(/PORCH_DIR=/); + // porch/prompts may appear only as the DEAD bucket (its measurement and its + // report label) — never as an input to the phase-task term. The tree still + // exists and the report must expose it; what must never return is it FEEDING + // the phase mean, which was defect 1. + const porchRefs = code.split('\n').filter((l) => l.includes('porch/prompts')); + expect(porchRefs.length).toBeGreaterThan(0); + for (const line of porchRefs) { + expect(line, `unexpected porch/prompts use: ${line}`).toMatch(/DEAD_[WF]=|\| DEAD /); + } + const phaseMeanFn = code.slice(code.indexOf('phase_mean()'), code.indexOf('consult_mean()')); + expect(phaseMeanFn).not.toMatch(/porch/); + }); + + it('agrees with loadPromptFile, which resolves protocols//prompts/', () => { + // Ground the assertion in the real loader rather than a hardcoded string: + // if porch's resolution path ever moves, this fails loudly. + const loader = fs.readFileSync( + path.join(repoRoot, 'packages/codev/src/commands/porch/prompts.ts'), + 'utf-8', + ); + expect(loader).toMatch(/protocols\/\$\{protocolName\}\/prompts\/\$\{promptFile\}/); + }); + + it('counts the spawn-inlined role file (defect 2)', () => { + const spawn = fs.readFileSync( + path.join(repoRoot, 'packages/codev/src/agent-farm/commands/spawn-worktree.ts'), + 'utf-8', + ); + expect(spawn).toContain('.builder-role.md'); // it really is injected at spawn + expect(fs.readFileSync(script, 'utf-8')).toMatch(/BUILDER_ROLE=.*roles\/builder\.md/); + }); + + it('adds the hot tier to CLAUDE.md rather than assuming it is inlined (defect 3)', () => { + const managed = fs.readFileSync( + path.join(repoRoot, 'packages/codev/src/lib/managed-block.ts'), + 'utf-8', + ); + expect(managed).toContain('@codev/resources/arch-critical.md'); // @import, not inlined + const src = fs.readFileSync(script, 'utf-8'); + expect(src).toMatch(/SHARED=\$\(\(\s*CLAUDE_MD \+ HOT\s*\)\)/); + // The stale claim must not survive as an ASSERTION about current behaviour. + // It may appear in the header, where it is quoted as a defect being corrected. + const header = src.slice(0, src.indexOf('set -euo pipefail')); + const body = src.slice(src.indexOf('set -euo pipefail')); + expect(body).not.toMatch(/already inlines/); + expect(header).toMatch(/TRANSCLUDES|@import/); // documents the real mechanism + }); +}); + +describe('T1b — per-file four-tier resolution, not directory-level selection', () => { + it('resolves each file at its own winning tier', () => { + const dir = makeFixture(); + // Override ONE prompt in .codev/ while its siblings stay in the skeleton. + fs.mkdirSync(path.join(dir, '.codev/protocols/spir/prompts'), { recursive: true }); + fs.writeFileSync( + path.join(dir, '.codev/protocols/spir/prompts/specify.md'), + 'aa bb cc dd ee ff gg hh\n', // 8 words vs 5 + ); + const out = run(dir); + // Directory-level selection would have missed the override entirely. + expect(out).toMatch(/\| spir \| \d+ \| 8 \| \d+ \|/); + }); + + it('prefers .codev/ over codev/ over codev-skeleton/', () => { + const src = fs.readFileSync(script, 'utf-8'); + const order = src.slice(src.indexOf('resolve() {'), src.indexOf('# SERVED words')); + expect(order.indexOf('.codev/')).toBeLessThan(order.indexOf('"codev/$1"')); + expect(order.indexOf('"codev/$1"')).toBeLessThan(order.indexOf('codev-skeleton/')); + }); +}); + +describe('T2 — phantom-savings proof: includes are expanded', () => { + it('moving text from a prompt into an included template changes nothing', () => { + const dir = makeFixture(); + const before = num(run(dir), 'ALWAYS_ON_WORDS'); + + // Same served content, different authored ownership. + fs.writeFileSync( + path.join(dir, 'codev-skeleton/protocols/spir/prompts/specify.md'), + 'aa bb {{> protocols/spir/templates/frag.md}}\n', + ); + fs.mkdirSync(path.join(dir, 'codev-skeleton/protocols/spir/templates'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'codev-skeleton/protocols/spir/templates/frag.md'), 'cc dd ee\n'); + + expect(num(run(dir), 'ALWAYS_ON_WORDS')).toBe(before); + }); + + it('expands non-markdown includes too — protocol.json delivery depends on it (P6)', () => { + const dir = makeFixture(); + const before = num(run(dir), 'ALWAYS_ON_WORDS'); + fs.writeFileSync( + path.join(dir, 'codev-skeleton/protocols/spir/protocol.md'), + 'protocol words here now\n```json\n{{> protocols/spir/protocol.json}}\n```\n', + ); + fs.writeFileSync( + path.join(dir, 'codev-skeleton/protocols/spir/protocol.json'), + '{ "a": 1, "b": 2 }\n', + ); + // The JSON's words must appear in the served count, not vanish. + expect(num(run(dir), 'ALWAYS_ON_WORDS')).toBeGreaterThan(before); + }); +}); + +describe('T3 — per-surface reporting completeness (not a ceiling)', () => { + it('reports every protocol found on disk, in either tree', () => { + const out = run(); + const onDisk = new Set(); + for (const tree of ['codev/protocols', 'codev-skeleton/protocols']) { + const p = path.join(repoRoot, tree); + if (!fs.existsSync(p)) continue; + for (const e of fs.readdirSync(p, { withFileTypes: true })) { + if (e.isDirectory()) onDisk.add(e.name); + } + } + expect(onDisk.size).toBeGreaterThan(0); + for (const name of onDisk) { + expect(out, `protocol "${name}" missing from the report`).toMatch( + new RegExp(`^\\| ${name} \\|`, 'm'), + ); + } + }); + + it('includes codev-only protocols with no skeleton twin (release)', () => { + expect(run()).toMatch(/^\| release \|/m); + }); +}); + +describe('T11 — buckets vs audience loads are reported on different bases', () => { + it('states the formulas and warns the audience loads overlap', () => { + const out = run(); + expect(out).toContain('ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p]'); + expect(out).toMatch(/OVERLAP by design/); + expect(out).toMatch(/these SUM/); + }); + + it('one bucket growing while another shrinks shows BOTH movements, not a netted zero', () => { + const dir = makeFixture(); + const before = run(dir); + const sharedBefore = Number(before.match(/\| SHARED [^|]*\| (\d+) \|/)![1]); + const archBefore = Number(before.match(/\| ARCHITECT [^|]*\| (\d+) \|/)![1]); + + fs.writeFileSync(path.join(dir, 'CLAUDE.md'), 'alpha\n'); // shrink + fs.writeFileSync(path.join(dir, 'codev/roles/architect.md'), 'a b c d e f g\n'); // grow + + const after = run(dir); + const sharedAfter = Number(after.match(/\| SHARED [^|]*\| (\d+) \|/)![1]); + const archAfter = Number(after.match(/\| ARCHITECT [^|]*\| (\d+) \|/)![1]); + + expect(sharedAfter).toBeLessThan(sharedBefore); + expect(archAfter).toBeGreaterThan(archBefore); + }); +}); + +describe('T15 — relocation is visible, never reported as deletion (M0c)', () => { + it('moving a block into a skill drops always-on but holds total-authored steady', () => { + const dir = makeFixture(); + fs.writeFileSync(path.join(dir, 'CLAUDE.md'), 'alpha bravo charlie delta echo foxtrot\n'); + const before = run(dir); + + // Relocate three words out of CLAUDE.md into a skill — the P3/P4 move. + fs.writeFileSync(path.join(dir, 'CLAUDE.md'), 'alpha bravo charlie\n'); + fs.mkdirSync(path.join(dir, '.claude/skills/afx'), { recursive: true }); + fs.writeFileSync(path.join(dir, '.claude/skills/afx/SKILL.md'), 'delta echo foxtrot\n'); + + const after = run(dir); + expect(num(after, 'ALWAYS_ON_WORDS')).toBeLessThan(num(before, 'ALWAYS_ON_WORDS')); + expect(num(after, 'TOTAL_AUTHORED_WORDS')).toBe(num(before, 'TOTAL_AUTHORED_WORDS')); + }); + + it('counts all four skill trees — one-tree counting would report relocation as deletion', () => { + const src = fs.readFileSync(script, 'utf-8'); + for (const tree of [ + '.claude/skills', + '.codex/skills', + 'codev-skeleton/.claude/skills', + 'codev-skeleton/.codex/skills', + ]) { + expect(src, `total-authored basis must include ${tree}`).toContain(tree); + } + }); +}); + +describe('T12 — determinism', () => { + it('emits byte-identical output twice at the same commit', () => { + expect(run()).toBe(run()); + }); +}); + +describe('the corrected baseline is what the spec claims', () => { + let out: string; + beforeAll(() => { out = run(); }); + + it('reproduces ALWAYS_ON_WORDS = 34,255 for a SPIR builder at I=10', () => { + // 34,235 — not the 34,255 quoted in the spec. The spec's figure came from the + // 1252 additive include model, which counted the `{{> path}}` directive's own + // tokens AND the content substituted for them (~2 words per include, x10 + // iterations = 20). This instrument substitutes, so the figure is 20 lower and + // more honest. Size is reporting-only under the amended charter, so no + // criterion moves; the delta is recorded in 1280-word-baseline.md. + expect(num(out, 'ALWAYS_ON_WORDS')).toBe(34235); + }); + + it('reproduces the architect load (8,599)', () => { + expect(out).toMatch(/\| Architect \(per session\) \| 8599 \|/); + }); + + it('honours PHASE_ITERS as a comparison constant', () => { + const one = num(run(repoRoot, { PHASE_ITERS: '1' }), 'ALWAYS_ON_WORDS'); + const two = num(run(repoRoot, { PHASE_ITERS: '2' }), 'ALWAYS_ON_WORDS'); + expect(two - one).toBe(736 + 1396); // HOT + spir phase mean (substituted) + }); +}); diff --git a/packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts b/packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts new file mode 100644 index 000000000..c6b6d8148 --- /dev/null +++ b/packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts @@ -0,0 +1,119 @@ +/** + * Spec 1280 — T16: phase-manifest completeness (the mechanical guard on M11). + * + * M11 requires the architect to inspect the old-vs-new diff of EVERY changed + * file, per phase, in batches of <=12. That review is the project's acceptance + * mechanism — but a human can only inspect what they are shown. A file changed + * in a phase and omitted from that phase's manifest is invisible to the review, + * so it fails the phase. + * + * This test is written in Phase 0, before the first manifest exists, precisely + * because the guard must predate the thing it guards. + */ +import { describe, it, expect } from 'vitest'; +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const repoRoot = path.resolve(import.meta.dirname, '../../../..'); +const manifestDir = path.join( + repoRoot, + 'codev/projects/1280-prompt-surface-judgment-not-ru/manifests', +); + +/** Files a manifest is responsible for listing: prompt-bearing surfaces only. */ +const PROMPT_BEARING = /^(CLAUDE\.md|AGENTS\.md|codev(-skeleton)?\/(protocols|roles)\/.*\.md)$/; + +interface Manifest { + file: string; + phase: string; + rows: { path: string; oldWords: string; newWords: string; principles: string }[]; +} + +function parseManifest(file: string): Manifest { + const body = fs.readFileSync(file, 'utf-8'); + const rows: Manifest['rows'] = []; + for (const line of body.split('\n')) { + // | path | old | new | principles | rationale | + const m = line.match(/^\|\s*`?([^`|]+?)`?\s*\|\s*(\d+)\s*\|\s*(\d+)\s*\|\s*([^|]*)\|/); + if (m && !/^-+$/.test(m[1].trim())) { + rows.push({ + path: m[1].trim(), + oldWords: m[2], + newWords: m[3], + principles: m[4].trim(), + }); + } + } + return { file, phase: path.basename(file, '.md'), rows }; +} + +function manifests(): Manifest[] { + if (!fs.existsSync(manifestDir)) return []; + return fs + .readdirSync(manifestDir) + .filter((f) => f.endsWith('.md')) + .map((f) => parseManifest(path.join(manifestDir, f))); +} + +describe('T16 — manifest completeness (M11)', () => { + it('the manifest directory is a known location, created when the first phase lands', () => { + // Phase 0 changes no prompt-bearing file, so an absent directory is valid here. + // From Phase 1 onward the per-phase check below does the real work. + expect(path.isAbsolute(manifestDir)).toBe(true); + }); + + it('every manifest row carries all four required fields', () => { + for (const m of manifests()) { + expect(m.rows.length, `${m.phase} has no rows`).toBeGreaterThan(0); + for (const r of m.rows) { + expect(r.path, `${m.phase}: empty path`).not.toBe(''); + expect(r.oldWords, `${m.phase}/${r.path}: old word count`).toMatch(/^\d+$/); + expect(r.newWords, `${m.phase}/${r.path}: new word count`).toMatch(/^\d+$/); + expect( + r.principles, + `${m.phase}/${r.path}: must name the principles applied (P1..P7 or "none")`, + ).toMatch(/P[1-7]|none/i); + } + } + }); + + it('no manifest declares a batch larger than 12 files', () => { + for (const m of manifests()) { + const body = fs.readFileSync(m.file, 'utf-8'); + // A manifest may declare several batches; each is capped independently. + const batches = body.split(/^##+\s+Batch\s/im).slice(1); + if (batches.length === 0) { + expect(m.rows.length, `${m.phase}: single-batch manifest over the cap`).toBeLessThanOrEqual(12); + } else { + for (const [i, b] of batches.entries()) { + const n = (b.match(/^\|\s*`?[^`|]+?`?\s*\|\s*\d+\s*\|/gm) || []).length; + expect(n, `${m.phase} batch ${i + 1} over the cap`).toBeLessThanOrEqual(12); + } + } + } + }); + + it('every prompt-bearing file changed on this branch appears in some manifest', () => { + let changed: string[]; + try { + changed = execFileSync('git', ['diff', '--name-only', 'origin/main...HEAD'], { + cwd: repoRoot, + encoding: 'utf-8', + }) + .split('\n') + .map((s) => s.trim()) + .filter((s) => PROMPT_BEARING.test(s)); + } catch { + return; // no origin/main to diff against (fresh clone / CI shallow) — skip + } + if (changed.length === 0) return; + + const listed = new Set(manifests().flatMap((m) => m.rows.map((r) => r.path))); + const missing = changed.filter((f) => !listed.has(f)); + expect( + missing, + `changed but absent from every manifest — the architect cannot inspect what is not listed:\n${missing.join('\n')}`, + ).toEqual([]); + }); +}); diff --git a/scripts/extract-capability-inventory.sh b/scripts/extract-capability-inventory.sh new file mode 100755 index 000000000..de1ff7089 --- /dev/null +++ b/scripts/extract-capability-inventory.sh @@ -0,0 +1,156 @@ +#!/usr/bin/env bash +# +# extract-capability-inventory.sh — Spec 1280, criterion M5. +# +# Extracts every CAPABILITY the prompt surface communicates to an agent, so a +# rewrite can be proven not to have silently dropped one. +# +# THE INVENTORY IS OVER SERVED PROMPT TEXT, NOT OVER protocol.json. +# ------------------------------------------------------------------ +# This distinction is the whole point (CMAP round 2 caught the earlier design +# doing the opposite). Extracting gate/check names from `protocol.json`, or +# notification names from `afx send` call sites, would report every capability +# as present even if every corresponding INSTRUCTION vanished from the prompts — +# because this project does not touch those files. `protocol.json` and the +# source tree supply the EXPECTED SET; the served prompt text is where each +# item must be found. +# +# REPRESENTATION (M5, reconciled with principle P6) +# ------------------------------------------------- +# P6 permits replacing narrated names with a reference to structured truth. So a +# capability counts as represented if EITHER: +# (a) it is named in served prompt text, OR +# (b) the served text carries a resolvable `{{> ...}}` include of the source +# that defines it (the include is expanded here before matching, so (b) +# reduces to (a) automatically). +# A naive "the name must appear in prose" rule would fail a CONFORMANT rewrite. +# +# DETECTION LIMIT — stated, not implied +# ------------------------------------- +# This detects DELETION of a capability. It does NOT detect INVERSION or +# GUTTING of the instruction attached to one: "a gate message is a notification +# to the human, not authorization" can collapse to a bare mention of the gate +# name and still pass. That gap is covered by M11 (architect reads every diff) +# and O4 (the A/B compliance checklist), not by this script. +# +# Usage: scripts/extract-capability-inventory.sh [repo-root] > inventory.json +set -euo pipefail + +ROOT="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +cd "$ROOT" + +resolve() { + if [ -f ".codev/$1" ]; then echo ".codev/$1" + elif [ -f "codev/$1" ]; then echo "codev/$1" + elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" + else echo /dev/null; fi +} + +# Served text of one file with {{> ...}} includes expanded (two levels). +expand() { + local f="$1"; [ -f "$f" ] || return 0 + local body; body="$(cat "$f")" + local inc ip + for inc in $(grep -o '{{> *[^} ]*' "$f" 2>/dev/null | sed 's/{{> *//'); do + ip="$(resolve "$inc")" + [ -f "$ip" ] || continue + body="$body +$(cat "$ip")" + local inc2 ip2 + for inc2 in $(grep -o '{{> *[^} ]*' "$ip" 2>/dev/null | sed 's/{{> *//'); do + ip2="$(resolve "$inc2")" + [ -f "$ip2" ] && body="$body +$(cat "$ip2")" + done + done + printf '%s\n' "$body" +} + +# ---- the served prompt surface: every prompt-bearing file, both trees ------- +served_corpus() { + local f + for f in CLAUDE.md AGENTS.md; do [ -f "$f" ] && expand "$f"; done + for f in $(find codev/roles codev-skeleton/roles -name '*.md' 2>/dev/null); do expand "$f"; done + for f in $(find codev/protocols codev-skeleton/protocols -name '*.md' 2>/dev/null); do expand "$f"; done +} + +CORPUS="$(served_corpus)" + +# ---- expected sets (from structured/source truth) -------------------------- +GATES=$(python3 - <<'PY' +import json,glob +g=set() +for f in glob.glob('codev*/protocols/*/protocol.json'): + try: d=json.load(open(f)) + except Exception: continue + for p in d.get('phases',[]): + v=p.get('gate') + if isinstance(v,str): g.add(v) + elif isinstance(v,dict) and isinstance(v.get('name'),str): g.add(v['name']) +print('\n'.join(sorted(g))) +PY +) +CHECKS=$(python3 - <<'PY' +import json,glob +c=set() +for f in glob.glob('codev*/protocols/*/protocol.json'): + try: d=json.load(open(f)) + except Exception: continue + for p in d.get('phases',[]): + c |= set((p.get('checks') or {}).keys()) +print('\n'.join(sorted(c))) +PY +) + +# Normalization: lowercase, strip backticks/quotes/punctuation noise. +norm() { tr '[:upper:]' '[:lower:]' | tr -d '`"'"'"'*'; } +CORPUS_N="$(printf '%s' "$CORPUS" | norm)" + +# NOTE: a here-string, deliberately NOT `printf ... | grep -q`. +# Under `set -o pipefail`, `grep -q` exits on its FIRST match and closes the +# pipe; the upstream printf then dies of SIGPIPE and the pipeline reports +# failure — so every capability that WAS found reported as absent. The exit +# status had two causes and the code read only one of them. (Same failure class +# this project has been recording: trust the authoritative signal, not the +# convenient one.) +present() { + local needle; needle="$(printf '%s' "$1" | norm)" + if grep -qF -- "$needle" <<<"$CORPUS_N"; then echo true; else echo false; fi +} + +# ---- signals, artifact paths, notification triggers: found IN the corpus ---- +SIGNALS=$(printf '%s' "$CORPUS" | grep -o ']*>[A-Z_]*' | grep -o '[A-Z_]\{4,\}' | sort -u || true) +ARTIFACTS=$(printf '%s' "$CORPUS" | grep -oE 'codev/(specs|plans|reviews|state)/[A-Za-z0-9{}_.*-]+' | sort -u || true) +NOTIFY=$(printf '%s' "$CORPUS" | grep -oE 'afx send [a-z:<>{}-]+' | sort -u || true) + +json_array() { # each stdin line -> {"name":..,"present":..} + local first=1 + while IFS= read -r item; do + [ -z "$item" ] && continue + [ $first -eq 1 ] || printf ',\n' + first=0 + printf ' {"name": %s, "present_in_served_prompts": %s}' \ + "$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$item")" "$(present "$item")" + done + [ $first -eq 1 ] || printf '\n' +} + +cat </dev/null || echo n/a)", + "gates": [ +$(printf '%s\n' "$GATES" | json_array) ], + "checks": [ +$(printf '%s\n' "$CHECKS" | json_array) ], + "signals": [ +$(printf '%s\n' "$SIGNALS" | json_array) ], + "artifact_paths": [ +$(printf '%s\n' "$ARTIFACTS" | json_array) ], + "notification_triggers": [ +$(printf '%s\n' "$NOTIFY" | json_array) ] +} +EOF diff --git a/scripts/measure-prompt-surface.sh b/scripts/measure-prompt-surface.sh index 5dda6c769..9963c02f2 100755 --- a/scripts/measure-prompt-surface.sh +++ b/scripts/measure-prompt-surface.sh @@ -1,14 +1,40 @@ #!/usr/bin/env bash # # measure-prompt-surface.sh — word-count measurement of Codev's prompt surface. -# Spec 1252, criterion M6 / test T1. +# Originally Spec 1252 (M6/T1); CORRECTED under Spec 1280 (M0). # -# Run before any trimming (Phase 1) and again after (Phase 7); the delta is the -# N1 figure. Emits a markdown table on stdout. +# WHAT THIS MEASURES, AND WHY YOU SHOULD DISTRUST THE PREVIOUS VERSION +# ------------------------------------------------------------------- +# Spec 1280 found three defects in the 1252 version. All three produced numbers +# that were precise and wrong: # -# The number that matters is ALWAYS-ON: content that enters an agent's context -# whether or not it is needed. Total authored surface is much larger (~150k -# words) but most of it is on-demand and already correctly tiered. +# 1. It derived the phase-task term from `codev-skeleton/porch/prompts/` — a +# dead Ralph-SPIR-era tree with NO runtime consumer. The live resolver +# (commands/porch/prompts.ts, loadPromptFile) loads +# `protocols//prompts/.md`. Real SPIR phase prompts average +# 1,398 words; the dead tree averaged 400. The metric was structurally +# blind to the largest always-on surface in the system. +# 2. It omitted `roles/builder.md` (1,837 words), which spawn-worktree.ts:854 +# writes to `.builder-role.md` for harness injection on EVERY builder spawn. +# 3. Its comment asserted CLAUDE.md "already inlines" the hot tier. Since #1119 +# (lib/managed-block.ts) CLAUDE.md carries `@import` lines, which Claude Code +# TRANSCLUDES at session launch. So `wc -w CLAUDE.md` EXCLUDES 736 words that +# are always loaded. +# +# Reported baseline went 21,702 -> 34,255 on correction. Spec 1280's principle: +# the instrument is part of the deliverable, and instruments get reviewed against +# what they claim to measure — not merely against whether they run. +# +# TWO REPORTING BASES — deliberately different, do not conflate +# ------------------------------------------------------------ +# ALWAYS-ON: what rides into an agent's context whether needed or not. +# Dedupes twins (AGENTS.md is CLAUDE.md's byte-identical twin; +# one loads per session) and EXPANDS @import / {{> ...}}. +# TOTAL AUTHORED: physical files on disk, each counted once, NO twin dedup and +# NO expansion. Its job is to detect RELOCATION — content moved +# out of an always-on file must still show up somewhere. Under +# an always-on-only metric, moving 3,900 words into a skill +# scores identically to deleting them (Spec 1280 M0c). # # Usage: scripts/measure-prompt-surface.sh [repo-root] set -euo pipefail @@ -26,115 +52,198 @@ fdir() { # count of .md files under a dir [ -d "$1" ] && find "$1" -name '*.md' | wc -l | tr -d ' ' || echo 0 } +# --- Four-tier resolution, PER FILE (Spec 1280 M0 item b) -------------------- +# Mirrors lib/skeleton.ts resolveCodevFile. The 1252 version did TWO-tier, +# DIRECTORY-level selection, so a project overriding a single prompt in .codev/ +# while the rest resolved from the skeleton was measured at the wrong tier. +# Tier 3 (runtime cache) is not present in a checkout and is skipped here; it is +# a cache of tier 4, so it cannot change the resolved CONTENT for measurement. +resolve() { + if [ -f ".codev/$1" ]; then echo ".codev/$1" + elif [ -f "codev/$1" ]; then echo "codev/$1" + elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" + else echo /dev/null; fi +} + +# SERVED words: {{> ...}} includes are expanded into the prompt, so the proxy +# must count them or dedup-by-include would claim phantom savings (it changes +# authored ownership, not served bytes). Extension-agnostic: the runtime +# resolver inlines any file type, including protocol.json (Spec 1280 P6). +# +# This performs REAL TEXTUAL SUBSTITUTION rather than adding the include's word +# count to the host file's. The 1252 version added, which double-counted the +# directive's own tokens (`{{>` and the path are 2 words to `wc -w`) on top of +# the content that replaces them — so every include over-reported by ~2 words +# and moving text into a template was not exactly neutral. Substituting makes +# the phantom-savings property exact, which is what T2 asserts. +# Mirrors lib/skeleton.ts resolveCodevIncludes EXACTLY: a regex replace of the +# directive *within* the surrounding text (not a line-wise swap — text sharing a +# line with an include must survive), recursive, depth-guarded at 5, with an +# unresolved include collapsing to empty. +expand_text() { + python3 - "$1" <<'PY' +import re, sys, os +DIRECTIVE = re.compile(r'\{\{>\s*([^}\s]+)\s*\}\}') + +def resolve(rel): + for base in ('.codev', 'codev', 'codev-skeleton'): + p = os.path.join(base, rel) + if os.path.isfile(p): + return p + return None + +def expand(text, depth=0): + if depth > 5: + return text + def sub(m): + p = resolve(m.group(1)) + if not p: + return '' + with open(p, encoding='utf-8', errors='replace') as fh: + return expand(fh.read(), depth + 1) + return DIRECTIVE.sub(sub, text) + +path = sys.argv[1] +if os.path.isfile(path): + with open(path, encoding='utf-8', errors='replace') as fh: + sys.stdout.write(expand(fh.read())) +PY +} +expanded_w() { [ -f "$1" ] && expand_text "$1" | wc -w | tr -d ' ' || echo 0; } + CLAUDE_MD=$(w CLAUDE.md) AGENTS_MD=$(w AGENTS.md) ARCH_CRIT=$(w codev/resources/arch-critical.md) LESS_CRIT=$(w codev/resources/lessons-critical.md) ARCH=$(w codev/resources/arch.md) LESS=$(w codev/resources/lessons-learned.md) +HOT=$(( ARCH_CRIT + LESS_CRIT )) -SKEL_W=$(wdir codev-skeleton); SKEL_F=$(fdir codev-skeleton) -SKEL_PROTO_W=$(wdir codev-skeleton/protocols) -SKEL_ROLES_W=$(wdir codev-skeleton/roles) -SHADOW_PROTO_W=$(wdir codev/protocols); SHADOW_PROTO_F=$(fdir codev/protocols) -SHADOW_ROLES_W=$(wdir codev/roles); SHADOW_ROLES_F=$(fdir codev/roles) -SKILLS_W=$(wdir .claude/skills); SKILLS_F=$(fdir .claude/skills) - -# Always-on = what rides into context regardless of need. -# CLAUDE.md already inlines the two hot-tier files, so they are NOT re-added -# here (double-counting them would inflate the baseline and flatter the result). -# AGENTS.md is the byte-identical twin for other tools — one or the other -# loads per session, never both, so it is excluded too. -# -# SPAWN_PROMPT and PHASE_TASK are DERIVED from the artifacts that compose them, -# resolved the way the runtime resolves (tier-2 codev/ first, then the shipped -# skeleton), so Phase-7 trims to any component show up in the rerun. Hardcoding -# the one-off measured values (4891 / 1395) broke reproducibility — a rerun -# after trimming would have reported the pre-trim numbers (caught by Codex at -# the Phase-1 review). -# -# SPAWN_PROMPT proxy = spir builder-prompt.md + spir protocol.md (inlined into -# every spawn prompt). The real spawn prompt adds the issue body (~170 words -# here), which varies per project and is not a trimmable prompt surface, so -# it is deliberately excluded from the proxy. -# -# PHASE_TASK proxy = hot tier (injected into every porch phase prompt) + the -# mean of the porch phase prompts. Task-JSON boilerplate varies per phase and -# is porch code, not prompt surface. -resolve() { # two-tier resolve: codev/ wins, else skeleton - if [ -f "codev/$1" ]; then echo "codev/$1" - elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" - else echo /dev/null; fi +# --- SHARED bucket ----------------------------------------------------------- +# CLAUDE.md @imports the hot tier (#1119) rather than inlining it, so the hot +# words are ADDED here. AGENTS.md is excluded: byte-identical twin, one loads. +SHARED=$(( CLAUDE_MD + HOT )) + +# --- ARCHITECT bucket -------------------------------------------------------- +ARCHITECT_ROLE=$(w "$(resolve roles/architect.md)") +ARCHITECT=$ARCHITECT_ROLE + +# --- Per-protocol buckets ---------------------------------------------------- +BUILDER_ROLE=$(w "$(resolve roles/builder.md)") # inlined at spawn (defect 2) +CONSULTANT_ROLE=$(w "$(resolve roles/consultant.md)") + +protocols() { # every protocol in EITHER tree, unioned, deduped + { ls -d codev/protocols/*/ 2>/dev/null; ls -d codev-skeleton/protocols/*/ 2>/dev/null; } \ + | xargs -n1 basename 2>/dev/null | sort -u } -PHASE_ITERS="${PHASE_ITERS:-10}" -# SERVED words, not authored: {{> partials/...}} includes are expanded into -# every spawn prompt, so the proxy must count them or dedup-by-include would -# claim phantom always-on savings (it changes authored ownership, not served -# bytes). -expanded_w() { # file + partial includes, two levels deep (partials may nest) - local f="$1"; local total; total=$(w "$f") - for inc in $(grep -o '{{> *[^} ]*' "$f" 2>/dev/null | sed 's/{{> *//'); do - local ip; ip="$(resolve "$inc")" - total=$(( total + $(w "$ip") )) - for inc2 in $(grep -o '{{> *[^} ]*' "$ip" 2>/dev/null | sed 's/{{> *//'); do - total=$(( total + $(w "$(resolve "$inc2")") )) - done + +spawn_words() { # BUILDER_SPAWN[p] = builder role + wrapper + protocol.md + local p="$1" + echo $(( BUILDER_ROLE \ + + $(expanded_w "$(resolve "protocols/$p/builder-prompt.md")") \ + + $(expanded_w "$(resolve "protocols/$p/protocol.md")") )) +} + +phase_mean() { # mean expanded phase prompt for protocol p, 0 if it has none + local p="$1" total=0 n=0 f + for f in $(ls "codev/protocols/$p/prompts/"*.md "codev-skeleton/protocols/$p/prompts/"*.md 2>/dev/null \ + | xargs -n1 basename 2>/dev/null | sort -u); do + total=$(( total + $(expanded_w "$(resolve "protocols/$p/prompts/$f")") )); n=$(( n + 1 )) + done + [ "$n" -gt 0 ] && echo $(( total / n )) || echo 0 +} + +consult_mean() { # mean consult-type prompt for protocol p, 0 if it has none + local p="$1" total=0 n=0 f + for f in $(ls "codev/protocols/$p/consult-types/"*.md "codev-skeleton/protocols/$p/consult-types/"*.md 2>/dev/null \ + | xargs -n1 basename 2>/dev/null | sort -u); do + total=$(( total + $(w "$(resolve "protocols/$p/consult-types/$f")") )); n=$(( n + 1 )) done - echo "$total" + [ "$n" -gt 0 ] && echo $(( total / n )) || echo 0 } -SPAWN_BP=$(expanded_w "$(resolve protocols/spir/builder-prompt.md)") -SPAWN_PROTO=$(w "$(resolve protocols/spir/protocol.md)") -SPAWN_PROMPT=$(( SPAWN_BP + SPAWN_PROTO )) -PORCH_PROMPT_MEAN=0 -PORCH_DIR="$( [ -d codev/porch/prompts ] && echo codev/porch/prompts || echo codev-skeleton/porch/prompts )" -if [ -d "$PORCH_DIR" ]; then - PORCH_N=$(find "$PORCH_DIR" -name '*.md' | wc -l | tr -d ' ') - if [ "$PORCH_N" -gt 0 ]; then - PORCH_TOTAL=0 - for pf in "$PORCH_DIR"/*.md; do PORCH_TOTAL=$(( PORCH_TOTAL + $(expanded_w "$pf") )); done - PORCH_PROMPT_MEAN=$(( PORCH_TOTAL / PORCH_N )) - fi -fi -PHASE_TASK=$(( ARCH_CRIT + LESS_CRIT + PORCH_PROMPT_MEAN )) -ALWAYS_ON=$(( CLAUDE_MD + SPAWN_PROMPT + PHASE_TASK * PHASE_ITERS )) + +# --- TOTAL AUTHORED (relocation detector, M0 item g) ------------------------- +# Physical files, no dedup, no expansion. Counts ALL FOUR skill trees: Spec 1280 +# found .claude/skills and .codex/skills are byte-identical and the skeleton +# ships its own copies of both. Counting only .claude/skills would report +# content relocated into a skill as DELETED — inverting M0c. +SKILLS_CLAUDE=$(wdir .claude/skills) +SKILLS_CODEX=$(wdir .codex/skills) +SKILLS_SKEL_CLAUDE=$(wdir codev-skeleton/.claude/skills) +SKILLS_SKEL_CODEX=$(wdir codev-skeleton/.codex/skills) +SKILLS_ALL=$(( SKILLS_CLAUDE + SKILLS_CODEX + SKILLS_SKEL_CLAUDE + SKILLS_SKEL_CODEX )) +SKILLS_F_ALL=$(( $(fdir .claude/skills) + $(fdir .codex/skills) \ + + $(fdir codev-skeleton/.claude/skills) + $(fdir codev-skeleton/.codex/skills) )) + +AUTH_PROTO_CODEV=$(wdir codev/protocols); AUTH_PROTO_SKEL=$(wdir codev-skeleton/protocols) +AUTH_ROLES_CODEV=$(wdir codev/roles); AUTH_ROLES_SKEL=$(wdir codev-skeleton/roles) +TOTAL_AUTHORED=$(( CLAUDE_MD + AGENTS_MD + AUTH_PROTO_CODEV + AUTH_PROTO_SKEL \ + + AUTH_ROLES_CODEV + AUTH_ROLES_SKEL + SKILLS_ALL )) + +# --- DEAD -------------------------------------------------------------------- +DEAD_W=$(wdir codev-skeleton/porch/prompts); DEAD_F=$(fdir codev-skeleton/porch/prompts) + +# --- Headline ---------------------------------------------------------------- +# ALWAYS_ON_WORDS == ALWAYS_ON(builder, spir, I). I is a COMPARISON CONSTANT, +# identical before and after; it is not a claim about any real project. +PHASE_ITERS="${PHASE_ITERS:-10}" +HEADLINE_PROTO="${HEADLINE_PROTO:-spir}" +SPAWN_SPIR=$(spawn_words "$HEADLINE_PROTO") +PHASE_MEAN_SPIR=$(phase_mean "$HEADLINE_PROTO") +ALWAYS_ON=$(( SHARED + SPAWN_SPIR + PHASE_ITERS * (HOT + PHASE_MEAN_SPIR) )) +ALWAYS_ON_ARCHITECT=$(( SHARED + ARCHITECT )) +ALWAYS_ON_CONSULTANT=$(( CONSULTANT_ROLE + $(consult_mean "$HEADLINE_PROTO") )) cat </dev/null || echo n/a)\` +Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. + +## Exclusive buckets (partition the authored surface; these SUM) + +| Bucket | Words | +|---|---:| +| SHARED (CLAUDE.md + transcluded hot tier) | $SHARED | +| ARCHITECT (roles/architect.md) | $ARCHITECT | +| DEAD (codev-skeleton/porch/prompts, $DEAD_F files) | $DEAD_W | + +## Per-protocol (resolved per file, four-tier) + +| Protocol | BUILDER_SPAWN | PHASE mean | CONSULT mean | +|---|---:|---:|---:| +EOF +for p in $(protocols); do + printf "| %s | %s | %s | %s |\n" "$p" "$(spawn_words "$p")" "$(phase_mean "$p")" "$(consult_mean "$p")" +done + +cat < Date: Fri, 31 Jul 2026 21:35:40 -0700 Subject: [PATCH 28/32] [Spec 1280][Phase: phase_0_instrument] Thread: Phase 0 built, two self-inflicted bugs recorded --- codev/state/spir-1280_thread.md | 39 +++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index c124b5d2b..2a25ca463 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -423,3 +423,42 @@ Phase count 10 → 11. Decisions still 67, max batch 11. **Notable**: Claude caught that my rollback mapping contradicted the spec I wrote — Phase 1 claimed G2/G6 while rewriting roles/builder.md (G3) and roles/consultant.md (G5). A G3 revert would have silently pulled Phase 1 work out and T10 would have rehearsed the wrong map. + +### plan-approval APPROVED → Phase 0 (2026-08-01) + +Waleed approved; ran `porch approve 1280 plan-approval` myself. Recorded the architect's +skills-drift ruling in the plan first: skills this project TOUCHES get four-tree parity (T17, +scoped to the touched set); pre-existing drift on untouched skills → separate architect-filed +issue, recorded-known-state, and must not fail T17. + +**Phase 0 (PR-1) built. Commit 9d8c2569. No prompt-surface file touched.** + +Corrected the three known defects; the reported baseline moves **21,702 → 34,235**. + +**A fourth inaccuracy found while writing the tests — mine, not 1252's.** Include expansion was +*additive*: it counted the `{{> path}}` directive's own tokens PLUS the content substituted for +them, over-reporting ~2 words per include. `expand_text` now does real substitution mirroring +`resolveCodevIncludes` (regex replace in place, recursive, depth-guarded, unresolved → empty). +That is why the figure is 34,235 and not the 34,255 the spec quotes — a 20-word delta across 10 +iterations. Size is reporting-only, so no criterion moves, but the spec's number is now +superseded and the baseline artifact says so explicitly. + +Two of my own bugs, both caught by testing rather than by reading: + +1. **`set -o pipefail` + `grep -q`** in the capability extractor reported **all 57 capabilities + as absent** — grep exits on first match, printf takes SIGPIPE, and the pipeline reports + failure *because the match succeeded*. An exit code with two causes, read as one: the exact + pattern this project has now logged six times. Fixed with a here-string. +2. **Line-wise include expansion dropped text sharing a line with a directive** — `aa bb {{> x}}` + lost `aa bb`. Only surfaced because T2 asserts exact neutrality. + +Frozen capability inventory: **57 capabilities, 47 present in served prompts, 10 absent +pre-existing** (porch delivers gates/checks via task JSON, not authored prompt text). That +asymmetry is baseline state, correctly captured — M5 compares post ⊇ pre, so nothing must be +invented. It also validates the instrument: an inventory reporting 100% present would have been +suspicious. + +Artifacts: `1280-word-baseline.md`, `1280-capability-inventory.json`, manifest format README, +`1252-word-*.md` annotated in place (originals preserved, marked superseded). Tests: 22 passing +(T1, T1b, T2, T3, T11, T12, T15, T16). T16 written before any manifest exists — the guard must +predate what it guards. From a248a7ed7711ef5131f35afea8a25d7bf0806d2f Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:38:15 -0700 Subject: [PATCH 29/32] [Spec 1280][Phase: phase_0_instrument] Thread: build is a suite precondition; skeleton is a build copy --- codev/state/spir-1280_thread.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index 2a25ca463..fce8ad0a2 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -462,3 +462,20 @@ Artifacts: `1280-word-baseline.md`, `1280-capability-inventory.json`, manifest f `1252-word-*.md` annotated in place (originals preserved, marked superseded). Tests: 22 passing (T1, T1b, T2, T3, T11, T12, T15, T16). T16 written before any manifest exists — the guard must predate what it guards. + +### Worktree environment note — `pnpm build` is a precondition for the suite + +58 test files / 116 tests failed on first full run. **Not my change.** Cause: +`Skeleton directory not found. Package may be corrupted.` from `getTemplatesDir` — this +worktree had never been built, and `pnpm build`'s `copy-skeleton` step is what creates +`packages/codev/skeleton` from `codev-skeleton/`. After building, the three representative +failing files (adopt, hot-tier-materialization, cold-tier-materialization) pass 19/19. + +**Carry-forward for every later phase**: `packages/codev/skeleton` is a BUILD COPY of +`codev-skeleton/`. When a phase rewrites skeleton files, the suite will keep testing the stale +copy until `pnpm build` re-runs. Any phase touching `codev-skeleton/` must rebuild before +claiming green, or it is asserting against pre-rewrite content — a phantom-green of exactly the +kind this project exists to stop being fooled by. + +I diagnosed this from an actual error message rather than the "probably needs a build" theory, +which is the same discipline the rest of the project has been applying to instruments. From 5c962b7a4e1d34dddaa6691e0ec02fa57403f33c Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:39:51 -0700 Subject: [PATCH 30/32] chore(porch): 1280 record PR #1319 --- .../1280-prompt-surface-judgment-not-ru/status.yaml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml index 5d59a25cb..ad3bbf00a 100644 --- a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -54,4 +54,9 @@ iteration: 1 build_complete: false history: [] started_at: '2026-08-01T02:46:36.457Z' -updated_at: '2026-08-01T04:26:12.253Z' +updated_at: '2026-08-01T04:39:51.924Z' +pr_history: + - phase: implement + pr_number: 1319 + branch: builder/1280-instrument + created_at: '2026-08-01T04:39:51.923Z' From e24c20ebb12a14a1eca0b6f9957899046b00619d Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:46:29 -0700 Subject: [PATCH 31/32] [Spec 1280][Phase: phase_0_instrument] fix: make the word count platform-independent CI caught what the local suite could not: PR #1319 measured 34,231 while my Mac measured 34,235. Same commit, same tree - verified by exporting the PR merge ref and measuring it locally, which reproduced 34,235. ROOT CAUSE: wc -w is not portable for this corpus. macOS/BSD wc in a UTF-8 locale counts the emoji U+26A0 WARNING SIGN + U+FE0F VARIATION SELECTOR-16 as TWO words; GNU wc on Linux, LC_ALL=C wc, and Python's str.split() all count ONE. spir/protocol.md carries four such banners, and 4 x 1 = the exact delta. Located by diffing per-line counts between BSD wc and Python rather than by guessing. WHY THIS IS A REAL DEFECT, NOT A TEST NUISANCE: this instrument exists to produce an honest before/after comparison. A count that depends on which machine ran it means measuring 'before' on a laptop and 'after' in CI yields a fictional delta - the phantom-savings failure the whole instrument was built to prevent, wearing a different hat. FIX: word counting is now DEFINED by the script (whitespace-delimited tokens of the UTF-8 decoded text, via Python str.split) rather than delegated to the host's wc. Verified identical across worktree, a clean export of the PR merge ref, and LC_ALL=C: all three now report 34,231. Two new regression tests: the script must not delegate to wc -w, and the total must be identical under LC_ALL=C and UTF-8. Baseline restated 34,235 -> 34,231 with both corrections documented in the artifact header (-20 additive-include model, -4 wc portability). Size is reporting-only under the charter amendment, so no acceptance criterion moves. Noted for the record: the characters that broke portability are the 'WARNING BLOCKING' worst-case-padding banners that principle P7 exists to delete. 24 tests passing. --- .../resources/1280-capability-inventory.json | 2 +- codev/resources/1280-word-baseline.md | 33 ++++++++------ .../spec-1280-measurement-instrument.test.ts | 43 +++++++++++++++---- scripts/measure-prompt-surface.sh | 28 ++++++++++-- 4 files changed, 80 insertions(+), 26 deletions(-) diff --git a/codev/resources/1280-capability-inventory.json b/codev/resources/1280-capability-inventory.json index c9a791271..fda837b80 100644 --- a/codev/resources/1280-capability-inventory.json +++ b/codev/resources/1280-capability-inventory.json @@ -3,7 +3,7 @@ "_criterion": "M5", "_basis": "served prompt text (includes expanded); protocol.json supplies the expected set only", "_detection_limit": "detects deletion, not inversion or gutting — see M11 and O4", - "_commit": "1056834d", + "_commit": "5c962b7a", "gates": [ {"name": "dev-approval", "present_in_served_prompts": true}, {"name": "experiment-complete", "present_in_served_prompts": false}, diff --git a/codev/resources/1280-word-baseline.md b/codev/resources/1280-word-baseline.md index 46d321a5a..64516453d 100644 --- a/codev/resources/1280-word-baseline.md +++ b/codev/resources/1280-word-baseline.md @@ -5,15 +5,22 @@ produced by an instrument with three defects (see the script header). Those files are annotated in place; their originals are preserved. - NOTE ON 34,235 vs the 34,255 quoted in spec 1280: the spec's figure came from the 1252 - ADDITIVE include model, which counted a `{{> path}}` directive's own tokens AND the content - substituted for them (~2 words per include, x10 iterations = 20). This instrument performs - real substitution, mirroring lib/skeleton.ts resolveCodevIncludes, so the figure is 20 lower - and more honest. Size is reporting-only under the amended charter (issue #1280 AMENDMENT - 2026-08-01), so no acceptance criterion moves. --> + NOTE ON 34,231 vs the 34,255 quoted in spec 1280 — two corrections, both making the + instrument more honest, neither moving an acceptance criterion (size is reporting-only + under the charter AMENDMENT of 2026-08-01): + + -20 The 1252 ADDITIVE include model counted a `{{> path}}` directive's own tokens AND + the content substituted for them. This instrument performs real substitution, + mirroring lib/skeleton.ts resolveCodevIncludes. + -4 `wc -w` is NOT portable for this corpus. macOS/BSD wc in a UTF-8 locale counts + `WARNING SIGN + VARIATION SELECTOR-16` as TWO words; GNU wc on Linux, `LC_ALL=C wc`, + and Python's str.split() count ONE. Four such banners in spir/protocol.md made the + same commit measure 34,235 on macOS and 34,231 in CI. Word counting is now DEFINED + (whitespace-delimited tokens of the UTF-8 decoded text) rather than delegated to the + host's wc. Caught by CI on PR #1319 — the local suite was green. --> # Prompt-surface measurement -Commit: `1056834d` +Commit: `5c962b7a` Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. ## Exclusive buckets (partition the authored surface; these SUM) @@ -37,7 +44,7 @@ Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. | release | 3463 | 0 | 0 | | research | 3671 | 0 | 0 | | spike | 3155 | 0 | 0 | -| spir | 6364 | 1396 | 430 | +| spir | 6360 | 1396 | 430 | ## Derived audience loads (these OVERLAP by design — never sum them) @@ -48,7 +55,7 @@ Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. | Audience | Words | |---|---:| -| **Builder (spir, I=10)** — the headline | **34235** | +| **Builder (spir, I=10)** — the headline | **34231** | | Architect (per session) | 8599 | | Consultant (per review, spir) | 682 | @@ -57,12 +64,12 @@ Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. | Component | Words | |---|---:| | CLAUDE.md + AGENTS.md (no twin dedup here) | 11630 | -| codev/protocols + codev-skeleton/protocols | 88475 | +| codev/protocols + codev-skeleton/protocols | 88461 | | codev/roles + codev-skeleton/roles | 8274 | | skills, ALL FOUR trees (42 files) | 44840 | -| **TOTAL_AUTHORED** | **153219** | +| **TOTAL_AUTHORED** | **153205** | Reference (on-demand, not always-on): arch.md 20367, lessons-learned.md 21270. -ALWAYS_ON_WORDS=34235 -TOTAL_AUTHORED_WORDS=153219 +ALWAYS_ON_WORDS=34231 +TOTAL_AUTHORED_WORDS=153205 diff --git a/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts b/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts index d1998a820..6b95c2de6 100644 --- a/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts +++ b/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts @@ -251,6 +251,29 @@ describe('T15 — relocation is visible, never reported as deletion (M0c)', () = }); }); +describe('portability — the count must not depend on the host', () => { + it('does not delegate word counting to `wc -w`', () => { + // BSD wc (macOS, UTF-8 locale) counts `⚠️` (U+26A0 U+FE0F) as two words; GNU wc + // and Python's str.split() count one. Four such banners in spir/protocol.md + // made the same commit measure 34,235 locally and 34,231 in CI. An instrument + // whose before/after must be comparable across machines cannot delegate its + // core definition to a platform-variant tool. + const code = fs + .readFileSync(script, 'utf-8') + .split('\n') + .filter((l) => !/^\s*#/.test(l)) + .join('\n'); + expect(code).not.toMatch(/wc -w/); + expect(code).toMatch(/_count\(\)/); + }); + + it('reports the same total under a C locale as under UTF-8', () => { + const utf8 = num(run(repoRoot, { LC_ALL: 'en_US.UTF-8' }), 'ALWAYS_ON_WORDS'); + const c = num(run(repoRoot, { LC_ALL: 'C' }), 'ALWAYS_ON_WORDS'); + expect(c).toBe(utf8); + }); +}); + describe('T12 — determinism', () => { it('emits byte-identical output twice at the same commit', () => { expect(run()).toBe(run()); @@ -261,14 +284,16 @@ describe('the corrected baseline is what the spec claims', () => { let out: string; beforeAll(() => { out = run(); }); - it('reproduces ALWAYS_ON_WORDS = 34,255 for a SPIR builder at I=10', () => { - // 34,235 — not the 34,255 quoted in the spec. The spec's figure came from the - // 1252 additive include model, which counted the `{{> path}}` directive's own - // tokens AND the content substituted for them (~2 words per include, x10 - // iterations = 20). This instrument substitutes, so the figure is 20 lower and - // more honest. Size is reporting-only under the amended charter, so no - // criterion moves; the delta is recorded in 1280-word-baseline.md. - expect(num(out, 'ALWAYS_ON_WORDS')).toBe(34235); + it('reproduces ALWAYS_ON_WORDS = 34,231 for a SPIR builder at I=10', () => { + // 34,231 — not the 34,255 quoted in the spec. Two corrections, both making the + // instrument more honest and neither moving an acceptance criterion (size is + // reporting-only under the amended charter): + // -20 the 1252 additive include model counted a `{{> path}}` directive's own + // tokens AND the content substituted for them; this one substitutes. + // -4 `wc -w` is not portable: BSD wc in a UTF-8 locale splits `⚠️` into two + // words where GNU wc and Python's split() see one. Counting is now + // defined explicitly rather than delegated to the platform's wc. + expect(num(out, 'ALWAYS_ON_WORDS')).toBe(34231); }); it('reproduces the architect load (8,599)', () => { @@ -278,6 +303,6 @@ describe('the corrected baseline is what the spec claims', () => { it('honours PHASE_ITERS as a comparison constant', () => { const one = num(run(repoRoot, { PHASE_ITERS: '1' }), 'ALWAYS_ON_WORDS'); const two = num(run(repoRoot, { PHASE_ITERS: '2' }), 'ALWAYS_ON_WORDS'); - expect(two - one).toBe(736 + 1396); // HOT + spir phase mean (substituted) + expect(two - one).toBe(736 + 1396); // HOT + spir phase mean }); }); diff --git a/scripts/measure-prompt-surface.sh b/scripts/measure-prompt-surface.sh index 9963c02f2..d96402a4c 100755 --- a/scripts/measure-prompt-surface.sh +++ b/scripts/measure-prompt-surface.sh @@ -42,11 +42,33 @@ set -euo pipefail ROOT="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" cd "$ROOT" +# WORD COUNT IS DEFINED HERE, NOT DELEGATED TO `wc -w`. +# +# `wc -w` is NOT portable for this corpus. macOS/BSD wc in a UTF-8 locale counts +# `⚠️` (U+26A0 WARNING SIGN + U+FE0F VARIATION SELECTOR-16) as TWO words; GNU wc +# on Linux, `LC_ALL=C wc`, and Python's str.split() all count it as one. There +# are four such banners in spir/protocol.md alone, so the same commit measured +# 34,235 on a developer's Mac and 34,231 in CI. +# +# For an instrument whose entire purpose is an honest before/after comparison, +# a platform-dependent count is a correctness defect: measure "before" on one +# machine and "after" on another and the delta is fiction. So the definition is +# made explicit and deterministic — a word is a whitespace-delimited token of +# the UTF-8 decoded text, per Python's str.split(). +# +# (The irony is recorded rather than smoothed over: the characters that broke +# portability are the `⚠️ BLOCKING` worst-case-padding banners that principle P7 +# exists to delete.) +_count() { # count words on stdin, deterministically + python3 -c 'import sys; print(len(sys.stdin.read().split()))' +} w() { # word count of a file, 0 if absent - [ -f "$1" ] && wc -w < "$1" | tr -d ' ' || echo 0 + [ -f "$1" ] || { echo 0; return; } + _count < "$1" } wdir() { # word count of all .md under a dir, 0 if absent - [ -d "$1" ] && find "$1" -name '*.md' -exec cat {} + 2>/dev/null | wc -w | tr -d ' ' || echo 0 + [ -d "$1" ] || { echo 0; return; } + find "$1" -name '*.md' -exec cat {} + 2>/dev/null | _count } fdir() { # count of .md files under a dir [ -d "$1" ] && find "$1" -name '*.md' | wc -l | tr -d ' ' || echo 0 @@ -109,7 +131,7 @@ if os.path.isfile(path): sys.stdout.write(expand(fh.read())) PY } -expanded_w() { [ -f "$1" ] && expand_text "$1" | wc -w | tr -d ' ' || echo 0; } +expanded_w() { [ -f "$1" ] && expand_text "$1" | _count || echo 0; } CLAUDE_MD=$(w CLAUDE.md) AGENTS_MD=$(w AGENTS.md) From fdf1650c7961ac511f1952e7eb351dbf673ead72 Mon Sep 17 00:00:00 2001 From: M Waleed Kadous Date: Fri, 31 Jul 2026 21:46:54 -0700 Subject: [PATCH 32/32] [Spec 1280][Phase: phase_0_instrument] Thread: CI-caught wc portability defect --- codev/state/spir-1280_thread.md | 39 +++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md index fce8ad0a2..44dad0141 100644 --- a/codev/state/spir-1280_thread.md +++ b/codev/state/spir-1280_thread.md @@ -479,3 +479,42 @@ kind this project exists to stop being fooled by. I diagnosed this from an actual error message rather than the "probably needs a build" theory, which is the same discipline the rest of the project has been applying to instruments. + +### CI caught a portability defect the local suite could not (2026-08-01) + +PR #1319 Unit Tests failed: **CI measured 34,231, my Mac measured 34,235.** Same commit. + +Diagnosed by narrowing rather than guessing: +1. Fetched `refs/pull/1319/merge`, diffed every measured file against my HEAD — **tracked + content identical**, so not a content delta. +2. Exported that merge ref to a clean tree and measured it locally — **still 34,235**, so not + the tree either. That left the platform. +3. Compared `wc -w` / `LC_ALL=C wc -w` / Python `split()` per file: `spir/protocol.md` gave + 3703 / 3699 / 3699. +4. Diffed per-LINE counts to find the exact characters. + +**Root cause: `⚠️` (U+26A0 WARNING SIGN + U+FE0F VARIATION SELECTOR-16).** macOS/BSD `wc -w` in +a UTF-8 locale splits it into two words; GNU `wc`, `LC_ALL=C wc`, and Python's `str.split()` +count one. Four such banners in `spir/protocol.md` — exactly the 4-word delta. + +**Why this mattered rather than being a test nuisance**: the instrument's entire purpose is an +honest before/after comparison. A count that depends on the host means measuring "before" on a +laptop and "after" in CI produces a fictional delta — the phantom-savings failure this whole +instrument exists to prevent, in a new disguise. It would have been invisible until the final +report, and then unfalsifiable. + +Fix: word counting is now **defined by the script** (whitespace-delimited tokens of UTF-8 +decoded text) rather than delegated to the host's `wc`. Worktree, clean merge-ref export, and +`LC_ALL=C` now all report **34,231**. Two regression tests added: no delegation to `wc -w`, and +identical totals under `LC_ALL=C` vs UTF-8. + +Baseline restated 34,255 → 34,231, with both corrections documented in the artifact header +(−20 additive-include model, −4 `wc` portability). Size is reporting-only, so no criterion moves. + +**The characters that broke portability are the `⚠️ BLOCKING` worst-case-padding banners that +principle P7 exists to delete.** Recorded rather than smoothed over. + +**Lesson for the ledger** (seventh instance of the family): *a green local suite is not a green +build.* The delegated tool — like the overloaded exit code, the truncated grep, the +skeleton-only enumeration, and the stale script comment before it — looked authoritative and +wasn't. CI was the authoritative signal here, and it existed all along.