diff --git a/codev/plans/1280-prompt-surface-judgment-not-ru.md b/codev/plans/1280-prompt-surface-judgment-not-ru.md new file mode 100644 index 000000000..29403dbbf --- /dev/null +++ b/codev/plans/1280-prompt-surface-judgment-not-ru.md @@ -0,0 +1,451 @@ +# Plan: Prompt surface — judgment-not-rules rewrite (principle conformance) + +## Metadata +- **ID**: plan-2026-07-31-prompt-surface-judgment-not-rules +- **Status**: draft (CMAP round 1 incorporated) +- **Specification**: [codev/specs/1280-prompt-surface-judgment-not-ru.md](../specs/1280-prompt-surface-judgment-not-ru.md) +- **Created**: 2026-07-31 +- **Issue**: #1280 (charter amended 2026-08-01: acceptance = principle conformance, size reporting-only) + +## Executive Summary + +Approach 1 from the spec: **in-place principle rewrite, surface by surface**, keeping file +layout, the four-tier resolver, and porch untouched. Approach 2 is rejected twice over — it +changes porch behaviour and defeats M11, since the architect cannot inspect old-vs-new diffs of +files that no longer exist as authored artifacts. + +**The binding constraint is M11, not the writing.** The architect inspects the old-vs-new diff of +every changed file, in batches of ≤12. Phase boundaries are drawn by *inspection load*. + +**Decision count: 67** (the spec says "~66"; **67 is the correct figure** and both CMAP reviewers +independently reproduced it — readers should not have to re-derive it): + +| Category | Count | Notes | +|---|---:|---| +| `protocol.md` | 10 | 9 skeleton + `release` (codev-local, no twin) | +| `builder-prompt.md` | 9 | `release` has none | +| `prompts/*.md` | 18 | spir 4 · aspir 4 · pir 3 · bugfix 3 · air 2 · maintain 2 | +| `templates/*.md` | 8 | spir 3 · experiment 1 · maintain 1 · spike 1 + 2 maintain codev-local | +| `consult-types/*.md` | 18 | spir 5 · aspir 5 · pir 2 · bugfix 2 · air 2 · maintain 2 | +| `roles/*.md` | 3 | architect · builder · consultant | +| `CLAUDE.md` + `AGENTS.md` | 1 | one decision, two byte-identical files | +| **Total** | **67** | | + +### Review batch — defined, because "one phase = one batch" was false + +A **review batch** is *every distinct file the architect reads in one sitting* — prompt +decisions **plus** the supporting test files, registry, retirements entries and manifest that +ride along. A phase may contain **more than one batch**; each batch is ≤12 and is reviewed +before the next begins. Phases below declare their batches explicitly. + +### Rollback groups are commit-pure, phases may span groups + +Phases are drawn by inspection load, so a phase can touch several rollback groups. The +invariant is at the **commit** level: **every commit belongs to exactly one group**, so any group +reverts cleanly regardless of which phase produced it. Each phase declares its groups; T10 +rehearses **every group the project touched**, not a sample. (CMAP round 1 correctly found the +first draft's group mapping was wrong — Phase 1 claimed G2/G6 while also rewriting +`roles/builder.md` (G3) and `roles/consultant.md` (G5).) + +**M6's dead-tree deletion is assigned to G4** (phase surfaces), which owns the `prompts/` axis. + +### Two mechanisms that must be settled before the work they gate + +**1. P6 delivery of `protocol.json` (gates Phase 3).** Verified: `protocol.md` is inlined at +spawn via `{{protocol_reference}}` (`spawn-roles.ts:112-124`), but **`protocol.json` is inlined +nowhere** — `spawn-roles.ts:267` only reads it for validation. In a fresh adopter project +`codev/protocols/

/protocol.json` does not exist on disk; it resolves from tier 4. So a prose +instruction "read `protocol.json`" is exactly the fetch-by-path of a framework file that +CLAUDE.md forbids and the spec's own constraint restates. + +**Decision: deliver it through the existing include resolver.** `resolveCodevIncludes` +(`skeleton.ts:108-119`) is **extension-agnostic** — verified — so `protocol.md` carries a fenced +```` ```json ```` block containing `{{> protocols/

/protocol.json}}`. This resolves through all +four tiers, works in fresh installs, requires **no porch change**, and is the literal expression +of P6 ("rich references"). Cost, stated honestly: it adds the JSON back as served words (spir +570, aspir 568, pir 375, others 77–282) — acceptable because size is reporting-only under the +amended charter, and it replaces narration with authoritative structured truth. +**Tested in both strict mode (porch-driven) and soft mode (builder reads `protocol.md` with no +porch)** — the asymmetry matters: strict-mode builders receive checks and gates as porch tasks, +soft-mode builders have only the prompt. + +**2. Skill relocation is a FOUR-tree sync (gates Phase 1).** Verified: skills exist in +`.claude/skills` (10), `.codex/skills` (10, **byte-identical** to `.claude`), +`codev-skeleton/.claude/skills` (7) and `codev-skeleton/.codex/skills` (7) — with **existing +drift** (`afx`, `porch` differ repo-vs-skeleton; `forge`, `skill-creator`, `team` are absent from +the skeleton). Consequences if unaddressed: content relocated out of CLAUDE.md into one copy +leaves **Codex agents without it**, leaves **adopters without it** after `codev update`, and is +reported as **deleted** by M0c/T15 — inverting the project's honesty artifact. Phase 0 widens +M0(g)'s basis to all four trees; Phase 1 adds **T17** (skills parity) and treats every relocation +as a four-copy write. + +## Success Metrics + +- [ ] **MP** — every file marked *rewritten* in the spec's disposition table conforms to + P1, P2, P3, P4, P6, P7 (P5 N/A with reason), judged per file by the architect +- [ ] **M11** — architect inspected every changed file, in ≤12-file batches, with a complete manifest +- [ ] **M0 / M0b / M0c** — corrected instrument landed early as PR-1; deleted vs relocated reported +- [ ] **M1 / M2** — before/after figures and per-file counts published (no threshold) +- [ ] **M2b** — CLAUDE.md human-readable, architect-confirmed +- [ ] **M3** — every surface enumerated from disk rewritten and inspected +- [ ] **M4** — eight scar canonicals byte-identical; count pinned at 8 +- [ ] **M5** — capability inventory over served prompt text; unlisted removals fail +- [ ] **M6** — dead tree deleted, Spec 987 test consumer handled +- [ ] **M7** — A/B SHIP verdict (verify phase) +- [ ] **M8** — behavioural baseline re-run (verify phase) +- [ ] **M9** — rollback rehearsed for every group touched +- [ ] **M10** — every retired prose-pinned assertion named with its originating spec +- [ ] **M12** — no release between merge and SHIP verdict (spans the verify phase) +- [ ] Suite green **at the end of every phase**, not only at the end + +## Phases (Machine Readable) + + + +```json +{ + "phases": [ + {"id": "phase_0_instrument", "title": "Corrected instrument + frozen capability inventory (PR-1, ships early)"}, + {"id": "phase_1_shared_skills", "title": "CLAUDE.md/AGENTS.md + four-tree skill relocation (G2)"}, + {"id": "phase_2_roles", "title": "Three role files (G6, G3, G5)"}, + {"id": "phase_3_protocol_md", "title": "protocol.md x10 with the P6 include mechanism (G3)"}, + {"id": "phase_4_builder_prompts", "title": "builder-prompt.md x9 + M10 test-retirement burden (G3)"}, + {"id": "phase_5_prompts_heavy", "title": "Phase prompts: spir, aspir, pir (G4)"}, + {"id": "phase_6_prompts_light_spir_templates", "title": "Phase prompts: bugfix, air, maintain + spir templates (G4)"}, + {"id": "phase_7_templates_consult_spir", "title": "Remaining templates + spir consult-types (G4, G5)"}, + {"id": "phase_8_consult_types_a", "title": "Consult-types: aspir, bugfix, air (G5)"}, + {"id": "phase_9_consult_registry_deadtree", "title": "Consult-types pir/maintain + scar registry + dead-tree deletion (G5, G7, G4)"}, + {"id": "phase_10_integration", "title": "Capability verification, measurement report, rollback rehearsal, governance docs"} + ] +} +``` + +## Phase Breakdown + +### Phase 0: Corrected instrument + frozen capability inventory (PR-1, ships early) +**Groups**: G1 · **Batches**: 1 (script + ~6 test files + 3 artifacts ≈ 10) +**Dependencies**: None. **Ships as its own PR before any prompt rewriting** (M0b). + +#### Deliverables +- [ ] `scripts/measure-prompt-surface.sh` corrected — all seven M0 items +- [ ] **M0(g) basis widened to all four skill trees** (`.claude/skills`, `.codex/skills`, and + both `codev-skeleton/` copies) — otherwise relocation reports as deletion +- [ ] First tests for the script: **T1, T1b, T2, T3, T11, T12, T15** +- [ ] **T16** (manifest completeness) implemented **now** — it is the mechanical guard on M11, + the project's binding constraint, and must exist before Phase 1 produces the first manifest +- [ ] **`codev/resources/1280-capability-inventory.json` extracted from the PRE-rewrite surface + and committed** — M5 requires a frozen pre-rewrite baseline, and Phase 3 onward asserts + against it +- [ ] `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline +- [ ] In-place annotation of `1252-word-baseline.md` / `1252-word-after-phase7.md` +- [ ] **Manifest format defined** at `codev/projects/1280-*/manifests/phase-N.md`: one row per + changed file — path · old words · new words · principles applied · one-line rationale + +#### PR-1 operational mechanics +1. Cut `builder/1280-instrument` from the current branch point +2. Open PR-1, architect review, merge +3. `git fetch origin main && git checkout -b builder/1280-rewrite origin/main` — **never** + `git checkout main` (a worktree cannot check out a branch checked out elsewhere) +4. Record: `porch done 1280 --pr --branch builder/1280-instrument`, then + `porch done 1280 --merged ` +5. Verify the rewrite branch contains **no duplicate Phase-0 commits** (`git log origin/main..HEAD`) + +#### Acceptance Criteria +- [ ] T1, T1b, T2, T3, T11, T12, T15, T16 pass; output deterministic +- [ ] Pre-rewrite capability inventory committed and non-empty +- [ ] Architect reviews the batch; PR-1 merged; suite green + +#### Rollback Strategy +Group **G1**. No prompt surface has changed. + +--- + +### Phase 1: CLAUDE.md/AGENTS.md + four-tree skill relocation (G2) +**Groups**: G2 · **Batches**: 1 (CLAUDE+AGENTS, ≤6 skill files across 4 trees, 2 test files ≈ 10) +**Dependencies**: Phase 0 merged. + +#### Deliverables +- [ ] `CLAUDE.md` + `AGENTS.md` rewritten (1 decision, 2 byte-identical files) +- [ ] Relocated how-to content written to **all four skill trees**, not one +- [ ] **T17 — skills parity, scoped by the plan-gate ruling (2026-08-01)**: every skill this + project **touches** must be four-tree consistent (`.claude/skills`, `.codex/skills`, and + both `codev-skeleton/` copies). Untouched skills are **exempt** — T17 asserts over the + touched set only, so pre-existing drift (`afx`, `porch`) and skeleton-absent skills + (`forge`, `skill-creator`, `team`) cannot fail it. Those go to the architect's separate + issue and are recorded here as known state +- [ ] All eight scar canonicals present byte-identically +- [ ] **M10 here, not Phase 4**: `spec-1273-wait-discipline-docs.test.ts:31` asserts on + `.claude/skills/afx/SKILL.md` + `.codex/skills/afx/SKILL.md` — relocation into `afx` breaks + it in **this** phase +- [ ] Manifest + architect review; suite green + +#### Implementation Details +- Governing principles: **P3**, **P4**, **P1**. +- **Scar verification here is against the ratified source, not T4.** T4 and the registry are + built in Phase 9, after the surface settles (Baked Decision 2). This phase verifies the eight + canonicals byte-for-byte against `builder/spir-1252:codev/resources/scar-rules.yaml` directly. + (CMAP round 1 caught the first draft asserting T4 in Phase 1 while creating it in Phase 8.) +- M2b: architect confirms CLAUDE.md is still navigable by a human. + +#### Acceptance Criteria +- [ ] Eight canonicals verified against the ratified YAML; T7, T17 pass +- [ ] Architect judges the decision conformant; M2b confirmed +- [ ] Every touched assertion re-baselined with its originating spec named +- [ ] Suite green + +#### Rollback Strategy +Group **G2**, commit-pure. + +--- + +### Phase 2: Three role files (G6, G3, G5) +**Groups**: G6 (`architect`), G3 (`builder`), G5 (`consultant`) — **three commits, one per group** +**Batches**: 1 (3 decisions × 2 trees + 1 test ≈ 7) + +#### Deliverables +- [ ] `roles/architect.md` rewritten (G6) — confirm nothing is load-bearing for multi-architect + coordination (Specs 755/786/823) **before** cutting +- [ ] `roles/builder.md` rewritten (G3) +- [ ] `roles/consultant.md` — *inspected-but-unchanged (expected)*; rewritten only if inspection + finds non-conformance (G5) +- [ ] **M10**: `spec-1273-wait-discipline-docs.test.ts:26` (`ROLE_DOCS` = `codev/roles/builder.md` + + skeleton twin) breaks here +- [ ] Manifest + architect review; suite green + +#### Rollback Strategy +Three group-pure commits: G6, G3, G5. + +--- + +### Phase 3: protocol.md ×10 with the P6 include mechanism (G3) +**Groups**: G3 · **Batches**: 1 (10 decisions + include-mechanism test ≈ 11) +**Dependencies**: the P6 mechanism decision above. + +#### Deliverables +- [ ] Ten `protocol.md` files rewritten (incl. `release`, codev-local, no twin) +- [ ] **P6 include mechanism implemented**: fenced ```` ```json ```` block containing + `{{> protocols/

/protocol.json}}` +- [ ] **T18 — P6 delivery, both modes**: strict (porch-driven spawn resolves the include) **and** + soft (a builder reading `protocol.md` with no porch still receives the structured source); + asserted against a simulated fresh-install resolution where `codev/protocols/` is absent +- [ ] Manifest + architect review; suite green + +#### Implementation Details +- `release/protocol.md` has no `protocol.json`; P6 does not apply — rewritten on P1/P3 alone. +- **P7**: delete worst-case padding except scar rules. +- Largest single cut in the project (`spir/protocol.md`, 3,703w); M5's contract-presence + assertions against the Phase-0 frozen inventory are the primary defence. + +#### Acceptance Criteria +- [ ] T5 passes against the **frozen pre-rewrite inventory** — every gate, check, signal and + artifact contract still represented by name or resolvable reference +- [ ] T18 passes in both modes +- [ ] Architect judges all 10 conformant; suite green + +--- + +### Phase 4: builder-prompt.md ×9 + M10 test-retirement burden (G3) +**Groups**: G3 · **Batches**: **2** — (A) 9 prompt decisions; (B) 4 test suites + retirements file +**Dependencies**: Phase 3 reviewed. + +#### Deliverables +- [ ] Nine `builder-prompt.md` files rewritten *(batch A)* +- [ ] **M10 executed and enumerated** *(batch B)*: `baked-decisions.test.ts:139-148` + (pure-addition diff on `protocols/{spir,aspir,air}/builder-prompt.md` — structurally + incompatible with rewriting them), `bugfix-744-spir-pr-strategy.test.ts`, + `bugfix-619-aspir-prompt.test.ts`, plus any `governance-sweep` / `framework-ref-audit` + assertions touched +- [ ] `codev/resources/1280-retirements.md` entries with architect approval *(batch B)* +- [ ] Manifest + architect review of **both** batches; suite green + +#### Implementation Details +For each assertion: name the originating spec, state whether the protected behaviour survives in +the rewritten prose, and either write the replacement assertion or record an architect-visible +retirement. Re-baselining a pure-addition baseline requires the originating spec named and the +new baseline committed **in the same commit**. + +#### Risks +The highest-risk phase: silently gutting a prior spec's protection to make the suite green. +Mitigated by M10 being an explicit deliverable with architect sign-off per assertion, reviewed as +its own batch rather than buried among prompt diffs. + +--- + +### Phase 5: Phase prompts — spir, aspir, pir (G4) +**Groups**: G4 · **Batches**: 1 (11 decisions) + +#### Deliverables +- [ ] spir ×4, aspir ×4, pir ×3 +- [ ] **M10**: `template-delivery.test.ts` if the include wiring is touched +- [ ] Manifest + architect review; suite green + +#### Implementation Details +- **P2** is the lever; the `{{> …}}` template includes make each prompt ~600 words heavier than + it reads. Templates themselves are rewritten in Phases 6–7. +- Two separate constraints, not to be conflated: porch's `REQUIRED_SPEC_SECTIONS` needs **4** + headings (`checks.ts:149-154`); the `spec-review` consult type advisorily expects 20. +- `` tags are capability-inventory items — preserved or retired explicitly. + +#### Acceptance Criteria +- [ ] T5 passes; porch checks pass on a scratch project driven specify→plan +- [ ] Architect judges all 11 conformant; suite green + +--- + +### Phase 6: Phase prompts — bugfix, air, maintain + spir templates (G4) +**Groups**: G4 · **Batches**: 1 (10 decisions) + +#### Deliverables +- [ ] bugfix ×3, air ×2, maintain ×2; spir templates `spec.md`/`plan.md`/`review.md` ×3 +- [ ] Manifest + architect review; suite green + +#### Implementation Details +- Lighter protocols are already closer to conformant (means 356–457w); **a file already + conformant passes unchanged** under the acceptance model. +- **`plan.md`'s machine-readable phases JSON block is a CAPABILITY, not an example** — porch's + `has_phases_json` and `min_two_phases` checks require it. It survives P2 untouched. + +#### Acceptance Criteria +- [ ] A plan generated from the rewritten template passes `has_phases_json` + `min_two_phases` +- [ ] T5, T6, T7 pass; architect judges all 10 conformant; suite green + +--- + +### Phase 7: Remaining templates + spir consult-types (G4, G5) +**Groups**: G4 (templates), G5 (consult-types) — **two commits, one per group** +**Batches**: 1 (10 decisions + `bugfix-742` test ≈ 11) + +#### Deliverables +- [ ] Templates: experiment 1, maintain 1, spike 1, **maintain codev-local ×2** (no skeleton twin) = 5 +- [ ] spir consult-types ×5 +- [ ] **M10**: `bugfix-742-consult-templates.test.ts:27-28` pins prose in + `spir/consult-types/{pr,impl}-review.md` — breaks here +- [ ] Manifest + architect review; suite green + +#### Implementation Details +- The two codev-local maintain templates have no skeleton twin — inspected once, excluded from + T7's twin-parity intersection. +- **The verdict format (`VERDICT: APPROVE | REQUEST_CHANGES | COMMENT`) is a CAPABILITY** — + `consult` parses it. Preserved exactly. + +#### Acceptance Criteria +- [ ] A live `consult --type spec-review` returns a parseable verdict +- [ ] T5, T6, T7 pass; architect judges all 10 conformant; suite green + +--- + +### Phase 8: Consult-types — aspir, bugfix, air (G5) +**Groups**: G5 · **Batches**: 1 (9 decisions + `bugfix-742` bugfix-side assertions ≈ 10) + +#### Deliverables +- [ ] aspir ×5, bugfix ×2, air ×2 +- [ ] **M10**: `bugfix-742-consult-templates.test.ts:25-26` pins `bugfix/consult-types/{pr,impl}-review.md` +- [ ] Manifest + architect review; suite green + +#### Risks +Mechanical application without judgment — "same as spir" is itself a rules-not-judgment failure +in a project about exactly that. Each file is judged on its own diff. + +--- + +### Phase 9: Consult-types pir/maintain + scar registry + dead-tree deletion (G5, G7, G4) +**Groups**: G5 (consult-types), G7 (registry), G4 (dead tree) — **three commits, one per group** +**Batches**: **2** — (A) 4 consult-type decisions; (B) registry + T4 + dead-tree deletion + T8 + routing test + +#### Deliverables +- [ ] pir ×2, maintain ×2 consult-types *(batch A)* +- [ ] `codev/resources/scar-rules.yaml` rebuilt — eight canonicals verbatim, `must_appear_on` + re-derived against the **post-rewrite** surface *(batch B)* +- [ ] **T4** scar enforcement test created here — first phase where it can be meaningful *(batch B)* +- [ ] `codev-skeleton/porch/prompts/` deleted (M6, group G4), with + `review-prompt-routing.test.ts:29` updated under M10 naming **Spec 987** *(batch B)* +- [ ] Manifest + architect review of both batches; suite green + +#### Implementation Details +- The registry is rebuilt **now** — Baked Decision 2 defers enforcement until the surface stops + moving; `must_appear_on` derived earlier would be stale. +- M6 verification is **not** a bare grep: an untruncated repo-wide search reconciled against the + full hit list. (A truncated grep in this project's own spec phase produced a false + "no consumers" claim — the failure this step exists to avoid.) + +#### Acceptance Criteria +- [ ] T4 passes with the rebuilt registry; count pinned at 8; reword/deletion fails +- [ ] T8 passes; Spec 987 protection preserved on remaining files +- [ ] Architect judges the 4 decisions conformant **and** ratifies `must_appear_on`; suite green + +--- + +### Phase 10: Capability verification, measurement report, rollback rehearsal, governance docs +**Groups**: all (verification only) · **Batches**: 1 (artifacts) + +#### Deliverables +- [ ] Post-rewrite capability inventory extracted and compared against the Phase-0 frozen + baseline (M5); any removal listed in `1280-retirements.md` with architect approval +- [ ] Measurement re-run: before/after, per-audience, **deleted vs relocated** across all four + skill trees (M0c, M1, M2) +- [ ] **T9** live spawn probe; **T10** rollback rehearsal for **every group touched** + (G1–G7), not a sample (M9) +- [ ] Governance docs routed by tier — including the **"trust the authoritative source, not the + convenient signal"** lesson (five instances in this project's spec phase alone) +- [ ] Suite green + +#### Note on scope +The **review document and PR belong to porch's `review` phase**, not here — the first draft +folded them into an implement sub-phase where porch's review phase would re-run over them. + +--- + +## Post-merge: the verify phase (M7, M8, M12, T13, T14) + +SPIR's `verify` phase is where the A/B lives; it is **not** an implement phase and is listed here +so it has an explicit home rather than being assumed. + +- [ ] **M12 release hold** in force from the moment the rewrite PR merges until the SHIP verdict +- [ ] **T14 pre-flight per pair**: every surface under test resolves from tier 2 — no skeleton + file lacks a `codev/` twin. Failure **voids the pair** rather than producing a comparison + that looks valid and is not +- [ ] **M7**: ≥6 issue-pairs, prompt-only overlay construction (both arms from source commit `S`; + control applies one overlay reverting G2–G6), both hashes recorded per run +- [ ] `codev/resources/1280-ab-results.md` — one row per run +- [ ] **M8 / T13**: `measure-prompt-behavior.ts` re-run, B1 compared directionally to 51.88% +- [ ] SHIP / HOLD / ROLLBACK verdict → `verify-approval` + +## Timeline & Dependencies + +Strictly sequential; each phase ends at the architect's per-file review and a green suite. + +``` +P0 (PR-1, merged) → P1 → P2 → P3 → P4 → P5 → P6 → P7 → P8 → P9 → P10 → [PR] → verify + 1 3 10 9 11 10 10 9 4 — + decisions = 67 +``` + +## Rollback Strategy (whole project) + +Seven groups per the spec: G1 instrument · G2 shared · G3 builder-spawn · G4 phase · +G5 consultant · G6 architect · G7 scar registry. **Every commit is group-pure**, so any group +reverts cleanly. Reverting **G7** requires reverting every group carrying scar text (G2, G3, G4, +G6); all others are mutually independent. T10 rehearses **every** group touched. + +## Open Questions Carried Into Implementation + +- **Batch size** — ≤12 is the spec's cap; smaller grows the phase count, not the batches. +- **`roles/consultant.md`** — *inspected-but-unchanged (expected)*; rewritten only on finding. +- **Hot tier** — out of scope by the spec's disposition table; a reviewer may argue P3 applies. +- ~~**Pre-existing skills drift**~~ — **RULED at the plan gate (2026-08-01)**: any skill this + project **touches** gets full four-tree parity as part of touching it (**T17** enforces). + Pre-existing drift on **untouched** skills (`afx`, `porch` repo-vs-skeleton) and the + skeleton-absence product question (`forge`, `skill-creator`, `team`) go to a **separate + issue**, filed by the architect. This plan treats them as **recorded known state, nothing + more** — they are not repaired here and their presence must not fail T17. + +## Notes + +**Why phases are drawn by inspection load.** The architect's per-file review is the throughput +constraint, so the boundary that matters is "a batch a human can review in one sitting." Phase 4 +is the exception that proves it: only 9 decisions, but it carries the entire M10 burden, so it is +split into two explicit batches. + +**Phase 0 is not optional sequencing.** Rewriting before the instrument is corrected — and before +the capability inventory is frozen — would make every subsequent measurement and every M5 +assertion unfalsifiable. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md new file mode 100644 index 000000000..66c5238e4 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-plan-iter1-rebuttals.md @@ -0,0 +1,113 @@ +# Plan iteration 1 response — Spec 1280 + +Codex REQUEST_CHANGES (HIGH, 8) · Claude REQUEST_CHANGES (HIGH, 7). **All accepted; no +disputes.** Both reviewers verified the plan's decision arithmetic independently and both +reproduced 67 — the accounting held; what failed was the *supporting* work riding alongside the +decisions, which the first draft under-specified because it drew boundaries purely by inspection +load. + +Two findings were **blocking mechanism gaps** that would have surfaced mid-implementation as +adopter breakage rather than as clean failures. + +--- + +## Blocking mechanism gaps (both reviewers, verified) + +### 1. P6's `protocol.json` reference had no adopter-resolvable mechanism + +Verified against source: `protocol.md` is inlined at spawn via `{{protocol_reference}}` +(`spawn-roles.ts:112-124`); **`protocol.json` is inlined nowhere** — `spawn-roles.ts:267` reads it +only for validation. In a fresh adopter project `codev/protocols/

/protocol.json` does not +exist on disk. So "read `protocol.json`" is exactly the fetch-by-path of a framework file that +CLAUDE.md forbids and the spec's own constraint restates — and it was carrying the largest single +cut in the project. + +**Resolved by verifying the resolver rather than guessing**: `resolveCodevIncludes` +(`skeleton.ts:108-119`) is **extension-agnostic**, so `protocol.md` carries a fenced ```json +block containing `{{> protocols/

/protocol.json}}`. Resolves through all four tiers, works in +fresh installs, needs **no porch change**, and is the literal expression of P6. Cost stated +honestly (spir +570 words back) — acceptable because size is reporting-only under the amended +charter. New **T18** tests both **strict** and **soft** mode, because the asymmetry is real: +strict-mode builders get checks/gates as porch tasks, soft-mode builders have only the prompt. + +### 2. Skill relocation is a FOUR-tree sync the instrument could only see one quarter of + +Claude's finding, verified: skills exist in `.claude/skills` (10), `.codex/skills` (10, +**byte-identical**), `codev-skeleton/.claude/skills` (7), `codev-skeleton/.codex/skills` (7) — +with **existing drift** (`afx`, `porch` differ repo-vs-skeleton; `forge`, `skill-creator`, `team` +skeleton-absent). M0(g) counted only `.claude/skills`, so relocating content there would have +left Codex agents and adopters without it **and reported it as deleted** by M0c/T15 — inverting +the project's honesty artifact. + +Fixed: Phase 0 widens M0(g) to all four trees; Phase 1 adds **T17** (skills parity) and treats +every relocation as a four-copy write. Pre-existing drift is recorded as known state rather than +silently "fixed" — with the scope question raised for the architect at the gate. + +--- + +## Codex + +**Pre-rewrite capability inventory never created — ACCEPTED.** Phase 2 asserted T5 while Phase 9 +first extracted the inventory. M5 requires a *committed pre-rewrite* baseline. Moved into Phase 0 +(PR-1) and frozen there; every later phase asserts against it. + +**Post-merge work has no executable home — ACCEPTED.** Added an explicit **verify phase** +section covering M7, M8, M12, T13, T14, `1280-ab-results.md`, and the SHIP/HOLD/ROLLBACK verdict. + +**T3, T13, T14, T16 unhomed — ACCEPTED.** T3 and **T16** into Phase 0 — T16 especially, since it +is the mechanical guard on M11 and must exist *before* Phase 1 produces the first manifest. T13 +and T14 into verify. Manifest format and location now specified. + +**"≤12 batches" didn't count supporting changes — ACCEPTED, and it would have broken the +architect's own mandate.** Added an explicit definition: a **review batch** is every distinct file +the architect reads, including tests, registry and retirements. Phases 4 and 9 now declare **two +batches each**. + +**Scar-test sequencing contradictory — ACCEPTED.** Phase 1 asserted T4 while Phase 8 created it. +Phase 1 now verifies the eight canonicals byte-for-byte against the ratified +`builder/spir-1252:scar-rules.yaml` directly; T4 is created in Phase 9 and applies from there. + +**Grouped rollback not achievable from the proposed commits — ACCEPTED** (Claude found the same +from the mapping side). Resolved with a commit-level invariant: **every commit is group-pure**; +phases may span groups and now declare them. M6's dead-tree deletion assigned to **G4**. T10 +rehearses **every** group touched, not a sample. + +**PR-1 mechanics unstated — ACCEPTED.** Added: branch cut, merge wait, re-branch via +`git fetch origin main && git checkout -b … origin/main` (never `git checkout main` from a +worktree), `porch done --pr` / `--merged` recording, and a duplicate-commit check. + +--- + +## Claude + +**Rollback mapping contradicted the spec — ACCEPTED.** Phase 1 claimed G2/G6 while rewriting +`roles/builder.md` (spec G3) and `roles/consultant.md` (spec G5); a G3 revert would have pulled +Phase 1 work out and T10 would have rehearsed the wrong map. Roles now live in their own phase +with **three group-pure commits** (G6/G3/G5). + +**M10 concentrated in one phase but collisions are spread — one misassigned — ACCEPTED, +verified.** `spec-1273-wait-discipline-docs.test.ts:26` targets `codev/roles/builder.md` + its +skeleton twin → breaks in the **roles** phase, not the builder-prompt phase; line 31 targets +`.claude`/`.codex` `afx/SKILL.md` → breaks in **Phase 1** on relocation. +`bugfix-742-consult-templates.test.ts:25-28` pins spir *and* bugfix consult-types → Phases 7 and +8. M10 is now a per-phase deliverable, assigned where each collision actually lands. + +**No per-phase green-suite requirement — ACCEPTED.** The branch could have sat red across eight +review batches while the architect inspected diffs on a broken tree. Every phase now ends green, +and it is in Success Metrics. + +**Reconcile "~66" vs 67 — ACCEPTED.** The plan states 67 is correct and notes both reviewers +reproduced it. + +**Phase 9 folded review-phase deliverables into an implement phase — ACCEPTED.** Review document +and PR moved out to porch's `review` phase, which would otherwise re-run over them. + +--- + +## Net + +Fifteen findings, none disputed. The plan's decision accounting survived both reviews intact; +everything that failed was work that *accompanies* the decisions — tests, inventories, group +purity, sync obligations — which is exactly what a plan drawn by inspection load will +under-specify if nobody checks. Phase count 10 → 11, with two phases now carrying explicit +double batches. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md new file mode 100644 index 000000000..403e9e580 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter1-rebuttals.md @@ -0,0 +1,174 @@ +# Iteration 1 response — Spec 1280 + +**No disputes filed.** Every REQUEST_CHANGES point from both reviewers was verified against +source and accepted. This document records what changed and where, so round 2 can check the +fixes rather than re-derive the findings. + +Two of the findings were factual errors in my own Current State. Both are the exact class +this spec's principle 7 ("the instrument is part of the deliverable") exists to catch, and +both are recorded in the spec rather than quietly patched. + +--- + +## Codex findings + +### 1. Scar-rule accounting contradicts M1/M2 — ACCEPTED + +Iteration 1 declared per-surface targets as served-word ceilings *and* said "targets above +are net of this [scar carriage]." Incompatible, and Claude independently computed the +consequence: gross landing 50.2–51.4%, so **M1 would have failed while every per-surface row +passed**. + +**Changed**: all ceilings restated **gross**, with a per-row "scar carriage inside" column. +Arithmetic rebuilt (`Desired State`). Carriage is now *exempt from rewriting but counted in +ceilings* — with the corollary stated explicitly: a ceiling a surface cannot meet while +carrying its scar rules is a wrong ceiling, raised deliberately, never met by trimming scar +text. Margin disclosed (3.2 points) and M2 made binding with M1 derived; a 50–52% landing is +a HOLD with named further-cut candidates. + +### 2. M0 does not match the runtime resolver — ACCEPTED + +Correct: `loadPromptFile` resolves each file independently through the four-tier chain, while +the script does two-tier, directory-level selection. Fixing the directory alone would +reproduce the same defect class one layer down. + +**Changed**: M0 now requires per-file four-tier resolution matching `resolveCodevFile`, and +**T1b** is a new test using a fixture with a `.codex`/`.codev/` override of *one* prompt while +others resolve from the skeleton. M0 also now names the hot-tier transclusion and the +segment reporting the architect's directive requires. + +### 3. "Gate friction" unchecked under Critical while the A/B assumes it — ACCEPTED + +**Changed**: resolved and moved out of Critical. O1 is scored prospectively by the architect +on a 3-item rubric with an explicit 0/1/2 scale, recorded in a committed results artifact at +scoring time, and **demoted to advisory-with-a-tripwire** rather than a SHIP gate. If scoring +is incomplete for any pair, O1 reports incomplete and SHIP rests on O2/O3/O4. (Claude raised +the same single-point-of-failure concern independently and suggested exactly this demotion.) + +### 4. A/B contamination controls — ACCEPTED + +**Changed**: new **Contamination controls** subsection — model ids/efforts and consult +backend versions pinned and recorded; arm isolation (second arm must not see the first arm's +branch, PR, or thread; sequential-with-unpushed-branch or isolated concurrent, recorded per +pair); alternating arm order; and a committed results artifact +(`codev/resources/1280-ab-results.md`) with one row per run including base commit, isolation +mode, every outcome, and any exclusion with its reason. + +### 5. Per-surface rollback independence overstated — ACCEPTED + +Correct — prompts, included templates, registry mappings and integrity tests are coupled. + +**Changed**: rollback restated as **seven groups** (instrument / shared / builder-spawn / +phase / consultant / architect / scar-registry), each internally consistent, with an explicit +dependency rule: reverting G7 requires reverting every group carrying scar text. T10 rehearses +by group. + +### 6. M5's inventory diff not deterministic — ACCEPTED + +**Changed**: M5 now specifies a committed pre-rewrite `capability-inventory.json` with +explicit recognition rules per element type (artifact paths, gate names from `protocol.json`, +`` tags, check ids, `afx send architect` call sites) and normalization (lowercase, +strip backticks/punctuation, dedupe). Post-rewrite must be a superset; removals fail and must +be justified as deliberate retirements. + +### 7. Future-dated provenance — ACCEPTED + +The architect's instructions carried UTC timestamps (`2026-08-01T02:50Z`, `02:59Z`); local +time was 19:50/19:59 on 2026-07-31. + +**Changed**: all dates normalized to **2026-07-31**, with the UTC/local explanation recorded +in the consultation log so the provenance is auditable rather than merely corrected. + +--- + +## Claude findings + +### 1. "All existing tests pass" is unsatisfiable — ACCEPTED (highest-value finding) + +Verified: `agent-farm/__tests__/baked-decisions.test.ts:143-148` enforces a pure-addition +diff against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md`, which +the cut plan takes 824 → ≤420. Also verified the shape across +`bugfix-744-spir-pr-strategy.test.ts`, `spec-1273-wait-discipline-docs.test.ts`, +`bugfix-619-aspir-prompt.test.ts`, `template-delivery.test.ts`, `framework-ref-audit.test.ts`, +`governance-sweep.test.ts`, `review-prompt-routing.test.ts`. + +The framing is the important part and I have adopted it verbatim in intent: **each assertion +is a prior spec's protection encoded as a grep, so retiring one is a governance act, not a +test fix.** + +**Changed**: new criterion **M10** — every modified or retired assertion listed in the review +with (i) the originating spec, (ii) whether the protected behaviour survives in the rewritten +prose, (iii) the replacement assertion or an explicit architect-visible retirement. +Pure-addition re-baselining only with the originating spec named and the new baseline +committed in the same commit. Silent deletion to go green is declared a project failure. Also +added as a Risks row (High/High) and a Dependencies entry. + +### 2. M1 and M2 stated on different bases — ACCEPTED + +Same as Codex 1; see above. Claude's gross computation (50.2–51.4%) is what made the severity +concrete, and the thin-margin warning is now in the spec as a HOLD rule. + +### 3. Two factual errors in Current State — ACCEPTED, both verified + +**(a) Hot tier is `@import`ed, not inlined.** Verified `CLAUDE.md:14-15` and +`managed-block.ts:59-67`: #1119 replaced Spec 987's verbatim inlining with `@import` lines +that Claude Code transcludes at session launch. So `wc -w CLAUDE.md` = 5,815 **excludes** the +736 hot-tier words the session actually loads. Baseline corrected **33,519 → 34,255**. + +Worth stating plainly: I inherited this from the measurement script's own stale comment +(lines 44–47) while writing a spec whose headline principle is that instruments get audited +against their claims. M0(d) now requires fixing that comment, and the incident is written +into the Problem Statement as principle 7 applied to this spec itself. + +**(b) "No code reads `porch/prompts`" is false.** Verified +`review-prompt-routing.test.ts:29` pushes `codev-skeleton/porch/prompts/review.md` onto its +assertion list (Spec 987 hot/cold routing). My cause: I ran `grep -rn … | head -20` and drew +a conclusion from truncated output — the full result is 48 lines. The tree remains dead as +*prompt surface* (no runtime consumer), but M6's stated verification method was wrong. + +**Changed**: Current State carries the retraction and its cause; M6 now requires an +untruncated search reconciled against the full hit list, and routes the test consumer through +M10 naming Spec 987. Recorded in the thread as a sweep-scope failure — the class 1252 +identified as its dominant review-iteration cost. + +### 4. O1 single point of failure — ACCEPTED + +See Codex 3. Adopted Claude's suggested demotion. + +### 5. A/B arms not as clean as "two checkouts" — ACCEPTED + +Sharp catch: for issues touching Codev's own prompt surface, the treatment arm's CLAUDE.md is +simultaneously instrument and subject. + +**Changed**: eligibility rule added — an issue is ineligible for the A/B if it modifies any +surface under test. + +### 6. Template pressure conflated — ACCEPTED + +Verified `checks.ts:149-154`: `REQUIRED_SPEC_SECTIONS` is four headings (Problem Statement, +Current State, Desired State, Success Criteria), not the template's 20; the 20-heading +pressure is the `spec-review` consult type, advisorily. + +**Changed**: named as two separate constraints in Current State, so the plan does not +over-preserve template surface for a check that does not require it. + +### 7. T3-vs-CI contradiction — ACCEPTED + +**Changed**: the Nice-to-Know open question withdrawn; T3 stands and is stated as the +anti-re-growth guard that runs in CI. + +--- + +## Also incorporated this iteration (not from review) + +**Architect scope directive** (2026-07-31): scope is the entire prompt surface — architect +role prompts, builder roles and spawn wrappers, consultant/CMAP prompts, phase prompts and +their template includes, `protocol.md` texts — and the instrument must **segment by audience** +so a cut concentrated in one segment while another grows is visible rather than averaged away. + +**Changed**: Current State now carries a full inventory of every prompt-bearing surface with +its resolver path and served word count, grouped SHARED / ARCHITECT / BUILDER / PHASE / +CONSULTANT / DEAD. Per-segment ceilings and post-rewrite figures added (builder −53.2%, +architect −61.2%, consultant −33.8%, no segment growing), M0(f) requires per-segment +subtotals, and **T11** proves with a fixture that one segment growing while another shrinks +is not netted to zero. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md new file mode 100644 index 000000000..77c486c19 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter2-rebuttals.md @@ -0,0 +1,105 @@ +# Iteration 2 response — Spec 1280 (CMAP round 2) + +**No disputes filed.** All nine findings verified and accepted. Both arithmetic claims were +independently recomputed before acceptance rather than taken on trust. + +Round 2 existed because the architect directed it — porch had advanced to the gate after the +round-1 rebuttal without re-reviewing. It found two arithmetic errors that would otherwise +have shipped, which settles the question of whether it was worth running. + +--- + +## Codex + +**C1. Segment arithmetic underspecified** — ACCEPTED. Audience loads overlap (SHARED rides in +both architect and builder) and use different multipliers, so T11's "subtotals sum to the +total" was false as written. Current State now separates **exclusive buckets** (a partition — +these sum) from **derived audience loads** (overlap by design), with the explicit +`ALWAYS_ON_WORDS` formula. M0(f) and T11 restated accordingly. + +**C2. "All ten protocols in both trees" is impossible** — ACCEPTED, verified on disk. +`codev/protocols/` has ten; `codev-skeleton/protocols/` has nine (`release` is project-local +by design). `experiment`/`research`/`spike`/`release` have no `prompts/`; +`experiment`/`research`/`spike`/`release` have no `consult-types/`. Coverage restated as +per-**surface**, enumerated from disk across both trees and unioned: absence never fails, +unmeasured presence does. M2/M3/T6 rewritten. + +**Consequence I owe you separately**: this exposed that `codev/protocols/release/protocol.md` +(1,626w) was missing from my inventory entirely, because I enumerated +`codev-skeleton/protocols/*/` and `release` lives only in `codev/`. Now in M3 and Current +State, with the cause recorded. Same root cause as round 1's truncated grep: enumerating from +a convenient source rather than the authoritative one. + +**C3. M5 does not prove prompt capability preservation** — ACCEPTED, and the sharpest finding +of the round. Gate and check names extracted from an unchanged `protocol.json`, and +notification names from unchanged call sites, remain present even if every corresponding +instruction disappears from the served prompts — the inventory would have reported success +while measuring files I am not touching. M5 now inventories the **resolved, expanded prompt +surface**, with a contract-presence assertion per capability, and is named as the primary +defence for the most aggressive row in the table (`protocol.md` −81%). + +**C4. M5 internally contradictory on removals** — ACCEPTED (Claude raised this independently). +Resolved on M10's pattern: hard fail unless the retired name is listed in a committed +`codev/resources/1280-retirements.md` in the same commit, with the reason and architect +approval. An approved listed retirement passes; anything else fails. + +**C5. O3 timing ambiguous** — ACCEPTED. "Any post-merge defect" cannot be evaluated at a +pre-merge SHIP decision. Split: pre-merge architect findings gate SHIP; post-merge defects are +a **14-day rollback signal**. Which arm merges is now answered under C5's sibling finding +below. + +--- + +## Claude + +**L1. Nothing distinguishes deleted words from relocated words** — ACCEPTED, and the most +important finding of the round. Principle 4 authorizes relocation to skills, and relocating +3,900 words scores identically to deleting them under an always-on-only metric. A −53.2% +headline is equally consistent with −30% deleted + −23% relocated, and only *deleted* content +satisfies Problem Statement claim 1 — relocated content still enters context when looked up. +This is the phantom-savings class T2 catches on the include axis, unmonitored on the +relocation axis, and my own principle 7 requires the instrument to show it. Added **M0(g)** +(report total authored surface), **M0c** (decompose the cut into deleted vs relocated), and +**T15** (fixture: moving a block to a skill must show always-on falling while total-authored +holds steady). + +**L2. A/B execution model undefined** — ACCEPTED. Three unanswered operational questions, now +answered in a new **Execution and sequencing** subsection: +- **M7 gates `verify-approval`, not the PR merge** — matching where the rollback triggers + already point, keeping a 12-run trial off the PR's critical path, and making "treatment arm + = what builders actually get" literally true. Consequence stated plainly: a SHIP failure + means rolling back a merged change, which is what the grouped rollback plan is for. +- **Arm disposition** — the treatment arm's PR is the merge candidate; the control arm's + closes unmerged after its outcomes are recorded. The cost defence is corrected: **~6 of 12 + runs produce merged work, not 12.** +- **Architect load** — ~24 gate approvals + 12 PR reviews by one person, each SPIR gate + requiring O1 rubric scoring at approval time. Named as the trial's binding scheduling + constraint and the reason the pair count is the architect's call. + +**L3. M5 severity contradiction** — same as C4. + +**L4. M1's HOLD branch is arithmetically unreachable** — ACCEPTED, verified by recomputation. +Meeting every ceiling yields ≤16,016 (−53.2%); the 50–52% band requires a total of +16,442–17,128, i.e. ceilings already exceeded and M2 already failing. The branch was dead +prose. Withdrawn and replaced with the *reachable* contingency: **denominator movement** — if +correcting the instrument surfaces always-on content not yet found (as it has twice), the +baseline and every ceiling are re-derived to preserve >50%, and that goes to the architect +rather than being absorbed silently. + +**Smaller notes, all adopted**: M2b added (nothing protected CLAUDE.md's human readability at +5,815 → 1,900 — humans are named stakeholders with no criterion behind them; the architect now +reviews usability at the gate); the fleet-wide consultant figure stated (≈20,500 words/project, +comparable to the entire builder load, so "−33.8% on 683" does not read as negligible); and the +hot-tier exemption quantified as a ceiling on the project (7,360 of 16,016 post-rewrite builder +words = 46%). + +--- + +## Net effect + +Nine findings, none disputed, two of them arithmetic errors caught only because the architect +insisted on a round the machinery had skipped. The spec's own principle 7 — instruments get +reviewed against what they claim to measure — has now caught four errors in this spec, three +sharing one root cause: enumerating from a convenient source instead of the authoritative one. +That pattern belongs in the review's lessons learned, and it is the argument for M3's +"enumerate from disk" requirement being a test rather than an instruction. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md new file mode 100644 index 000000000..c9df851fa --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter3-rebuttals.md @@ -0,0 +1,108 @@ +# Iteration 3 response — Spec 1280 (CMAP round 3, acceptance-model revision) + +Codex REQUEST_CHANGES (HIGH, 5 findings) · Claude COMMENT (HIGH, 6 findings). **All accepted; +no disputes.** Both reviewers independently caught the same inventory error, and both landed on +M5's weakness from different angles. + +--- + +## Both reviewers + +**Inventory mislabel — ACCEPTED, verified, and mine.** I reported "3 `codev/protocols` copies +that differ." **Zero differ.** All three (`maintain/templates/audit-report.md`, +`maintain/templates/lessons-learned.md`, `release/protocol.md`) are **local-only with no +skeleton twin**. Cause: my `cmp -s` loop treated a nonzero exit as "differs", but `cmp` also +exits nonzero when a file is absent. I read an exit code without distinguishing its two causes. + +Fixed: table relabelled with a separate "no skeleton twin" row, and **T7 now operates on the +intersection of files that have twins**, so the three local-only files can never be spurious +failures. + +This is the **fifth** self-audit finding of this spec phase and the fourth sharing one root +cause — *trusting a convenient signal instead of checking the authoritative thing* (truncated +grep; skeleton-only enumeration; the script's stale comment; an overloaded exit code). It is +now well-evidenced enough to belong in `lessons-learned.md` as its own entry, which the review +phase will route. + +**M5 is weaker than it reads — ACCEPTED from both angles, and they compose.** + +- *Codex*: **M5 conflicts with P6.** P6 permits replacing narrated gate/check names with a + reference to structured truth; M5 as written demanded those names remain in served prose. A + *conformant* P6 rewrite would have failed M5. +- *Claude*: **M5 detects deletion, not inversion.** "A gate message is a notification to the + human, not authorization" could collapse to a bare mention of the gate name and still pass. + +Fixed together: **representation** is now defined as *either* the name appearing in served text +*or* an explicit resolvable reference to the structured source that still defines it — which +makes P6 and M5 compatible. And M5's **detection limit is stated outright**: it is a deletion +detector, not a meaning detector; the gap is assigned to M11 (architect reads the diff) and O4 +(zero-tolerance compliance), plus a short hand-curated set of **semantic invariants** asserted +as behaviour rather than name presence. + +--- + +## Codex + +**Scope contradiction — ACCEPTED.** MP/M3 said "every prompt-bearing file" while the hot tier +was left unchanged and `.claude/skills/` was measured, received relocated content, and had an +unresolved conformance status. Fixed with an authoritative **per-surface disposition table**: +every category marked **rewritten**, **inspected-but-unchanged**, or **excluded with reason**. +Scope is now exactly that table. (This also delivers the issue-mandated cut plan — see Claude 1.) + +**A/B arms cannot use both "the same base commit" and "pre-/post-rewrite commits" — ACCEPTED.** +Genuinely incompatible as written, and the naive reading also let later pairs inherit source +changes the pinned control commit lacked. Rebuilt as a **prompt-only overlay**: both arms branch +from the same source commit `S`; treatment uses `S`; control applies one overlay commit +reverting rollback groups G2–G6 and nothing else; each run records **both** the source hash and +a **prompt-surface hash** over every file in the disposition table. Source is identical within a +pair, `S` may advance between pairs, and "no code differs" becomes literally true. + +**"Total authored surface" ambiguous — ACCEPTED.** M0(g) now defines it as **physical files on +disk**, each counted once, **no twin deduplication and no transclusion expansion** — +deliberately a different basis from the always-on buckets (which dedupe and expand), because its +job is to detect relocation. Both figures are reported side by side and labelled with their +basis, so T11 and T15 have deterministic expected values. + +--- + +## Claude + +**1. The per-surface cut plan is missing — ACCEPTED, and the sharpest process catch.** Issue +#1280's Protocol section requires the spec phase to produce it. Word *targets* are withdrawn by +the architect's redirect, but the **disposition mapping survives that redirect** and was absent. +Entering implement with only "apply P1–P7" — with the architect as the throughput bottleneck — +invites rewrite→reject churn. Added as the disposition table above, with dominant +non-conformance, governing principles, and relocation destination per bucket. + +**2. Issue #1280's body now contradicts the spec — ACCEPTED, flagged to the architect.** Title +and Goal still say ">50% reduction… measured with 1252's committed measurement script" — a +script this spec disqualifies, and a goal the acceptance-model redirect replaced. The issue is +the charter CMAP reviewers load, so it will keep generating "doesn't meet the stated goal" +findings. **I have not edited the charter myself** — that is the architect's artifact. Raised +for them to update, citing the 2026-07-31 ruling. + +**4. No release-hold between merge and the SHIP verdict — ACCEPTED.** M7 gates +`verify-approval`, so the rewritten skeleton reaches `main` and is shippable to adopters before +the A/B validates it; "pin the prior version" is reactive. Added **M12**: no `@cluesmith/codev` +release between the rewrite merge and the SHIP verdict; if one must cut, it ships from a commit +predating the merge. + +**5. A/B isolation rests on an unstated resolver property — ACCEPTED, and verified.** Every +`codev-skeleton/protocols/**` and `roles/*.md` file has a `codev/` twin (**0 lack one** — +confirmed), so tier 2 shadows tier 4 for every surface under test and the control arm genuinely +serves the old surface. But **deleting a `codev/` file while keeping its skeleton twin would +silently drop the control arm through to the new skeleton** — a comparison that looks valid and +is not. Stated as a precondition; **T14 now asserts it pre-flight per pair and voids the pair on +failure**. + +**6. Minor — ACCEPTED.** Twin mislabel fixed (above). `builder/spir-1252` confirmed present on +`origin` (`ee310a64`), so the "sole source" risk is downgraded Low/High → Low/Medium. + +--- + +## Net + +Eleven findings across two reviewers, all accepted, none disputed. The two structural ones — +scope contradiction and the A/B's impossible arm construction — would both have surfaced during +implementation as confusion rather than as a clean defect, which is the case for having run a +third round on a spec that had already passed two. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md b/codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md new file mode 100644 index 000000000..438c01072 --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/manifests/README.md @@ -0,0 +1,37 @@ +# Per-phase inspection manifests (Spec 1280, M11) + +The architect inspects the **old-vs-new diff of every changed file**, per phase, before porch +advances. A manifest is what makes that inspection possible: **the architect cannot inspect what +is not listed**, so a changed prompt-bearing file absent from its phase's manifest fails the +phase (**T16**, `spec-1280-phase-manifest.test.ts`). + +## File + +`phase--.md`, one per implement phase, committed with that phase's work. + +## Required shape + +A manifest has one row per changed file with **all four fields**. Phases with more than 12 +files declare explicit `## Batch N` sections, each ≤12 — the cap is on *what the architect reads +in one sitting*, which includes tests, registry and retirements entries, not only prompt files. + +```markdown +# Phase 3 — protocol.md x10 (G3) + +## Batch 1 + +| File | Old | New | Principles | Rationale | +|---|---:|---:|---|---| +| `codev-skeleton/protocols/spir/protocol.md` | 3703 | 690 | P6, P1, P7 | State machine now references protocol.json via `{{> }}`; deleted phase-body checklists | +| `codev-skeleton/protocols/air/protocol.md` | 643 | 380 | P1, P7 | Deleted worst-case padding; kept artifact contract | +``` + +- **Old / New** — served word counts from `scripts/measure-prompt-surface.sh`. +- **Principles** — which of P1–P7 were applied (or `none` for an inspected-but-unchanged file). + P5 is N/A project-wide, with reason, per the spec. +- **Rationale** — one line. What was cut, and why it was safe to cut. + +## What a manifest is not + +It is not a substitute for reading the diff. It is the index that makes the diff review +complete and bounded. diff --git a/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml new file mode 100644 index 000000000..ad3bbf00a --- /dev/null +++ b/codev/projects/1280-prompt-surface-judgment-not-ru/status.yaml @@ -0,0 +1,62 @@ +id: '1280' +title: prompt-surface-judgment-not-ru +protocol: spir +phase: implement +plan_phases: + - id: phase_0_instrument + title: Corrected instrument + frozen capability inventory (PR-1, ships early) + status: in_progress + - id: phase_1_shared_skills + title: CLAUDE.md/AGENTS.md + four-tree skill relocation (G2) + status: pending + - id: phase_2_roles + title: Three role files (G6, G3, G5) + status: pending + - id: phase_3_protocol_md + title: protocol.md x10 with the P6 include mechanism (G3) + status: pending + - id: phase_4_builder_prompts + title: builder-prompt.md x9 + M10 test-retirement burden (G3) + status: pending + - id: phase_5_prompts_heavy + title: 'Phase prompts: spir, aspir, pir (G4)' + status: pending + - id: phase_6_prompts_light_spir_templates + title: 'Phase prompts: bugfix, air, maintain + spir templates (G4)' + status: pending + - id: phase_7_templates_consult_spir + title: Remaining templates + spir consult-types (G4, G5) + status: pending + - id: phase_8_consult_types_a + title: 'Consult-types: aspir, bugfix, air (G5)' + status: pending + - id: phase_9_consult_registry_deadtree + title: Consult-types pir/maintain + scar registry + dead-tree deletion (G5, G7, G4) + status: pending + - id: phase_10_integration + title: Capability verification, measurement report, rollback rehearsal, governance docs + status: pending +current_plan_phase: phase_0_instrument +gates: + spec-approval: + status: approved + requested_at: '2026-08-01T03:06:40.784Z' + approved_at: '2026-08-01T03:48:22.599Z' + plan-approval: + status: approved + requested_at: '2026-08-01T04:00:12.126Z' + approved_at: '2026-08-01T04:26:10.219Z' + pr: + status: pending + verify-approval: + status: pending +iteration: 1 +build_complete: false +history: [] +started_at: '2026-08-01T02:46:36.457Z' +updated_at: '2026-08-01T04:39:51.924Z' +pr_history: + - phase: implement + pr_number: 1319 + branch: builder/1280-instrument + created_at: '2026-08-01T04:39:51.923Z' diff --git a/codev/resources/1252-word-after-phase7.md b/codev/resources/1252-word-after-phase7.md index f36be3fe7..8339dae81 100644 --- a/codev/resources/1252-word-after-phase7.md +++ b/codev/resources/1252-word-after-phase7.md @@ -56,3 +56,30 @@ largest block (protocol.md, 3,703 words) is protocol semantics — excluded by the spec's Non-goals. The 20% arithmetic assumed more of the surface was duplicated rules than measurement bore out. Whether even −7% moved behaviour is the verify phase's question (M12b). + +--- + +## SUPERSEDED by Spec 1280 (2026-08-01) + +**The figures above were produced by an instrument with three defects and should not be +cited.** They are preserved unaltered because the record of what was believed, and when, +is part of the history — not because they are correct. + +The instrument (`scripts/measure-prompt-surface.sh` as of Spec 1252): + +1. derived its phase-task term from `codev-skeleton/porch/prompts/`, a dead Ralph-SPIR-era + tree with **no runtime consumer**, while the live resolver + (`commands/porch/prompts.ts`, `loadPromptFile`) loads `protocols/

/prompts/`. Real SPIR + phase prompts average ~1,396 words; the dead tree averaged 400; +2. omitted `roles/builder.md` (1,837 words), which `spawn-worktree.ts:854` injects into + **every** builder spawn; +3. asserted in its own comments that `CLAUDE.md` "already inlines" the hot tier. Since #1119 + it carries `@import` lines, which are **transcluded** at session launch — so `wc -w + CLAUDE.md` excludes 736 always-loaded words. + +**Corrected pre-rewrite baseline: `codev/resources/1280-word-baseline.md` +(ALWAYS_ON_WORDS = 34,235 for a SPIR builder at I=10, versus the 21,702 implied here).** + +The behavioural baseline in `1252-behavior-baseline.md` is **unaffected** — it uses a +different instrument (`measure-prompt-behavior.ts`) and remains the valid "before" for +Spec 1280's M8. diff --git a/codev/resources/1252-word-baseline.md b/codev/resources/1252-word-baseline.md index 99aa8abe5..08dd93707 100644 --- a/codev/resources/1252-word-baseline.md +++ b/codev/resources/1252-word-baseline.md @@ -37,3 +37,30 @@ session, not both), and the hot-tier files appear inside CLAUDE.md's count (they ARE counted once per phase task, where porch injects them separately). ALWAYS_ON_WORDS=21856 + +--- + +## SUPERSEDED by Spec 1280 (2026-08-01) + +**The figures above were produced by an instrument with three defects and should not be +cited.** They are preserved unaltered because the record of what was believed, and when, +is part of the history — not because they are correct. + +The instrument (`scripts/measure-prompt-surface.sh` as of Spec 1252): + +1. derived its phase-task term from `codev-skeleton/porch/prompts/`, a dead Ralph-SPIR-era + tree with **no runtime consumer**, while the live resolver + (`commands/porch/prompts.ts`, `loadPromptFile`) loads `protocols/

/prompts/`. Real SPIR + phase prompts average ~1,396 words; the dead tree averaged 400; +2. omitted `roles/builder.md` (1,837 words), which `spawn-worktree.ts:854` injects into + **every** builder spawn; +3. asserted in its own comments that `CLAUDE.md` "already inlines" the hot tier. Since #1119 + it carries `@import` lines, which are **transcluded** at session launch — so `wc -w + CLAUDE.md` excludes 736 always-loaded words. + +**Corrected pre-rewrite baseline: `codev/resources/1280-word-baseline.md` +(ALWAYS_ON_WORDS = 34,235 for a SPIR builder at I=10, versus the 21,702 implied here).** + +The behavioural baseline in `1252-behavior-baseline.md` is **unaffected** — it uses a +different instrument (`measure-prompt-behavior.ts`) and remains the valid "before" for +Spec 1280's M8. diff --git a/codev/resources/1280-capability-inventory.json b/codev/resources/1280-capability-inventory.json new file mode 100644 index 000000000..fda837b80 --- /dev/null +++ b/codev/resources/1280-capability-inventory.json @@ -0,0 +1,69 @@ +{ + "_spec": "1280", + "_criterion": "M5", + "_basis": "served prompt text (includes expanded); protocol.json supplies the expected set only", + "_detection_limit": "detects deletion, not inversion or gutting — see M11 and O4", + "_commit": "5c962b7a", + "gates": [ + {"name": "dev-approval", "present_in_served_prompts": true}, + {"name": "experiment-complete", "present_in_served_prompts": false}, + {"name": "maintain-complete", "present_in_served_prompts": false}, + {"name": "plan-approval", "present_in_served_prompts": true}, + {"name": "pr", "present_in_served_prompts": true}, + {"name": "research-complete", "present_in_served_prompts": false}, + {"name": "scope-approval", "present_in_served_prompts": true}, + {"name": "spec-approval", "present_in_served_prompts": true}, + {"name": "verify-approval", "present_in_served_prompts": true} ], + "checks": [ + {"name": "build", "present_in_served_prompts": true}, + {"name": "e2e_tests", "present_in_served_prompts": false}, + {"name": "has_phases_json", "present_in_served_prompts": false}, + {"name": "min_two_phases", "present_in_served_prompts": false}, + {"name": "plan_exists", "present_in_served_prompts": false}, + {"name": "pr_exists", "present_in_served_prompts": true}, + {"name": "regression_test", "present_in_served_prompts": false}, + {"name": "review_has_arch_updates", "present_in_served_prompts": true}, + {"name": "review_has_lessons_updates", "present_in_served_prompts": true}, + {"name": "spec_exists", "present_in_served_prompts": false}, + {"name": "spec_has_required_sections", "present_in_served_prompts": false}, + {"name": "tests", "present_in_served_prompts": true} ], + "signals": [ + {"name": "AWAITING_INPUT", "present_in_served_prompts": true}, + {"name": "BLOCKED", "present_in_served_prompts": true}, + {"name": "PHASE_COMPLETE", "present_in_served_prompts": true}, + {"name": "PLAN_DRAFTED", "present_in_served_prompts": true}, + {"name": "PR_READY", "present_in_served_prompts": true}, + {"name": "REVIEW_COMPLETE", "present_in_served_prompts": true}, + {"name": "SPEC_DRAFTED", "present_in_served_prompts": true}, + {"name": "TOO_COMPLEX", "present_in_served_prompts": true} ], + "artifact_paths": [ + {"name": "codev/plans/{{artifact_name}}.md", "present_in_served_prompts": true}, + {"name": "codev/plans/0042-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/plans/42-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/plans/XXXX-*.md", "present_in_served_prompts": true}, + {"name": "codev/plans/XXXX-name.md", "present_in_served_prompts": true}, + {"name": "codev/reviews/{{artifact_name}}.md", "present_in_served_prompts": true}, + {"name": "codev/reviews/42-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/reviews/XXXX-name.md", "present_in_served_prompts": true}, + {"name": "codev/specs/...", "present_in_served_prompts": true}, + {"name": "codev/specs/{{artifact_name}}.md", "present_in_served_prompts": true}, + {"name": "codev/specs/{{project_id}}-*.md", "present_in_served_prompts": true}, + {"name": "codev/specs/0042-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/specs/1-test-infrastructure.md", "present_in_served_prompts": true}, + {"name": "codev/specs/42-feature-name.md", "present_in_served_prompts": true}, + {"name": "codev/specs/42-feature.md", "present_in_served_prompts": true}, + {"name": "codev/specs/spec-file.md", "present_in_served_prompts": true}, + {"name": "codev/specs/XXXX-*.md", "present_in_served_prompts": true}, + {"name": "codev/specs/XXXX-name.md", "present_in_served_prompts": true}, + {"name": "codev/state/*.md", "present_in_served_prompts": true}, + {"name": "codev/state/spir-823_thread.md", "present_in_served_prompts": true} ], + "notification_triggers": [ + {"name": "afx send ", "present_in_served_prompts": true}, + {"name": "afx send ", "present_in_served_prompts": true}, + {"name": "afx send :architect", "present_in_served_prompts": true}, + {"name": "afx send ", "present_in_served_prompts": true}, + {"name": "afx send architect", "present_in_served_prompts": true}, + {"name": "afx send architect:", "present_in_served_prompts": true}, + {"name": "afx send architect:ob-refine", "present_in_served_prompts": true}, + {"name": "afx send marketmaker:architect", "present_in_served_prompts": true} ] +} diff --git a/codev/resources/1280-word-baseline.md b/codev/resources/1280-word-baseline.md new file mode 100644 index 000000000..64516453d --- /dev/null +++ b/codev/resources/1280-word-baseline.md @@ -0,0 +1,75 @@ + +# Prompt-surface measurement + +Commit: `5c962b7a` +Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. + +## Exclusive buckets (partition the authored surface; these SUM) + +| Bucket | Words | +|---|---:| +| SHARED (CLAUDE.md + transcluded hot tier) | 6551 | +| ARCHITECT (roles/architect.md) | 2048 | +| DEAD (codev-skeleton/porch/prompts, 10 files) | 4009 | + +## Per-protocol (resolved per file, four-tier) + +| Protocol | BUILDER_SPAWN | PHASE mean | CONSULT mean | +|---|---:|---:|---:| +| air | 3017 | 456 | 437 | +| aspir | 3467 | 1396 | 430 | +| bugfix | 2965 | 377 | 683 | +| experiment | 3330 | 0 | 0 | +| maintain | 4158 | 356 | 406 | +| pir | 4801 | 1435 | 491 | +| release | 3463 | 0 | 0 | +| research | 3671 | 0 | 0 | +| spike | 3155 | 0 | 0 | +| spir | 6360 | 1396 | 430 | + +## Derived audience loads (these OVERLAP by design — never sum them) + + HOT = arch-critical(416) + lessons-critical(320) = 736 + ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p] + I x (HOT + mean PHASE[p]) + ALWAYS_ON(architect) = SHARED + ARCHITECT + ALWAYS_ON(consultant,p) = roles/consultant.md + mean CONSULT-type[p] + +| Audience | Words | +|---|---:| +| **Builder (spir, I=10)** — the headline | **34231** | +| Architect (per session) | 8599 | +| Consultant (per review, spir) | 682 | + +## Total authored surface (relocation detector — different basis, see header) + +| Component | Words | +|---|---:| +| CLAUDE.md + AGENTS.md (no twin dedup here) | 11630 | +| codev/protocols + codev-skeleton/protocols | 88461 | +| codev/roles + codev-skeleton/roles | 8274 | +| skills, ALL FOUR trees (42 files) | 44840 | +| **TOTAL_AUTHORED** | **153205** | + +Reference (on-demand, not always-on): arch.md 20367, lessons-learned.md 21270. + +ALWAYS_ON_WORDS=34231 +TOTAL_AUTHORED_WORDS=153205 diff --git a/codev/specs/1280-prompt-surface-judgment-not-ru.md b/codev/specs/1280-prompt-surface-judgment-not-ru.md new file mode 100644 index 000000000..c7691ed1a --- /dev/null +++ b/codev/specs/1280-prompt-surface-judgment-not-ru.md @@ -0,0 +1,875 @@ +# Specification: Prompt surface — judgment-not-rules rewrite (principle conformance) + + + +## Metadata +- **ID**: spec-2026-07-31-prompt-surface-judgment-not-rules +- **Status**: draft (acceptance model revised to principle conformance) +- **Created**: 2026-07-31 +- **Issue**: #1280 +- **Protocol**: SPIR + +## Clarifying Questions Asked + +Issue #1280 carries a complete charter, so no clarifying questions were put to the architect +before drafting. Five were resolved against the repository or by architect ruling: + +1. **What is the always-on surface, as served?** — Read the live composition path + (`commands/porch/prompts.ts`, `lib/skeleton.ts`, `lib/managed-block.ts`, + `agent-farm/commands/spawn-worktree.ts`) and measured this builder's own served artifacts + (`.builder-prompt.txt` 4,921w, `.builder-role.md` 1,837w). See **Inventory**. +2. **Is the committed measurement script fit for this project?** — No; three defects, one + disqualifying. Criterion **M0**; architect verified against source and endorsed. +3. **How wide is the rewrite target?** — The **entire** prompt surface: architect roles, + builder roles and spawn wrappers, consultant/CMAP prompts, phase prompts and their template + includes, `protocol.md` texts. +4. **Where are the eight ratified scar rules?** — Recovered verbatim from + `builder/spir-1252:codev/resources/scar-rules.yaml`. +5. **What is the acceptance criterion?** — **Principle conformance, not size** (Waleed, above). + Measurement is retained for honesty, not for grading. + +## Problem Statement + +Codev's prompt surface was written for a fleet that could not be trusted to infer process. It +tells agents how to be agents: ordered procedures, all-caps prohibitions, checklists restating +the phase body, annotated templates. A builder consumes **34,255 served always-on words** +before reading a line of the code it was spawned to change. + +Spec 1252 proved the obvious remedy fails: deduplication yields **−7.0%**, because the surface +is not duplicated, it is **over-instructed**. Anthropic's published account of the +Claude-5-generation rewrite reports: + +> "We removed over 80% of Claude Code's system prompt for models like Claude Opus 5 and Claude +> Fable 5 with no measurable loss on our coding evaluations." + +Three compounding costs: + +1. **It crowds out judgment.** A model given a 3,703-word procedure follows the procedure; + given a contract and a goal, it reasons about the goal. +2. **Nobody reads it, so it rots.** 1252 found the *served* SPIR builder prompt had silently + lost its entire `Verify Phase` section, with a detector reporting the drift, unread, for + months. A surface too large to read is too large to maintain. +3. **It makes its own success unmeasurable.** The committed measurement script scores a + directory the runtime never loads. + +Deletion on judgment-trust grounds was an explicit **Non-goal** of Spec 1252. + +### Principle: the instrument is part of the deliverable + +This is the second measurement defect in the 1252 lineage — the first being that 1252 shipped +without a measurement plan at all, caught at a human gate, not by CMAP. Neither was found by +reading the instrument's code; both by asking *what does this claim to measure, and does it?* + +> **The instrument is part of the deliverable, and instruments get reviewed against what they +> claim to measure — not merely against whether they run.** "Deterministic and committed" is +> not "correct." + +Under the revised acceptance model the instrument no longer *grades* the work — but it still +keeps the project honest about what actually happened, which is why **M0**, **M0b** and +**M0c** survive the demotion of every word target. + +## Current State + +### Inventory — every prompt-bearing surface, by audience + +Served and expanded words (`{{> …}}` includes resolved `codev/` → `codev-skeleton/`), captured +2026-07-31 at `047f92f7`. **Reported for observability; no figure here is a target.** + +**SHARED — every agent in this repo** + +| Surface | Words | How served | +|---|---:|---| +| `CLAUDE.md` | 5,815 | session, harness auto-load | +| ↳ `@codev/resources/arch-critical.md` | 416 | **transcluded at session launch** (#1119) | +| ↳ `@codev/resources/lessons-critical.md` | 320 | same | +| `AGENTS.md` | 5,815 | byte-identical twin; one loads per session, never both | + +**Session shared total: 6,551.** + +**ARCHITECT** — `roles/architect.md` 2,048 (read at `arch-init`); `.claude/skills/*/SKILL.md` +×10 = 6,672 (on-demand — progressive disclosure already working as intended). + +**BUILDER — spawn, once per builder** + +| Surface | Words | | Surface | Words | +|---|---:|---|---|---:| +| `roles/builder.md` (inlined) | 1,837 | | `protocols/pir/protocol.md` | 2,066 | +| `protocols/spir/protocol.md` | 3,703 | | `protocols/maintain/protocol.md` | 1,949 | +| `protocols/spir/builder-prompt.md` | 824 | | `protocols/release/protocol.md` | 1,626 | +| `protocols/pir/builder-prompt.md` | 898 | | `protocols/research/protocol.md` | 1,278 | +| `protocols/aspir/builder-prompt.md` | 820 | | `protocols/experiment/protocol.md` | 1,023 | +| `protocols/research/builder-prompt.md` | 556 | | `protocols/spike/protocol.md` | 920 | +| `protocols/air/builder-prompt.md` | 537 | | `protocols/aspir/protocol.md` | 810 | +| `protocols/experiment/builder-prompt.md` | 472 | | `protocols/bugfix/protocol.md` | 699 | +| `protocols/bugfix/builder-prompt.md` | 429 | | `protocols/air/protocol.md` | 643 | +| `protocols/spike/builder-prompt.md` | 400 | | `protocols/maintain/builder-prompt.md` | 374 | + +**SPIR builder spawn total: 6,364.** + +**PHASE — per porch task delivery, ×I.** Hot tier (736) rides on *every* phase prompt. + +| Protocol | Prompts (expanded) | Mean | +|---|---|---:| +| spir / aspir | specify 1,402 · plan 1,169 · implement 1,065 · review 1,957 | 1,398 | +| pir | review 2,414 · implement 1,151 · plan 741 | 1,435 | +| bugfix | pr 491 · fix 352 · investigate 290 | 378 | +| air | pr 471 · implement 442 | 457 | +| maintain | maintain 402 · review 310 | 356 | + +**CONSULTANT — per CMAP review.** `roles/consultant.md` 252 + one consult-type: spir/aspir spec +514 · impl 421 · phase 421 · plan 406 · pr 392; bugfix pr 726 / impl 641; pir pr 475 / impl +507; air pr 455 / impl 420; maintain 421 / 392. **SPIR per review: 683.** Fleet-wide ≈ 683 × 3 +models × ~10 reviews ≈ **20,500 words/project**. + +**DEAD** — `codev-skeleton/porch/prompts/**`, 10 files, 4,009 words, no runtime consumer. + +### How the reported figures are composed + +Buckets are **exclusive** and partition the authored surface: `SHARED` (6,551) · `ARCHITECT` +(2,048) · `BUILDER_SPAWN[p]` · `PHASE[p]` · `CONSULTANT[p]` · `DEAD` (4,009). Audience loads +are **derived** and deliberately overlap — so they are reported separately, never summed: + +``` +HOT = arch-critical + lessons-critical = 736 +ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p] + I × (HOT + mean PHASE[p]) +ALWAYS_ON(architect) = SHARED + ARCHITECT +ALWAYS_ON(consultant,p) = roles/consultant.md + mean CONSULTANT-type[p] + +ALWAYS_ON_WORDS ≡ ALWAYS_ON(builder, spir, 10) + = 6,551 + 6,364 + 10 × (736 + 1,398) = 34,255 +``` + +Architect load 8,599; consultant 683. `I = 10` is 1252's proxy — a comparison constant, +identical before and after. These definitions exist so M0's report is unambiguous; **none of +them is a target.** + +### File counts — what the architect will personally inspect + +| | Count | +|---|---:| +| Prompt-bearing `.md` files, both trees + `CLAUDE.md`/`AGENTS.md` | **131** | +| `codev/protocols` copies **byte-identical** to their skeleton twin | **60** | +| `codev/protocols` copies that **differ** from a twin | **0** | +| `codev/protocols` files with **no skeleton twin** (local-only) | 3 — `maintain/templates/audit-report.md`, `maintain/templates/lessons-learned.md`, `release/protocol.md` | +| `roles/*.md` — all three byte-identical across trees | 3 pairs | +| `CLAUDE.md` ≡ `AGENTS.md` | verified identical | +| **Distinct content decisions** | **~66** | + +This matters for M11: reviewing all 131 diffs would mean re-reading ~65 byte-identical copies. +The inspection is over **distinct content decisions**, with twin sync verified mechanically +(**T7**), which must therefore operate on the **intersection of files that have twins** — the +three local-only files are inspected once and are not twin-parity candidates. + +**Resolver precondition (load-bearing for the A/B).** Every `codev-skeleton/protocols/**` and +`roles/*.md` file currently has a `codev/` twin — verified: **0 skeleton files lack one** — so +tier 2 shadows the installed-package skeleton (tier 4) for every surface under test. The +control arm only genuinely serves the old surface while this holds. **Deleting a `codev/` file +while keeping its skeleton twin would silently drop the control arm through to the new +skeleton**, invalidating the comparison without any error. Asserted as a pre-flight in T14. + +### Coverage is per-surface, not per-protocol × surface-type + +`codev/protocols/` holds **ten** protocols, `codev-skeleton/protocols/` **nine** — `release` is +project-local by design, has only `protocol.md` (1,626w, no `protocol.json`, no +`builder-prompt.md`), is human-invoked prose an agent reads, and is in scope but not +porch-orchestrated. `experiment`, `research`, `spike` and `release` have no `prompts/` or +`consult-types/`; those absences are intentional. Coverage criteria therefore apply to **each +surface that exists after resolution**, enumerated from disk across both trees and unioned. + +### The measurement defects (why M0 exists) + +1. **Dead directory.** `measure-prompt-surface.sh:89` derives `PORCH_PROMPT_MEAN` from + `codev-skeleton/porch/prompts/*.md` (mean 400). The live resolver + (`commands/porch/prompts.ts:78`, `loadPromptFile`) loads + `protocols//prompts/.md`, mean **1,398**. The dead tree is a Ralph-SPIR-era + leftover ("You are the **Spec Writer** hat in a Ralph-SPIR loop"). +2. **Omits the inlined role.** `spawn-worktree.ts:854` writes `roles/builder.md` to + `.builder-role.md` for harness injection — 1,837 always-on words, uncounted. +3. **Stale hot-tier accounting.** The script's comment asserts CLAUDE.md "already inlines" the + hot files. Since #1119 (`managed-block.ts:59-67`) it carries `@import` lines, transcluded at + session launch, so `wc -w CLAUDE.md` **excludes** 736 always-loaded words. + +Net: the reported baseline (21,702) understates the phase term ~3.5×, omits the role file, and +under-counts the session term by 736. The dead tree has no *runtime* consumer but does have a +**test** consumer: `review-prompt-routing.test.ts:29` (a Spec 987 hot/cold-routing protection). + +### What landed, what is deferred + +On `main` from the 1252 harvest: drift reconciliation, the audit, two word baselines, the +behavioural baseline (B1 = 51.88% REQUEST_CHANGES, n=160, self-excluded; B2 1.12 rounds/phase), +and the measurement tooling. Deferred here by architect ruling: the **scar registry and its +eight ratified wordings**, with enforcement rebuilt *after* the shrink. + +Issue #1279 is partly overtaken — SPIR prompts now inline templates via `{{> …}}`. Two separate +constraints govern template shape and must not be conflated: `checks.ts:149-154` +(`REQUIRED_SPEC_SECTIONS`) requires only **four** headings; the 20-heading pressure comes from +the `spec-review` consult type, advisorily. + +## Desired State + +**Every prompt-bearing file conforms to the blog's principles.** Not "is smaller" — conformant. +A file that is principle-conformant at more words passes; a file that hits any size one might +have hoped for while still narrating procedure fails. + +### The principles, verbatim + +Quoted from *The new rules of context engineering for Claude-5-generation models*. These are +the acceptance basis; each is restated as a per-file question the architect can answer from a +diff. + +| # | Blog transition | Verbatim rationale | Per-file conformance question | +|---|---|---|---| +| **P1** | "Give Claude rules" → **"Let Claude use judgement"** | "newer models have better judgement and can handle these decisions well without explicit rules." | Does this file state a *contract* (what must be true) rather than a *procedure* (what steps to take)? Is every remaining rule one a frontier model would get wrong without it? | +| **P2** | "Give Claude examples" → **"Design interfaces"** | "giving examples actually constrains them to a certain exploration space. Instead of using examples, think more about the design of your tools, scripts and files." | Has each illustrative example been replaced by an interface — a heading skeleton, a schema, a named artifact contract? | +| **P3** | "Put it all upfront" → **"Use progressive disclosure"** | "Claude Code has gotten very competent at using progressive disclosure- loading the right context at the right times." | Is everything in this always-on file needed *every* time? Has look-it-up content moved to a skill or on-demand file, addressed by name? | +| **P4** | "Repeat yourself" → **"Simple tool descriptions"** | "We found we could delete these repeat examples and put instructions on how to use tools in the tool descriptions rather than the system prompt." | Do CLI/tool how-tos live with the tool (skills, `--help`) rather than in the prompt? Is anything repeated here that another surface already owns? | +| **P5** | "Memory in CLAUDE.md files" → **"Auto-memory"** | "Claude now automatically saves memories that are relevant to the work and to you." | **N/A for this project, with reason** — auto-memory is a Claude Code harness feature, and Baked Decision 1's fleet includes GPT 5.6 and Gemini 3.6 consumers with no equivalent. Codev's governance content stays in the hot tier rather than relying on it. Declared rather than silently skipped. | +| **P6** | "Simple specs" → **"Rich references"** | "Claude can handle increasingly more complicated references. Instead of simple markdown files, Claude can reference HTML artifacts." | Where prose restates machine-readable truth (`protocol.json` gates, checks, phases), does the file reference the structured source instead of narrating it? | +| **P7** | **"Unhobbling"** — worst-case guardrails | Old guardrails existed because "we needed to be sure that Claude avoided worst case scenarios, such as deleting files." | Has defensive padding written for weaker models been deleted — **except** the ratified scar rules (below)? | + +**P7 and the scar rules — the one deliberate exception.** The blog's worst-case example +("deleting files") is precisely the class Codev's scar rules guard. Baked Decision 2 keeps all +eight verbatim. This is a knowing departure from P7, not an oversight: the blog's guardrails +protected against *bad output*, which judgment now handles; scar rules protect against +*irreversible acts* — destroyed worktrees, killed sessions, bypassed human gates — where the +cost of being wrong once is unbounded and no amount of judgment makes the wager sensible. Every +other P7 candidate goes. + +### Per-surface disposition and cut plan + +Issue #1280's Protocol section requires the spec phase to produce "the per-surface cut plan…, +the A/B eval design, and the scar-rule carriage plan." The *word targets* are withdrawn by the +acceptance-model redirect; the **disposition mapping survives it**, and is the authoritative +answer to "which surfaces are in scope, and what is expected to change in each." + +Every category is marked **rewritten**, **inspected-but-unchanged**, or **excluded with +reason** — no category is left implicit. + +| Bucket | Disposition | Dominant non-conformance today | Governing principles | Relocation destination | +|---|---|---|---|---| +| `CLAUDE.md` / `AGENTS.md` | **rewritten** | CLI walkthroughs, worktree recipes, protocol-selection prose — all needed rarely, loaded always | P3, P4, P1 | `.claude/skills/` (afx, codev, porch, consult), `--help` | +| hot tier (`arch-critical`, `lessons-critical`) | **inspected-but-unchanged** | none — already capped, judgment-shaped, displacement-enforced (Spec 987) | — | — | +| `roles/architect.md` | **rewritten** | procedure narration for coordination already covered by skills | P1, P3 | `arch-init` / `afx` skills | +| `roles/builder.md` | **rewritten** | ordered procedure + repeated prohibitions | P1, P7 | — | +| `roles/consultant.md` | **inspected-but-unchanged (expected)** | already lean at 252w; rewritten only if inspection finds non-conformance | P1 | — | +| `protocols/*/protocol.md` | **rewritten** | narrates the state machine that `protocol.json` already defines; checklists restate phase bodies | **P6**, P1, P7 | reference `protocol.json` | +| `protocols/*/builder-prompt.md` | **rewritten** | worst-case padding, all-caps prohibitions | P1, P7 | — | +| `protocols/*/prompts/*.md` | **rewritten** | step-by-step process; annotated templates inlined via `{{> …}}` | **P2**, P1 | heading interfaces | +| `protocols/*/templates/*.md` | **rewritten** | annotated examples with filler prose | **P2** | heading interfaces | +| `protocols/*/consult-types/*.md` | **rewritten** | process prose around a rubric + verdict contract | P1, P2 | — | +| `.claude/skills/**` | **excluded from rewrite; in scope for measurement** | on-demand already — P3 working as intended. Receives relocated content, so it is measured (M0(g)) and grows by design | — | — | +| `codev-skeleton/porch/prompts/**` | **deleted** (M6) | dead — no runtime consumer | — | — | +| `codev/protocols/release/protocol.md` | **rewritten** | human-invoked prose, no skeleton twin, missed by earlier inventories | P1, P3 | — | + +**Scope is exactly this table.** MP and M3 apply to every row marked *rewritten*; rows marked +*inspected-but-unchanged* are still inspected under M11 (the architect confirms conformance +rather than approving a diff); the one *excluded* row is excluded for a stated reason and is +still measured. + +### Conformance is judged per file, by the architect + +Acceptance is not a number and not a CMAP verdict. Each rewritten file carries a +**conformance record** — principles applied, what was cut and why, old and new word counts — +and the architect inspects the actual old-vs-new diff. See **M11**. + +### Word counts: measured, reported, never a gate + +The corrected instrument still runs before and after, still reports per-audience loads, and +still separates deletion from relocation (**M0c**) — because a project that deletes 20,000 +words should be able to say truthfully where they went. **No criterion passes or fails on any +of these numbers.** The rewrite's own projection, kept purely so the reported figures have +something to be compared against, is roughly 34,255 → ~16,000 for a SPIR builder; if principle +conformance lands somewhere else, the number moves and the spec does not. + +### Scar-rule carriage plan + +The eight rules ratified 2026-07-28 ship **verbatim** (~188 words): `git-add-explicit`, +`never-destroy-worktrees`, `no-destructive-git`, `human-gates`, `no-hand-edit-status`, +`afx-from-root`, `shellper-verified-orphan`, `tower-restart-permission`. + +- The registry is **rebuilt after the shrink** — each rule's `must_appear_on` re-derived against + the post-rewrite surface. +- Enforcement is a byte-identical-presence test, pinned at 8 rules and their ids. +- A scar rule may be compressed **only by architect ratification**, never by a builder applying + P1 or P7. + +### Rollout: the corrected instrument lands on `main` first (M0b) + +**No prompt-surface word is rewritten before the corrected instrument is on `main`.** 1252's +published baselines cite dead-tree figures while being shared knowledge other work reads. + +**PR-1 (early, standalone) contains exactly:** + +| In | Out (deliberately) | +|---|---| +| Corrected `measure-prompt-surface.sh` — all seven M0 items | Any edit to any prompt surface | +| Its tests (T1, T1b, T2, T11, T12, T15). The script has **no test at all** today, which is how three defects survived in a "committed and reproducible" instrument | The scar registry (rebuilt after the shrink) | +| `codev/resources/1280-word-baseline.md` — corrected, segmented pre-rewrite baseline | The dead-tree deletion (has a test consumer → M10 governance) | +| In-place annotation of `1252-word-baseline.md` and `1252-word-after-phase7.md`: originals **preserved**, marked superseded, reason + pointer | Re-derivation of 1252's behavioural baseline (B1 stands; M8 re-runs post-merge) | + +**Verified safe**: no test asserts on either 1252 word-count artifact (the frozen-sample test at +`prompt-behavior-metrics.test.ts:184` pins the *behavioural* sample, a different instrument). + +## Stakeholders + +- **Primary Users**: builder agents; CMAP reviewer agents; architect agents. +- **Secondary Users**: humans who must read and maintain the surface (M2b); downstream adopters + receiving it via `codev update`. +- **Technical Team**: this builder; the architect, who personally inspects every changed file. +- **Business Owners**: Waleed — sets the acceptance model, ratifies scar wordings, rules on the + A/B verdict. + +## Success Criteria + +**Acceptance basis: MP1–MP7 (principle conformance) plus M11 (architect inspection). The +measurement criteria M0/M0b/M0c exist for honesty; M1 and M2 are reporting obligations that +cannot fail on a number.** + +- [ ] **MP — every prompt-bearing file conforms to P1, P2, P3, P4, P6 and P7** (P5 declared + N/A with reason), judged per file by the architect against the verbatim principle table. + A file passes on conformance regardless of its word count. Non-conformance at any size is + a failure. +- [ ] **M11 — architect personal inspection of every changed file.** Each implement phase ends + with a **per-file manifest**: path · old word count · new word count · principles applied · + one-line rationale for what was cut. The architect reviews **actual old-vs-new diffs, file + by file** — not samples, not summaries, not CMAP-mediated — before the phase advances. + Batches are humanly sized: **≤12 distinct files per review batch**. Inspection is over the + **~66 distinct content decisions**, not all 131 file-diffs — 60 `codev/protocols` copies + are byte-identical mirrors of their skeleton twins, so reviewing both would be re-reading + the same bytes; twin sync is verified mechanically by **T7** instead. Any file the + architect judges non-conformant returns to the builder before the phase advances. +- [ ] **M0 — the metric measures what is served.** The corrected script (a) sources phase + prompts from the directory `loadPromptFile` resolves; (b) resolves **per-file through the + full four-tier chain** as `resolveCodevFile` does; (c) counts the inlined + `roles/builder.md`; (d) counts hot-tier `@import` transclusion **and corrects the stale + inlining comment**; (e) expands `{{> …}}` includes; (f) reports exclusive bucket subtotals + and derived audience loads separately; (g) reports **total authored prompt-surface words**, defined + unambiguously as **physical files on disk** — every `.md` under `codev/protocols`, + `codev-skeleton/protocols`, `codev/roles`, `codev-skeleton/roles`, `.claude/skills`, plus + `CLAUDE.md` and `AGENTS.md`, each counted once, **no deduplication of twins and no + transclusion expansion**. This is deliberately a *different* basis from the always-on + buckets (which dedupe twins and expand `@import`/`{{> …}}`), because its job is to detect + relocation — content moved out of an always-on file must still show up somewhere. Both + figures are reported side by side and labelled with their basis, so T11 and T15 have + deterministic expected values. Tests assert (a) and (b) against the real resolver. +- [ ] **M0b — the corrected instrument and baseline land on `main` early**, as a small + standalone PR (precedent #1290), per **Rollout**. +- [ ] **M0c — deleted words are distinguished from relocated words.** P3 authorizes moving + content to skills, and relocation scores identically to deletion under an always-on-only + metric. The review **decomposes the reduction into deleted vs relocated**, evidenced by + M0(g). This is a reporting obligation, not a threshold. +- [ ] **M1 — before/after figures measured and published.** Same corrected script both sides; + both committed as generated artifacts; per-audience loads reported. **No pass/fail + threshold attaches to any of them.** +- [ ] **M2 — per-file word counts appear in every manifest** (M11), so the architect sees the + size effect of each decision while judging conformance. **Reporting only; no ceilings.** +- [ ] **M2b — CLAUDE.md stays human-readable.** The rewritten file retains a navigable heading + structure and is reviewed by the architect for human usability — twin-parity bytes are not + a readability check. +- [ ] **M3 — sweep completeness.** Surfaces enumerated from disk (both trees, unioned), never a + hardcoded list; every existing surface is rewritten and inspected. Includes + `codev/protocols/release/protocol.md`, which has no skeleton twin. +- [ ] **M4 — scar rules intact.** Eight canonicals byte-identical on every registered surface; + test fails on reword or deletion; count pinned at 8. +- [ ] **M5 — no capability lost, proven against the prompt text.** A committed + `capability-inventory.json` extracted pre-rewrite with explicit recognition rules — + artifact paths, gate names, signal names (`` tags), porch check names, + notification triggers — normalized (lowercase, strip backticks/punctuation, dedupe). + + **The inventory is over the resolved, expanded prompt surface, not over `protocol.json` or + source call sites** — extracting gate names from an unchanged `protocol.json` would report + every capability present even if every corresponding instruction vanished from the served + prompts. + + **Representation, defined so M5 does not contradict P6.** P6 explicitly permits replacing + narrated gate/check/phase names with a reference to the structured source. A naive + "every extracted name must still appear in prose" rule would make a *conformant* P6 + rewrite fail. A capability is therefore **represented** if either (a) it is named in + served prompt text, **or** (b) the served text carries an explicit, resolvable reference + to the structured source that defines it (e.g. "gates, checks and phase order are defined + in `protocol.json`; read it") **and** that source still defines it. (b) satisfies M5. + + **Detection limit, stated rather than implied.** Set-inclusion over names catches the + *deletion* of a capability; it does **not** catch *inversion or gutting* of the + instruction attached to one. "A gate message is a notification to the human, not + authorization" could collapse to a bare mention of the gate name and still pass M5 and + T6. That gap is covered by **M11** (the architect reads the actual diff) and **O4** (the + A/B's zero-tolerance compliance checklist), not by M5. To narrow it further, a small + hand-curated set of **semantic invariants** — human-gate semantics, artifact-contract + obligations, the scar prohibitions — is asserted as *behaviour present in the served + text*, not as name presence. This is a short list by design; the honest claim is that M5 + is a deletion detector, not a meaning detector. + + **Severity**: a removal is a hard failure **unless** the retired name appears in a + committed `codev/resources/1280-retirements.md` in the same commit, naming the capability, + why it is obsolete, and the architect approval. +- [ ] **M6 — the dead tree is gone, with its consumer handled.** + `codev-skeleton/porch/prompts/` deleted. Verification is **not** a bare grep: an + untruncated repo-wide search reconciled against the full hit list shows zero *runtime* + consumers, and the one **test** consumer (`review-prompt-routing.test.ts:29`, a Spec 987 + protection) is updated under M10 naming Spec 987. +- [ ] **M7 — A/B non-inferiority passes** per the pre-registered decision rule; gates + `verify-approval`. Mandatory per the charter: behavioural outcomes are the evidence the + principles are working. +- [ ] **M8 — behavioural baseline re-run.** `measure-prompt-behavior.ts` re-run and committed; + B1 compared directionally to 51.88% (n=160). +- [ ] **M9 — rollback rehearsed** by group, per **Rollback Plan**. +- [ ] **M10 — prose-pinned test re-baselining is deliberate and enumerated.** ~25 test files + assert exact prose in the surfaces being rewritten; the hardest is + `agent-farm/__tests__/baked-decisions.test.ts:143-148`, enforcing a **pure-addition diff** + against committed baselines for `protocols/{spir,aspir,air}/builder-prompt.md`. Also + `bugfix-744-spir-pr-strategy.test.ts`, `spec-1273-wait-discipline-docs.test.ts`, + `bugfix-619-aspir-prompt.test.ts`, `template-delivery.test.ts`, + `framework-ref-audit.test.ts`, `governance-sweep.test.ts`, + `review-prompt-routing.test.ts`. **Each assertion is a prior spec's protection encoded as a + grep, so retiring one is a governance act** — a principle matter, not a size matter. Every + modified or retired assertion is listed in the review with (i) the originating spec, + (ii) whether the protected behaviour survives in the rewritten prose, (iii) the replacement + assertion, or an explicit architect-visible retirement. Silent deletion to make the suite + green is a project failure, not a test fix. +- [ ] **M12 — no release between the rewrite merge and the SHIP verdict.** M7 gates + `verify-approval`, not merge, so the rewritten skeleton is on `main` — and therefore + shippable to adopters via `codev update` — before the A/B has validated it. "Pin the prior + version" is a reactive remedy for a problem this criterion prevents. If a release must cut + inside the window, it ships from a commit predating the rewrite merge. +- [ ] All tests pass **after M10's enumerated re-baselining**; no coverage reduction. New tests + cover M0, M3, M4, M5. +- [ ] Documentation routed by tier; `CLAUDE.md`/`AGENTS.md` byte-identical. + +## Constraints + +### Technical Constraints + +Verbatim from issue #1280's **Baked Decisions** — fixed, not re-litigated by this spec, the +plan, or CMAP reviewers: + +- **All prompt consumers are frontier models** (Claude 5, GPT 5.6, Gemini 3.6 class). No + weak-model tier, no fallback scaffolding variant, no tiering mechanism. One form. +- **Scar rules are exempt and verbatim** — the eight compressed canonicals developed in Spec + 1252 Phase 5 (six repo rules + shellper verified-orphan + Tower-restart permission) ship with + the rewrite; the registry/enforcement concept from 1252 is rebuilt fit-for-purpose around the + post-shrink surface, not before it. +- **Validation is A/B, not observational**: same issues executed by builders on old vs new + prompts, compared on outcomes (gate friction, review rounds, correctness). Spec 1252's M12 + established that observational baselines (n=17) can only detect large regressions — + insufficient at deletion scale. The A/B design is a first-class spec section. +- Spec must define a rollback story (prompt surfaces are files; reverting is cheap — say so + concretely). + +Arising from the repository and architect rulings: + +- **Principle conformance is the acceptance criterion; size is not.** +- **Both trees**; `CLAUDE.md` ≡ `AGENTS.md`. +- **Four-tier resolution, per file.** No fetch-by-path instruction for a file that may not exist + on disk (deliver-don't-fetch). +- **No porch behaviour changes** — content rewrite plus a measurement-script fix. +- **Template shape is governed by two separate constraints** (porch's 4 headings; the consult + type's advisory 20) which must not be conflated. + +### Business Constraints + +- Two human gates (`spec-approval`, `plan-approval`) plus `pr`; the A/B verdict is the + architect's call. +- **The architect personally inspects every changed file** (M11) — this is the binding + throughput constraint on phase design. +- Adopters consume the skeleton via `codev update`, so rollback is a revertible unit per group. +- Scar wordings are architect-ratified; a builder may not compress them. +- **The corrected instrument ships early, as its own PR** (M0b). + +## Assumptions + +- The eight scar wordings on `builder/spir-1252` remain the ratified set. +- Frontier-model behaviour is stable across the A/B window (mitigated by pairing, same base + commit, pinned model/config versions). +- `builder/spir-1252` stays undeleted — sole source of the ratified registry. +- CMAP reviewers are blind to the builder's prompt surface by construction. +- The architect has capacity for ~66 file inspections across the implement phases; if not, the + phase count grows rather than the batch size. + +## Solution Approaches + +### Approach 1: In-place principle rewrite, surface by surface (RECOMMENDED) + +Rewrite each file to P1–P7, keeping file layout, resolver and porch untouched. Templates become +interfaces (P2); how-to content moves to skills (P3, P4); prose restating `protocol.json` +becomes a reference to it (P6); worst-case padding goes except the scar rules (P7). The +instrument is corrected first. + +**Pros**: zero mechanism risk; every change is a reviewable text diff — which is exactly what +M11's per-file inspection requires; rollback granularity equals cut granularity. +**Cons**: discipline-dependent; ~66 distinct files to inspect; collides with ~25 prose-pinned +test files (M10). +**Complexity**: Medium · **Risk**: Low-Medium + +### Approach 2: Generate prompts from `protocol.json` + +Synthesize phase prompts at runtime from the state machine, with a small per-phase prose delta. + +**Pros**: the purest expression of P6; structurally prevents drift. +**Cons**: introduces a code path between authoring and serving, changes porch behaviour (an +explicit constraint), and **defeats M11** — the architect cannot inspect old-vs-new diffs of +files that no longer exist as authored artifacts. Couples the rewrite to a mechanism change, +making any A/B regression un-attributable. +**Verdict**: right idea, wrong project — and now also incompatible with the inspection mandate. + +### Approach 3: Shared kernel + per-protocol deltas + +One protocol-agnostic builder kernel included by every protocol, plus a short per-protocol delta. + +**Pros**: directly serves P4 (stop repeating yourself across ten protocols). +**Cons**: 1252 proved dedup buys ~7% on *served* words — an include expands, so it changes +ownership, not the reader's bill. Risks re-creating the shadow-tree bug class, where one edit +silently changes ten protocols' served prompts — which also complicates per-file inspection. +**Verdict**: adopt selectively *inside* Approach 1 where a kernel genuinely serves P4, and only +where the architect can still see what each protocol serves. + +## Open Questions + +### Critical (Blocks Progress) + +*None outstanding.* + +### Important (Affects Design) + +- [ ] **Batch size for M11.** ≤12 distinct files per review batch is this spec's proposal, + giving ~6–7 inspection batches. The architect may want smaller. +- [ ] **Do SPIR templates survive as interfaces, or disappear?** P2 says design interfaces; + porch requires only 4 headings, while the `spec-review` consult type advisorily expects 20. + Recommendation: survive as heading interfaces. +- [ ] **Is the hot tier in scope?** It is the one surface already written to these principles. + This spec leaves it unchanged; a reviewer may argue P3 applies to it too. +- [ ] **`roles/architect.md`** — outside 1252's analysis entirely. Confirm nothing in it is + load-bearing for multi-architect coordination (Specs 755/786/823) before rewriting. +- [ ] **A/B sample size.** ≥6 pairs specified; the architect sets the ceiling given the + inspection load already on them. + +### Nice-to-Know (Optimization) + +- [ ] Does rewriting consult-type prompts move CMAP verdict *quality*, or only cost? +- [ ] Are the ~17,000 words of `.claude/skills/` the right destination for P3/P4 relocations, + or does that surface need its own conformance pass? + +## Performance Requirements + +No runtime-performance requirement. Reporting obligations only: + +- Before/after `ALWAYS_ON_WORDS` and per-audience loads, measured by the corrected script and + committed (M1). +- Deleted-vs-relocated decomposition (M0c). +- Per-file word counts in every manifest (M2). +- Measurement runtime <5s and deterministic — same commit ⇒ byte-identical output. + +None of these is a threshold. + +## Security Considerations + +- **The scar rules are the security surface.** All eight guard irreversible acts. Weakening one + is the highest-severity failure this project can produce — hence verbatim carriage, the + explicit P7 exception, byte-identical enforcement, and a hard rollback trigger on any observed + violation. +- **Human-gate integrity.** The rewrite must not weaken "a gate message is a notification to the + human, not authorization." Gate semantics are content, and this project edits content. +- **No secrets in prompt surfaces** — existing property, re-verified. +- **Adopter blast radius.** Skeleton changes ship on `codev update`; a weakened prohibition + propagates silently. Hence grouped, rehearsed rollback. + +## Test Scenarios + +Tests verify *mechanical* properties. **Principle conformance is judged by the architect (M11), +not asserted by a test** — that is the point of the revised acceptance model. + +### Functional Tests + +1. **T1 — Instrument sources the served directory** (M0 item a), asserted against the real + resolver. +2. **T1b — Instrument resolves per-file, four-tier** (M0 item b). A fixture with a `.codev/` + override of *one* prompt while others resolve from the skeleton. +3. **T2 — Include expansion (phantom-savings proof).** Moving text from prompt into template + produces **zero** change in the reported total. +4. **T3 — Word-count reporting.** The script emits a per-file table covering every resolved + surface. **Asserts completeness of the report, not any ceiling.** +5. **T4 — Scar integrity (M4).** Every canonical byte-identical on every registered surface; + count pinned at 8; reword or deletion fails. +6. **T5 — Capability inventory (M5).** Post-rewrite extraction over served prompt text ⊇ + pre-rewrite; unlisted removals fail. +7. **T6 — Sweep completeness (M3).** Surfaces enumerated from disk across both trees and + unioned; absence of `prompts/`/`consult-types/` for a protocol that has none must **not** + fail; a new surface fails until rewritten and inspected. Covers `release`. +8. **T7 — Twin parity.** `CLAUDE.md` ≡ `AGENTS.md`; every `codev/protocols` file **that has a + skeleton twin** is byte-identical to it — the assertion runs on the intersection, so the + three local-only files (`maintain/templates/audit-report.md`, + `maintain/templates/lessons-learned.md`, `release/protocol.md`) are not failures. + **Load-bearing for M11**: it is what makes inspecting ~66 files instead of 131 sound. +9. **T8 — Dead-tree removal (M6).** Tree absent; no runtime reference; the Spec 987 routing test + updated per M10. +10. **T9 — Live spawn probe.** A builder spawned end-to-end on the rewritten surface receives a + spawn prompt containing every element of the artifact contract, and its first `porch next` + returns a well-formed task. ("It compiled" is not "it works.") +11. **T10 — Rollback rehearsal (M9).** Reverting a rollback group restores it byte-for-byte and + leaves the suite green. +12. **T11 — Bucket and audience reporting** (M0 item f). Exclusive bucket subtotals sum to the + authored total; derived audience loads are asserted against the stated formulas, never a + naive sum. +13. **T15 — Relocation visibility (M0c).** A fixture moving a block from an always-on surface + into `.claude/skills/` shows always-on falling **and** total-authored holding steady. +14. **T16 — Manifest completeness (M11).** Every file changed in a phase appears in that phase's + manifest with all four fields. A changed file missing from the manifest fails the phase — + the architect cannot inspect what is not listed. + +### Non-Functional Tests + +1. **T12 — Determinism.** Two runs at the same commit emit byte-identical output. +2. **T13 — Behavioural re-measurement (M8).** `measure-prompt-behavior.ts` re-run post-merge with + self-exclusion; B1/B2/B4 committed and compared directionally. +3. **T14 — A/B execution (M7).** The full pre-registered protocol below, including a + **pre-flight assertion per pair**: every surface under test resolves from tier 2 (`codev/`), + i.e. no skeleton file lacks a `codev/` twin. If that fails, the control arm would silently + serve the *new* skeleton and the pair is void — so the pre-flight aborts the pair rather than + producing a comparison that looks valid and is not. + +## A/B Validation Design + +*(First-class section per Baked Decision 3, and unchanged by the acceptance-model revision: the +A/B is how we learn whether principle conformance actually holds up behaviourally. A +**non-inferiority** trial — the claim under test is "a principle-conformant prompt surface does +not degrade outcomes", not "it improves them".)* + +### Unit and arms + +The unit is an **issue-pair**: one GitHub issue executed twice, by two freshly-spawned builders +in separate worktrees. + +**Arms are a prompt-only overlay on one source snapshot — not two different commits.** An +earlier draft said both "from the same base commit" *and* "control = pre-rewrite commit, +treatment = post-rewrite commit." Those are incompatible, and the naive reading also lets later +pairs inherit source changes the pinned control commit does not have. The construction is: + +1. Both arms branch from the **same source commit** `S` (current `main` at pair start). +2. The **treatment** arm uses `S` unmodified — the rewritten prompt surface. +3. The **control** arm applies one **prompt-only overlay commit** on top of `S`: the rollback + groups (G2–G6) reverted, restoring the pre-rewrite prompt surface and touching nothing else. +4. Each run records **both hashes** — source commit `S` and a `prompt-surface hash` (a digest + over every file in the disposition table) — so any later audit can prove the arms differed + in prompts and only in prompts. + +This keeps source identical within a pair, lets `S` advance between pairs without contaminating +comparisons, and makes "no code differs" literally true rather than approximately true. + +**Precondition**: control-arm isolation depends on tier 2 (`codev/`) shadowing the installed +skeleton (tier 4) for every surface under test — true today (0 skeleton files lack a `codev/` +twin) but silently breakable if the rewrite *deletes* a `codev/` file while keeping its +skeleton twin. Asserted pre-flight in T14. + +### Sample and eligibility + +- **≥6 pairs (12 runs)**, stratified: ≥3 SPIR/ASPIR and ≥3 lighter protocols (BUGFIX/AIR). +- Issues drawn from the backlog, selected **before** either arm runs, then frozen. +- **Eligibility exclusion**: an issue is ineligible if it modifies any surface under test — + otherwise the treatment arm's prompt surface is simultaneously instrument and subject. + +### Contamination controls + +- **Pin the environment**: model ids and reasoning efforts, consult backend versions, and + `.codev/config.json` frozen for the trial window and recorded with the results. +- **Arm isolation**: the second arm must not see the first arm's branch, PR, or thread. + Sequential with the intervening branch unpushed, or concurrent in isolated worktrees; which + was used is recorded per pair. +- **Arm order alternates** per pair. +- **Recording**: one committed artifact (`codev/resources/1280-ab-results.md`), a row per run — + pair id, arm, protocol, issue, base commit, order, isolation mode, every outcome, and any + exclusion with its reason. + +### Execution and sequencing + +- **M7 gates `verify-approval`, not the PR merge.** The rewrite PR merges on principle + conformance (MP, M11) plus the mechanical criteria; the A/B then runs against merged `main` as + treatment and a pinned pre-rewrite commit as control. A SHIP failure means rolling back a + merged change — which is what the grouped, rehearsed rollback plan is for. +- **Arm disposition**: the treatment arm's PR is the merge candidate; the control arm's closes + unmerged once outcomes are recorded. **~6 of 12 runs produce merged work, not 12.** +- **Architect load**: 6 pairs with ≥3 SPIR-class implies up to **~24 gate approvals and 12 PR + reviews**, each SPIR gate requiring O1 rubric scoring at approval time — *on top of* M11's ~66 + file inspections. This is the project's binding constraint and why the pair count is the + architect's call. +- **Release hold (M12).** Because M7 gates `verify-approval` rather than merge, the rewritten + skeleton reaches `main` before the A/B runs, and adopters would consume it on the next npm + release. "Pin the prior version" is purely reactive. Therefore: **no `@cluesmith/codev` + release between the rewrite merge and the SHIP verdict.** If a release must cut inside that + window, it ships from a commit predating the rewrite merge. + +### Pre-registered outcomes + +| ID | Outcome | Instrument | Role | +|---|---|---|---| +| **O1** | Gate friction | Architect scores each gate at approval time on a 3-item rubric — *artifact complete as specified? / rework required before approval? / clarifying message needed?* — each 0 (none) / 1 (minor) / 2 (blocking) | advisory + tripwire | +| **O2** | Review rounds | Iterations to terminal state per phase from `status.yaml`; CMAP REQUEST_CHANGES rate (comparable to B1 = 51.88%) | gate | +| **O3** | Correctness | **At the SHIP decision**: architect PR-review findings by severity, observable pre-merge on both arms. **Post-merge defects are excluded from the SHIP gate** and act as a **14-day rollback signal** | gate (pre-merge) + rollback signal (post-merge) | +| **O4** | Protocol compliance | Binary per-run checklist: required artifacts with required headings · stopped at every human gate · no `status.yaml` hand-edit · no `git add -A` · no scar violation · thread committed | **zero tolerance** | +| **O5** | Cost & duration | Tokens, wall-clock, `consult stats` delta | advisory | + +### Blinding + +CMAP reviewers are blind by construction. The architect is not and cannot be; mitigations: O2 +and O4 are extracted mechanically from committed artifacts, and O1/O3 are scored against a +rubric written **before** any run. + +### Decision rule (pre-registered) + +**SHIP** iff all of: + +1. **O4 = zero violations** in the treatment arm. Any scar violation, skipped gate, or missing + required artifact is an immediate hard stop. +2. **O2**: treatment mean review rounds ≤ control **+ 0.5 rounds/phase**, and treatment + REQUEST_CHANGES rate ≤ control **+ 10 percentage points**. +3. **O3 (pre-merge part only)**: no treatment-arm finding of severity ≥ "would block merge" + absent from its paired control run. +4. **O1 tripwire**: no pair where the treatment arm scored **2 (blocking)** at a gate its control + scored 0, for the same reason. If O1 scoring is incomplete, O1 reports incomplete and SHIP + rests on 1–3. + +Otherwise **HOLD** (fix and re-run the failing pairs) or **ROLLBACK**. + +### Honest power statement + +With n=6 pairs this detects only **large** effects — roughly a doubling of review rounds or a +≥20-point REQUEST_CHANGES shift. It cannot certify the absence of a subtle regression, and this +spec does not claim it can. It is nonetheless stronger than 1252's observational baseline, +because each pair is matched on the issue itself. **O4's zero-tolerance criterion is where the +real protection lives**: compliance is binary, observable in every run, and is the failure mode +an aggressive rewrite would plausibly cause. + +## Rollback Plan + +Prompts, included templates, registry mappings and integrity tests are coupled, so rollback is by +**group** — each internally consistent and independently revertible: + +| Group | Contents | +|---|---| +| **G1 instrument** | measurement script + its tests + baseline artifacts | +| **G2 shared** | `CLAUDE.md` + `AGENTS.md` + hot-tier wiring | +| **G3 builder-spawn** | `roles/builder.md` + all `builder-prompt.md` + all `protocol.md` + their prose-pinned tests | +| **G4 phase** | all `prompts/*.md` + their `templates/*.md` + porch check expectations | +| **G5 consultant** | `roles/consultant.md` + all `consult-types/*.md` | +| **G6 architect** | `roles/architect.md` + relocated skill content | +| **G7 scar registry** | `scar-rules.yaml` + its enforcement test | + +- **Dependency rule**: reverting **G7** requires reverting every group carrying scar text (G2, + G3, G4, G6). All other groups are mutually independent. +- **Mechanism**: `git revert` restores prior bytes. No migration, state, schema, or data. + Rehearsed under T10 before the PR merges. +- **Blast radius**: effective for the next spawned builder — in-flight builders keep the surface + they were spawned with. For adopters the revert ships in the next release; an adopter can also + pin the prior `@cluesmith/codev` version. +- **Triggers**: (a) any O4 violation — immediate; (b) an observed scar violation in any real + project post-merge; (c) O2/O3 outside pre-registered margins; (d) the 14-day post-merge O3 + window; (e) architect judgment at any gate. +- **Partial rollback is the expected shape** — revert the offending group, keep the rest. +- **Cost**: one revert, one release. No irreversible step exists anywhere in this project. + +## Dependencies + +- **External Services**: none. (`gh` for issue/PR reads during the A/B; consult backends + unchanged.) +- **Internal Systems**: four-tier resolver (`lib/skeleton.ts`); porch prompt composition + (`commands/porch/prompts.ts`) — read, not modified; managed-block hot-tier wiring + (`lib/managed-block.ts`); role injection (`agent-farm/commands/spawn-worktree.ts`); + consult-type resolution (`commands/consult/index.ts`); both measurement scripts. +- **Artifacts**: `builder/spir-1252` (ratified registry — must not be deleted); + `codev/resources/1252-*.md` baselines. +- **Test suites**: the ~25 prose-pinned files in M10 — the rewrite cannot land without + deliberately re-baselining them. +- **Architect availability**: M11's per-file inspection is a hard dependency on one person's + time, not a background activity. +- **Libraries/Frameworks**: none new. + +## References + +- Issue #1280 (charter); #1279 (dead spec/review templates); #1276, #1277 (filed by 1252 — + superseded here or out of scope); #1032/#1033 (agy `--type` review limitation). +- PR #1278 (Spec 1252, closed unmerged) and `builder/spir-1252`; PR #1290 (early-PR precedent). +- `codev/reviews/1252-prompt-architecture-single-own.md`; `codev/resources/1252-*.md`; + `codev/state/spir-1252_thread.md`. +- *The new rules of context engineering for Claude-5-generation models* — + https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models + (P1–P7 quoted verbatim in **Desired State**). +- `codev/resources/arch.md`; `codev/resources/lessons-learned.md`. + +## Risks and Mitigation + +| Risk | Probability | Impact | Mitigation Strategy | +|------|------------|--------|-------------------| +| "Principle conformance" is judged inconsistently across ~66 files | **High** | High | P1–P7 quoted verbatim with a per-file conformance question each; one architect judges all of them; the manifest records which principles were applied per file | +| A deleted instruction was load-bearing; loss is silent | Medium | High | M5 inventory over served prompt text; O4 zero-tolerance; M11 per-file inspection; grouped rollback | +| A scar rule weakened or dropped under P1/P7 | Low | **Critical** | The P7 exception is stated explicitly; verbatim carriage; byte-identical enforcement pinned at 8; architect-only rewording; hard rollback trigger | +| Prose-pinned tests silently gutted to go green | **High** | **High** | M10: every retired assertion named with its originating spec and its behaviour re-asserted or explicitly retired | +| M11 inspection load stalls the project | **High** | Medium | ≤12 files per batch; inspection scoped to ~66 distinct decisions via T7 twin parity, not 131 diffs; phase count grows rather than batch size | +| Sweep misses a protocol, tree, or surface | High | Medium | T6 enumerates from disk (both trees, unioned); T7 twin parity; T16 manifest completeness | +| Relocation reported as deletion | Medium | Low | M0c + M0(g) + T15 (now a reporting-honesty concern, not a grading one) | +| A/B underpowered; subtle regression ships | Medium | Medium | Power statement; O4 binary compliance; T13 post-merge behavioural re-measurement | +| A further instrument defect ships undetected | Medium | Medium | T1/T1b/T11/T15 assert the instrument against the live resolver; M0b puts it under public review early; the script gets its first tests | +| Surface re-grows after the project | Medium | Medium | No ceiling test exists under the revised model; re-growth is caught by the same principle review at the next MAINTAIN | +| `builder/spir-1252` deleted, losing the registry | Low | Medium | The branch exists on `origin` (verified), not only locally; registry content is also quoted in this project's thread; rebuilt registry committed to `main` early | + +## Expert Consultation + +**Round 1** — 2026-07-31 · Codex (GPT-5.6 Sol) + Claude Opus 5 · both REQUEST_CHANGES (HIGH) · +**13 findings, none disputed**. +**Round 2** — 2026-07-31, architect-directed re-review · same models · both REQUEST_CHANGES +(HIGH) · **9 findings, none disputed**. +**Round 3** — 2026-07-31, on the acceptance-model revision · Codex REQUEST_CHANGES (HIGH, 5) · +Claude COMMENT (HIGH, 6) · **11 findings, none disputed**. Both independently caught a +twin-file mislabel in this spec's own inventory (0 files differ; 3 are twinless — my `cmp -s` +loop conflated "differs" with "absent"), and both landed on M5's weakness from different angles +(P6 conflict / inversion-not-detected), which composed into one fix. + +Every finding was verified against source (both arithmetic claims independently recomputed) +before acceptance; all are folded into the sections above. The finding-by-finding record is in +`codev/projects/1280-prompt-surface-judgment-not-ru/1280-specify-iter{1,2}-rebuttals.md`. + +Five corrections were errors in this spec's own analysis, four sharing one root cause — +**trusting a convenient signal instead of checking the authoritative thing**: a truncated grep; +skeleton-only protocol enumeration; the measurement script's stale comment; and an overloaded +`cmp` exit code read as "differs" when it also means "absent". That is the sweep-scope class +1252 named as its dominant review cost, it is why M3's "enumerate from disk" is a **test** +rather than an instruction, and at five instances it is a `lessons-learned.md` entry the review +phase will route. + +Gemini/`agy` did not participate: the known `--type` review limitation (#1032/#1033). Per +current lane policy this 2-way review is correct. + +## Approval +- [ ] Technical Lead Review +- [ ] Product Owner Review +- [ ] Stakeholder Sign-off +- [ ] Expert AI Consultation Complete + +## Notes + +**On the acceptance model.** An earlier draft made ">50% reduction" the headline criterion, with +per-surface word ceilings as the binding test. Waleed's redirect removed that: *"I don't think +the goal should be a particular size. That's not the right criteria. It should be to stick to +the principles outlined in the blog post."* This is a better criterion for a reason worth +recording — a word ceiling can be met by a file that still narrates procedure, and can be missed +by a file that is perfectly conformant but genuinely needs the words. Size was a proxy; +conformance is the thing. + +**What this project deliberately does not do.** No prompt generator (Approach 2 — now also +incompatible with M11), no tiering of any kind (Baked Decision 1), no porch behaviour changes, +and not 1252's full enforcement machinery — only the scar-integrity check the rewrite makes +necessary. Enforcement built around a still-moving surface is enforcement built twice. + +**On the deferred decision from 1252.** The architect's pr-gate ruling was that structural +machinery is not worth carrying for a surface about to be rewritten. That sequencing is honoured: +rewrite first, then enforce what remains. diff --git a/codev/state/spir-1280_thread.md b/codev/state/spir-1280_thread.md new file mode 100644 index 000000000..44dad0141 --- /dev/null +++ b/codev/state/spir-1280_thread.md @@ -0,0 +1,520 @@ +# spir-1280 — Prompt surface: judgment-not-rules rewrite (>50% always-on reduction) + +## Specify phase — opening survey (2026-07-31) + +Read the required prior art before drafting: `codev/reviews/1252-prompt-architecture-single-own.md`, +`1252-word-baseline.md`, `1252-word-after-phase7.md`, `1252-behavior-baseline.md`, +`scripts/measure-prompt-surface.sh`, issue #1279, and the ratified scar registry from +`builder/spir-1252:codev/resources/scar-rules.yaml` (all eight rules recovered verbatim). + +### Finding that reshapes the spec: the committed measurement script measures a dead directory + +`scripts/measure-prompt-surface.sh` computes `PORCH_PROMPT_MEAN` over +`codev-skeleton/porch/prompts/*.md` (10 files, mean 400w). The live code +(`packages/codev/src/commands/porch/prompts.ts:78`, `loadPromptFile`) resolves +`protocols//prompts/.md` — a *different* directory. Repo-wide grep +(excluding node_modules/dist/.git) finds no code reading `porch/prompts`; the only hits +are historical spec/plan prose. That tree is a Ralph-SPIR-era leftover: its `specify.md` +opens "You are the **Spec Writer** hat in a Ralph-SPIR loop." + +Consequences: +- The real SPIR phase prompts (expanded with their `{{> templates/...}}` includes) are + specify 1402, plan 1169, implement 1065, review 1957 → **mean 1398**, not 400. +- The proxy also omits `roles/builder.md` (1837w), which the spawn wrapper inlines + verbatim (verified against this worktree's own `.builder-role.md`). +- **The metric is blind to this project's single biggest target.** Cutting the phase + prompts would not move `ALWAYS_ON_WORDS` at all under the current script. + +Corrected always-on model (SPIR, I=10 task deliveries): 5,815 + 6,364 + 21,340 = **33,519** +vs the script's reported 21,702. Same methodology (served/expanded words), corrected inputs. +Spec makes fixing this M0 — before/after both measured with the corrected script, so the +>50% target is unaffected in kind, only in denominator. + +Notified the architect; not blocking on it (the fix serves the stated intent of the goal +rather than contradicting a Baked Decision). + +### Per-surface sizes captured for the cut plan + +| Surface | Words | Notes | +|---|---:|---| +| CLAUDE.md / AGENTS.md | 5,815 each | byte-identical twins; hot tier inlined | +| roles/builder.md | 1,837 | inlined into every spawn | +| spir/protocol.md | 3,703 | inlined into every spawn | +| spir/builder-prompt.md | 824 | spawn wrapper | +| spir/prompts (expanded) | 1,398 mean | ×I per project — the dominant term | +| spir/templates | 632/649/641 | pulled in by specify/plan/review prompts | +| hot tier | 736 | capped, judgment-shaped, keep | +| spir/consult-types | 2,154 (5 files) | reviewer-side always-on, unmeasured today | + +Cut plan and A/B design go in the spec. + +### Architect ruling (2026-08-01) — M0 endorsed, two additions + +Architect independently verified both claims against source (script line 89; the live +resolver at `prompts.ts:78`) and endorsed M0 as specced: fix the instrument first, +measure before AND after on it, >50% target unchanged against the corrected 33,519-word +baseline. Two additions folded into the spec: + +1. **M0b** — the corrected script + corrected baseline land on `main` in a small early + standalone PR (precedent #1290), not at the end of the branch, because the 1252 + baseline artifacts cite the wrong figure and are shared knowledge other work reads. + Recorded as an architect-requested PR under the issue's PR strategy. +2. **Principle 7, "the instrument is part of the deliverable"** — written into the spec's + Problem Statement with the lineage: this is the SECOND measurement defect in the 1252 + line (the first: 1252 originally shipped with no measurement plan at all, caught at a + human gate, not by CMAP). Neither was caught by reading the instrument's code; both by + asking what it claims to measure. + +Spec drafted (5,800w) — carries the three architect-mandated designs (per-surface cut +plan with word targets, A/B non-inferiority design with a pre-registered decision rule, +scar-rule carriage plan) plus the rollback story. Headline: 33,519 → ≤15,900 (−52.7%), +with the phase-task term (71% of the post-rewrite budget) dominated by the hot tier, +which is explicitly exempt from cuts. Signalling SPEC_DRAFTED to porch for 3-way review. + +### Iteration 2 — CMAP round 1 + scope directive (2026-07-31) + +**Both reviewers REQUEST_CHANGES, both HIGH confidence, both worth every word.** Verified +each factual claim against source before acting (lessons-critical: summaries are evidence, +not ground truth). All findings accepted; **no rebuttal filed** — nothing was a false +positive. + +Two findings were errors in *my own* Current State, and both are the class this spec's +principle 7 exists to catch: + +1. **Stale hot-tier claim.** I wrote "CLAUDE.md (hot tier inlined)". Since #1119 + (`managed-block.ts:59-67`) it carries `@import` lines that Claude Code transcludes at + session launch. So `wc -w CLAUDE.md` = 5,815 **excludes** 736 always-loaded words. + I inherited this from the measurement script's own stale comment (lines 44-47) — i.e. I + trusted the instrument while writing a spec about auditing instruments. Baseline + 33,519 → **34,255**. +2. **Truncated grep → false claim.** I asserted "no code reads `porch/prompts`" from a + `grep -rn ... | head -20`. The 48-line full output contains + `review-prompt-routing.test.ts:29`, a Spec 987 protection that reads + `codev-skeleton/porch/prompts/review.md`. Tree is still dead as prompt surface, but M6's + verification method was wrong. **Never conclude "no consumers" from truncated output** — + this is the sweep-scope failure class 1252 named as its dominant review cost. + +Claude's biggest structural catch: **~25 test files pin exact prose in the surfaces being +cut**, worst being `baked-decisions.test.ts:143-148` — a pure-addition diff against +committed baselines for three `builder-prompt.md` files, logically incompatible with +824 → ≤420. Each assertion is a prior spec's protection encoded as a grep, so retiring one +is a governance act. Became **M10**: every modified/retired assertion named with its +originating spec, its protected behaviour re-asserted or explicitly retired. Without this a +builder under pressure quietly deletes assertions to go green — the exact silent-capability +-loss this project exists to avoid. + +Codex's sharpest: ceilings were stated net-of-scar while M1 was gross (Claude independently +found the same, computing the gross landing at 50.2–51.4% — i.e. M1 would have *failed* +while every per-surface row passed). All ceilings are now gross; M2 is binding and M1 +derived; a 50–52% landing is a HOLD with named further-cut candidates. + +**Architect scope directive** (same window): scope is the ENTIRE prompt surface — architect +roles, builder roles + spawn wrappers, consultant/CMAP prompts, phase prompts + template +includes, protocol.md texts — and the instrument must segment by audience so a cut in one +segment masking growth in another is visible, not averaged away. Built the full inventory +(every prompt-bearing surface, resolver path, served word count) and folded it into Current +State; added per-segment targets and T11 (segment reporting, with a fixture proving one +segment growing while another shrinks is not netted to zero). + +Post-rewrite, by segment: builder 34,255 → 16,016 (−53.2%) · architect 8,599 → 3,336 +(−61.2%) · consultant 683 → 452 (−33.8%). No segment grows. Consultant cuts least because +`roles/consultant.md` is already lean at 252 — reported, not averaged away. + +Spec now 6,942w. Iteration 2 ready for CMAP round 2. + +### spec-approval gate reached (2026-07-31) + +`porch done` → `porch gate 1280`. **STOPPED, waiting for human approval.** Not running +`porch approve` — human-only gate. + +Two caveats flagged to the architect rather than silently accepted: + +1. **No CMAP round 2 ran.** Porch advanced straight from the iteration-1 rebuttal to the + gate, though `specify` has `max_iterations: 3` and only iteration 1 was consumed. The + revised spec (6,942w, substantially restructured — gross ceilings, M10, segmented + inventory, rollback groups) has therefore not been reviewed by anyone but me. Offered to + re-run round 2 if the architect wants it before the plan phase. +2. **Gemini/agy did not participate.** Porch's model set for this consult was codex+claude + only — the known `--type` review limitation (#1032/#1033). This was a 2-way, not a 3-way + review. Recorded in the spec's Expert Consultation section rather than left implicit. + +State at the gate: spec + review-response committed; worktree clean apart from the +harness-generated `.builder-*` files. + +### Pre-gate directives (2026-07-31) + +Architect directed two items before the gate goes to Waleed: + +1. **Run CMAP round 2** on the revised spec — budget allows (1 of 3 consumed), the revision + absorbed 13 findings incl. two factual corrections and a structural net→gross ceiling + change, and blast radius is every prompt every agent reads. Porch advancing without + round 2 was permissive machinery, not a judgment. Launched codex + claude in parallel + against the post-revision spec (7,306w). 2-way is correct per current lane policy; + gemini's absence needs no remedy. +2. **Answer the early-landing directive in the spec**, which I had carried only as criterion + M0b with no concrete plan. Fair catch — a criterion is not a rollout. Added + **Desired State → Rollout**: PR-1 contains the corrected script, its **first-ever tests** + (the instrument has none today — which is how three defects survived in something + "committed and reproducible"), the corrected segmented baseline artifact, and in-place + annotation of the two 1252 artifacts (originals preserved, marked superseded, reason + + pointer). Deliberately excluded: any prompt-surface edit, the scar registry, the dead-tree + deletion. Timing: end of first implement phase, before any cut. + + Verified the annotation is safe: no test asserts on either 1252 word-count artifact — the + frozen-sample test (`prompt-behavior-metrics.test.ts:184`) pins the *behavioural* sample, + a different instrument. So no re-baselining cost. + + Considered pushback and rejected it: the only argument against early landing is that the + corrected baseline is meaningful only alongside the cuts it scores, and that fails — + the baseline's value is that it is *pre-cut*, and publishing early is what makes the + eventual −53.2% claim checkable by someone who didn't watch it being produced. + +Porch remains at the spec-approval gate throughout; running consults manually does not move +state, and I am not touching the gate. + +### CMAP round 2 (2026-07-31) — the round porch skipped, and why it mattered + +Both REQUEST_CHANGES (HIGH). **Nine findings, none disputed, two of them arithmetic errors +that would otherwise have shipped.** The architect's insistence on running a round porch had +skipped is fully vindicated. + +The two that would have shipped: + +1. **M1's HOLD branch was unreachable.** M1 is fully derived from the M2 ceilings: meeting + every ceiling yields ≤16,016 (−53.2%), while the "HOLD at 50–52%" band needs + 16,442–17,128 — i.e. ceilings already exceeded and M2 already failing. Recomputed and + confirmed before accepting. Dead prose replaced with the reachable contingency + (denominator movement). +2. **M5 proved nothing about prompts.** It extracted gate/check names from `protocol.json` + and notification names from source call sites — files this project does not touch. Every + capability would have reported present even if every corresponding instruction vanished + from the served prompts. Now inventories the resolved, expanded *prompt surface* with a + contract-presence assertion per capability. + +Claude's best structural catch: **the metric cannot tell deletion from relocation.** +Principle 4 authorizes moving how-to content to skills, and relocation scores identically to +deletion under an always-on-only metric — the phantom-savings class T2 catches on the include +axis, unmonitored on the relocation axis. −53.2% is equally consistent with −30% deleted + +−23% relocated, and only deleted content supports the "crowds out judgment" claim. Added M0(g), +M0c, T15. + +Codex's best: **"all ten protocols in both trees" is impossible** — `release` is project-local +(10 in `codev/`, 9 in the skeleton), and four protocols intentionally have no `prompts/` or +`consult-types/`. Which exposed that **my inventory missed `release/protocol.md` (1,626w) +entirely**, because I enumerated `codev-skeleton/protocols/*/`. + +**Fourth self-audit finding of this spec phase, and three share one root cause**: enumerating +from a convenient source instead of the authoritative one (truncated grep → false "no +consumers"; skeleton-only enumeration → missing protocol; script's stale comment → wrong +baseline). This is exactly the sweep-scope class 1252 named as its dominant review cost, and +it is the argument for making M3's "enumerate from disk" a *test* rather than an instruction. +Belongs in the review's lessons learned. + +Spec now 9,415w (iteration 3). Porch remains at spec-approval; I have not touched the gate. + +### Compression for signal (2026-07-31) — 9,415 → 6,860 words (−27%) + +Architect-directed before the gate; I had flagged the length myself. Rule applied: **fold, +don't drop.** + +**Kept whole**: all 14 M-criteria, all 16 tests, 5 outcomes, 7 rollback groups, the full +per-audience inventory, the rollout contract, the A/B design, and the four Baked Decisions +verbatim. + +**Cut**: the narration of how the spec got here — the two round-by-round finding tables +(~1,400w), "(round-2 finding)" asides threaded through the criteria, the superseded 33,519 +figure, and the withdrawn HOLD-band arithmetic. That record lives in the iter1/iter2 rebuttal +files and this thread, which is where the architect said it belongs. + +**Self-verified mechanically** (diffed old vs new): 14/14 criteria, 16/16 tests, 5/5 outcomes, +7/7 groups, all 20 template headings in order, porch's `spec_has_required_sections` passes. +Diffed every number too — eleven appeared to vanish; ten were from the deleted narration or +the withdrawn HOLD band (correctly gone). **One was a real loss**: the margin figure (−53.2% +clears >50% by 3.2 points ≈ 1,100 words), which was Claude's thin-margin warning and is real +information for an approver. Folded back into M1 as a clause rather than left dropped. + +That number-level diff is the compression analogue of principle 7 — "I kept the criteria" is +not the same claim as "I kept the content," and only the diff distinguishes them. + +Gate remains pending; still not touched. + +### Acceptance model redirected (2026-07-31) — principles, not size + +**Gate NOT APPROVED.** Waleed: *"I don't think the goal should be a particular size. That's not +the right criteria. It should be to stick to the principles outlined in the blog post."* + +Rewrote the acceptance model (commit 0821c7ab). Fetched the blog rather than paraphrasing from +the issue charter, and quoted its principles **verbatim** as P1–P7, each restated as a per-file +question answerable from a diff. + +Two honest complications I surfaced rather than smoothed over: + +- **P5 (auto-memory) does not apply.** It is a Claude Code harness feature; Baked Decision 1's + fleet includes GPT 5.6 / Gemini 3.6 consumers with no equivalent. Declared N/A with reason + rather than listing six principles and hoping nobody counted seven. +- **P7 collides with the scar-rule exemption.** The blog deletes worst-case guardrails and its + own example is "such as deleting files" — exactly what our scar rules guard. Named the + collision: the blog's guardrails protected against *bad output* (judgment now handles it); + scar rules protect against *irreversible acts*, where being wrong once is unbounded. + +Demoted to observability: >50%, all ceilings, per-segment goals, thin-margin analysis. M1/M2 +are reporting obligations that cannot fail on a number. M0/M0b/M0c survive — they keep us +honest about what happened. + +**M11** (architect per-file inspection) added with the load sized rather than assumed, and +**T16** fails a phase whose manifest omits a changed file. + +### CMAP round 3 — Codex: fifth self-audit finding, same root cause + +REQUEST_CHANGES (HIGH), five findings. The factual one lands squarely on me again: + +**My twin-file table was wrong.** I reported "3 codev/protocols copies that differ". In fact +**zero differ** — all three (`maintain/templates/audit-report.md`, +`maintain/templates/lessons-learned.md`, `release/protocol.md`) are **local-only, no skeleton +twin**. Cause: my `cmp -s` loop treated a nonzero exit as "differs", but `cmp` also exits +nonzero when a file is missing. I read an exit code without distinguishing its two causes. + +That is the **fifth** self-audit finding of this spec phase and the fourth sharing one root +cause: *trusting a convenient signal instead of checking the authoritative thing* (truncated +grep; skeleton-only enumeration; the script's stale comment; now an overloaded exit code). The +pattern is now well-evidenced enough that it belongs in lessons-learned as its own entry, not +just as this project's review note. + +Three other findings are sharp and structural: MP/M3 scope contradicts the hot-tier and skills +exclusions; the A/B cannot both use "the same base commit" and "pre-/post-rewrite commits"; +and **M5 conflicts with P6** — P6 permits replacing narrated gate/check names with a reference +to structured truth, while M5 demands those names remain in served prose, so a conformant P6 +rewrite would fail M5. Fixing after Claude's round-3 lands. + +### CMAP round 3 complete — 11 findings, none disputed (2026-07-31) + +Codex REQUEST_CHANGES (HIGH, 5) · Claude COMMENT (HIGH, 6). Commit be95ef40. + +**The two structural ones would have surfaced in implement as confusion, not as clean defects:** + +1. **The A/B was impossible as written.** It said both "both arms from the same base commit" + AND "control = pre-rewrite commit, treatment = post-rewrite commit." Rebuilt as a + **prompt-only overlay**: both arms branch from source commit `S`; control applies one overlay + reverting G2–G6 and nothing else; each run records source hash AND prompt-surface hash. Also + fixes the latent bug Codex spotted — later pairs would otherwise inherit source changes the + pinned control commit lacked. +2. **M5 conflicted with P6.** P6 permits replacing narrated gate/check names with a reference to + structured truth; M5 demanded the names stay in prose. A *conformant* rewrite would have + failed the capability check. Representation now means name-in-text OR an explicit resolvable + reference to a source that still defines it. Claude hit the same criterion from the other + side — it detects deletion, not inversion — so both fixes composed: the limit is now stated + outright and the gap assigned to M11/O4 plus a short set of semantic invariants. + +**Claude's best process catch**: the issue-mandated **per-surface cut plan** was missing. Word +targets died with the redirect; the *disposition mapping* survived it and I had dropped both. +Added as a disposition table marking every category rewritten / inspected-but-unchanged / +excluded-with-reason — which also resolved Codex's scope contradiction (hot tier, .claude/skills). + +**Also**: M12 (no release between merge and SHIP verdict — adopters would otherwise consume an +unvalidated skeleton); the A/B's unstated resolver precondition (tier-2 shadows tier-4; verified +0 skeleton files lack a codev/ twin) now asserted pre-flight in T14, voiding a pair on failure +rather than producing a comparison that looks valid and isn't. + +**Not actioned by me**: issue #1280's title/Goal still state ">50% reduction… measured with +1252's committed measurement script" — superseded on both counts. That is the architect's +artifact; flagged, not edited. Left unchanged it will keep drawing "doesn't meet the stated +goal" findings from every future CMAP round. + +Spec 8,212w. Gate remains pending and untouched. + +### Charter amended by the architect (2026-08-01 UTC / 2026-07-31 local) + +Issue #1280 retitled "Prompt-surface judgment rewrite: acceptance = blog-principles conformance +per file (size is reporting-only)" with a dated AMENDMENT block superseding the original Goal. +Original Goal preserved above it and marked superseded — history stays honest. + +**Verified rather than assumed** (the spec quotes Baked Decisions verbatim, so an amendment +touching them would silently invalidate a Constraints section): + +- All **four Baked Decisions are byte-intact** — diffed the issue's bullets against the spec's + Constraints, all four match. No re-quoting needed. +- The amendment cites the spec's P7-vs-scar-rules resolution and states architect inspection as + charter-level, matching what the spec specifies. + +One residual, not worth raising: the Protocol section still reads "the per-surface cut plan with +word targets." Amendment point 2 (size reporting-only) supersedes the "word targets" clause, and +the spec already handles it explicitly — targets withdrawn, disposition mapping retained. No +action. + +Fifth self-audit instance confirmed for the lessons ledger by the architect: **"trusting a +convenient signal over the authoritative thing"** — `cmp -s` exit codes joining truncated greps +and skeleton-only enumeration. Named pattern; the review phase routes it to lessons-learned. + +**Gate presentation to Waleed is out with the architect's recommendation. Nothing is pending on +me. Not touching the gate; waiting.** + +### spec-approval APPROVED → plan phase (2026-07-31) + +Waleed approved; architect relayed. Ran `porch approve 1280 spec-approval` myself per the flow +(porch required the `--a-human-explicitly-approved-this` flag — correct guard). Advanced to plan. + +**Plan drafted: 10 phases, 67 decisions, max batch 11 (cap 12).** + +Phase boundaries are drawn by **inspection load, not subsystem elegance** — M11 makes the +architect's per-file review the throughput constraint, so the boundary that matters is "a batch +a human can review in one sitting." + +``` +P0 (PR-1) → P1(4) → P2(10) → P3(9) → P4(11) → P5(10) → P6(10) → P7(9) → P8(4) → P9 + sum = 67 +``` + +Verified the enumeration against disk rather than trusting the spec's "~66": protocol.md 10 +(9 skeleton + release local) · builder-prompt 9 · prompts 18 · templates 8 (6 + 2 codev-local) · +consult-types 18 · roles 3 · CLAUDE/AGENTS 1 = **67**. + +Design decisions worth recording: + +- **P0 ships as PR-1 before any prompt word changes.** Not administrative sequencing — rewriting + first would make every later measurement unfalsifiable (principle 7). +- **P1 is deliberately small (4).** Highest blast radius (CLAUDE.md carries all 8 scar rules), + and it calibrates the architect's conformance standard for the seven phases after it. +- **P3 is only 9 decisions but is the riskiest phase**, because it carries the whole M10 burden: + `baked-decisions.test.ts` enforces a pure-addition diff on three `builder-prompt.md` files, + structurally incompatible with rewriting them. Kept separate from P1 for that reason alone. +- **P8 rebuilds the scar registry last**, against the settled surface — Baked Decision 2 defers + enforcement until the surface stops moving, and `must_appear_on` derived earlier would be stale. +- **Capability risks named per phase** rather than deferred: the plan template's phases-JSON block + (P5) and the consult verdict format (P6/P7) are *capabilities*, not examples, and would be + plausible casualties of P2 applied carelessly. Both get live integration checks, not fixtures. + +Porch checks pass: plan_exists, has_phases_json, min_two_phases (10). + +### Plan CMAP round 1 — 15 findings, none disputed (2026-07-31) + +Codex REQUEST_CHANGES (8) · Claude REQUEST_CHANGES (7). Both independently reproduced the 67 +decision count — **the accounting held; what failed was everything riding alongside it.** A plan +drawn purely by inspection load under-specifies the supporting work, and that is precisely what +both reviewers found. + +**Two blocking mechanism gaps, both verified against source before accepting:** + +1. **P6 had no adopter-resolvable mechanism.** `protocol.md` is inlined at spawn via + `{{protocol_reference}}` (spawn-roles.ts:112-124); **protocol.json is inlined nowhere** (:267 + reads it only for validation). In a fresh adopter project the file isn't on disk. So my + "reference protocol.json" was the fetch-by-path CLAUDE.md forbids — carrying the largest cut + in the project. Resolved by *checking the resolver rather than guessing*: + `resolveCodevIncludes` is extension-agnostic, so a fenced ```json block with + `{{> protocols/

/protocol.json}}` resolves through all four tiers with no porch change. + New T18 tests strict AND soft mode — soft-mode builders have only the prompt, no porch tasks. + +2. **Skill relocation is a FOUR-tree sync** (Claude). `.claude/skills` (10), `.codex/skills` (10, + byte-identical), and both skeleton copies (7 each) — with existing drift (afx, porch) and + three skeleton-absent skills. M0(g) counted only `.claude/skills`, so relocated content would + have left Codex agents and adopters without it **and been reported as deleted** — inverting + the project's own honesty artifact. Phase 0 widens the basis; new T17 asserts parity. + +**Other structural fixes**: pre-rewrite capability inventory frozen in Phase 0 (was first +extracted in the final phase, while Phase 2 already asserted against it); verify phase given an +explicit home for M7/M8/M12/T13/T14; T3/T16 into Phase 0 (T16 guards M11 and had to exist before +the first manifest); "review batch" *defined* to include tests/registry/retirements, with Phases +4 and 9 declaring two batches each; scar canonicals verified against the ratified YAML in Phase 1 +with T4 deferred to Phase 9; commit-level group purity replacing the broken phase-level mapping; +per-phase green suite; M10 reassigned to where each collision actually lands +(spec-1273 → roles phase and Phase 1, not Phase 4; bugfix-742 → Phases 7/8). + +Phase count 10 → 11. Decisions still 67, max batch 11. + +**Notable**: Claude caught that my rollback mapping contradicted the spec I wrote — Phase 1 +claimed G2/G6 while rewriting roles/builder.md (G3) and roles/consultant.md (G5). A G3 revert +would have silently pulled Phase 1 work out and T10 would have rehearsed the wrong map. + +### plan-approval APPROVED → Phase 0 (2026-08-01) + +Waleed approved; ran `porch approve 1280 plan-approval` myself. Recorded the architect's +skills-drift ruling in the plan first: skills this project TOUCHES get four-tree parity (T17, +scoped to the touched set); pre-existing drift on untouched skills → separate architect-filed +issue, recorded-known-state, and must not fail T17. + +**Phase 0 (PR-1) built. Commit 9d8c2569. No prompt-surface file touched.** + +Corrected the three known defects; the reported baseline moves **21,702 → 34,235**. + +**A fourth inaccuracy found while writing the tests — mine, not 1252's.** Include expansion was +*additive*: it counted the `{{> path}}` directive's own tokens PLUS the content substituted for +them, over-reporting ~2 words per include. `expand_text` now does real substitution mirroring +`resolveCodevIncludes` (regex replace in place, recursive, depth-guarded, unresolved → empty). +That is why the figure is 34,235 and not the 34,255 the spec quotes — a 20-word delta across 10 +iterations. Size is reporting-only, so no criterion moves, but the spec's number is now +superseded and the baseline artifact says so explicitly. + +Two of my own bugs, both caught by testing rather than by reading: + +1. **`set -o pipefail` + `grep -q`** in the capability extractor reported **all 57 capabilities + as absent** — grep exits on first match, printf takes SIGPIPE, and the pipeline reports + failure *because the match succeeded*. An exit code with two causes, read as one: the exact + pattern this project has now logged six times. Fixed with a here-string. +2. **Line-wise include expansion dropped text sharing a line with a directive** — `aa bb {{> x}}` + lost `aa bb`. Only surfaced because T2 asserts exact neutrality. + +Frozen capability inventory: **57 capabilities, 47 present in served prompts, 10 absent +pre-existing** (porch delivers gates/checks via task JSON, not authored prompt text). That +asymmetry is baseline state, correctly captured — M5 compares post ⊇ pre, so nothing must be +invented. It also validates the instrument: an inventory reporting 100% present would have been +suspicious. + +Artifacts: `1280-word-baseline.md`, `1280-capability-inventory.json`, manifest format README, +`1252-word-*.md` annotated in place (originals preserved, marked superseded). Tests: 22 passing +(T1, T1b, T2, T3, T11, T12, T15, T16). T16 written before any manifest exists — the guard must +predate what it guards. + +### Worktree environment note — `pnpm build` is a precondition for the suite + +58 test files / 116 tests failed on first full run. **Not my change.** Cause: +`Skeleton directory not found. Package may be corrupted.` from `getTemplatesDir` — this +worktree had never been built, and `pnpm build`'s `copy-skeleton` step is what creates +`packages/codev/skeleton` from `codev-skeleton/`. After building, the three representative +failing files (adopt, hot-tier-materialization, cold-tier-materialization) pass 19/19. + +**Carry-forward for every later phase**: `packages/codev/skeleton` is a BUILD COPY of +`codev-skeleton/`. When a phase rewrites skeleton files, the suite will keep testing the stale +copy until `pnpm build` re-runs. Any phase touching `codev-skeleton/` must rebuild before +claiming green, or it is asserting against pre-rewrite content — a phantom-green of exactly the +kind this project exists to stop being fooled by. + +I diagnosed this from an actual error message rather than the "probably needs a build" theory, +which is the same discipline the rest of the project has been applying to instruments. + +### CI caught a portability defect the local suite could not (2026-08-01) + +PR #1319 Unit Tests failed: **CI measured 34,231, my Mac measured 34,235.** Same commit. + +Diagnosed by narrowing rather than guessing: +1. Fetched `refs/pull/1319/merge`, diffed every measured file against my HEAD — **tracked + content identical**, so not a content delta. +2. Exported that merge ref to a clean tree and measured it locally — **still 34,235**, so not + the tree either. That left the platform. +3. Compared `wc -w` / `LC_ALL=C wc -w` / Python `split()` per file: `spir/protocol.md` gave + 3703 / 3699 / 3699. +4. Diffed per-LINE counts to find the exact characters. + +**Root cause: `⚠️` (U+26A0 WARNING SIGN + U+FE0F VARIATION SELECTOR-16).** macOS/BSD `wc -w` in +a UTF-8 locale splits it into two words; GNU `wc`, `LC_ALL=C wc`, and Python's `str.split()` +count one. Four such banners in `spir/protocol.md` — exactly the 4-word delta. + +**Why this mattered rather than being a test nuisance**: the instrument's entire purpose is an +honest before/after comparison. A count that depends on the host means measuring "before" on a +laptop and "after" in CI produces a fictional delta — the phantom-savings failure this whole +instrument exists to prevent, in a new disguise. It would have been invisible until the final +report, and then unfalsifiable. + +Fix: word counting is now **defined by the script** (whitespace-delimited tokens of UTF-8 +decoded text) rather than delegated to the host's `wc`. Worktree, clean merge-ref export, and +`LC_ALL=C` now all report **34,231**. Two regression tests added: no delegation to `wc -w`, and +identical totals under `LC_ALL=C` vs UTF-8. + +Baseline restated 34,255 → 34,231, with both corrections documented in the artifact header +(−20 additive-include model, −4 `wc` portability). Size is reporting-only, so no criterion moves. + +**The characters that broke portability are the `⚠️ BLOCKING` worst-case-padding banners that +principle P7 exists to delete.** Recorded rather than smoothed over. + +**Lesson for the ledger** (seventh instance of the family): *a green local suite is not a green +build.* The delegated tool — like the overloaded exit code, the truncated grep, the +skeleton-only enumeration, and the stale script comment before it — looked authoritative and +wasn't. CI was the authoritative signal here, and it existed all along. diff --git a/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts b/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts new file mode 100644 index 000000000..6b95c2de6 --- /dev/null +++ b/packages/codev/src/__tests__/spec-1280-measurement-instrument.test.ts @@ -0,0 +1,308 @@ +/** + * Spec 1280 — Phase 0: the corrected measurement instrument. + * + * The Spec 1252 instrument was committed, deterministic, and wrong: it derived + * its phase-task term from `codev-skeleton/porch/prompts/`, a dead tree with no + * runtime consumer, while the live resolver loads `protocols/

/prompts/`. It + * also omitted the spawn-inlined `roles/builder.md` and mis-stated how the hot + * tier reaches CLAUDE.md. + * + * These tests exist because "deterministic and committed" is not "correct". + * They assert the instrument against the REAL resolver and the REAL runtime + * loader, so a future edit cannot quietly reintroduce any of the three defects. + * + * Covers T1, T1b, T2, T3, T11, T12, T15. + */ +import { describe, it, expect, beforeAll } from 'vitest'; +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; + +const repoRoot = path.resolve(import.meta.dirname, '../../../..'); +const script = path.join(repoRoot, 'scripts/measure-prompt-surface.sh'); + +function run(root: string = repoRoot, env: Record = {}): string { + return execFileSync('bash', [script, root], { + encoding: 'utf-8', + env: { ...process.env, ...env }, + maxBuffer: 20 * 1024 * 1024, + }); +} + +function num(output: string, key: string): number { + const m = output.match(new RegExp(`^${key}=(\\d+)$`, 'm')); + if (!m) throw new Error(`${key} not found in measurement output`); + return Number(m[1]); +} + +/** Minimal fixture repo: only what the script reads. */ +function makeFixture(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'spec1280-')); + const mk = (rel: string, body: string) => { + fs.mkdirSync(path.join(dir, path.dirname(rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); + }; + mk('CLAUDE.md', 'alpha bravo charlie\n'); + mk('AGENTS.md', 'alpha bravo charlie\n'); + mk('codev/resources/arch-critical.md', 'one two\n'); + mk('codev/resources/lessons-critical.md', 'three four\n'); + mk('codev/roles/builder.md', 'role word here\n'); + mk('codev/roles/architect.md', 'architect role words\n'); + mk('codev/roles/consultant.md', 'consultant words\n'); + mk('codev-skeleton/protocols/spir/builder-prompt.md', 'wrapper words here\n'); + mk('codev-skeleton/protocols/spir/protocol.md', 'protocol words here now\n'); + mk('codev-skeleton/protocols/spir/prompts/specify.md', 'aa bb cc dd ee\n'); + mk('codev-skeleton/protocols/spir/consult-types/spec-review.md', 'rubric words\n'); + return dir; +} + +describe('T1 — the instrument sources the directory the runtime actually loads', () => { + it('reads protocols/

/prompts/, not the dead porch/prompts tree', () => { + const src = fs.readFileSync(script, 'utf-8'); + // Assert on EXECUTABLE lines only. The header comment legitimately names the + // dead tree while explaining the defect, and a blanket string ban would + // forbid documenting the very bug this test guards. + const code = src + .split('\n') + .filter((l) => !/^\s*#/.test(l) && l.trim() !== '') + .join('\n'); + expect(code).toMatch(/protocols\/\$p\/prompts\//); + expect(code).not.toMatch(/PORCH_DIR=/); + // porch/prompts may appear only as the DEAD bucket (its measurement and its + // report label) — never as an input to the phase-task term. The tree still + // exists and the report must expose it; what must never return is it FEEDING + // the phase mean, which was defect 1. + const porchRefs = code.split('\n').filter((l) => l.includes('porch/prompts')); + expect(porchRefs.length).toBeGreaterThan(0); + for (const line of porchRefs) { + expect(line, `unexpected porch/prompts use: ${line}`).toMatch(/DEAD_[WF]=|\| DEAD /); + } + const phaseMeanFn = code.slice(code.indexOf('phase_mean()'), code.indexOf('consult_mean()')); + expect(phaseMeanFn).not.toMatch(/porch/); + }); + + it('agrees with loadPromptFile, which resolves protocols//prompts/', () => { + // Ground the assertion in the real loader rather than a hardcoded string: + // if porch's resolution path ever moves, this fails loudly. + const loader = fs.readFileSync( + path.join(repoRoot, 'packages/codev/src/commands/porch/prompts.ts'), + 'utf-8', + ); + expect(loader).toMatch(/protocols\/\$\{protocolName\}\/prompts\/\$\{promptFile\}/); + }); + + it('counts the spawn-inlined role file (defect 2)', () => { + const spawn = fs.readFileSync( + path.join(repoRoot, 'packages/codev/src/agent-farm/commands/spawn-worktree.ts'), + 'utf-8', + ); + expect(spawn).toContain('.builder-role.md'); // it really is injected at spawn + expect(fs.readFileSync(script, 'utf-8')).toMatch(/BUILDER_ROLE=.*roles\/builder\.md/); + }); + + it('adds the hot tier to CLAUDE.md rather than assuming it is inlined (defect 3)', () => { + const managed = fs.readFileSync( + path.join(repoRoot, 'packages/codev/src/lib/managed-block.ts'), + 'utf-8', + ); + expect(managed).toContain('@codev/resources/arch-critical.md'); // @import, not inlined + const src = fs.readFileSync(script, 'utf-8'); + expect(src).toMatch(/SHARED=\$\(\(\s*CLAUDE_MD \+ HOT\s*\)\)/); + // The stale claim must not survive as an ASSERTION about current behaviour. + // It may appear in the header, where it is quoted as a defect being corrected. + const header = src.slice(0, src.indexOf('set -euo pipefail')); + const body = src.slice(src.indexOf('set -euo pipefail')); + expect(body).not.toMatch(/already inlines/); + expect(header).toMatch(/TRANSCLUDES|@import/); // documents the real mechanism + }); +}); + +describe('T1b — per-file four-tier resolution, not directory-level selection', () => { + it('resolves each file at its own winning tier', () => { + const dir = makeFixture(); + // Override ONE prompt in .codev/ while its siblings stay in the skeleton. + fs.mkdirSync(path.join(dir, '.codev/protocols/spir/prompts'), { recursive: true }); + fs.writeFileSync( + path.join(dir, '.codev/protocols/spir/prompts/specify.md'), + 'aa bb cc dd ee ff gg hh\n', // 8 words vs 5 + ); + const out = run(dir); + // Directory-level selection would have missed the override entirely. + expect(out).toMatch(/\| spir \| \d+ \| 8 \| \d+ \|/); + }); + + it('prefers .codev/ over codev/ over codev-skeleton/', () => { + const src = fs.readFileSync(script, 'utf-8'); + const order = src.slice(src.indexOf('resolve() {'), src.indexOf('# SERVED words')); + expect(order.indexOf('.codev/')).toBeLessThan(order.indexOf('"codev/$1"')); + expect(order.indexOf('"codev/$1"')).toBeLessThan(order.indexOf('codev-skeleton/')); + }); +}); + +describe('T2 — phantom-savings proof: includes are expanded', () => { + it('moving text from a prompt into an included template changes nothing', () => { + const dir = makeFixture(); + const before = num(run(dir), 'ALWAYS_ON_WORDS'); + + // Same served content, different authored ownership. + fs.writeFileSync( + path.join(dir, 'codev-skeleton/protocols/spir/prompts/specify.md'), + 'aa bb {{> protocols/spir/templates/frag.md}}\n', + ); + fs.mkdirSync(path.join(dir, 'codev-skeleton/protocols/spir/templates'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'codev-skeleton/protocols/spir/templates/frag.md'), 'cc dd ee\n'); + + expect(num(run(dir), 'ALWAYS_ON_WORDS')).toBe(before); + }); + + it('expands non-markdown includes too — protocol.json delivery depends on it (P6)', () => { + const dir = makeFixture(); + const before = num(run(dir), 'ALWAYS_ON_WORDS'); + fs.writeFileSync( + path.join(dir, 'codev-skeleton/protocols/spir/protocol.md'), + 'protocol words here now\n```json\n{{> protocols/spir/protocol.json}}\n```\n', + ); + fs.writeFileSync( + path.join(dir, 'codev-skeleton/protocols/spir/protocol.json'), + '{ "a": 1, "b": 2 }\n', + ); + // The JSON's words must appear in the served count, not vanish. + expect(num(run(dir), 'ALWAYS_ON_WORDS')).toBeGreaterThan(before); + }); +}); + +describe('T3 — per-surface reporting completeness (not a ceiling)', () => { + it('reports every protocol found on disk, in either tree', () => { + const out = run(); + const onDisk = new Set(); + for (const tree of ['codev/protocols', 'codev-skeleton/protocols']) { + const p = path.join(repoRoot, tree); + if (!fs.existsSync(p)) continue; + for (const e of fs.readdirSync(p, { withFileTypes: true })) { + if (e.isDirectory()) onDisk.add(e.name); + } + } + expect(onDisk.size).toBeGreaterThan(0); + for (const name of onDisk) { + expect(out, `protocol "${name}" missing from the report`).toMatch( + new RegExp(`^\\| ${name} \\|`, 'm'), + ); + } + }); + + it('includes codev-only protocols with no skeleton twin (release)', () => { + expect(run()).toMatch(/^\| release \|/m); + }); +}); + +describe('T11 — buckets vs audience loads are reported on different bases', () => { + it('states the formulas and warns the audience loads overlap', () => { + const out = run(); + expect(out).toContain('ALWAYS_ON(builder,p,I) = SHARED + BUILDER_SPAWN[p]'); + expect(out).toMatch(/OVERLAP by design/); + expect(out).toMatch(/these SUM/); + }); + + it('one bucket growing while another shrinks shows BOTH movements, not a netted zero', () => { + const dir = makeFixture(); + const before = run(dir); + const sharedBefore = Number(before.match(/\| SHARED [^|]*\| (\d+) \|/)![1]); + const archBefore = Number(before.match(/\| ARCHITECT [^|]*\| (\d+) \|/)![1]); + + fs.writeFileSync(path.join(dir, 'CLAUDE.md'), 'alpha\n'); // shrink + fs.writeFileSync(path.join(dir, 'codev/roles/architect.md'), 'a b c d e f g\n'); // grow + + const after = run(dir); + const sharedAfter = Number(after.match(/\| SHARED [^|]*\| (\d+) \|/)![1]); + const archAfter = Number(after.match(/\| ARCHITECT [^|]*\| (\d+) \|/)![1]); + + expect(sharedAfter).toBeLessThan(sharedBefore); + expect(archAfter).toBeGreaterThan(archBefore); + }); +}); + +describe('T15 — relocation is visible, never reported as deletion (M0c)', () => { + it('moving a block into a skill drops always-on but holds total-authored steady', () => { + const dir = makeFixture(); + fs.writeFileSync(path.join(dir, 'CLAUDE.md'), 'alpha bravo charlie delta echo foxtrot\n'); + const before = run(dir); + + // Relocate three words out of CLAUDE.md into a skill — the P3/P4 move. + fs.writeFileSync(path.join(dir, 'CLAUDE.md'), 'alpha bravo charlie\n'); + fs.mkdirSync(path.join(dir, '.claude/skills/afx'), { recursive: true }); + fs.writeFileSync(path.join(dir, '.claude/skills/afx/SKILL.md'), 'delta echo foxtrot\n'); + + const after = run(dir); + expect(num(after, 'ALWAYS_ON_WORDS')).toBeLessThan(num(before, 'ALWAYS_ON_WORDS')); + expect(num(after, 'TOTAL_AUTHORED_WORDS')).toBe(num(before, 'TOTAL_AUTHORED_WORDS')); + }); + + it('counts all four skill trees — one-tree counting would report relocation as deletion', () => { + const src = fs.readFileSync(script, 'utf-8'); + for (const tree of [ + '.claude/skills', + '.codex/skills', + 'codev-skeleton/.claude/skills', + 'codev-skeleton/.codex/skills', + ]) { + expect(src, `total-authored basis must include ${tree}`).toContain(tree); + } + }); +}); + +describe('portability — the count must not depend on the host', () => { + it('does not delegate word counting to `wc -w`', () => { + // BSD wc (macOS, UTF-8 locale) counts `⚠️` (U+26A0 U+FE0F) as two words; GNU wc + // and Python's str.split() count one. Four such banners in spir/protocol.md + // made the same commit measure 34,235 locally and 34,231 in CI. An instrument + // whose before/after must be comparable across machines cannot delegate its + // core definition to a platform-variant tool. + const code = fs + .readFileSync(script, 'utf-8') + .split('\n') + .filter((l) => !/^\s*#/.test(l)) + .join('\n'); + expect(code).not.toMatch(/wc -w/); + expect(code).toMatch(/_count\(\)/); + }); + + it('reports the same total under a C locale as under UTF-8', () => { + const utf8 = num(run(repoRoot, { LC_ALL: 'en_US.UTF-8' }), 'ALWAYS_ON_WORDS'); + const c = num(run(repoRoot, { LC_ALL: 'C' }), 'ALWAYS_ON_WORDS'); + expect(c).toBe(utf8); + }); +}); + +describe('T12 — determinism', () => { + it('emits byte-identical output twice at the same commit', () => { + expect(run()).toBe(run()); + }); +}); + +describe('the corrected baseline is what the spec claims', () => { + let out: string; + beforeAll(() => { out = run(); }); + + it('reproduces ALWAYS_ON_WORDS = 34,231 for a SPIR builder at I=10', () => { + // 34,231 — not the 34,255 quoted in the spec. Two corrections, both making the + // instrument more honest and neither moving an acceptance criterion (size is + // reporting-only under the amended charter): + // -20 the 1252 additive include model counted a `{{> path}}` directive's own + // tokens AND the content substituted for them; this one substitutes. + // -4 `wc -w` is not portable: BSD wc in a UTF-8 locale splits `⚠️` into two + // words where GNU wc and Python's split() see one. Counting is now + // defined explicitly rather than delegated to the platform's wc. + expect(num(out, 'ALWAYS_ON_WORDS')).toBe(34231); + }); + + it('reproduces the architect load (8,599)', () => { + expect(out).toMatch(/\| Architect \(per session\) \| 8599 \|/); + }); + + it('honours PHASE_ITERS as a comparison constant', () => { + const one = num(run(repoRoot, { PHASE_ITERS: '1' }), 'ALWAYS_ON_WORDS'); + const two = num(run(repoRoot, { PHASE_ITERS: '2' }), 'ALWAYS_ON_WORDS'); + expect(two - one).toBe(736 + 1396); // HOT + spir phase mean + }); +}); diff --git a/packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts b/packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts new file mode 100644 index 000000000..c6b6d8148 --- /dev/null +++ b/packages/codev/src/__tests__/spec-1280-phase-manifest.test.ts @@ -0,0 +1,119 @@ +/** + * Spec 1280 — T16: phase-manifest completeness (the mechanical guard on M11). + * + * M11 requires the architect to inspect the old-vs-new diff of EVERY changed + * file, per phase, in batches of <=12. That review is the project's acceptance + * mechanism — but a human can only inspect what they are shown. A file changed + * in a phase and omitted from that phase's manifest is invisible to the review, + * so it fails the phase. + * + * This test is written in Phase 0, before the first manifest exists, precisely + * because the guard must predate the thing it guards. + */ +import { describe, it, expect } from 'vitest'; +import { execFileSync } from 'node:child_process'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const repoRoot = path.resolve(import.meta.dirname, '../../../..'); +const manifestDir = path.join( + repoRoot, + 'codev/projects/1280-prompt-surface-judgment-not-ru/manifests', +); + +/** Files a manifest is responsible for listing: prompt-bearing surfaces only. */ +const PROMPT_BEARING = /^(CLAUDE\.md|AGENTS\.md|codev(-skeleton)?\/(protocols|roles)\/.*\.md)$/; + +interface Manifest { + file: string; + phase: string; + rows: { path: string; oldWords: string; newWords: string; principles: string }[]; +} + +function parseManifest(file: string): Manifest { + const body = fs.readFileSync(file, 'utf-8'); + const rows: Manifest['rows'] = []; + for (const line of body.split('\n')) { + // | path | old | new | principles | rationale | + const m = line.match(/^\|\s*`?([^`|]+?)`?\s*\|\s*(\d+)\s*\|\s*(\d+)\s*\|\s*([^|]*)\|/); + if (m && !/^-+$/.test(m[1].trim())) { + rows.push({ + path: m[1].trim(), + oldWords: m[2], + newWords: m[3], + principles: m[4].trim(), + }); + } + } + return { file, phase: path.basename(file, '.md'), rows }; +} + +function manifests(): Manifest[] { + if (!fs.existsSync(manifestDir)) return []; + return fs + .readdirSync(manifestDir) + .filter((f) => f.endsWith('.md')) + .map((f) => parseManifest(path.join(manifestDir, f))); +} + +describe('T16 — manifest completeness (M11)', () => { + it('the manifest directory is a known location, created when the first phase lands', () => { + // Phase 0 changes no prompt-bearing file, so an absent directory is valid here. + // From Phase 1 onward the per-phase check below does the real work. + expect(path.isAbsolute(manifestDir)).toBe(true); + }); + + it('every manifest row carries all four required fields', () => { + for (const m of manifests()) { + expect(m.rows.length, `${m.phase} has no rows`).toBeGreaterThan(0); + for (const r of m.rows) { + expect(r.path, `${m.phase}: empty path`).not.toBe(''); + expect(r.oldWords, `${m.phase}/${r.path}: old word count`).toMatch(/^\d+$/); + expect(r.newWords, `${m.phase}/${r.path}: new word count`).toMatch(/^\d+$/); + expect( + r.principles, + `${m.phase}/${r.path}: must name the principles applied (P1..P7 or "none")`, + ).toMatch(/P[1-7]|none/i); + } + } + }); + + it('no manifest declares a batch larger than 12 files', () => { + for (const m of manifests()) { + const body = fs.readFileSync(m.file, 'utf-8'); + // A manifest may declare several batches; each is capped independently. + const batches = body.split(/^##+\s+Batch\s/im).slice(1); + if (batches.length === 0) { + expect(m.rows.length, `${m.phase}: single-batch manifest over the cap`).toBeLessThanOrEqual(12); + } else { + for (const [i, b] of batches.entries()) { + const n = (b.match(/^\|\s*`?[^`|]+?`?\s*\|\s*\d+\s*\|/gm) || []).length; + expect(n, `${m.phase} batch ${i + 1} over the cap`).toBeLessThanOrEqual(12); + } + } + } + }); + + it('every prompt-bearing file changed on this branch appears in some manifest', () => { + let changed: string[]; + try { + changed = execFileSync('git', ['diff', '--name-only', 'origin/main...HEAD'], { + cwd: repoRoot, + encoding: 'utf-8', + }) + .split('\n') + .map((s) => s.trim()) + .filter((s) => PROMPT_BEARING.test(s)); + } catch { + return; // no origin/main to diff against (fresh clone / CI shallow) — skip + } + if (changed.length === 0) return; + + const listed = new Set(manifests().flatMap((m) => m.rows.map((r) => r.path))); + const missing = changed.filter((f) => !listed.has(f)); + expect( + missing, + `changed but absent from every manifest — the architect cannot inspect what is not listed:\n${missing.join('\n')}`, + ).toEqual([]); + }); +}); diff --git a/scripts/extract-capability-inventory.sh b/scripts/extract-capability-inventory.sh new file mode 100755 index 000000000..de1ff7089 --- /dev/null +++ b/scripts/extract-capability-inventory.sh @@ -0,0 +1,156 @@ +#!/usr/bin/env bash +# +# extract-capability-inventory.sh — Spec 1280, criterion M5. +# +# Extracts every CAPABILITY the prompt surface communicates to an agent, so a +# rewrite can be proven not to have silently dropped one. +# +# THE INVENTORY IS OVER SERVED PROMPT TEXT, NOT OVER protocol.json. +# ------------------------------------------------------------------ +# This distinction is the whole point (CMAP round 2 caught the earlier design +# doing the opposite). Extracting gate/check names from `protocol.json`, or +# notification names from `afx send` call sites, would report every capability +# as present even if every corresponding INSTRUCTION vanished from the prompts — +# because this project does not touch those files. `protocol.json` and the +# source tree supply the EXPECTED SET; the served prompt text is where each +# item must be found. +# +# REPRESENTATION (M5, reconciled with principle P6) +# ------------------------------------------------- +# P6 permits replacing narrated names with a reference to structured truth. So a +# capability counts as represented if EITHER: +# (a) it is named in served prompt text, OR +# (b) the served text carries a resolvable `{{> ...}}` include of the source +# that defines it (the include is expanded here before matching, so (b) +# reduces to (a) automatically). +# A naive "the name must appear in prose" rule would fail a CONFORMANT rewrite. +# +# DETECTION LIMIT — stated, not implied +# ------------------------------------- +# This detects DELETION of a capability. It does NOT detect INVERSION or +# GUTTING of the instruction attached to one: "a gate message is a notification +# to the human, not authorization" can collapse to a bare mention of the gate +# name and still pass. That gap is covered by M11 (architect reads every diff) +# and O4 (the A/B compliance checklist), not by this script. +# +# Usage: scripts/extract-capability-inventory.sh [repo-root] > inventory.json +set -euo pipefail + +ROOT="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" +cd "$ROOT" + +resolve() { + if [ -f ".codev/$1" ]; then echo ".codev/$1" + elif [ -f "codev/$1" ]; then echo "codev/$1" + elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" + else echo /dev/null; fi +} + +# Served text of one file with {{> ...}} includes expanded (two levels). +expand() { + local f="$1"; [ -f "$f" ] || return 0 + local body; body="$(cat "$f")" + local inc ip + for inc in $(grep -o '{{> *[^} ]*' "$f" 2>/dev/null | sed 's/{{> *//'); do + ip="$(resolve "$inc")" + [ -f "$ip" ] || continue + body="$body +$(cat "$ip")" + local inc2 ip2 + for inc2 in $(grep -o '{{> *[^} ]*' "$ip" 2>/dev/null | sed 's/{{> *//'); do + ip2="$(resolve "$inc2")" + [ -f "$ip2" ] && body="$body +$(cat "$ip2")" + done + done + printf '%s\n' "$body" +} + +# ---- the served prompt surface: every prompt-bearing file, both trees ------- +served_corpus() { + local f + for f in CLAUDE.md AGENTS.md; do [ -f "$f" ] && expand "$f"; done + for f in $(find codev/roles codev-skeleton/roles -name '*.md' 2>/dev/null); do expand "$f"; done + for f in $(find codev/protocols codev-skeleton/protocols -name '*.md' 2>/dev/null); do expand "$f"; done +} + +CORPUS="$(served_corpus)" + +# ---- expected sets (from structured/source truth) -------------------------- +GATES=$(python3 - <<'PY' +import json,glob +g=set() +for f in glob.glob('codev*/protocols/*/protocol.json'): + try: d=json.load(open(f)) + except Exception: continue + for p in d.get('phases',[]): + v=p.get('gate') + if isinstance(v,str): g.add(v) + elif isinstance(v,dict) and isinstance(v.get('name'),str): g.add(v['name']) +print('\n'.join(sorted(g))) +PY +) +CHECKS=$(python3 - <<'PY' +import json,glob +c=set() +for f in glob.glob('codev*/protocols/*/protocol.json'): + try: d=json.load(open(f)) + except Exception: continue + for p in d.get('phases',[]): + c |= set((p.get('checks') or {}).keys()) +print('\n'.join(sorted(c))) +PY +) + +# Normalization: lowercase, strip backticks/quotes/punctuation noise. +norm() { tr '[:upper:]' '[:lower:]' | tr -d '`"'"'"'*'; } +CORPUS_N="$(printf '%s' "$CORPUS" | norm)" + +# NOTE: a here-string, deliberately NOT `printf ... | grep -q`. +# Under `set -o pipefail`, `grep -q` exits on its FIRST match and closes the +# pipe; the upstream printf then dies of SIGPIPE and the pipeline reports +# failure — so every capability that WAS found reported as absent. The exit +# status had two causes and the code read only one of them. (Same failure class +# this project has been recording: trust the authoritative signal, not the +# convenient one.) +present() { + local needle; needle="$(printf '%s' "$1" | norm)" + if grep -qF -- "$needle" <<<"$CORPUS_N"; then echo true; else echo false; fi +} + +# ---- signals, artifact paths, notification triggers: found IN the corpus ---- +SIGNALS=$(printf '%s' "$CORPUS" | grep -o ']*>[A-Z_]*' | grep -o '[A-Z_]\{4,\}' | sort -u || true) +ARTIFACTS=$(printf '%s' "$CORPUS" | grep -oE 'codev/(specs|plans|reviews|state)/[A-Za-z0-9{}_.*-]+' | sort -u || true) +NOTIFY=$(printf '%s' "$CORPUS" | grep -oE 'afx send [a-z:<>{}-]+' | sort -u || true) + +json_array() { # each stdin line -> {"name":..,"present":..} + local first=1 + while IFS= read -r item; do + [ -z "$item" ] && continue + [ $first -eq 1 ] || printf ',\n' + first=0 + printf ' {"name": %s, "present_in_served_prompts": %s}' \ + "$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$item")" "$(present "$item")" + done + [ $first -eq 1 ] || printf '\n' +} + +cat </dev/null || echo n/a)", + "gates": [ +$(printf '%s\n' "$GATES" | json_array) ], + "checks": [ +$(printf '%s\n' "$CHECKS" | json_array) ], + "signals": [ +$(printf '%s\n' "$SIGNALS" | json_array) ], + "artifact_paths": [ +$(printf '%s\n' "$ARTIFACTS" | json_array) ], + "notification_triggers": [ +$(printf '%s\n' "$NOTIFY" | json_array) ] +} +EOF diff --git a/scripts/measure-prompt-surface.sh b/scripts/measure-prompt-surface.sh index 5dda6c769..d96402a4c 100755 --- a/scripts/measure-prompt-surface.sh +++ b/scripts/measure-prompt-surface.sh @@ -1,14 +1,40 @@ #!/usr/bin/env bash # # measure-prompt-surface.sh — word-count measurement of Codev's prompt surface. -# Spec 1252, criterion M6 / test T1. +# Originally Spec 1252 (M6/T1); CORRECTED under Spec 1280 (M0). # -# Run before any trimming (Phase 1) and again after (Phase 7); the delta is the -# N1 figure. Emits a markdown table on stdout. +# WHAT THIS MEASURES, AND WHY YOU SHOULD DISTRUST THE PREVIOUS VERSION +# ------------------------------------------------------------------- +# Spec 1280 found three defects in the 1252 version. All three produced numbers +# that were precise and wrong: # -# The number that matters is ALWAYS-ON: content that enters an agent's context -# whether or not it is needed. Total authored surface is much larger (~150k -# words) but most of it is on-demand and already correctly tiered. +# 1. It derived the phase-task term from `codev-skeleton/porch/prompts/` — a +# dead Ralph-SPIR-era tree with NO runtime consumer. The live resolver +# (commands/porch/prompts.ts, loadPromptFile) loads +# `protocols//prompts/.md`. Real SPIR phase prompts average +# 1,398 words; the dead tree averaged 400. The metric was structurally +# blind to the largest always-on surface in the system. +# 2. It omitted `roles/builder.md` (1,837 words), which spawn-worktree.ts:854 +# writes to `.builder-role.md` for harness injection on EVERY builder spawn. +# 3. Its comment asserted CLAUDE.md "already inlines" the hot tier. Since #1119 +# (lib/managed-block.ts) CLAUDE.md carries `@import` lines, which Claude Code +# TRANSCLUDES at session launch. So `wc -w CLAUDE.md` EXCLUDES 736 words that +# are always loaded. +# +# Reported baseline went 21,702 -> 34,255 on correction. Spec 1280's principle: +# the instrument is part of the deliverable, and instruments get reviewed against +# what they claim to measure — not merely against whether they run. +# +# TWO REPORTING BASES — deliberately different, do not conflate +# ------------------------------------------------------------ +# ALWAYS-ON: what rides into an agent's context whether needed or not. +# Dedupes twins (AGENTS.md is CLAUDE.md's byte-identical twin; +# one loads per session) and EXPANDS @import / {{> ...}}. +# TOTAL AUTHORED: physical files on disk, each counted once, NO twin dedup and +# NO expansion. Its job is to detect RELOCATION — content moved +# out of an always-on file must still show up somewhere. Under +# an always-on-only metric, moving 3,900 words into a skill +# scores identically to deleting them (Spec 1280 M0c). # # Usage: scripts/measure-prompt-surface.sh [repo-root] set -euo pipefail @@ -16,125 +42,230 @@ set -euo pipefail ROOT="${1:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" cd "$ROOT" +# WORD COUNT IS DEFINED HERE, NOT DELEGATED TO `wc -w`. +# +# `wc -w` is NOT portable for this corpus. macOS/BSD wc in a UTF-8 locale counts +# `⚠️` (U+26A0 WARNING SIGN + U+FE0F VARIATION SELECTOR-16) as TWO words; GNU wc +# on Linux, `LC_ALL=C wc`, and Python's str.split() all count it as one. There +# are four such banners in spir/protocol.md alone, so the same commit measured +# 34,235 on a developer's Mac and 34,231 in CI. +# +# For an instrument whose entire purpose is an honest before/after comparison, +# a platform-dependent count is a correctness defect: measure "before" on one +# machine and "after" on another and the delta is fiction. So the definition is +# made explicit and deterministic — a word is a whitespace-delimited token of +# the UTF-8 decoded text, per Python's str.split(). +# +# (The irony is recorded rather than smoothed over: the characters that broke +# portability are the `⚠️ BLOCKING` worst-case-padding banners that principle P7 +# exists to delete.) +_count() { # count words on stdin, deterministically + python3 -c 'import sys; print(len(sys.stdin.read().split()))' +} w() { # word count of a file, 0 if absent - [ -f "$1" ] && wc -w < "$1" | tr -d ' ' || echo 0 + [ -f "$1" ] || { echo 0; return; } + _count < "$1" } wdir() { # word count of all .md under a dir, 0 if absent - [ -d "$1" ] && find "$1" -name '*.md' -exec cat {} + 2>/dev/null | wc -w | tr -d ' ' || echo 0 + [ -d "$1" ] || { echo 0; return; } + find "$1" -name '*.md' -exec cat {} + 2>/dev/null | _count } fdir() { # count of .md files under a dir [ -d "$1" ] && find "$1" -name '*.md' | wc -l | tr -d ' ' || echo 0 } +# --- Four-tier resolution, PER FILE (Spec 1280 M0 item b) -------------------- +# Mirrors lib/skeleton.ts resolveCodevFile. The 1252 version did TWO-tier, +# DIRECTORY-level selection, so a project overriding a single prompt in .codev/ +# while the rest resolved from the skeleton was measured at the wrong tier. +# Tier 3 (runtime cache) is not present in a checkout and is skipped here; it is +# a cache of tier 4, so it cannot change the resolved CONTENT for measurement. +resolve() { + if [ -f ".codev/$1" ]; then echo ".codev/$1" + elif [ -f "codev/$1" ]; then echo "codev/$1" + elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" + else echo /dev/null; fi +} + +# SERVED words: {{> ...}} includes are expanded into the prompt, so the proxy +# must count them or dedup-by-include would claim phantom savings (it changes +# authored ownership, not served bytes). Extension-agnostic: the runtime +# resolver inlines any file type, including protocol.json (Spec 1280 P6). +# +# This performs REAL TEXTUAL SUBSTITUTION rather than adding the include's word +# count to the host file's. The 1252 version added, which double-counted the +# directive's own tokens (`{{>` and the path are 2 words to `wc -w`) on top of +# the content that replaces them — so every include over-reported by ~2 words +# and moving text into a template was not exactly neutral. Substituting makes +# the phantom-savings property exact, which is what T2 asserts. +# Mirrors lib/skeleton.ts resolveCodevIncludes EXACTLY: a regex replace of the +# directive *within* the surrounding text (not a line-wise swap — text sharing a +# line with an include must survive), recursive, depth-guarded at 5, with an +# unresolved include collapsing to empty. +expand_text() { + python3 - "$1" <<'PY' +import re, sys, os +DIRECTIVE = re.compile(r'\{\{>\s*([^}\s]+)\s*\}\}') + +def resolve(rel): + for base in ('.codev', 'codev', 'codev-skeleton'): + p = os.path.join(base, rel) + if os.path.isfile(p): + return p + return None + +def expand(text, depth=0): + if depth > 5: + return text + def sub(m): + p = resolve(m.group(1)) + if not p: + return '' + with open(p, encoding='utf-8', errors='replace') as fh: + return expand(fh.read(), depth + 1) + return DIRECTIVE.sub(sub, text) + +path = sys.argv[1] +if os.path.isfile(path): + with open(path, encoding='utf-8', errors='replace') as fh: + sys.stdout.write(expand(fh.read())) +PY +} +expanded_w() { [ -f "$1" ] && expand_text "$1" | _count || echo 0; } + CLAUDE_MD=$(w CLAUDE.md) AGENTS_MD=$(w AGENTS.md) ARCH_CRIT=$(w codev/resources/arch-critical.md) LESS_CRIT=$(w codev/resources/lessons-critical.md) ARCH=$(w codev/resources/arch.md) LESS=$(w codev/resources/lessons-learned.md) +HOT=$(( ARCH_CRIT + LESS_CRIT )) -SKEL_W=$(wdir codev-skeleton); SKEL_F=$(fdir codev-skeleton) -SKEL_PROTO_W=$(wdir codev-skeleton/protocols) -SKEL_ROLES_W=$(wdir codev-skeleton/roles) -SHADOW_PROTO_W=$(wdir codev/protocols); SHADOW_PROTO_F=$(fdir codev/protocols) -SHADOW_ROLES_W=$(wdir codev/roles); SHADOW_ROLES_F=$(fdir codev/roles) -SKILLS_W=$(wdir .claude/skills); SKILLS_F=$(fdir .claude/skills) - -# Always-on = what rides into context regardless of need. -# CLAUDE.md already inlines the two hot-tier files, so they are NOT re-added -# here (double-counting them would inflate the baseline and flatter the result). -# AGENTS.md is the byte-identical twin for other tools — one or the other -# loads per session, never both, so it is excluded too. -# -# SPAWN_PROMPT and PHASE_TASK are DERIVED from the artifacts that compose them, -# resolved the way the runtime resolves (tier-2 codev/ first, then the shipped -# skeleton), so Phase-7 trims to any component show up in the rerun. Hardcoding -# the one-off measured values (4891 / 1395) broke reproducibility — a rerun -# after trimming would have reported the pre-trim numbers (caught by Codex at -# the Phase-1 review). -# -# SPAWN_PROMPT proxy = spir builder-prompt.md + spir protocol.md (inlined into -# every spawn prompt). The real spawn prompt adds the issue body (~170 words -# here), which varies per project and is not a trimmable prompt surface, so -# it is deliberately excluded from the proxy. -# -# PHASE_TASK proxy = hot tier (injected into every porch phase prompt) + the -# mean of the porch phase prompts. Task-JSON boilerplate varies per phase and -# is porch code, not prompt surface. -resolve() { # two-tier resolve: codev/ wins, else skeleton - if [ -f "codev/$1" ]; then echo "codev/$1" - elif [ -f "codev-skeleton/$1" ]; then echo "codev-skeleton/$1" - else echo /dev/null; fi +# --- SHARED bucket ----------------------------------------------------------- +# CLAUDE.md @imports the hot tier (#1119) rather than inlining it, so the hot +# words are ADDED here. AGENTS.md is excluded: byte-identical twin, one loads. +SHARED=$(( CLAUDE_MD + HOT )) + +# --- ARCHITECT bucket -------------------------------------------------------- +ARCHITECT_ROLE=$(w "$(resolve roles/architect.md)") +ARCHITECT=$ARCHITECT_ROLE + +# --- Per-protocol buckets ---------------------------------------------------- +BUILDER_ROLE=$(w "$(resolve roles/builder.md)") # inlined at spawn (defect 2) +CONSULTANT_ROLE=$(w "$(resolve roles/consultant.md)") + +protocols() { # every protocol in EITHER tree, unioned, deduped + { ls -d codev/protocols/*/ 2>/dev/null; ls -d codev-skeleton/protocols/*/ 2>/dev/null; } \ + | xargs -n1 basename 2>/dev/null | sort -u } -PHASE_ITERS="${PHASE_ITERS:-10}" -# SERVED words, not authored: {{> partials/...}} includes are expanded into -# every spawn prompt, so the proxy must count them or dedup-by-include would -# claim phantom always-on savings (it changes authored ownership, not served -# bytes). -expanded_w() { # file + partial includes, two levels deep (partials may nest) - local f="$1"; local total; total=$(w "$f") - for inc in $(grep -o '{{> *[^} ]*' "$f" 2>/dev/null | sed 's/{{> *//'); do - local ip; ip="$(resolve "$inc")" - total=$(( total + $(w "$ip") )) - for inc2 in $(grep -o '{{> *[^} ]*' "$ip" 2>/dev/null | sed 's/{{> *//'); do - total=$(( total + $(w "$(resolve "$inc2")") )) - done + +spawn_words() { # BUILDER_SPAWN[p] = builder role + wrapper + protocol.md + local p="$1" + echo $(( BUILDER_ROLE \ + + $(expanded_w "$(resolve "protocols/$p/builder-prompt.md")") \ + + $(expanded_w "$(resolve "protocols/$p/protocol.md")") )) +} + +phase_mean() { # mean expanded phase prompt for protocol p, 0 if it has none + local p="$1" total=0 n=0 f + for f in $(ls "codev/protocols/$p/prompts/"*.md "codev-skeleton/protocols/$p/prompts/"*.md 2>/dev/null \ + | xargs -n1 basename 2>/dev/null | sort -u); do + total=$(( total + $(expanded_w "$(resolve "protocols/$p/prompts/$f")") )); n=$(( n + 1 )) + done + [ "$n" -gt 0 ] && echo $(( total / n )) || echo 0 +} + +consult_mean() { # mean consult-type prompt for protocol p, 0 if it has none + local p="$1" total=0 n=0 f + for f in $(ls "codev/protocols/$p/consult-types/"*.md "codev-skeleton/protocols/$p/consult-types/"*.md 2>/dev/null \ + | xargs -n1 basename 2>/dev/null | sort -u); do + total=$(( total + $(w "$(resolve "protocols/$p/consult-types/$f")") )); n=$(( n + 1 )) done - echo "$total" + [ "$n" -gt 0 ] && echo $(( total / n )) || echo 0 } -SPAWN_BP=$(expanded_w "$(resolve protocols/spir/builder-prompt.md)") -SPAWN_PROTO=$(w "$(resolve protocols/spir/protocol.md)") -SPAWN_PROMPT=$(( SPAWN_BP + SPAWN_PROTO )) -PORCH_PROMPT_MEAN=0 -PORCH_DIR="$( [ -d codev/porch/prompts ] && echo codev/porch/prompts || echo codev-skeleton/porch/prompts )" -if [ -d "$PORCH_DIR" ]; then - PORCH_N=$(find "$PORCH_DIR" -name '*.md' | wc -l | tr -d ' ') - if [ "$PORCH_N" -gt 0 ]; then - PORCH_TOTAL=0 - for pf in "$PORCH_DIR"/*.md; do PORCH_TOTAL=$(( PORCH_TOTAL + $(expanded_w "$pf") )); done - PORCH_PROMPT_MEAN=$(( PORCH_TOTAL / PORCH_N )) - fi -fi -PHASE_TASK=$(( ARCH_CRIT + LESS_CRIT + PORCH_PROMPT_MEAN )) -ALWAYS_ON=$(( CLAUDE_MD + SPAWN_PROMPT + PHASE_TASK * PHASE_ITERS )) + +# --- TOTAL AUTHORED (relocation detector, M0 item g) ------------------------- +# Physical files, no dedup, no expansion. Counts ALL FOUR skill trees: Spec 1280 +# found .claude/skills and .codex/skills are byte-identical and the skeleton +# ships its own copies of both. Counting only .claude/skills would report +# content relocated into a skill as DELETED — inverting M0c. +SKILLS_CLAUDE=$(wdir .claude/skills) +SKILLS_CODEX=$(wdir .codex/skills) +SKILLS_SKEL_CLAUDE=$(wdir codev-skeleton/.claude/skills) +SKILLS_SKEL_CODEX=$(wdir codev-skeleton/.codex/skills) +SKILLS_ALL=$(( SKILLS_CLAUDE + SKILLS_CODEX + SKILLS_SKEL_CLAUDE + SKILLS_SKEL_CODEX )) +SKILLS_F_ALL=$(( $(fdir .claude/skills) + $(fdir .codex/skills) \ + + $(fdir codev-skeleton/.claude/skills) + $(fdir codev-skeleton/.codex/skills) )) + +AUTH_PROTO_CODEV=$(wdir codev/protocols); AUTH_PROTO_SKEL=$(wdir codev-skeleton/protocols) +AUTH_ROLES_CODEV=$(wdir codev/roles); AUTH_ROLES_SKEL=$(wdir codev-skeleton/roles) +TOTAL_AUTHORED=$(( CLAUDE_MD + AGENTS_MD + AUTH_PROTO_CODEV + AUTH_PROTO_SKEL \ + + AUTH_ROLES_CODEV + AUTH_ROLES_SKEL + SKILLS_ALL )) + +# --- DEAD -------------------------------------------------------------------- +DEAD_W=$(wdir codev-skeleton/porch/prompts); DEAD_F=$(fdir codev-skeleton/porch/prompts) + +# --- Headline ---------------------------------------------------------------- +# ALWAYS_ON_WORDS == ALWAYS_ON(builder, spir, I). I is a COMPARISON CONSTANT, +# identical before and after; it is not a claim about any real project. +PHASE_ITERS="${PHASE_ITERS:-10}" +HEADLINE_PROTO="${HEADLINE_PROTO:-spir}" +SPAWN_SPIR=$(spawn_words "$HEADLINE_PROTO") +PHASE_MEAN_SPIR=$(phase_mean "$HEADLINE_PROTO") +ALWAYS_ON=$(( SHARED + SPAWN_SPIR + PHASE_ITERS * (HOT + PHASE_MEAN_SPIR) )) +ALWAYS_ON_ARCHITECT=$(( SHARED + ARCHITECT )) +ALWAYS_ON_CONSULTANT=$(( CONSULTANT_ROLE + $(consult_mean "$HEADLINE_PROTO") )) cat </dev/null || echo n/a)\` +Instrument: corrected under Spec 1280 (M0). Supersedes the Spec 1252 version. + +## Exclusive buckets (partition the authored surface; these SUM) + +| Bucket | Words | +|---|---:| +| SHARED (CLAUDE.md + transcluded hot tier) | $SHARED | +| ARCHITECT (roles/architect.md) | $ARCHITECT | +| DEAD (codev-skeleton/porch/prompts, $DEAD_F files) | $DEAD_W | + +## Per-protocol (resolved per file, four-tier) + +| Protocol | BUILDER_SPAWN | PHASE mean | CONSULT mean | +|---|---:|---:|---:| +EOF +for p in $(protocols); do + printf "| %s | %s | %s | %s |\n" "$p" "$(spawn_words "$p")" "$(phase_mean "$p")" "$(consult_mean "$p")" +done + +cat <