BigSimmo · BigSimmo · Jul 13, 2026 · Jul 13, 2026 · Jul 13, 2026 · Jul 13, 2026
@@ -392,3 +392,9 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-13 | origin/codex/rag-canary-completion | d1b7647bde73e7d0558472ad6aa19c3b49d94639 | branch-cleanup | Retained for open PR #612 and active worktree activity. | Fresh GitHub open-PR query and moving-head observation; deletion prohibited. |
 | 2026-07-13 | origin/main | 528a1752f41cd29a518ca9341c93c724030173ae | branch-cleanup | Protected base branch retained. | Final refreshed origin/main snapshot before the ledger PR. |
 | 2026-07-14 | PR #629 / codex/eval-canary-quota-handling | 8fe1be6d0c58c60afeeb82f720b03c90ce57c2cf | review-followup | One P2 structured-error retry defect confirmed and fixed on `codex/eval-canary-structured-errors`. | GitHub connector thread inspection; `tests/eval-utils.test.ts` 14/14; focused ESLint and Prettier. |
+| 2026-07-14 | claude/repo-next-steps-e53523 | 3f7d6d76f597f2a0311af1052480f84b68ecc259 | launch-readiness and RAG-performance follow-up | Branch changes were already squash-merged by PR #513. Fresh review found one P1 governance inconsistency (Singapore app/worker processing omitted from the PIA's cross-border account) and three P2 follow-ups: degraded-answer SLO overcounting, a second blocking shared-cache miss diagnostic query, and stale launch/RAG backlog status. No source fix was applied in this review. | `git diff --check c3828ceb9f3812abeebc1b653361fc254dda9f5e..3f7d6d76f597f2a0311af1052480f84b68ecc259`; focused Vitest 92/92; `npm run eval:rag:offline` 36 fixtures and 59/59 contract tests; `npm run verify:cheap` passed runtime, action pins, sitemap, type scale, lint, typecheck, and 1,672 passed/1 skipped tests. Provider-backed Supabase/OpenAI, browser, release, drift, and live retrieval-quality checks were not run. |
+| 2026-07-14 | codex/rag-performance-followups | 5502fd498ea2069f810795a8659f98ab3abf8c80 | release-readiness review | The local release review found no P0-P2 defect after correcting one stale PIA statement; the later hosted review follow-up is recorded below. The scoped RAG round-trip, SLO, retention, privacy, and documentation changes were ready for PR handoff. Highest residual risk is the known live hybrid-RPC latency tail; model experiments remain blocked by provider quota and legal execution remains operator/counsel work. | Rebased onto `origin/main`; runtime and full Prettier check; ESLint; TypeScript; Vitest 2,213 passed/1 skipped; Next.js production build (636 pages) plus client-bundle secret scan; offline RAG 36 fixtures and 277/277 contract tests; `git diff --check`. Live retention jobs 13/16 were already verified during this workstream. |
+| 2026-07-14 | PR #632 / codex/rag-performance-followups | aa264e92c44b42fdcceeac6292011ba51169b862 | review-followup | One P2 SLO-classification defect was confirmed: the broad source-only `degraded` flag also included healthy extractive answers. Fixed by persisting and counting a separate `provider_generation_degraded` flag derived only from `generation_fallback`. | GitHub connector plus UTF-8 thread-aware review inspection; focused ESLint; 45/45 targeted Vitest tests; TypeScript; offline RAG 36 fixtures and 277/277 contract tests; `git diff --check`. |
+| 2026-07-14 | PR #632 / codex/rag-performance-followups | 07cac8f4e094bf4b5ccd264feb6420fbd4e35e2b | review-followup | CodeRabbit's APP 1/5 governance contradiction was confirmed and corrected; Railway's pending DPA and Singapore processor basis are now explicit checklist/status items. The optional cache-miss granularity suggestion was not adopted because it would undo the reviewed single-read cold-miss design or require a broader schema/RPC change. | GitHub review-thread inspection; focused Prettier and `git diff --check` passed. Local production-readiness inspection reported the expected missing provider env; the Railway production-context check had already passed on this workstream. |
+| 2026-07-14 | PR #632 / codex/rag-performance-followups | 0c792b8c10516c85d18832fc96562dd70a671b5f | review-followup | One P1 privacy-disclosure defect was confirmed: OpenAI retrieval embedding can receive query text even when answer synthesis remains source-only. Fixed the PIA and public `/privacy` copy to distinguish retrieval-embedding egress from model-backed synthesis egress, with focused copy coverage. | GitHub connector review-thread inspection; focused privacy UI test, ESLint, TypeScript, Prettier, production-readiness, and `git diff --check`. |
+| 2026-07-14 | PR #632 / codex/rag-performance-followups | a2eb6db0efbef983e1b3242261d5cc6b2b9d839d | review-followup | One late P2 rollout-compatibility defect was confirmed: the provider-fallback SLO would omit recent rows written before `provider_generation_degraded` existed. Fixed the count predicate to include the new flag or legacy `generation_fallback:` reasons while continuing to exclude intentional extractive routes. | GitHub connector review-thread inspection; focused answer-SLO test, ESLint, TypeScript, Prettier, and `git diff --check`. |
@@ -61,11 +61,13 @@ properties made it the pragmatic core platform:
 **The trade Railway forces — and why it was accepted:** Railway has **no
 Australian region** (its regions are US West, US East, Amsterdam, Singapore),
 while Supabase is in Sydney. The app therefore pays a cross-region hop to the
-database (quantified in §2.1). The penalty is real but bounded, falls almost
-entirely on _novel_ (cache-miss) answers, is dwarfed by OpenAI generation time,
-and is fully addressable within Railway (a Singapore read replica — §2.1). Given
-that shipping a real environment was the priority, a live-in-Singapore deployment
-beat an indefinitely blocked "optimal" one.
+database (quantified in §2.1). Warm responses show that this hop can be bounded,
+but 2026-07-14 cold probes also exposed database-execution outliers far larger
+than network RTT. A Singapore replica would copy those query plans rather than
+fix them, so the current first lever is RPC/query-plan work and, if needed, a
+small primary-compute comparison. Given that shipping a real environment was the
+priority, a live-in-Singapore deployment beat an indefinitely blocked "optimal"
+one.
 
 ### Why a long-lived container and not serverless (Vercel et al.)
 
@@ -116,6 +118,15 @@ inherent per RPC.
 | Authenticated PostgREST round-trip | Real per-RPC cost the app pays                          | **~145 ms best, ~340 ms typical**                     |
 | Production novel answer            | `supabase_rpc_latency_ms` (retrieval, 3 query variants) | **~4.4 s**; total ~25 s incl. ~19 s OpenAI generation |
 
+**Fresh production comparison, 2026-07-14:** one warmed repeat reported
+`supabase_rpc_latency_ms=0`, while two cold synthesis probes reported
+**48.5–49.4 s** of Supabase RPC time and **51–53 s** total. A six-case approved
+live-database retrieval run on the current local code preserved perfect fixture
+recall/hit-rate with **1.8 s median / 47.3 s p90** latency. Database statistics
+also show large temporary-file I/O in the slow hybrid RPC families. These
+outliers supersede the earlier assumption that generation or cross-region RTT is
+always the dominant latency.
+
 **What multiplies, what doesn't.** Retrieval fans out _wide_ but the RPCs within
 a stage run in parallel (`Promise.all`), so fan-out width costs ~1×RTT, not N×.
 What multiplies RTT is the sequential **depth** — ~5–8 serial DB round-trips on a
@@ -134,17 +145,22 @@ shared Postgres cache tier still pay ~one origin round-trip.
 - keep-alive connection reuse (undici default) removes repeated client→edge
   handshakes on the hot path.
 
-**Mitigations if the penalty starts to bite (not yet needed):**
+**Mitigations, in current priority order:**
 
-1. **Supabase read replica in Singapore** (biggest win). Co-locating a read
-   replica with Railway serves the read-heavy retrieval RPCs locally (~5 ms),
-   collapsing the penalty to near-parity with full co-location. Writes (telemetry,
-   cache) stay on the Sydney primary and are off the answer critical path. Cost:
-   the paid replica add-on **plus** app work to route reads to the replica
-   endpoint. This is the recommended first lever.
+1. **Profile and optimise the slow hybrid RPC plans**, then compare the Sydney
+   primary on Micro versus Small compute if execution remains resource-bound.
+   The live primary currently exposes the Micro connection ceiling (60 direct
+   connections) and a small `work_mem`; scaling the primary tests query-memory
+   headroom without adding read routing or replica staleness.
 2. **Reduce sequential DB depth** in the answer path (batch the cache-version +
    shared-cache probes, collapse hydration round-trips). App change; measure
    `latencyTimings.supabase_rpc_latency_ms` before/after.
+3. **Reconsider a Singapore read replica only after the query plans are fast and
+   network time is again material.** Supabase currently requires at least Small
+   compute for read replicas; a same-size replica is asynchronous/read-only and
+   adds compute/storage cost plus read-routing and freshness validation. It is
+   not justified by the current evidence because it would reproduce the observed
+   execution outliers.
 
 The OpenAI leg is region-agnostic: OpenAI is US-hosted, so app→OpenAI RTT is
 comparable (~200 ms) from Singapore or Sydney and does not favour either host.

@@ -5,9 +5,14 @@
 runs automatically — every **⏸ PAUSE** is a provider-touching action (Supabase / Railway / OpenAI /
 GitHub) that needs your explicit go-ahead, per the AGENTS.md provider boundary.
 
+**Current-state note (2026-07-14):** production app and worker deployment is already recorded as live.
+Use this as a verification/release sequence, not evidence that staging, migrations, or first deployment
+are still pending. Confirm current provider state before repeating any historical action.
+
 Host note: production app + worker run on **Railway** (user directive 2026-07-12), not Fly. The image is
 host-agnostic. Railway has no Sydney region (closest Singapore); data at rest stays in Supabase Sydney,
-so this is a latency/SLO tradeoff only — confirm answer-p95 in the staging soak (step 4).
+so this creates both a latency/SLO tradeoff and overseas processing that must be covered by the approved
+privacy/processor record. Confirm answer-p95 in the staging soak (step 4).
 
 Legend: **⏸ PAUSE** = provider action, needs your approval · **✅ verify** = check to run after.
 
@@ -17,9 +22,9 @@ Legend: **⏸ PAUSE** = provider action, needs your approval · **✅ verify** =
 
 ```text
 0. Pre-flight identity check
-1. Apply pending live migrations  (July-8 batch + PIA-4 + drift-codify)   [Supabase]
-2. Run the full release gate                                              [live keys]
-3. Provision staging + seed                                               [Supabase + Railway]
+1. Confirm completed migrations; apply only any explicitly unresolved control [Supabase]
+2. Run the full release gate before the next release                     [live keys]
+3. Provision staging + seed, only if still absent                         [Supabase + Railway]
 4. Staging soak + rollback rehearsal                                      [Railway]
 5. Production deploy                                                       [Railway]
 6. Post-deploy: worker, registry seed, auth conn cap, observability wiring
@@ -35,26 +40,27 @@ npx supabase migration list --linked
 npm run reindex:health             # note jobs_pending / jobs_processing (needed for step 1 R17)
 ```
 
-## 1. Apply pending live migrations 🧑 Supabase
+## 1. Confirm migration state; apply only unresolved controls 🧑 Supabase
+
+The July-8 ingestion/tenancy batch and the retrieval drift-codification work are recorded as applied and
+verified on 2026-07-13. Their detailed procedures remain for staging/disaster recovery; do **not** reapply
+them merely because they appear below. First compare linked migration history and verify the remaining
+PIA-4 retention migrations. **Do not redeploy the worker until `20260708130000` is confirmed live.**
 
-Detailed runbook: [operator-apply-july8-batch.md](operator-apply-july8-batch.md). Apply **in this order**
-when the ingestion queue is quiet. **Do not redeploy the worker until step `20260708130000` is live.**
+| Group | Migration/control                                                                                | Recorded status                                     | Operator action                                                                                            |
+| ----- | ------------------------------------------------------------------------------------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
+| a–f   | July-8 ingestion/tenancy batch in [operator-apply-july8-batch.md](operator-apply-july8-batch.md) | **Verified live 2026-07-13**                        | Verify only; redeploy the worker if that recorded remaining action is still open.                          |
+| g     | `20260708120000_rag_query_misses_retention`                                                      | **Applied and verified live 2026-07-14**            | Job 13 is active with the 90-day retention window.                                                         |
+| h     | `20260713201542_consolidate_rag_response_cache_retention`                                        | **Applied and verified live 2026-07-14**            | Job 16 is active and bounded; obsolete duplicate job confirmed absent.                                     |
+| i     | Retrieval RPC forward-codification (`20260713062107`…`20260713062139`)                           | **Applied and drift/readiness verified 2026-07-13** | Verify only; see [forward-codify-retrieval-rpcs-workorder.md](forward-codify-retrieval-rpcs-workorder.md). |
 
-| #   | Migration                                             | Note                                        |
-| --- | ----------------------------------------------------- | ------------------------------------------- |
-| a   | `20260708140000_drop_ingestion_job_stages_job_id_fk`  | no-op on live                               |
-| b   | `20260708130000_ingestion_concurrency_rpc_hardening`  | **worker-redeploy blocker**                 |
-| c   | `20260708150000_ensure_retrieval_owner_matches`       | helper before fail-closed                   |
-| d   | `20260708160001_retrieval_owner_matches_fail_closed`  | tenancy fail-closed (#409)                  |
-| e   | `20260708310000_r5_document_metadata_merge`           | R5 deep-merge (#408)                        |
-| f   | `20260708170000_ingestion_jobs_one_open_per_document` | R17 — approved manual `CONCURRENTLY` path   |
-| g   | `20260708120000_rag_query_misses_retention`           | **PIA-4** purge cron                        |
-| h   | `<drift-codify-forward>`                              | **only after task 1.2 lands** — see step 1b |
+**⏸ PAUSE:** if and only if linked history shows migration `20260708120000` or `20260713201542` absent,
+apply the reviewed committed migration through the normal guarded workflow. Do not use this status
+reconciliation as authority to replay the verified July-8 or forward-codification groups.
 
-**⏸ PAUSE:** apply via `supabase db push` (queue quiet) or the R17 manual `CREATE UNIQUE INDEX CONCURRENTLY`
-path in the July-8 doc. R17 manual path is an approved exception to the live-change guardrail; record
-the migration history entry and reconcile schema.sql after manual execution to prevent untracked drift.
-R17 uses its own version so history/repair can't collide with `20260708160001`.
+After applying, query `cron.job` and expect exactly these active retention jobs: `purge-rag-query-misses`
+at `45 3 * * *` and `purge-rag-response-cache` at `15 * * * *`. The obsolete
+`purge-expired-rag-response-cache` name must be absent.
 
 **✅ verify:**
 
@@ -65,20 +71,17 @@ npm run check:indexing              # search_schema_health() ok
 npm run eval:retrieval:quality      # must stay 36/36 (retrieval-affecting: step d + drift-codify)
 ```
 
-### 1b. Drift-codify apply (task 1.2)
+### 1b. Drift-codify status (task 1.2) — complete
 
-The forward-codify migration (live-diverged `match_document_chunks` `hnsw.ef_search=100` wrapper + `*_text`
-multi-strategy bodies) is authored + validated with normalized fingerprint comparison vs a Docker replay
-before it reaches you, so its apply is an **idempotent no-op on live**. **This step is blocked until the
-migration artifact is committed and execution-time live fingerprint recapture is completed.** Before
-applying, recapture live fingerprints using the exact committed capture query and compare normalized md5s
-against the committed table. Abort on mismatch. Apply as step 1h only after verification, then re-run
-`check:drift` + `eval:retrieval:quality` (36/36). Background:
-[database-drift-detection.md](database-drift-detection.md).
+The reviewed forward-codify migrations were applied on 2026-07-13 after scratch replay and fingerprint
+validation. `check:drift` passed and production readiness reported READY. The authoritative evidence and
+historical replay procedure are in
+[forward-codify-retrieval-rpcs-workorder.md](forward-codify-retrieval-rpcs-workorder.md). No further apply
+is pending unless a new drift check identifies a new, separately reviewed difference.
 
 ## 2. Full release gate 🧑 live keys
 
-Clears the accumulated verification debt (universal search, cross-mode links, rag.ts decomp).
+Run before the next release when provider-backed verification is explicitly approved.
 
 **⏸ PAUSE** (bounded OpenAI spend):
 
@@ -161,6 +164,6 @@ no scale-to-zero, health `/api/health`. I'll prep the Railway service config via
 ## Standing guardrails
 
 - Never raw-SQL against live — committed migration + `schema.sql` reconciliation only.
-- Worker redeploy is blocked until `20260708130000` is live.
+- Never redeploy a worker image that expects the hardened completion RPC until `20260708130000` is confirmed live.
 - Any retrieval/ranking change re-runs `eval:retrieval:quality` 36/36 before it ships.
 - Each environment gets separate service-role + OpenAI keys (per-env blast radius).
@@ -118,7 +118,12 @@ means the OpenAI path is broken while users still get 200s.
 - **Warn:** > 20 % over 1 h. **Page:** > 50 % over 1 h (generation is
   effectively down).
 
-Measure via `metadata->>'fallback_reason'` / `answer_model_demoted`.
+Measure provider-generation degradation via the explicit
+`metadata->>'provider_generation_degraded' = 'true'` flag. During rollout, the
+probe also counts legacy `fallback_reason ILIKE '%generation_fallback:%'` rows so
+recent pre-flag provider failures remain visible until they age out. Keep `degraded` for
+the broader source-only UI state and `fallback_reason` as diagnostic detail;
+neither is narrow enough for provider health on its own.
 
 ## 3. Nightly production eval canary
 
@@ -177,7 +182,8 @@ header (same operator gate as the Supabase probe) — returns two counter blocks
 - **`slo`** — `answerSloSnapshot` (`src/lib/observability/answer-slo.ts`) counts
   `rag_queries` over the trailing `windowMinutes` (60) and reports
   `hybridRpcErrorQueries` / `hybridRpcErrorRate` (the §2 silent-RPC-death guard)
-  and `degradedQueries` / `degradedRate` (the source-only/fallback guard). These
+  and `degradedQueries` / `degradedRate` (the explicit provider-generation
+  fallback guard, excluding intentional extractive and unsupported/source-gap outcomes). These
   are windowed rates straight from the persisted telemetry.
 
 - **`cache`** — `cacheMetricsSnapshot` (`src/lib/observability/cache-metrics.ts`)