fix(text): repair extraction glyph artifacts app-wide and route all document-derived text through formatters

[BigSimmo]

Summary

• Fixes poorly formatted document-derived text across the app: PDF-extraction glyph artifacts (typographic ligatures fi fl, soft hyphens, zero-width/control characters) no longer reach any rendered surface.
• Adds one shared, lossless primitive — normalizeExtractedGlyphs in source-text-sanitizer.ts — wired into the base compactWhitespace/readableWhitespace cleaners so all ~20 downstream formatters inherit the repair at once. Deliberately does not rejoin line-break hyphenation: a soft-wrap hyphen is indistinguishable from a real compound hyphen (low-dose, twice-daily), so fusing would corrupt clinical compounds (caught by adversarial review, with regression tests).
• Routes previously-raw render surfaces through formatters: exact quote body and quote clipboard copy (sourceTextForVerbatimQuote — verbatim wording, artifacts repaired), citation labels (formatCitationLabel/formatCompactCitationLabel now glyph-clean titles centrally, fixing mobile labels + ward-output/smart-rag/clinical-safety), document titles (cleanDisplayTitle), visual-evidence labels, table snippets, image captions/alt, evidence-map details, and AccessibleTable cells/captions.
• Provenance rows drop unknown-filler segments ("Publisher unknown · Jurisdiction unknown · review Unknown") while always keeping clinical governance warnings; the clipboard provenance line stays fully explicit (audit artifact — reconciled with the main-branch test asserting this).
• Ingestion (buildChunks) normalizes page text once, keeping stored chunk content and source-span offsets consistent so citation anchoring is unaffected.
• New scripts/backfill-text-normalization.ts (npm run backfill:text-normalization): in-place stored-text cleanup, no re-index, no re-embed (vectors frozen → retrieval unchanged by construction), dry-run by default, requires --write --confirm, writes a revertible JSON backup, refuses to run against the wrong Supabase project. Not yet run against production.
• Centralizes static UI copy (empty states, error messages, starter prompts) in src/lib/ui-copy.ts; adds tests/rendered-text-formatting.test.ts as a static guard against reintroducing raw interpolations; conventions documented in docs/process-hardening.md.
• Merged latest main (mode homes, Clinical White restyle, relevance-first selection) with conflicts resolved so design changes win on structure/styling and formatter wiring is preserved (AnswerEmptyState now feeds ModeHomeTemplate from ui-copy.ts).

Verification

• npm run verify:cheap (runtime check, eslint, typecheck, 787 unit tests) — pass on the merged branch
• npm run verify:ui — not run locally (no env in this worktree); relying on the CI Chromium UI gate
• npm run verify:release — not claimed
• npm run format:check — all files touched by this PR are clean; 9 remaining warnings are pre-existing drift from main (rag.ts, rag-provider.ts, docs, unrelated tests), intentionally not reformatted here
• npm run check:production-readiness — not run locally (no env in this worktree); no environment, Supabase config, or deployment behavior changes are included
• npm run check:deployment-readiness — n/a, no deployment behavior changes

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — no changes to grounding/verification logic; display formatters only repair extraction artifacts and never alter clinical meaning (guard tests prove dose strings, ranges, and comparison symbols survive untouched)
• No patient-identifiable document workflow was introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — the backfill script hard-fails on any other project
• Service-role keys and private document access remain server-only — backfill runs server-side via the admin client; no client-side changes to access
• Demo/synthetic content remains clearly separated — Synthetic title prefix is stripped only for display labels, not from stored data
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — governance warnings ("Review status unknown", "Not locally validated") are always shown even when filler segments are dropped; the copied provenance line remains fully explicit for audit
• Deployment classification/TGA SaMD impact — no clinical decision-support behavior changed; this is display formatting, copy centralization, and a lossless stored-text repair

Notes

• The stored-text backfill is included but not executed; run order when ready: npm run check:supabase-project → npm run backfill:text-normalization (dry-run prints changed-chunk count + samples) → --write --confirm.
• Follow-ups tracked in docs/process-hardening.md: stale index_quality_issues/extraction_quality metadata recompute; source-metadata governance debt expires 2026-07-31.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR standardizes handling of document-derived text across the application by introducing a shared glyph-normalization primitive and routing previously raw render surfaces through existing formatter helpers, while also centralizing static UI chrome copy and adding regression tests + an optional stored-text backfill script.

Changes:

• Adds normalizeExtractedGlyphs and wires it into base whitespace cleaners so downstream formatters inherit PDF-extraction glyph repair.
• Updates multiple UI surfaces (quotes, titles, citations, tables, image captions, provenance) to avoid raw interpolation of extracted text.
• Introduces a dry-run-by-default backfill script plus new tests and documentation to prevent regressions.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/source-text-sanitizer.test.ts	Adds unit coverage for glyph normalization and the new verbatim-quote cleaner.
tests/source-metadata.test.ts	Updates provenance-summary expectations (drop unknown filler segments; keep governance warnings).
tests/rendered-text-formatting.test.ts	Adds static “no raw interpolation” guards for known document-derived render surfaces.
tests/citations.test.ts	Adds coverage for glyph repair + Synthetic prefix stripping in citation labels.
src/lib/ui-copy.ts	Introduces a central module for static UI chrome copy (empty states, errors, labels).
src/lib/source-text-sanitizer.ts	Adds normalizeExtractedGlyphs, integrates into cleaners, and introduces sourceTextForVerbatimQuote.
src/lib/source-metadata.ts	Drops “unknown” filler segments from visible provenance summaries while keeping governance warnings.
src/lib/citations.ts	Cleans citation titles centrally (glyph repair + Synthetic prefix removal) before building labels.
src/lib/chunking.ts	Normalizes extracted glyphs during chunk building so stored chunk text and offsets stay consistent.
src/components/ui-primitives.tsx	Adjusts provenance UI to drop unknown filler segments while retaining governance signals.
src/components/DocumentViewer.tsx	Routes image/table captions/headings through compact display formatting and improves alt text.
src/components/DocumentManagementActions.tsx	Documents intentional use of raw stored document title for destructive confirmation UX.
src/components/ClinicalDashboard.tsx	Routes quotes/titles/snippets through formatters; centralizes empty/error copy usage.
src/components/clinical-dashboard/master-search-header.tsx	Uses cleanDisplayTitle for scoped-document display strings and tooltips.
src/components/clinical-dashboard/DocumentManagerPanel.tsx	Centralizes empty/error strings and cleans displayed document titles.
src/components/clinical-dashboard/display-text.ts	Adds cleanDisplayTitle using glyph normalization; reuses it for source titles.
src/components/clinical-dashboard/answer-status.tsx	Moves empty-state and button copy to ui-copy.ts.
src/components/AccessibleTable.tsx	Normalizes extracted glyphs in table cell text cleaning.
scripts/backfill-text-normalization.ts	Adds an opt-in, safety-guarded stored-text backfill script (no re-embed).
package.json	Adds backfill:text-normalization script entry.
docs/process-hardening.md	Documents formatter/copy conventions and the new backfill + regression-test guard.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 143ec9f220

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/components/DocumentManagementActions.tsx:203

• The delete-confirmation gate compares against document.title byte-for-byte, but titles can contain invisible extraction artifacts (soft hyphens / zero-width / control chars) that users cannot reliably see or type. That can make deletion impossible even when the user types what appears to be the exact title, and it also reintroduces glyph artifacts into a rendered surface. Consider displaying a cleaned title and comparing confirmation using normalizeExtractedGlyphs() on both sides (or allowing either exact-match or normalized-match).

            {/* Deliberately the RAW stored title: the confirmation input is compared
                against document.title verbatim, so the user must see the exact string. */}
            <p className={cn("break-words text-xs font-semibold", textMuted)}>{document.title}</p>
            <div className="flex flex-wrap justify-end gap-2">
              <button type="button" className={floatingControl} onClick={closeDialog} disabled={pending}>
                Cancel
              </button>
              <button
                type="submit"
                className={cn(primaryControl, "bg-[color:var(--danger)] hover:bg-[color:var(--danger)]")}
                disabled={pending || deleteConfirmation !== document.title}
              >
                {pending ? <Loader2 className="h-4 w-4 animate-spin" /> : <Trash2 className="h-4 w-4" />}

[chatgpt-codex-connector]

💡 Codex Review

Database/src/lib/source-text-sanitizer.ts

Lines 403 to 405 in 9e2c89c

	export function sourceTextForIndexedPage(text: string) {
	return text
	.replace(/\r/g, "\n")

Normalize page viewer text before rendering

When the selected page is rendered through sourceTextForIndexedPage, this path still starts from the raw stored page text and never calls normalizeExtractedGlyphs, unlike the other display helpers. Existing document_pages.text values containing ligatures, soft hyphens, or zero-width/control characters will therefore still show those extraction artifacts in the DocumentViewer page text even though this change cleans chunks/citations; normalize the text at the start of this helper while preserving its fixed-width spacing behavior.

---

Database/src/components/DocumentViewer.tsx

Lines 425 to 426 in 9e2c89c

	{!hasStructuredTable && image.tableTextSnippet ? (
	<p className={cn("text-sm leading-6", textMuted)}>{image.tableTextSnippet}</p>

Clean visual table snippets in DocumentViewer

When an image row has tableTextSnippet but no structured table (or the snippet is passed as markdown with clinicalOnly false), the DocumentViewer still renders the extracted table text raw here. Snippets containing fi, soft hyphens, zero-width controls, or internal image/table markers will leak on the document page despite the new cleaned caption/title path, so route tableTextSnippet through the same compact/display formatter before passing or rendering it.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Sanitize evidence-map passage samples

This only cleans the row detail, but EvidenceMapTable renders the separate bestLinkedPassage as the “Passage sample”; for render-model evidence rows that field is still assigned from the raw quote/snippet/reason just below. When a cited quote or snippet contains ligatures, soft hyphens, or internal image markers, switching to the evidence-map view still exposes those artifacts, so apply the same compact/verbatim formatter to the passage sample field too.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Normalize section paths in the backfill

When existing chunks have glyph artifacts in extracted headings, this backfill normalizes section_heading but leaves document_chunks.section_path untouched because it is not selected or patched. Search/ranking paths still combine section_path into section text/haystacks (for example src/lib/clinical-search.ts joins result.section_path), so a confirmed write run can leave the old ligatures/soft hyphens leaking into retrieval context and facets even after the heading itself was cleaned.

Useful? React with 👍 / 👎.