Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions docs/scripts-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,16 +144,28 @@ These reach live providers — they need explicit confirmation before running.
Completed migration/batch helpers kept only for provenance; retire to the `scripts/archive/`
subfolder once the underlying migration is confirmed live (work order L1).

**Already archived** (in `scripts/archive/`, still runnable as live-DB re-verification probes via
`npm run check:m13-migration` / `npm run check:july8-live-batch`): `check-m13-migration.ts`,
`check-july8-live-batch.ts`. Their `.test.ts` companion lives in `scripts/archive/` too so it stays
out of the `tests/**` run.
**Already archived** (in `scripts/archive/`, still runnable): `check-m13-migration.ts` /
`check-july8-live-batch.ts` — live-DB re-verification probes via `npm run check:m13-migration` /
`npm run check:july8-live-batch`. Their `.test.ts` companion lives in `scripts/archive/` too so it
stays out of the `tests/**` run. `backfill-document-tags.ts` / `backfill-enrichment.ts` — completed
one-time backfills, still invocable via `npm run tags:backfill` / `npm run enrich:backfill` (both
repointed at the archive path). `backfill-document-covers.mjs` — completed one-time backfill for
documents indexed before the ingestion pipeline started generating cover thumbnails directly (see
`worker/main.ts`, `worker/python/extract_pdf_assets.py`); no npm script, invoke directly via
`node scripts/archive/backfill-document-covers.mjs`. `tests/document-cover-thumbnails.test.ts`
still asserts its pagination logic by reading the archived source (`ledger #194`, 2026-08-14).

Ledger `#194` also named `backfill-gold-document-labels.ts` and `backfill-smart-index.ts` as
one-shot candidates, but both are already classified as `[live]` ongoing tooling elsewhere in this
doc (Document intelligence & governance / Ingestion, indexing & reindex) and in
`docs/codebase-index.md` — not completed migrations. Left in `scripts/` on that basis; re-flag only
if their live classification is deliberately revisited.

**Remaining candidates** (still in `scripts/`, retire once each is confirmed retired):
`check-retrieval-owner-migration.ts`, `backfill-source-metadata.ts`, `backfill-text-normalization.ts`,
`backfill-visual-intelligence.ts`, `backfill-document-tags.ts`, `backfill-enrichment.ts`,
`derive-unknown-status.ts`, `reindex-image-generation-metadata.ts`, `measure-wrapped-dose-prevalence.ts`,
`decompose-indexing-v3.mjs`, `repro-coalesce-poison-race.mjs` (a one-off race reproducer).
`backfill-visual-intelligence.ts`, `derive-unknown-status.ts`, `reindex-image-generation-metadata.ts`,
`measure-wrapped-dose-prevalence.ts`, `decompose-indexing-v3.mjs`, `repro-coalesce-poison-race.mjs`
(a one-off race reproducer).

## Workflow planners [infra]

Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@
"import:docs:20": "node scripts/run-tsx.mjs scripts/import-documents.ts --queue-batch-size 20",
"measure:wrapped-dose-units": "node scripts/run-tsx.mjs scripts/measure-wrapped-dose-prevalence.ts",
"enrich:documents": "node scripts/run-tsx.mjs scripts/enrich-documents.ts",
"enrich:backfill": "node scripts/run-tsx.mjs scripts/backfill-enrichment.ts",
"enrich:backfill": "node scripts/run-tsx.mjs scripts/archive/backfill-enrichment.ts",
"classify:documents": "node scripts/run-tsx.mjs scripts/classify-documents.ts",
"audit:source-governance": "node scripts/run-tsx.mjs scripts/audit-source-governance.ts",
"audit:source-governance:release": "npm run audit:source-governance -- --debt-policy docs/release-source-metadata-debt-2026-06-30.json",
Expand All @@ -150,7 +150,7 @@
"check:july8-live-batch": "node scripts/run-tsx.mjs scripts/archive/check-july8-live-batch.ts",
"backfill:gold-labels": "node scripts/run-tsx.mjs scripts/backfill-gold-document-labels.ts",
"backfill:smart-v2-labels": "npm run classify:documents -- --all-owners --limit 5000 --only-missing-smart-v2",
"tags:backfill": "node scripts/run-tsx.mjs scripts/backfill-document-tags.ts",
"tags:backfill": "node scripts/run-tsx.mjs scripts/archive/backfill-document-tags.ts",
"index:backfill": "node scripts/run-tsx.mjs scripts/backfill-smart-index.ts",
"visual:backfill": "node scripts/run-tsx.mjs scripts/backfill-visual-intelligence.ts",
"backfill:text-normalization": "node scripts/run-tsx.mjs scripts/backfill-text-normalization.ts",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,15 @@
* - Never marks covers searchable / never attaches them to chunks
* - Prefers documents that do not already have a cover_page image
*
* One-shot backfill: retired from scripts/ once the ingestion pipeline started
* generating cover thumbnails for new documents directly (see worker/main.ts,
* worker/python/extract_pdf_assets.py). Kept here for provenance and as a
* re-runnable repair tool if a targeted document is ever missing its cover.
*
* Usage:
* node scripts/backfill-document-covers.mjs
* node scripts/backfill-document-covers.mjs --apply --limit 25
* node scripts/backfill-document-covers.mjs --apply --document-id <uuid>
* node scripts/archive/backfill-document-covers.mjs
* node scripts/archive/backfill-document-covers.mjs --apply --limit 25
* node scripts/archive/backfill-document-covers.mjs --apply --document-id <uuid>
*/

import { createHash, randomUUID } from "node:crypto";
Expand Down
5 changes: 4 additions & 1 deletion tests/document-cover-thumbnails.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,10 @@ import type { RelatedDocument } from "@/lib/types";

const extractorSource = readFileSync(new URL("../worker/python/extract_pdf_assets.py", import.meta.url), "utf8");
const workerSource = readFileSync(new URL("../worker/main.ts", import.meta.url), "utf8");
const backfillSource = readFileSync(new URL("../scripts/backfill-document-covers.mjs", import.meta.url), "utf8");
const backfillSource = readFileSync(
new URL("../scripts/archive/backfill-document-covers.mjs", import.meta.url),
"utf8",
);

describe("document cover thumbnails", () => {
it("pages cover backfill with stable ORDER BY for PostgREST range scans", () => {
Expand Down
Loading