Add M2 atomic reindex staged generation recovery - #91
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e6ac510623
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const applied = await supabase.rpc("cleanup_abandoned_document_index_generations", { | ||
| p_document_id: args.documentId, | ||
| p_limit: limit, | ||
| p_dry_run: false, | ||
| }); |
There was a problem hiding this comment.
When there are more eligible documents than p_limit, the dry-run RPC selects a limited candidate set, but this second RPC recomputes candidates instead of applying to the document_ids just returned. Because the SQL limit has no deterministic ordering and data can change between the two calls, apply mode can delete a different set of staged rows than the counts the operator confirmed; pass the dry-run IDs through to the apply step or otherwise make the selection stable.
Useful? React with 👍 / 👎.
| and c.document_id = any(target_document_ids) | ||
| and c.index_generation_id is not null | ||
| and c.index_generation_id::text is distinct from nullif(coalesce(d.metadata, '{}'::jsonb)->>'index_generation_id', ''); |
There was a problem hiding this comment.
If cleanup runs while workers/API requests can start reindexing, a document can be added to target_document_ids while it has no pending/processing job, then receive a new active job before these DELETE statements execute. Since the delete predicates only check the saved document IDs and generation mismatch, they can remove rows from the newly running staged generation; repeat the active-job guard in the delete phase or lock the target document/job rows before applying.
Useful? React with 👍 / 👎.
| delete from public.document_images a | ||
| using public.documents d | ||
| where d.id = a.document_id | ||
| and a.document_id = any(target_document_ids) | ||
| and nullif(coalesce(a.metadata, '{}'::jsonb)->>'index_generation_id', '') is not null | ||
| and nullif(coalesce(a.metadata, '{}'::jsonb)->>'index_generation_id', '') is distinct from nullif(coalesce(d.metadata, '{}'::jsonb)->>'index_generation_id', ''); |
There was a problem hiding this comment.
For failed staged generations that uploaded extracted images, deleting document_images here removes the only stored storage_path for those blobs; unlike document deletion, this path never enqueues a storage_cleanup_jobs entry or calls storage removal, so the generated image files remain orphaned in SUPABASE_IMAGE_BUCKET and cannot be cleaned by cleanup:storage afterward. Capture the paths before deleting and enqueue/remove them as part of the cleanup.
Useful? React with 👍 / 👎.
Records the five follow-ups that surfaced during PR #1316 and would otherwise be lost with this session's context: the band's inability to express a partial-source failure (and the favourites mask that papers over it), the refetch pulse deferred on auth-backed registries by the identity-clearing invariant, the Next streaming-clone strict-mode flake, the design-system gates that assert structure rather than rendered effect, and `PR required` reporting concurrency cancellations as failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Akwz3Sdms8uJ5AkDt3CduY
1 participant
Summary
npm run reindex:cleanup-stagedVerification
npm run test -- tests/reindex-pipeline.test.tsnpm run test -- tests/supabase-schema.test.tsnpm run typechecknpm run verify:cheapnpm run check:supabase-projectnpm run check:production-readinessNotes