fix: generation-versioned relation persistence

[phernandez]

Why

Concurrent indexing currently replaces each source entity's outgoing relations with a
delete-then-insert transaction. Mutual links can make those transactions acquire source entity,
relation, foreign-key, and uniqueness locks in opposite orders, producing the production
PostgreSQL deadlocks captured in #1213. Retrying identical work does not change that lock graph.

Closes #1213

What Changed

• Added a non-null relation.generation column, backfill migration, and source/generation index.
• Replaced relation delete-then-insert writers with bounded, generation-versioned upsert chunks
  followed by a separate stale-generation cleanup statement.
• Claimed the accepted note generation before relation publication and moved accepted-note
  relation publication after the entity/content transaction commits.
• Persisted relations unresolved (to_id = NULL) and made the shared resolver exclusively own
  generation-fenced to_id backfill.
• Adopted the generation-aware publisher and resolver in batch, local, accepted-write, and
  EntityService paths.
• Removed all four generation-zero legacy writer APIs after caller enumeration found no deployed
  exemption.
• Added SQLite/PostgreSQL race coverage, a real-PostgreSQL concurrent mutual-link test, migration
  coverage, retry-loop regressions, and the existing 10,000-target resolver budget test on the
  generation-aware setup.

Implementation Details

Ordered generations and source fences

The generation token is the existing ordered note_content.db_version, not a content checksum.
The publisher needs a monotonic ordering to reject stale work; a checksum can establish identity
but cannot say which write is newer.

Every relation upsert chunk and cleanup repeats a source-level
note_content.db_version = generation predicate. The per-row conflict guard alone is not enough:
without the source fence, an older publisher could still insert a relation key that is absent from
the newer generation.

Name-identity upserts use the existing unique constraint on
(from_id, to_name, relation_type) as the ON CONFLICT arbiter. Upserts accept only an equal or
newer row generation, reset to_id to NULL, and run in deterministic bounded chunks. Cleanup is
a separate transaction and deletes only older generations while the same source-generation fence
still holds.

SQLAlchemy renders the source-row FOR UPDATE fence on PostgreSQL and omits it on SQLite. SQLite's
single-writer serialization preserves the fence's meaning; both older/newer race directions run
against both backends.

Publication and resolution boundaries

Entity/content reconciliation now claims the accepted db_version before publication. Accepted
create, update, edit, and move flows commit their entity/content/search/observation state first,
then publish relations in sorted bounded transactions. A deferred reconciliation means the file
lineage is older than accepted content, so both retry loops now finish successfully without
publishing instead of retrying identical bytes five times.

Relations remain unresolved during persistence. Resolver plans carry the observed source
generation, preserve customer-authored to_name, and update only to_id under the source
generation predicate. The real mutual-link test exposed an explicit target-row lock inversion;
resolution now relies on the target external-ID predicate and the foreign-key key-share lock
acquired by the guarded mutation instead of taking the conflicting explicit target lock.

Legacy caller disposition

• replace_accepted_outgoing_relations: no production caller after the accepted-write inversion;
  removed.
• update_entity_relations: reachable only through undeployed upsert_entity_from_markdown;
  removed, with the supported EntityService write surface moved to generation publication.
• delete_outgoing_relations_from_entity: no source caller; removed.
• add_all_ignore_duplicates: no source caller, only legacy tests/scaling setup; removed. It also
  silently discarded Relation.generation, so it always wrote generation 0.

There is no single-writer exemption. A generation-0 row written by a legacy path would be deleted
as superseded by a later generation cleanup on the same database, so leaving any mixed writer was
a correctness hole.

Testing

Automated

• just fast-check: passed (Ruff formatting/lint and type checking).
• just test: returned 0 under the repository gate. SQLite unit and integration completed; the
  SQLite integration result was 466 passed, 9 skipped, and 29 deselected. PostgreSQL unit
  completed. PostgreSQL integration reached 30% with no assertion failure before the recipe's
  intentional 600-second timeout sent signal 9; test-int-postgres explicitly accepts exit 137
  because of the linked FastMCP issue, so this is not represented as all 474 cases passing.
• Focused relation-generation suite on SQLite: 67 passed, with the PostgreSQL-only concurrency
  case skipped.
• Focused relation-generation suite on PostgreSQL: 85 passed, including both concurrent mutual-link
  batch publication and local retry flows.
• Focused deferred-generation regression coverage: SQLite 23 passed; PostgreSQL 10 passed.
• Focused repository/session-isolation/scaling coverage: SQLite 32 passed, 1 skipped; PostgreSQL
  33 passed. The real-PostgreSQL 10,000-target resolver test stayed within its unchanged 30-query
  budget.
• just doctor: passed on the final head, including migrations and the file-to-database indexing
  loop.

Risks / Follow-ups

• Rollout order is required: migrate the schema first, then pause/drain every old delete/insert
  indexing worker before enabling generation publishers. Full generation guarantees begin only
  after no legacy writer remains.
• The cloud repository needs no separate persistence implementation; it composes these Core
  services in-process.
• The PostgreSQL integration just target retains its existing accepted 600-second timeout. The
  issue-owned PostgreSQL concurrency, repository, resolver, and scaling paths were run directly
  and passed as described above.

Reviewer Guide

The commit boundaries are deliberate. In particular, review the generation-claim inversion in
8d1db15c and the accepted-content-first publication change in 47b1cd52 independently from the
schema/repository foundation and resolver changes.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 43bfb5b0c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d7897ba7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1fe16ef022

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c21102d570

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2f72e9594

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 48398846d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 66b74dc590

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7dc9a23922

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

Windows CI triage for job 93312513075:

Group A — inherited from main, not branch-owned. The failed-job log contains 12 (not 11) tests/test_claude_issue_triage.py failures, all with the same Git Bash was not found at C:\Program Files\Git\mingw64\bin\bash.exe error. Main's own Tests run 31333892924, Windows unit job 93296433058, has the identical 12 failures and error. PR #1221 is now merged and its Windows unit job is green, so I will rebase #1220 onto current main before the pending single push.

Group B — branch-owned, but not a path-separator identity bug. All three actual assertions fail because entity.outgoing_relations is empty; the log shows no backslash-bearing relation identity, permalink mismatch, or failed target lookup. The invariant added on this branch compares the parsed publication payload with the raw bytes re-read before claiming the generation. On Windows, frontmatter normalization persisted CRLF bytes but FileService.update_frontmatter_with_result returned its LF in-memory template. Reconciliation therefore (correctly) treated the payload and claim as different snapshots and withheld relation publication, producing exactly these three empty-relation failures.

The fix is at that producing boundary: after normalization/formatting, return content and checksum from one binary read of the bytes that actually reached disk. The regression simulates Windows CRLF persistence and asserts both returned content bytes and checksum match disk. This preserves the strict snapshot/claim fence; it does not weaken equality or normalize the tests. The three failed end-to-end paths are in the pending verification set and will rerun in Windows CI after the push.

[phernandez]

Exhaustive relation-generation race audit at 6d11ed32

I swept the codebase for both recurring bug classes: (a) any reachable transaction that can take an Entity or Relation lock before NoteContent when both domains are touched, using current_relation_generation_statement as the canonical lock-order authority, and (b) any path that can pair parsed bytes with a generation claimed across a commit boundary.

Lock-order paths checked:

• Relation publisher: safe. It locks the source NoteContent generation first and rechecks that fence in every sorted relation-upsert chunk.
• Production resolver (RepositoryRelationResolutionRuntime.apply_resolved_targets): safe. It locks sorted source NoteContent rows before Relation rows and applies the source-generation guard.
• Accepted note create/update/edit/move/delete: safe. Update/edit/move lock NoteContent before Entity; create has no pre-existing NoteContent row and publishes only after commit; delete locks NoteContent first.
• Note materialization: safe. NoteContent precedes Entity.
• Directory delete: fixed here. The discovery query is a non-locking snapshot; the mutation transaction locks all affected NoteContent rows in sorted entity-id order immediately before any Relation/Entity DML.
• External watcher delete: safe after this audit. NoteContent is locked before relation-target capture and Entity deletion.
• Project-index delete: safe after this audit. NoteContent is locked before relation-target capture and Entity deletion.
• Project-index move: safe after this audit. The complete union of moved and replacement entity ids is sorted and fenced before Entity mutation or replacement deletion.
• Project hard delete: safe after this audit. All project NoteContent rows are locked in sorted order before the Project cascade can reach Entity/Relation.
• Project soft delete: safe/not applicable. It mutates Project only and cannot reach Entity/NoteContent/Relation in that transaction.
• Note-file vacate: safe/not applicable. It locks Entity but only plain-reads NoteContent and performs no NoteContent or Relation mutation, so it creates no opposing lock edge.
• Entity upsert/index: safe by transaction boundary. Entity/Observation persistence commits before NoteContent reconciliation and relation publication.
• Compatibility forward-reference runtime: audit found and fixed one direct Relation updater. It now locks sorted source NoteContent rows first, even though current production composition uses the repository resolver.
• Legacy EntityService.delete_entity, delete_entity_by_file_path, delete_directory, and orphan-cleanup helpers: no production src/ callers remain; only tests/internal compatibility references. They are not reachable relation-generation writer paths.

Snapshot/generation paths checked:

• Accepted DB-first writes: safe. The immutable publication payload and accepted NoteContent db_version are built in the same transaction.
• EntityService compatibility writer: safe. It parses persisted read-back bytes and claims/publishes the generation of those exact bytes.
• Single-file local index: safe. One SyncedMarkdownFile supplies content and relations; supersession defers/discards the publication.
• Batch local index: safe. It rereads raw bytes and requires exact snapshot equality; it may claim fresh content but withholds stale parsed relations.
• Cloud/no-reader batch index: safe. The scan snapshot itself is the claimed/published snapshot.
• Relation publisher: safe. Generation and payload are immutable, with a source-generation fence per chunk.
• Resolver: safe/not a parsed-byte claim. It captures relation identity plus source generation and performs a generation-guarded CAS.
• Search refresh: fixed here. One SQL statement loads a generation-coherent Entity/Observation/Relation graph and the exact marker ids. If generation N is no longer current, the N flow is a terminal no-op and leaves the marker for N+1; it never rereads newer content into the N flow.
• Resolver search repair: safe. It uses captured marker ids; a newer generation's markers survive.
• Read repair, materialization, and project moves: not applicable; they do not publish relations parsed from bytes under a claimed generation.
• Frontmatter write boundary: fixed here for Windows. After formatting/writing, FileService performs one binary reread and derives both returned markdown and checksum from those exact persisted bytes. This prevents Windows CRLF translation from making the strict publication snapshot comparison reject its own write.

Windows CI triage is recorded separately in the earlier comment. Group A reproduced identically on main and is fixed by merged PR #1221, now included by this rebase. Group B's three assertions all showed empty outgoing relations; the cause was the LF in-memory/CRLF persisted-byte mismatch above, not separator-sensitive relation identity or permalink construction.

Verification on the rebased head:

• just fast-check
• just doctor
• just fast-test: 905 passed, 8 skipped
• Exact Windows-suspect SQLite tests: 3 passed
• Focused SQLite audit/regression set: 7 passed
• Focused Postgres concurrency/repository set: 5 passed
• Broader focused relation suite before the no-conflict rebase: 135 passed, 4 skipped

[phernandez]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

basic-memory/src/basic_memory/indexing/directory_delete_runner.py

Line 197 in 6d11ed3

entity_mtime=float(row.mtime) if row.mtime is not None else None,

Revalidate directory membership after acquiring locks

When a note is moved after this unlocked snapshot is read but before delete_directory_entities() acquires its NoteContent locks, the move can commit the entity's new path and the delete then still removes that entity by the stale snapshotted ID. The active /delete-directory flow consequently deletes DB state for a note that no longer belongs to the requested directory and queues cleanup using its old path; depending on materialization timing, this can also lose the moved note. After locking NoteContent, re-query and lock the entities using the directory predicate, then delete only those that still match.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

Addressed Codex review 4892873946 in 6987500d.

The directory snapshot remains intentionally unlocked, and this adds no new lock construct. After the existing sorted NoteContent source fence, the store revalidates the snapshotted IDs against the current project and directory prefix without locking. The final Entity mutation repeats all three predicates in-statement:

• Entity.id IN <snapshotted ids>
• Entity.project_id = <requested project>
• Entity.file_path LIKE <escaped requested directory>/%

It uses DELETE ... RETURNING Entity.id, and the accepted result carries those actual deleted IDs so post-commit file cleanup is queued only for rows the guarded statement removed. A note moved out during the snapshot-to-delete window therefore remains intact and is not treated as accepted cleanup work. Relation/search/vector cleanup is scoped to the post-fence membership revalidation, while the lock budget remains exactly the NoteContent source fence.

Regression coverage runs on both backends and proves the stale snapshotted ID becomes a no-op after move-out, preserving both Entity and NoteContent at the destination. The SQL-shape test separately asserts that the final DELETE itself contains the ID, project, and directory predicates.

Verification:

• focused SQLite: 19 passed, 1 PostgreSQL-only skip
• focused PostgreSQL move-out + publication/delete overlap: 2 passed
• just fast-check
• just fast-test: 51 passed, 1 skipped
• just doctor

GitHub emitted 4892873946 as a review summary rather than an inline thread, so there is no review-thread node to resolve. The live PR currently has zero unresolved review threads; the two prior pending threads are resolved.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6987500df2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[phernandez]

Windows CI follow-up for job 93320769061, fixed in 659842f:\n\nThe two CRLF failures were test-I/O normalization, not production permalink-rewrite churn. On Windows the fixture Path.write_text call writes CRLF and the indexing path reads and preserves those bytes. The assertions then called file_service.read_file_content, whose text-mode universal-newline read normalized the persisted CRLF to LF. The affected assertions now compare indexed markdown with persisted bytes, matching the contract they intend to verify.\n\nLocal verification: both affected tests pass; just fast-check, focused SQLite and PostgreSQL suites, just fast-test, and just doctor are green.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 659842f06c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".