diff --git a/docs/REDIS_READ_CACHE_PLAN.md b/docs/REDIS_READ_CACHE_PLAN.md new file mode 100644 index 000000000..3207d4803 --- /dev/null +++ b/docs/REDIS_READ_CACHE_PLAN.md @@ -0,0 +1,591 @@ +# Redis Read Cache Plan + +## Status + +Accepted implementation plan. Basic Memory owns semantic read caching; Basic Memory Cloud owns +tenant and principal rate limiting. The datasets are logically isolated even when they share a +Redis instance. + +## Goals + +- Reduce latency and database work for repeated entity and note reads. +- Keep Redis optional so the default local-first installation has no external service + requirement. +- Make cache semantics, typed serialization, TTLs, and invalidation part of Basic Memory. +- Let a host such as Basic Memory Cloud inject an existing Redis client and opaque namespace. +- Preserve independent clients, key prefixes, metrics, and failure behavior for read caching and + rate limiting. + +## Ownership Boundary + +Basic Memory owns: + +- cacheable read operations; +- canonical request keys; +- typed serialization; +- TTL and payload-size policy; +- project-scoped invalidation; +- the model-bound read-through facade and Redis read-cache implementation. + +Cloud owns: + +- tenant and principal identity; +- rate-limit policy and enforcement; +- the Redis deployment topology; +- the opaque cache namespace supplied to Basic Memory; +- capacity, eviction, and availability decisions when Redis is shared. + +Physical topology is deliberately outside the Basic Memory cache contract. Cloud may point the +read cache and limiter at separate Redis instances or at separately configured clients on one +instance. + +### Tenant isolation on shared Redis + +Cloud does not create a Redis instance or connection pool per tenant. It reuses one long-lived, +Basic Memory-specific Redis client and constructs a lightweight `RedisReadCache` adapter from +trusted request or worker context: + +- `namespace`: the stable tenant/workspace UUID; +- `project_id`: the Basic Memory project external UUID; +- `prefix`: the Basic Memory read-cache keyspace, separate from rate limiting. + +The adapter hashes `(namespace, project_id)` into the Redis Cluster scope. Two tenants with the +same project identifier therefore cannot share a generation or data key. Tenant isolation must +not rely on project UUID uniqueness alone: every Cloud read and invalidation supplies the +authenticated tenant/workspace UUID, even when project UUIDs are globally unique in practice. + +The namespace is host-owned isolation context, not an API input. Never derive it from a display +name or slug, accept it from a caller, or include API keys and other secrets in it. API requests +and background workers must use one canonical tenant-to-namespace function so an index worker +invalidates the exact scope populated by the request path. + +### Cloud host requirements + +Basic Memory Cloud must satisfy all of these requirements before enabling cached reads for a +tenant: + +1. Derive one stable namespace from authenticated Cloud context. The preferred value is the + tenant UUID or workspace UUID already owned by Cloud. Project UUID alone is not a tenant + boundary, even if current database constraints make it globally unique. +1. Complete authorization and tenant database/schema selection before cache lookup. The + namespace prevents key collisions; it does not replace Cloud's access-control boundary. +1. Override the low-level `get_read_cache` dependency at the Cloud composition root. Construct + `RedisReadCache(client=shared_basic_memory_client, namespace=trusted_namespace)` as a + lightweight request-scoped adapter; reuse the long-lived client and connection pool. + FastAPI route dependencies then create lightweight `ModelReadCache` facades from that shared + backend and bind Basic Memory's response type, TTL, and payload-size policy. Cloud should not + duplicate those policy constants or construct Redis clients per response model. +1. Pass the trusted tenant/workspace identity through internal queue payloads, or include enough + trusted identifiers for workers to derive the exact same namespace. Never copy a namespace + from a public request field. +1. Inject the namespace-bound cache into every mutation-producing runtime: accepted note + materialization, object-storage events, direct and project indexing, directory moves/deletes, + watcher-detected paired moves, relation-resolution workers, and startup recovery or + reconciliation. Request-path invalidation alone is insufficient because a worker can update a + cached entity after the request returns. +1. Treat each asynchronous state transition as a separate freshness boundary. Invalidate after + the accepted-note transaction commits, immediately after terminal materialization/status + publication before indexing begins, again after indexing, and again after relation resolution + completes. This prevents a read filled between phases from surviving the later worker commit + or remaining pending throughout a slow index. +1. Make committed-mutation invalidation cancellation-safe. Accepted-note writes, directory + deletion, and each file in a directory move can be cancelled after their database commit + succeeds but before the transaction context returns. Finish the namespace-bound generation + bump before re-propagating cancellation so committed state cannot remain hidden behind the + previous generation. +1. Invalidate after each committed project-index move, delete, and file-index batch, while + retaining the final failure-safe completion boundary for later-phase errors. Local inline file + batches invalidate when their runner returns; queued Cloud file batches invalidate in the child + worker after its durable commit, not merely when the coordinator enqueues the job. +1. Put full search reindexing in a failure-safe invalidation scope owned by the worker that runs + the rebuild. Reindexing drops and repopulates search rows incrementally, so fuzzy resolutions + filled during a partial or completed rebuild must not survive its final generation bump. +1. Put direct single-file and watcher file-index invalidation in failure-safe boundaries. Entity + transactions can commit before search refresh or note-content reconciliation raises, so a + failed index attempt can still publish cache-relevant state. Wrap the transaction-bearing + watcher index and entity-delete operations themselves in cancellation-safe invalidation because + cancellation can land during transaction exit before their completion callbacks run. Once a + watcher callback runs, retain its generation bump before relation and search cleanup. +1. Pass the namespace-bound cache into hosted note-content read repair. A resource or entity read + can bootstrap a missing accepted-content row. Put cancellation-safe invalidation around the + transaction-bearing repair call, not only after it returns, so cancellation during transaction + exit still advances the generation before the repaired response can reach read-through + storage. +1. Pass the namespace-bound cache into pre-mutation content freshening. Freshening can index an + externally edited file before the accepted mutation begins, so invalidate after every + freshening attempt that may have published state, including when the later mutation is + rejected or raises. +1. Inject the namespace-bound cache into import endpoints and workers. Invalidate after every + attempted file write before the next item, including writes that raise after partial progress, + and retain the final failure-safe invalidation around the complete import attempt. +1. Invalidate directory deletion immediately after its acceptance transaction commits, then + again after file cleanup and surviving-relation refresh. A slow or failed cleanup must not + keep deleted entities reachable through the pre-acceptance generation. +1. Invalidate every committed watcher-detected move batch, then retain the final invalidation + after search refresh. Paired delete/create events are consumed by move processing and therefore + bypass the ordinary watcher callbacks; a large watcher run commits bounded batches before the + final refresh. +1. Invalidate directory moves after each individual file/database move commits, then again after + the final search and relation follow-ups. Directory moves are incremental batches, so a long or + partially failed request must not keep earlier files under the pre-move generation. +1. Inject the namespace-bound cache into project-root path mutations. A root change preserves the + project and entity UUIDs while changing the filesystem source behind resource reads, so + invalidate in a failure-safe boundary around the committed path update. +1. If startup recovery or reconciliation attempts can publish materialization, vacate, index, or + relation state, invalidate through the same namespace-bound cache before releasing the serving + barrier or resuming tenant traffic. Treat materialization and move-vacate recovery as separate + freshness phases: invalidate a completed first phase before starting the second so a later + setup/query failure cannot skip the earlier generation bump. Terminal conflict and failure + publication count even when the recovery did not produce a written file. Put each + transaction-bearing phase inside a cancellation-safe invalidation scope so shutdown cannot + interrupt the generation bump after recovery commits. +1. Keep `bm:read:v1` separate from rate-limit and Cloud control-plane prefixes, metrics, + timeouts, and failure policies. The clients may target one Redis deployment, but a read-cache + timeout must bypass while a rate-limit decision keeps its Cloud-owned security behavior. +1. Enable reads only after request and worker invalidation use the same namespace in the target + environment. Roll out by tenant cohort, watch hit/bypass/invalidation outcomes and database + queries, then remove overlapping Cloud gateway response caches only after parity. +1. Coordinate rolling deployments around the `bm:read:v1` payload/key contract. Bump the prefix + for incompatible serialized response changes so mixed application versions never interpret + one another's payloads with different schemas. + +If Cloud ever changes the namespace source, treat that as a cache-key migration. A new namespace +is safe because it cannot read the old tenant scope, but old keys remain until their TTLs expire +and every worker must switch atomically enough to avoid missing invalidations. + +## Architecture + +```mermaid +flowchart LR + H["Cloud or standalone API host"] -->|"optional client plus opaque namespace"| B["Raw ReadCache backend"] + B --> F["Typed ModelReadCache facades"] + API["Basic Memory read routes"] --> F + F -->|"hit"| API + F -->|"miss"| DB["Services, repositories, and storage"] + DB -->|"successful result"| F + W["Writes, indexing, recovery, and storage events"] -->|"invalidate after commit"| B + + RL["Cloud tenant rate limiter"] --> RLD["Cloud rate-limit keyspace"] + B --> BMD["Basic Memory read-cache keyspace"] + RLD -. "same or separate instance" .-> R["Redis"] + BMD -. "same or separate instance" .-> R +``` + +## Core Contract + +Introduce `src/basic_memory/read_cache/` with: + +- a narrow `ReadCache` protocol; +- a narrower `ReadCacheInvalidator` protocol for mutation and repair paths; +- immutable request/key values; +- canonical key construction; +- a generic `ModelReadCache[ModelT]` facade that owns one Pydantic response type and policy; +- an optional `RedisReadCache` adapter. + +The raw backend is namespace-bound at construction. Its public operations are: + +- `lookup(key)`, which returns the generation observed with a hit or miss; +- `store(key, lookup, payload, ttl)`, which reports stored or superseded; +- `invalidate_project(project_id)`. + +Cloud can create a lightweight namespace-bound adapter around a long-lived, Basic +Memory-specific async Redis client. Basic Memory then creates separate model-bound facades for +entity, resolution, and resource responses over that same adapter. Facades do not own clients or +connections. Basic Memory does not receive tenant, subscription, or rate-limit concepts. + +## Keys And Invalidation + +Use versioned, cluster-compatible keys: + +```text +bm:read:v1:{scope_digest}:generation +bm:read:v1:{scope_digest}:: +``` + +`scope_digest` hashes the host-supplied tenant/workspace namespace and project external ID. The +Redis Cluster hash tag keeps that tenant-project scope's generation and data keys in one slot. + +Each value records the random generation token under which it was created: + +1. Read the generation and data key together. +1. Accept the cached value only when its embedded generation matches. +1. After a successful mutation commit, replace the generation with a new random token. +1. Give generation metadata a bounded TTL, renew it to at least each stored response's remaining + lifetime, and reset that TTL on invalidation. A lookup also migrates a legacy persistent + generation key to a bounded TTL. +1. Let unreachable entries and inactive-project generation metadata expire; never scan or + bulk-delete keys. + +Random tokens prevent an evicted generation key from returning to an old integer generation and +reviving stale data. A read that fills after concurrent invalidation also remains safe because its +old token no longer matches. + +## Initial Cache Surface + +Phase one: + +| Operation | Initial TTL | Constraints | +| --------------------------- | ----------: | --------------------------------------------------- | +| Entity by external ID | 60 seconds | Cache validated `EntityResponseV2` JSON | +| Identifier resolution | 60 seconds | Include body and workspace context | +| Markdown note resource | 60 seconds | Cache only below an explicit size limit | +| Directory structure | 60 seconds | Folder-only tree; two MiB payload cap | +| Directory tree | 60 seconds | Full hierarchy; two MiB measured payload cap | +| Paginated directory listing | 60 seconds | Include path, depth, glob, page, and page-size keys | + +Phase two, after measuring phase one: + +| Operation | Initial TTL | Constraints | +| --------------------------- | ------------: | ---------------------------------------------- | +| Search | 30 seconds | Canonicalize the complete query and pagination | +| Context and recent activity | 15-30 seconds | Normalize or bound time-relative inputs | + +Do not initially cache failures, missing entities, graph/orphan responses, large or arbitrary +binary resources, schema inference, writes, or Cloud control-plane data. + +Caching is semantic rather than HTTP-method based. The POST identifier-resolution and search +operations can be cached without changing their public API. Identifier resolution depends on the +request-local workspace permalink context, so its request digest includes the workspace slug and +workspace type in addition to the validated request body. + +## Placement + +Cache typed boundary values rather than SQLAlchemy models. Use an explicit read-through scope in +the API routes so hit, miss, serialization, and fallback behavior remain visible. FastAPI injects +a route-specific `ModelReadCache[ResponseType] | None`; its provider binds the optional backend +to Basic Memory's response type, TTL, and payload-size policy at the dependency boundary. Routes +keep the authoritative read inline inside a Python async context manager instead of constructing +loader callbacks: + +```python +cache_key = ReadCacheKey( + project_id=project_external_id, + operation=ReadCacheOperation.entity, + request_digest=read_cache_request_digest(entity_id), +) +cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[EntityResponseV2]()) +) +async with cache_scope as cached: + if cached.value is not None: + return cached.value + + entity = await entity_repository.get_by_external_id(session, entity_id) + result = EntityResponseV2.model_validate(entity) + cached.value = result + return result +``` + +The context manager performs lookup before entering the body and stores an eligible miss when the +body exits normally. Exceptions and cancellation propagate without storing. When no backend is +configured, callers do not invoke cache lookup, store, or invalidation; there is no disabled cache +result or no-op implementation. A route that performs read repair passes the model-bound facade +itself through the narrow `ReadCacheInvalidator` capability when present; it never reaches through +the facade to a Redis/backend attribute. + +Mutation and indexing code uses the same direct scope pattern when invalidation is unconditional: + +```python +async with invalidate_cache(read_cache, project_id): + await importer.import_data(...) +``` + +Callers enter this scope only when a backend is present. Conditional and multi-phase invalidation +remains explicit so the freshness boundary is visible. The scope finishes its generation bump +before re-propagating cancellation, which makes it safe around operations whose transaction can +commit during async context-manager exit. + +Primary integration points: + +- `src/basic_memory/api/container.py` +- `src/basic_memory/api/app.py` +- `src/basic_memory/deps/read_cache.py` +- `src/basic_memory/api/v2/routers/directory_router.py` +- `src/basic_memory/api/v2/routers/knowledge_router.py` +- `src/basic_memory/api/v2/routers/resource_router.py` +- later, `src/basic_memory/api/v2/routers/search_router.py` + +Invalidation belongs at portable mutation and indexing completion boundaries, not only in +FastAPI routes. It must cover accepted note writes, terminal deferred materialization and status +publication, direct file indexing, filesystem watcher updates, project indexing, directory +mutations, imports, watcher-detected paired moves, startup recovery or reconciliation, Cloud +storage events, full search reindexing, and relation-resolution changes that affect cached +responses. Each later phase invalidates again so a value filled after an earlier generation bump +cannot outlive the state that phase publishes. Hosted read repair invalidates after bootstrapping +accepted content and before a repaired entity or resource is stored, including cancellation +during the repair transaction's exit. Directory moves invalidate after every committed file plus +the final reindex; directory deletion invalidates after acceptance and after cleanup. Imports +invalidate after every attempted file write and again around the complete attempt so partial +failures cannot escape. Project indexing and full search reindexing invalidate even after a +partial failure. Direct single-file and watcher file indexing invalidate even when a follow-up +fails after the entity commit. Watcher move batches invalidate after every commit, and watcher +index and entity-delete operations wrap their transaction-bearing executors so cancellation cannot +escape between a commit and completion callback. Watcher index/delete completion callbacks retain +a later generation bump before relation and search cleanup. Recovery phases invalidate +independently before the serving barrier is released, include terminal conflict or failure +publication, and finish their generation bump before startup cancellation propagates. +Deferred accepted-note materialization similarly advances the generation immediately after +terminal status publication, before potentially slow indexing, and retains the outer post-index +bump. + +## Dependency And Lifecycle + +Use the official asynchronous `redis-py` client behind the Basic Memory protocol. Add it only as +an optional package extra. A host may instead supply a compatible, already-owned client. + +The Core `ApiContainer` carries `ReadCache | None` and defaults to `None`. A managed host +activates caching by injecting or dependency-overriding a namespace-bound implementation and owns +that client's lifecycle; Cloud therefore reuses its long-lived Basic Memory cache client. Local +CLI, MCP in-process ASGI routing, and the standalone API simply skip cache work in the first +rollout. A later standalone Redis setting can create and close a client in the FastAPI lifespan +without changing the cache contract. + +`get_read_cache` is the host override point and returns `ReadCache | None`. Core-owned, +route-specific FastAPI providers call `create_model_read_cache` to return a correctly typed facade +when that backend exists. Portable mutation and indexing runtimes continue to depend only on the +optional backend or narrower invalidation capability; they do not depend on FastAPI. + +The FastAPI Redis SDK is not the foundational dependency for this work. The cache contract must +also participate in portable indexing and hosted storage-event invalidation, and Basic Memory's +local ASGI transport does not run FastAPI lifespan. + +## Cloud Production Calibration + +A bounded production snapshot from 2026-07-30 00:00-19:03 UTC changed directory reads from a +phase-two idea into a phase-one requirement: + +| Cloud web route | Calls | Average | p95 | +| --------------------------- | ----: | -------: | -------: | +| `GET /api/v2/projects/tree` | 1,073 | 1,794 ms | 4,656 ms | +| `GET /api/v2/notes` | 789 | 1,688 ms | 3,295 ms | +| `GET /api/v2/projects` | 145 | 1,536 ms | 3,269 ms | + +Cloud's existing user-scoped gateway response cache already proved that the directory work is +cacheable, but also exposed the misses that remain expensive: + +| Gateway route family | Attempts | Hits | Hit rate | Tenant dispatches | +| --------------------- | -------: | ----: | -------: | ----------------: | +| `directory_structure` | 14,163 | 8,845 | 62.5% | 5,318 | +| `directory_list` | 722 | 166 | 23.0% | 556 | +| `directory_tree` | 88 | 13 | 14.8% | 75 | +| `note_entity` | 2,779 | 877 | 31.6% | 1,896 | + +Across the preceding seven days, cached directory-tree response bodies had a p99 of 573 KiB and a +maximum of 1.29 MiB. Directory-node facades therefore use a two MiB payload cap rather than the +one MiB default; otherwise the largest and usually slowest tree in the measured workload would +always bypass storage. Paginated listings retain the one MiB default—their measured maximum was +183 KiB. + +For folder file navigation, a directory-list miss plus a tenant project-list dispatch averaged +2,407 ms; when both dependencies avoided tenant dispatch, the same composed endpoint averaged +636 ms. Project-tree requests with no directory miss and no project-list dispatch averaged +673 ms, while requests with six or more directory misses plus a project-list dispatch averaged +3,571 ms. These are associations within the snapshot, not a controlled benchmark, but they show +that cache locality materially changes end-to-end latency. + +The same snapshot contained 6,338 hosted MCP tool calls. `read_note` accounted for 2,636 calls at +2,782 ms average, `search` for 723 at 3,386 ms, and `list_directory` for 231 at 1,960 ms. Their +instrumented Basic Memory API dependencies included 3,947 identifier resolutions, 3,064 resource +reads, and 1,140 searches. This preserves resolution and resource as the primary MCP targets while +adding directory reads for the web explorer and `list_directory`. + +Project enumeration remains a Cloud-owned concern. `GET /api/v2/projects` combines access to +multiple workspaces, user visibility, project soft-delete state, and tenant database selection. +Even after loading a cached project-list body, the current Cloud service opens each tenant +database and queries active project IDs. Basic Memory's project-scoped semantic cache must not +absorb that authorization-aware composition. Cloud should optimize that active-project +reconciliation and its own project-list cache independently. + +Directory caching in Basic Memory is intended to replace overlapping route families after Cloud +reaches namespace, invalidation, and observability parity. During rollout, the inner cache can +also share tenant-project directory results across already-authorized users while Cloud's current +outer key remains user-specific. Do not keep both response-cache layers as the final design. + +## Failure Behavior + +- Operational Redis command failures, including connection, timeout, capacity, replica-read-only, + and server response failures, are represented explicitly as cache-unavailable outcomes. +- Reads bypass Redis and use the authoritative path when the cache is unavailable. +- Cache-store failures do not fail an otherwise successful read. +- Cache-invalidation failures do not fail committed writes, but they emit prominent telemetry. +- Short initial TTLs bound stale-data exposure after an invalidation failure and Redis recovery. +- Redis client-input errors plus local serialization, decoding, and programming errors fail fast + rather than masquerading as cache misses. + +Rate-limit failure behavior remains entirely Cloud-owned. + +## Observability + +Record: + +- hit, miss, bypass, store, invalidation, unavailable, and oversize outcomes; +- operation name without tenant or project metric labels; +- Redis operation latency; +- cached payload size; +- authoritative read latency on misses; +- hashed scope and request identifiers on diagnostic spans only. + +Do not add public cache headers in the first version. + +## Integration Tests + +Redis behavior must be tested against a real Redis server, not a mocked or in-memory substitute. +Integration tests will start Redis through testcontainers or use an explicitly configured CI +Redis URL. + +Run the focused suite with: + +```bash +LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -p pytest_mock --no-cov -q test-int/read_cache +``` + +`BASIC_MEMORY_TEST_REDIS_URL` selects an externally managed test server. Otherwise the fixture +starts `redis:8.8-alpine`; `BASIC_MEMORY_TEST_REDIS_IMAGE` can override that image without +changing the test contract. + +The real-Redis suite must prove: + +- namespace, project, operation, and request isolation; +- distinct model-bound facades share one raw backend while retaining their own response type; +- deterministic canonical keys; +- cache hit and TTL expiry behavior; +- project invalidation; +- a fill that completes after invalidation is never served; +- loss of the generation key cannot revive an older value; +- generation metadata expires for inactive projects, including legacy persistent keys, and stores + keep it alive at least as long as their response data; +- Redis restart or unavailability produces explicit bypass behavior; +- no invalidation operation touches keys outside the Basic Memory prefix; +- payload size limits; +- repeated API entity reads use the real cached representation; +- successful writes invalidate; a rejected write also invalidates when pre-write freshening may + already have published external file state, while a rolled-back transaction without such a + publication does not; +- cancellation after a real accepted-note, directory-delete, or per-file directory-move + transaction commits cannot interrupt the real Redis generation bump, including repeated + cancellation while invalidation is in progress; +- cancellation during a real hosted read-repair transaction's commit exit cannot interrupt the + real Redis generation bump; +- cancellation during watcher index or delete completion cannot interrupt the first real Redis + generation bump after the durable event; +- cancellation during a watcher entity-delete transaction's commit exit cannot escape before the + real Redis generation advances, even when the completion callback is never reached; +- cancellation during a watcher file-index transaction's commit exit cannot escape before the + real Redis generation advances, even when the completion callback is never reached; +- real Redis no-eviction capacity failures bypass cache storage and cannot fail committed-write + invalidation; +- authoritative read exceptions propagate without populating the missed cache key; +- watcher-detected paired moves invalidate even though their events bypass ordinary callbacks; +- consecutive watcher move batches each advance the real Redis generation before the next batch, + while retaining the final post-refresh bump; +- a partial or completed full search reindex invalidates fuzzy resolutions filled while the + search index was being rebuilt; +- startup recovery that publishes written, conflict, or failed materialization state invalidates + before serving resumes; +- accepted-note materialization advances the generation after terminal status publication while + a following index is still blocked, then advances it again after indexing; +- cancellation after materialization or move-vacate recovery commits cannot interrupt the + phase-specific real Redis generation bump; +- project-index failures invalidate any earlier committed batches; +- consecutive project-index move, delete, and inline file batches each advance the real Redis + generation before the next batch begins; +- direct and watcher file-index failures invalidate any entity state committed before failed + search or reconciliation follow-ups; +- hosted read repair invalidates cached entity metadata before storing the repaired resource; +- multi-item imports advance the real Redis generation after every attempted file write, while + retaining the final success and partial-failure bump; +- directory moves invalidate after each committed file and again after final reindexing; +- project-root path changes invalidate cached resources whose project/entity UUID keys remain + stable; +- directory deletion invalidates before cleanup starts and again after cleanup completes. +- startup materialization recovery remains invalidated when a later move-vacate phase fails. + +Run route behavior against both SQLite and Postgres where persistence behavior differs. Redis +semantics themselves are asserted only against the real Redis integration fixture. + +## Delivery Sequence + +### 1. Cache infrastructure + +- Add the protocol, key values, Redis adapter, typed facade, optional dependency, + telemetry, and real Redis integration tests. +- Do not cache production routes yet. + +### 2. Hot semantic and directory reads + +- Cache entity, resolution, bounded markdown-resource, directory tree, directory structure, and + paginated directory-list reads behind default-off configuration. +- Wire project invalidation through accepted writes and indexing paths. +- Add full-stack API, repeated `read_note`, and directory refresh integration coverage. + +### 3. Cloud rollout + +- Inject a Basic Memory-specific Redis client and tenant namespace. +- Derive that namespace from trusted request and worker context with one canonical function. +- Invalidate after accepted-note commit, immediately after terminal materialization/status + publication before indexing, again after indexing, and after relation-resolution workers using + the same tenant namespace as the request path. +- Invalidate watcher-detected paired moves at move completion, and invalidate any recovery or + reconciliation attempt that can publish terminal state before releasing the serving barrier or + resuming tenant traffic. +- Invalidate project indexing from a failure-safe completion boundary, and invalidate directory + deletion both after acceptance commit and after cleanup/relation refresh. +- Invalidate full search reindexing from the worker that executes the rebuild so partial or + completed search rows cannot leave cached fuzzy resolutions behind. +- Invalidate direct and watcher file indexing from failure-safe boundaries because entity commits + precede some search and reconciliation follow-ups. +- In Cloud's `build_cloud_index_file_runtime`, decorate the transaction-bearing `FileIndexer` + with `InvalidatingIndexFileExecutor` using the namespace-bound cache and canonical project UUID. + Both single-file jobs and every child of `index_file_batch` then invalidate at the same + committed-file boundary as the local filesystem watcher. +- Retain Cloud's existing post-entrypoint `GatewayCache` invalidation until directory/entity + route-family overlap is removed, but do not treat that later live-update side effect as the + Basic Memory cache correctness boundary. +- Wrap the watcher index and entity-delete transactions themselves in cancellation-safe + invalidation because cancellation can escape before their completion callbacks are reached. +- Finish watcher index and delete completion invalidation before cancellation propagates from the + post-event callback. +- Invalidate every committed watcher move batch before the next batch starts, then retain the + final post-refresh invalidation. +- Wrap hosted note-content read repair in cancellation-safe invalidation so cancellation during + transaction exit cannot skip the bump before a repaired entity or resource reaches read-through. +- Invalidate pre-mutation content freshening even when a later accepted mutation is rejected or + fails, because the freshening index may already have committed external file state. +- Invalidate every imported file write before the next item, and retain whole-import invalidation + for partial failures. +- Finish directory-delete acceptance invalidation before re-propagating cancellation that lands + after the delete transaction may have committed. +- Finish each directory-move file invalidation before re-propagating cancellation that lands + after its move transaction may have committed, and invalidate again after search/relation + follow-ups. +- Invalidate project-root path changes in a failure-safe boundary because the filesystem source + can change while every cache identity remains stable. +- Put each startup recovery phase inside a cancellation-safe invalidation scope before beginning + the next phase. +- Enable reads for a tenant only after every request, worker, partial-index, direct-index, + read-repair, import, accepted-delete, move, and recovery boundary has namespace and invalidation + parity. +- Start with shadow telemetry or a limited tenant cohort. +- Compare hit rate, Redis latency, database query volume, and end-to-end tool latency. + +### 4. Expand from evidence + +- Add search and graph-context reads when measured reuse supports them. +- Refine project-wide invalidation only if unrelated writes materially reduce the entity hit + rate. + +### 5. Remove overlap + +- Remove matching Cloud gateway response-cache families once Basic Memory caching reaches + behavioral and observability parity. + +## Verification Gates + +- Focused unit and real Redis integration tests. +- Entity/read API integration tests on SQLite and Postgres. +- `just fast-check`. +- `just doctor`. +- The appropriate broader SQLite and Postgres suites before opening a pull request. +- A before/after benchmark showing cache-hit latency and reduced authoritative reads. diff --git a/pyproject.toml b/pyproject.toml index 8e8d8650b..7d1a322b2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -68,6 +68,9 @@ milvus = [ "pymilvus>=3.0.0,<4; sys_platform == 'win32'", "pymilvus[milvus-lite]>=3.0.0,<4; sys_platform != 'win32'", ] +redis = [ + "redis>=8.0.0,<9", +] [project.urls] Homepage = "https://github.com/basicmachines-co/basic-memory" @@ -103,6 +106,7 @@ markers = [ "benchmark: Performance benchmark tests (deselect with '-m \"not benchmark\"')", "slow: Slow-running tests (deselect with '-m \"not slow\"')", "postgres: Tests that run against Postgres backend (deselect with '-m \"not postgres\"')", + "redis: Tests that run against a real Redis server", "windows: Windows-specific tests (deselect with '-m \"not windows\"')", "smoke: Fast end-to-end smoke tests for MCP flows", "semantic: Tests requiring semantic dependencies (fastembed, sqlite-vec, openai)", @@ -131,6 +135,7 @@ dev = [ "ruff>=0.16.0", "freezegun>=1.5.5", "testcontainers[postgres]>=4.0.0", + "redis>=8.0.0,<9", "psycopg>=3.2.0", "pyright>=1.1.408", "pytest-testmon>=2.2.0", diff --git a/src/basic_memory/api/container.py b/src/basic_memory/api/container.py index 4bf18da38..c0c585835 100644 --- a/src/basic_memory/api/container.py +++ b/src/basic_memory/api/container.py @@ -17,6 +17,7 @@ from basic_memory import db from basic_memory.config import BasicMemoryConfig, ConfigManager +from basic_memory.read_cache import ReadCache from basic_memory.runtime.mode import RuntimeMode, resolve_runtime_mode if TYPE_CHECKING: # pragma: no cover @@ -39,6 +40,11 @@ class ApiContainer: engine: AsyncEngine | None = None session_maker: async_sessionmaker[AsyncSession] | None = None + # --- Optional semantic read cache --- + # Hosts inject a namespace-bound implementation; local and off-lifespan + # ASGI requests deliberately stay dependency-free by default. + read_cache: ReadCache | None = None + @classmethod def create(cls) -> "ApiContainer": # pragma: no cover """Create container by reading ConfigManager. @@ -91,6 +97,7 @@ def create_watch_coordinator(self) -> "WatchCoordinator": # pragma: no cover config=self.config, should_watch=self.should_watch_files, skip_reason=self.watch_skip_reason, + read_cache=self.read_cache, ) # --- Database Factory --- diff --git a/src/basic_memory/api/v2/routers/directory_router.py b/src/basic_memory/api/v2/routers/directory_router.py index 1f8637fed..c8c53518b 100644 --- a/src/basic_memory/api/v2/routers/directory_router.py +++ b/src/basic_memory/api/v2/routers/directory_router.py @@ -9,11 +9,24 @@ - Better performance through indexed queries """ -from typing import Optional +from contextlib import nullcontext +from typing import Annotated -from fastapi import APIRouter, Query, Path +from fastapi import APIRouter, Depends, Path, Query -from basic_memory.deps import DirectoryServiceV2ExternalDep +from basic_memory.deps import ( + create_model_read_cache, + DirectoryServiceV2ExternalDep, + ReadCacheDep, +) +from basic_memory.read_cache import ( + ModelReadCache, + ReadCacheKey, + ReadCacheOperation, + ReadCacheScope, + read_cache_request_digest, +) +from basic_memory.read_cache.policy import DIRECTORY_READ_CACHE_MAX_PAYLOAD_BYTES from basic_memory.schemas.directory import ( DEFAULT_DIRECTORY_PAGE_SIZE, MAX_DIRECTORY_PAGE_SIZE, @@ -24,11 +37,42 @@ router = APIRouter(prefix="/directory", tags=["directory-v2"]) +def get_directory_node_read_cache( + read_cache: ReadCacheDep, +) -> ModelReadCache[DirectoryNode] | None: + """Bind directory trees and structures to the optional cache backend.""" + return create_model_read_cache( + read_cache, + DirectoryNode, + max_payload_bytes=DIRECTORY_READ_CACHE_MAX_PAYLOAD_BYTES, + ) + + +DirectoryNodeReadCacheDep = Annotated[ + ModelReadCache[DirectoryNode] | None, + Depends(get_directory_node_read_cache), +] + + +def get_directory_list_read_cache( + read_cache: ReadCacheDep, +) -> ModelReadCache[DirectoryListResponse] | None: + """Bind paginated directory listings to the optional cache backend.""" + return create_model_read_cache(read_cache, DirectoryListResponse) + + +DirectoryListReadCacheDep = Annotated[ + ModelReadCache[DirectoryListResponse] | None, + Depends(get_directory_list_read_cache), +] + + @router.get("/tree", response_model=DirectoryNode, response_model_exclude_none=True) async def get_directory_tree( directory_service: DirectoryServiceV2ExternalDep, + read_cache: DirectoryNodeReadCacheDep, project_id: str = Path(..., description="Project external UUID"), -): +) -> DirectoryNode: """Get hierarchical directory structure from the knowledge base. Args: @@ -38,18 +82,31 @@ async def get_directory_tree( Returns: DirectoryNode representing the root of the hierarchical tree structure """ - # Get a hierarchical directory tree for the specific project - tree = await directory_service.get_directory_tree() + cache_key = ReadCacheKey( + project_id=project_id, + operation=ReadCacheOperation.directory_tree, + request_digest=read_cache_request_digest("tree"), + ) + cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[DirectoryNode]()) + ) + async with cache_scope as cached: + if cached.value is not None: + return cached.value - # Return the hierarchical tree - return tree + result = await directory_service.get_directory_tree() + cached.value = result + return result @router.get("/structure", response_model=DirectoryNode, response_model_exclude_none=True) async def get_directory_structure( directory_service: DirectoryServiceV2ExternalDep, + read_cache: DirectoryNodeReadCacheDep, project_id: str = Path(..., description="Project external UUID"), -): +) -> DirectoryNode: """Get folder structure for navigation (no files). Optimized endpoint for folder tree navigation. Returns only directory nodes @@ -62,8 +119,23 @@ async def get_directory_structure( Returns: DirectoryNode tree containing only folders (type="directory") """ - structure = await directory_service.get_directory_structure() - return structure + cache_key = ReadCacheKey( + project_id=project_id, + operation=ReadCacheOperation.directory_structure, + request_digest=read_cache_request_digest("structure"), + ) + cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[DirectoryNode]()) + ) + async with cache_scope as cached: + if cached.value is not None: + return cached.value + + result = await directory_service.get_directory_structure() + cached.value = result + return result @router.get( @@ -73,12 +145,11 @@ async def get_directory_structure( ) async def list_directory( directory_service: DirectoryServiceV2ExternalDep, + read_cache: DirectoryListReadCacheDep, project_id: str = Path(..., description="Project external UUID"), dir_name: str = Query("/", description="Directory path to list"), depth: int = Query(1, ge=1, le=10, description="Recursion depth (1-10)"), - file_name_glob: Optional[str] = Query( - None, description="Glob pattern for filtering file names" - ), + file_name_glob: str | None = Query(None, description="Glob pattern for filtering file names"), page: int = Query(1, ge=1, description="One-indexed result page"), page_size: int = Query( DEFAULT_DIRECTORY_PAGE_SIZE, @@ -86,7 +157,7 @@ async def list_directory( le=MAX_DIRECTORY_PAGE_SIZE, description="Number of nodes per page", ), -): +) -> DirectoryListResponse: """List directory contents with filtering and depth control. Args: @@ -101,11 +172,32 @@ async def list_directory( Returns: Bounded page of DirectoryNode objects matching the criteria """ - # Get directory listing with filtering - return await directory_service.list_directory( - dir_name=dir_name, - depth=depth, - file_name_glob=file_name_glob, - page=page, - page_size=page_size, + cache_key = ReadCacheKey( + project_id=project_id, + operation=ReadCacheOperation.directory_list, + request_digest=read_cache_request_digest( + dir_name, + str(depth), + file_name_glob or "", + str(page), + str(page_size), + ), + ) + cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[DirectoryListResponse]()) ) + async with cache_scope as cached: + if cached.value is not None: + return cached.value + + result = await directory_service.list_directory( + dir_name=dir_name, + depth=depth, + file_name_glob=file_name_glob, + page=page, + page_size=page_size, + ) + cached.value = result + return result diff --git a/src/basic_memory/api/v2/routers/importer_router.py b/src/basic_memory/api/v2/routers/importer_router.py index 52291b8b1..aa5452b73 100644 --- a/src/basic_memory/api/v2/routers/importer_router.py +++ b/src/basic_memory/api/v2/routers/importer_router.py @@ -6,8 +6,9 @@ import json import logging +from contextlib import nullcontext -from fastapi import APIRouter, Form, HTTPException, UploadFile, status, Path +from fastapi import APIRouter, Form, HTTPException, Path, UploadFile, status from basic_memory.deps import ( AppConfigDep, @@ -15,8 +16,10 @@ ClaudeConversationsImporterV2ExternalDep, ClaudeProjectsImporterV2ExternalDep, MemoryJsonImporterV2ExternalDep, + ReadCacheDep, ) from basic_memory.importers import Importer +from basic_memory.read_cache import ReadCache, invalidate_cache from basic_memory.schemas.importer import ( ChatImportResult, EntityImportResult, @@ -45,6 +48,7 @@ async def import_chatgpt( importer: ChatGPTImporterV2ExternalDep, config: AppConfigDep, file: UploadFile, + read_cache: ReadCacheDep, project_id: str = Path(..., description="Project external UUID"), directory: str = Form("conversations"), ) -> ChatImportResult: @@ -63,7 +67,14 @@ async def import_chatgpt( HTTPException: If import fails. """ logger.info(f"V2 Importing ChatGPT conversations for project {project_id}") - return await import_file(importer, file, directory, config.import_upload_max_bytes) + return await import_file( + importer, + file, + directory, + config.import_upload_max_bytes, + read_cache=read_cache, + project_external_id=project_id, + ) @router.post("/claude/conversations", response_model=ChatImportResult) @@ -71,6 +82,7 @@ async def import_claude_conversations( importer: ClaudeConversationsImporterV2ExternalDep, config: AppConfigDep, file: UploadFile, + read_cache: ReadCacheDep, project_id: str = Path(..., description="Project external UUID"), directory: str = Form("conversations"), ) -> ChatImportResult: @@ -89,7 +101,14 @@ async def import_claude_conversations( HTTPException: If import fails. """ logger.info(f"V2 Importing Claude conversations for project {project_id}") - return await import_file(importer, file, directory, config.import_upload_max_bytes) + return await import_file( + importer, + file, + directory, + config.import_upload_max_bytes, + read_cache=read_cache, + project_external_id=project_id, + ) @router.post("/claude/projects", response_model=ProjectImportResult) @@ -97,6 +116,7 @@ async def import_claude_projects( importer: ClaudeProjectsImporterV2ExternalDep, config: AppConfigDep, file: UploadFile, + read_cache: ReadCacheDep, project_id: str = Path(..., description="Project external UUID"), directory: str = Form("projects"), ) -> ProjectImportResult: @@ -115,7 +135,14 @@ async def import_claude_projects( HTTPException: If import fails. """ logger.info(f"V2 Importing Claude projects for project {project_id}") - return await import_file(importer, file, directory, config.import_upload_max_bytes) + return await import_file( + importer, + file, + directory, + config.import_upload_max_bytes, + read_cache=read_cache, + project_external_id=project_id, + ) @router.post("/memory-json", response_model=EntityImportResult) @@ -123,6 +150,7 @@ async def import_memory_json( importer: MemoryJsonImporterV2ExternalDep, config: AppConfigDep, file: UploadFile, + read_cache: ReadCacheDep, project_id: str = Path(..., description="Project external UUID"), directory: str = Form("conversations"), ) -> EntityImportResult: @@ -149,7 +177,13 @@ async def import_memory_json( json_data = json.loads(line) file_data.append(json_data) - result = await importer.import_data(file_data, directory) + result = await run_import_with_invalidation( + importer, + file_data, + directory, + read_cache=read_cache, + project_external_id=project_id, + ) if not result.success: # pragma: no cover raise HTTPException( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, @@ -171,6 +205,9 @@ async def import_file[ImportResultT: ImportResult]( file: UploadFile, destination_directory: str, max_bytes: int, + *, + read_cache: ReadCache | None, + project_external_id: str, ) -> ImportResultT: """Helper function to import a file using an importer instance. @@ -190,7 +227,13 @@ async def import_file[ImportResultT: ImportResult]( # Process file upload_bytes = await read_import_upload(file, max_bytes) json_data = json.loads(upload_bytes) - result = await importer.import_data(json_data, destination_directory) + result = await run_import_with_invalidation( + importer, + json_data, + destination_directory, + read_cache=read_cache, + project_external_id=project_external_id, + ) if not result.success: # pragma: no cover raise HTTPException( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, @@ -207,3 +250,24 @@ async def import_file[ImportResultT: ImportResult]( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Import failed: {str(e)}", ) + + +async def run_import_with_invalidation[ImportResultT: ImportResult]( + importer: Importer[ImportResultT], + source_data: object, + destination_directory: str, + *, + read_cache: ReadCache | None, + project_external_id: str, +) -> ImportResultT: + """Run one import attempt and invalidate files it may have written.""" + # Importers write files one at a time and may return a failed result after + # earlier writes. Invalidate every attempted import so cached file-first + # resources cannot survive either success or partial failure. + invalidation_scope = ( + invalidate_cache(read_cache, project_external_id) + if read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + return await importer.import_data(source_data, destination_directory) diff --git a/src/basic_memory/api/v2/routers/knowledge_router.py b/src/basic_memory/api/v2/routers/knowledge_router.py index 3ecf5874b..05fcffd31 100644 --- a/src/basic_memory/api/v2/routers/knowledge_router.py +++ b/src/basic_memory/api/v2/routers/knowledge_router.py @@ -11,12 +11,13 @@ """ from collections.abc import Mapping +from contextlib import nullcontext from hashlib import sha256 import os import pathlib from typing import Annotated -from fastapi import APIRouter, Header, HTTPException, Response, Path, status +from fastapi import APIRouter, Depends, Header, HTTPException, Response, Path, status from loguru import logger import logfire @@ -30,6 +31,7 @@ should_ignore_path, ) from basic_memory.deps import ( + create_model_read_cache, EntityServiceV2ExternalDep, FileServiceV2ExternalDep, SearchServiceV2ExternalDep, @@ -44,12 +46,21 @@ EntityRepositoryV2ExternalDep, RelationRepositoryV2ExternalDep, ProjectExternalIdPathDep, + ReadCacheDep, IndexFileExecutorV2ExternalDep, EntityVectorSyncSchedulerDep, RelationResolutionSchedulerDep, SessionDep, SessionMakerDep, ) +from basic_memory.read_cache import ( + ModelReadCache, + ReadCacheKey, + ReadCacheOperation, + ReadCacheScope, + invalidate_cache, + read_cache_request_digest, +) from basic_memory.runtime.note_content import ( NOTE_CONTENT_BASE_CHECKSUM_HEADER, runtime_note_content_payload_as_dict, @@ -71,12 +82,39 @@ OrphanEntitiesResponse, IndexFileRequest, ) +from basic_memory.workspace_context import current_workspace_permalink_context from basic_memory.schemas.response import DirectoryMoveResult from basic_memory.utils import validate_project_path router = APIRouter(prefix="/knowledge", tags=["knowledge-v2"]) +def get_resolve_read_cache( + read_cache: ReadCacheDep, +) -> ModelReadCache[EntityResolveResponse] | None: + """Bind identifier-resolution responses to the optional cache backend.""" + return create_model_read_cache(read_cache, EntityResolveResponse) + + +ResolveReadCacheDep = Annotated[ + ModelReadCache[EntityResolveResponse] | None, + Depends(get_resolve_read_cache), +] + + +def get_entity_read_cache( + read_cache: ReadCacheDep, +) -> ModelReadCache[EntityResponseV2] | None: + """Bind entity responses to the optional cache backend.""" + return create_model_read_cache(read_cache, EntityResponseV2) + + +EntityReadCacheDep = Annotated[ + ModelReadCache[EntityResponseV2] | None, + Depends(get_entity_read_cache), +] + + def _schedule_post_write_followups( *, vector_sync_scheduler, @@ -230,11 +268,15 @@ async def get_orphan_entities( @router.post("/resolve", response_model=EntityResolveResponse) async def resolve_identifier( project_id: ProjectExternalIdPathDep, + project_external_id: Annotated[ + str, Path(alias="project_id", description="Project external UUID") + ], data: EntityResolveRequest, link_resolver: LinkResolverV2ExternalDep, entity_repository: EntityRepositoryV2ExternalDep, project_repository: ProjectRepositoryDep, session: SessionDep, + read_cache: ResolveReadCacheDep, ) -> EntityResolveResponse: """Resolve a string identifier (external_id, permalink, title, or path) to entity info. @@ -273,59 +315,82 @@ async def resolve_identifier( ): logger.info(f"API v2 request: resolve_identifier for '{data.identifier}'") - entity = await entity_repository.get_by_external_id(session, data.identifier) - resolution_method = "external_id" if entity else "search" - - if not entity: - try: - entity = await link_resolver.resolve_link( - data.identifier, - source_path=data.source_path, - strict=data.strict, - session=session, - ) - except AmbiguousIdentifierError as exc: - # A strict resolve refused to guess between several same-title notes (#1148). - # Surface it as 409 so edit/move report the ambiguity and ask for an exact id. - raise HTTPException( - status_code=status.HTTP_409_CONFLICT, - detail=str(exc), - ) from exc - if entity: - if entity.permalink == data.identifier: - resolution_method = "permalink" - elif entity.title == data.identifier: - resolution_method = "title" - elif entity.file_path == data.identifier: - resolution_method = "path" - else: - resolution_method = "search" - - if not entity: - raise HTTPException(status_code=404, detail=f"Entity not found: '{data.identifier}'") - - owner_project = await project_repository.get_by_id(session, entity.project_id) - if not owner_project: # pragma: no cover - raise HTTPException( - status_code=500, - detail="Resolved entity references an unknown project", - ) - - result = EntityResolveResponse( - external_id=entity.external_id, - entity_id=entity.id, - project_external_id=owner_project.external_id, - permalink=entity.permalink, - file_path=entity.file_path, - title=entity.title, - resolution_method=resolution_method, + workspace_context = current_workspace_permalink_context() + cache_key = ReadCacheKey( + project_id=project_external_id, + operation=ReadCacheOperation.resolve, + request_digest=read_cache_request_digest( + data.model_dump_json(), + workspace_context.workspace_slug if workspace_context else "", + workspace_context.workspace_type if workspace_context else "", + ), ) - - logger.debug( - f"API v2 response: resolved '{data.identifier}' to external_id={result.external_id} via {resolution_method}" + cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[EntityResolveResponse]()) ) + async with cache_scope as cached: + if cached.value is not None: + return cached.value + + entity = await entity_repository.get_by_external_id(session, data.identifier) + resolution_method = "external_id" if entity else "search" + + if not entity: + try: + entity = await link_resolver.resolve_link( + data.identifier, + source_path=data.source_path, + strict=data.strict, + session=session, + ) + except AmbiguousIdentifierError as exc: + # A strict resolve refused to guess between several same-title notes (#1148). + # Surface it as 409 so edit/move report ambiguity and ask for an exact id. + raise HTTPException( + status_code=status.HTTP_409_CONFLICT, + detail=str(exc), + ) from exc + if entity: + if entity.permalink == data.identifier: + resolution_method = "permalink" + elif entity.title == data.identifier: + resolution_method = "title" + elif entity.file_path == data.identifier: + resolution_method = "path" + + if not entity: + raise HTTPException( + status_code=404, + detail=f"Entity not found: '{data.identifier}'", + ) - return result + owner_project = await project_repository.get_by_id(session, entity.project_id) + if not owner_project: # pragma: no cover + raise HTTPException( + status_code=500, + detail="Resolved entity references an unknown project", + ) + + result = EntityResolveResponse( + external_id=entity.external_id, + entity_id=entity.id, + project_external_id=owner_project.external_id, + permalink=entity.permalink, + file_path=entity.file_path, + title=entity.title, + resolution_method=resolution_method, + ) + logger.debug( + f"API v2 response: resolved '{data.identifier}' " + f"to external_id={result.external_id} via {resolution_method}" + ) + # Cross-project references depend on two projects. Keep phase-one + # generation invalidation exact by caching only local resolutions. + cached.cacheable = result.project_external_id == project_external_id + cached.value = result + return result ## Single-file indexing endpoint @@ -381,6 +446,9 @@ def _canonical_file_path(home: pathlib.Path, segments: list[str]) -> str | None: async def index_file( data: IndexFileRequest, project_id: ProjectExternalIdPathDep, + project_external_id: Annotated[ + str, Path(alias="project_id", description="Project external UUID") + ], file_service: FileServiceV2ExternalDep, file_indexer: IndexFileExecutorV2ExternalDep, entity_repository: EntityRepositoryV2ExternalDep, @@ -388,6 +456,7 @@ async def index_file( search_service: SearchServiceV2ExternalDep, app_config: AppConfigDep, session_maker: SessionMakerDep, + read_cache: ReadCacheDep, ) -> EntityResponseV2: """Index a single markdown file that exists on disk but is not indexed yet. @@ -487,7 +556,16 @@ async def index_file( detail=f"Only markdown files can be indexed: '{data.file_path}'", ) - indexed = await file_indexer.index_file(file_path, source="api-index-file") + # The file indexer commits entity state before search and reconciliation + # follow-ups finish. Invalidate even when a later phase raises so those + # partial commits cannot remain reachable through the old generation. + invalidation_scope = ( + invalidate_cache(read_cache, project_external_id) + if read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + indexed = await file_indexer.index_file(file_path, source="api-index-file") async with db.scoped_session(session_maker) as session: entity = await entity_repository.get_by_id(session, indexed.entity_id) if entity is None: # pragma: no cover @@ -518,10 +596,13 @@ async def index_file( @router.get("/entities/{entity_id}", response_model=EntityResponseV2) async def get_entity_by_id( project_id: ProjectExternalIdPathDep, - project_repository: ProjectRepositoryDep, + project_external_id: Annotated[ + str, Path(alias="project_id", description="Project external UUID") + ], entity_repository: EntityRepositoryV2ExternalDep, note_content_query_service: NoteContentQueryServiceDep, session: SessionDep, + read_cache: EntityReadCacheDep, entity_id: str = Path(..., description="Entity external ID (UUID)"), ) -> EntityResponseV2: """Get an entity by its external ID (UUID). @@ -546,30 +627,45 @@ async def get_entity_by_id( ): logger.info(f"API v2 request: get_entity_by_id entity_id={entity_id}") - project = await project_repository.get_by_id(session, project_id) - if project is None: # pragma: no cover - raise HTTPException(status_code=404, detail=f"Project with ID {project_id} not found") - - note_payload = await note_content_query_service.get_note_entity_payload_with_read_repair( - project_external_id=project.external_id, - entity_external_id=entity_id, - session=session, + cache_key = ReadCacheKey( + project_id=project_external_id, + operation=ReadCacheOperation.entity, + request_digest=read_cache_request_digest(entity_id), ) - if note_payload is not None: - result = entity_response_from_note_content_payload(note_payload) - logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'") - return result - - entity = await entity_repository.get_by_external_id(session, entity_id) - if not entity: - raise HTTPException( - status_code=404, detail=f"Entity with external_id '{entity_id}' not found" + cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[EntityResponseV2]()) + ) + async with cache_scope as cached: + if cached.value is not None: + return cached.value + + note_payload = ( + await note_content_query_service.get_note_entity_payload_with_read_repair( + project_external_id=project_external_id, + entity_external_id=entity_id, + session=session, + read_cache=read_cache, + ) ) + if note_payload is not None: + result = entity_response_from_note_content_payload(note_payload) + logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'") + cached.value = result + return result + + entity = await entity_repository.get_by_external_id(session, entity_id) + if not entity: + raise HTTPException( + status_code=404, + detail=f"Entity with external_id '{entity_id}' not found", + ) - result = EntityResponseV2.model_validate(entity) - logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'") - - return result + result = EntityResponseV2.model_validate(entity) + logger.info(f"API v2 response: external_id={entity_id}, title='{result.title}'") + cached.value = result + return result ## Create endpoints @@ -907,6 +1003,9 @@ async def move_entity( async def move_directory( data: MoveDirectoryRequestV2, project_id: ProjectExternalIdPathDep, + project_external_id: Annotated[ + str, Path(alias="project_id", description="Project external UUID") + ], entity_service: EntityServiceV2ExternalDep, project_config: ProjectConfigV2ExternalDep, app_config: AppConfigDep, @@ -914,6 +1013,7 @@ async def move_directory( vector_sync_scheduler: EntityVectorSyncSchedulerDep, relation_resolution_scheduler: RelationResolutionSchedulerDep, session_maker: SessionMakerDep, + read_cache: ReadCacheDep, ) -> DirectoryMoveResult: """Move all entities in a directory to a new location. @@ -945,23 +1045,34 @@ async def move_directory( destination_directory=data.destination_directory, project_config=project_config, app_config=app_config, + project_external_id=project_external_id, + read_cache=read_cache, ) - # Reindex moved entities - for file_path in result.moved_files: - async with db.scoped_session(session_maker) as session: - entity = await entity_service.link_resolver.resolve_link( - file_path, session=session - ) - if entity: - await search_service.index_entity(entity) - _schedule_post_write_followups( - vector_sync_scheduler=vector_sync_scheduler, - relation_resolution_scheduler=relation_resolution_scheduler, - app_config=app_config, - entity_id=entity.id, - project_id=project_id, - ) + # Reindexing can alter entity responses after the move was first + # invalidated. Close that fill window even after partial + # follow-up failure. + invalidation_scope = ( + invalidate_cache(read_cache, project_external_id) + if read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + # Reindex moved entities + for file_path in result.moved_files: + async with db.scoped_session(session_maker) as session: + entity = await entity_service.link_resolver.resolve_link( + file_path, session=session + ) + if entity: + await search_service.index_entity(entity) + _schedule_post_write_followups( + vector_sync_scheduler=vector_sync_scheduler, + relation_resolution_scheduler=relation_resolution_scheduler, + app_config=app_config, + entity_id=entity.id, + project_id=project_id, + ) logger.info( f"API v2 response: move_directory " @@ -984,6 +1095,7 @@ async def delete_directory( str, Path(alias="project_id", description="Project external UUID") ], directory_delete_service: DirectoryDeleteServiceDep, + read_cache: ReadCacheDep, ) -> Response: """Delete all entities in a directory. @@ -1010,6 +1122,7 @@ async def delete_directory( result = await directory_delete_service.delete_directory( project_external_id=project_external_id, directory=data.directory, + read_cache=read_cache, ) payload = result.to_response_payload() logger.info( diff --git a/src/basic_memory/api/v2/routers/project_router.py b/src/basic_memory/api/v2/routers/project_router.py index 3fd00fc9d..2ade0c1d2 100644 --- a/src/basic_memory/api/v2/routers/project_router.py +++ b/src/basic_memory/api/v2/routers/project_router.py @@ -10,6 +10,7 @@ - Consistent with v2 entity operations """ +from contextlib import nullcontext import os from typing import Literal, Optional @@ -24,10 +25,12 @@ ProjectIndexCommandDep, ProjectIndexObserverDep, ProjectExternalIdPathDep, + ReadCacheDep, SessionDep, SessionMakerDep, ) from basic_memory.index.local_project import ProjectIndexRouteRequest +from basic_memory.read_cache import invalidate_cache from basic_memory.schemas import ProjectIndexStatusResponse from basic_memory.models import Project from basic_memory.repository.project_repository import ProjectRepository @@ -408,6 +411,7 @@ async def update_project_by_id( project_service: ProjectServiceDep, session_maker: SessionMakerDep, project_repository: ProjectRepositoryDep, + read_cache: ReadCacheDep, project_id: str = Path(..., description="Project external ID (UUID)"), path: Optional[str] = Body(None, description="New absolute path for the project"), is_active: Optional[bool] = Body(None, description="Status of the project (active/inactive)"), @@ -454,7 +458,17 @@ async def update_project_by_id( # Update using project name (service layer still uses names internally) if path: - await project_service.move_project(old_project.name, path) + # A path update changes the filesystem source behind every + # resource key while project and entity UUIDs stay stable. The + # service can update config before its DB follow-up completes, + # so invalidate on every attempted move completion path. + invalidation_scope = ( + invalidate_cache(read_cache, project_id) + if read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + await project_service.move_project(old_project.name, path) elif is_active is not None: await project_service.update_project(old_project.name, is_active=is_active) diff --git a/src/basic_memory/api/v2/routers/resource_router.py b/src/basic_memory/api/v2/routers/resource_router.py index fbdebe2c8..4ab0b19aa 100644 --- a/src/basic_memory/api/v2/routers/resource_router.py +++ b/src/basic_memory/api/v2/routers/resource_router.py @@ -9,31 +9,70 @@ storage-event indexing pipeline. No API endpoint writes resource files inline. """ +from contextlib import nullcontext from pathlib import Path as PathLib +from typing import Annotated -from fastapi import APIRouter, HTTPException, Response, Path +from fastapi import APIRouter, Depends, HTTPException, Response, Path from loguru import logger +from pydantic import BaseModel, ConfigDict import logfire from basic_memory import db from basic_memory.deps import ( + create_model_read_cache, ProjectConfigV2ExternalDep, FileServiceV2ExternalDep, EntityRepositoryV2ExternalDep, NoteContentQueryServiceDep, + ReadCacheDep, SessionMakerDep, ) +from basic_memory.read_cache import ( + ModelReadCache, + ReadCacheKey, + ReadCacheOperation, + ReadCacheScope, + read_cache_request_digest, +) from basic_memory.utils import validate_project_path router = APIRouter(prefix="/resource", tags=["resources-v2"]) +class CachedResourceResponse(BaseModel): + """Typed wire value for one cacheable resource response.""" + + content: bytes + media_type: str + + model_config = ConfigDict(ser_json_bytes="base64", val_json_bytes="base64") + + +def get_resource_read_cache( + read_cache: ReadCacheDep, +) -> ModelReadCache[CachedResourceResponse] | None: + """Bind resource responses to the optional cache backend.""" + return create_model_read_cache(read_cache, CachedResourceResponse) + + +ResourceReadCacheDep = Annotated[ + ModelReadCache[CachedResourceResponse] | None, + Depends(get_resource_read_cache), +] + + +def _is_markdown_resource(resource: CachedResourceResponse) -> bool: + return resource.media_type.partition(";")[0].strip().lower() == "text/markdown" + + @router.get("/{entity_id}") async def get_resource_content( config: ProjectConfigV2ExternalDep, entity_repository: EntityRepositoryV2ExternalDep, file_service: FileServiceV2ExternalDep, note_content_query_service: NoteContentQueryServiceDep, + read_cache: ResourceReadCacheDep, session_maker: SessionMakerDep, project_id: str = Path(..., description="Project external UUID"), entity_id: str = Path(..., description="Entity external UUID"), @@ -61,69 +100,104 @@ async def get_resource_content( ): logger.debug(f"V2 Getting content for project {project_id}, entity_id: {entity_id}") - # Keep the DB session open only for the lookups; close it before the - # filesystem I/O below so large/slow resource reads don't pin a pooled - # connection (and an open read transaction on Postgres) for their duration. - async with db.scoped_session(session_maker) as session: - note_resource = await note_content_query_service.get_note_resource_with_read_repair( - project_external_id=project_id, - entity_external_id=entity_id, - session=session, - ) - if note_resource is not None: + cache_key = ReadCacheKey( + project_id=project_id, + operation=ReadCacheOperation.resource, + request_digest=read_cache_request_digest(entity_id), + ) + cache_scope = ( + read_cache.read(key=cache_key) + if read_cache is not None + else nullcontext(ReadCacheScope[CachedResourceResponse]()) + ) + async with cache_scope as cached: + if cached.value is not None: return Response( - content=note_resource.content, - media_type=note_resource.content_type, + content=cached.value.content, + media_type=cached.value.media_type, + ) + + # Keep the DB session open only for the lookups; close it before the + # filesystem I/O below so large/slow resource reads don't pin a pooled + # connection (and an open read transaction on Postgres) for their duration. + async with db.scoped_session(session_maker) as session: + note_resource = await note_content_query_service.get_note_resource_with_read_repair( + project_external_id=project_id, + entity_external_id=entity_id, + session=session, + read_cache=read_cache, ) + if note_resource is not None: + resource = CachedResourceResponse( + content=note_resource.content.encode("utf-8"), + media_type=note_resource.content_type, + ) + cached.value = resource + return Response( + content=resource.content, + media_type=resource.media_type, + ) + + with logfire.span( + "api.resource.get_content.load_entity", + domain="resource", + action="get_content", + phase="load_entity", + ): + entity = await entity_repository.get_by_external_id(session, entity_id) + if not entity: + raise HTTPException( + status_code=404, + detail=f"Entity {entity_id} not found", + ) + # Copy the scalar columns needed for file I/O so the session can close. + entity_file_path = entity.file_path + entity_db_id = entity.id with logfire.span( - "api.resource.get_content.load_entity", + "api.resource.get_content.validate_path", domain="resource", action="get_content", - phase="load_entity", + phase="validate_path", ): - entity = await entity_repository.get_by_external_id(session, entity_id) - if not entity: - raise HTTPException(status_code=404, detail=f"Entity {entity_id} not found") - # Copy the scalar columns needed for file I/O so the session can close. - entity_file_path = entity.file_path - entity_db_id = entity.id - - with logfire.span( - "api.resource.get_content.validate_path", - domain="resource", - action="get_content", - phase="validate_path", - ): - project_path = PathLib(config.home) - if not validate_project_path(entity_file_path, project_path): - logger.error( # pragma: no cover - f"Invalid file path in entity {entity_db_id}: {entity_file_path}" - ) - raise HTTPException( # pragma: no cover - status_code=500, - detail="Entity contains invalid file path", - ) + project_path = PathLib(config.home) + if not validate_project_path(entity_file_path, project_path): + logger.error( # pragma: no cover + f"Invalid file path in entity {entity_db_id}: {entity_file_path}" + ) + raise HTTPException( # pragma: no cover + status_code=500, + detail="Entity contains invalid file path", + ) - with logfire.span( - "api.resource.get_content.ensure_exists", - domain="resource", - action="get_content", - phase="ensure_exists", - ): - if not await file_service.exists(entity_file_path): - raise HTTPException( # pragma: no cover - status_code=404, - detail=f"File not found: {entity_file_path}", - ) + with logfire.span( + "api.resource.get_content.ensure_exists", + domain="resource", + action="get_content", + phase="ensure_exists", + ): + if not await file_service.exists(entity_file_path): + raise HTTPException( # pragma: no cover + status_code=404, + detail=f"File not found: {entity_file_path}", + ) - with logfire.span( - "api.resource.get_content.read_content", - domain="resource", - action="get_content", - phase="read_content", - ): - content = await file_service.read_file_bytes(entity_file_path) - content_type = file_service.content_type(entity_file_path) + with logfire.span( + "api.resource.get_content.read_content", + domain="resource", + action="get_content", + phase="read_content", + ): + content = await file_service.read_file_bytes(entity_file_path) + content_type = file_service.content_type(entity_file_path) - return Response(content=content, media_type=content_type) + resource = CachedResourceResponse( + content=content, + media_type=content_type, + ) + cached.cacheable = _is_markdown_resource(resource) + cached.value = resource + return Response( + content=resource.content, + media_type=resource.media_type, + ) diff --git a/src/basic_memory/deps/__init__.py b/src/basic_memory/deps/__init__.py index 82dbe7cbd..d45c6e18f 100644 --- a/src/basic_memory/deps/__init__.py +++ b/src/basic_memory/deps/__init__.py @@ -38,6 +38,12 @@ ProjectConfigV2ExternalDep, ) +from basic_memory.deps.read_cache import ( + create_model_read_cache, + get_read_cache, + ReadCacheDep, +) + from basic_memory.deps.repositories import ( get_entity_repository_v2_external, EntityRepositoryV2ExternalDep, @@ -123,6 +129,10 @@ "ProjectExternalIdPathDep", "get_project_config_v2_external", "ProjectConfigV2ExternalDep", + # Read cache + "create_model_read_cache", + "get_read_cache", + "ReadCacheDep", # Repositories "get_entity_repository_v2_external", "EntityRepositoryV2ExternalDep", diff --git a/src/basic_memory/deps/importers.py b/src/basic_memory/deps/importers.py index fc1c013bb..44c51d77e 100644 --- a/src/basic_memory/deps/importers.py +++ b/src/basic_memory/deps/importers.py @@ -12,6 +12,7 @@ from fastapi import Depends from basic_memory.deps.projects import ProjectConfigV2ExternalDep +from basic_memory.deps.read_cache import ReadCacheDep from basic_memory.deps.services import ( FileServiceV2ExternalDep, MarkdownProcessorV2ExternalDep, @@ -20,10 +21,24 @@ ChatGPTImporter, ClaudeConversationsImporter, ClaudeProjectsImporter, + ImportReadCache, MemoryJsonImporter, ) +async def get_import_read_cache( + read_cache: ReadCacheDep, + project_id: str, +) -> ImportReadCache | None: + """Bind the optional host cache to the requested project.""" + if read_cache is None: + return None + return ImportReadCache(backend=read_cache, project_id=project_id) + + +ImportReadCacheDep = Annotated[ImportReadCache | None, Depends(get_import_read_cache)] + + # --- ChatGPT Importer --- @@ -31,6 +46,7 @@ async def get_chatgpt_importer_v2_external( project_config: ProjectConfigV2ExternalDep, markdown_processor: MarkdownProcessorV2ExternalDep, file_service: FileServiceV2ExternalDep, + read_cache: ImportReadCacheDep, ) -> ChatGPTImporter: """Create ChatGPTImporter with v2 external_id dependencies.""" return ChatGPTImporter( @@ -38,6 +54,7 @@ async def get_chatgpt_importer_v2_external( markdown_processor, file_service, project_name=project_config.name, + read_cache=read_cache, ) @@ -51,6 +68,7 @@ async def get_claude_conversations_importer_v2_external( project_config: ProjectConfigV2ExternalDep, markdown_processor: MarkdownProcessorV2ExternalDep, file_service: FileServiceV2ExternalDep, + read_cache: ImportReadCacheDep, ) -> ClaudeConversationsImporter: """Create ClaudeConversationsImporter with v2 external_id dependencies.""" return ClaudeConversationsImporter( @@ -58,6 +76,7 @@ async def get_claude_conversations_importer_v2_external( markdown_processor, file_service, project_name=project_config.name, + read_cache=read_cache, ) @@ -73,6 +92,7 @@ async def get_claude_projects_importer_v2_external( project_config: ProjectConfigV2ExternalDep, markdown_processor: MarkdownProcessorV2ExternalDep, file_service: FileServiceV2ExternalDep, + read_cache: ImportReadCacheDep, ) -> ClaudeProjectsImporter: """Create ClaudeProjectsImporter with v2 external_id dependencies.""" return ClaudeProjectsImporter( @@ -80,6 +100,7 @@ async def get_claude_projects_importer_v2_external( markdown_processor, file_service, project_name=project_config.name, + read_cache=read_cache, ) @@ -95,6 +116,7 @@ async def get_memory_json_importer_v2_external( project_config: ProjectConfigV2ExternalDep, markdown_processor: MarkdownProcessorV2ExternalDep, file_service: FileServiceV2ExternalDep, + read_cache: ImportReadCacheDep, ) -> MemoryJsonImporter: """Create MemoryJsonImporter with v2 external_id dependencies.""" return MemoryJsonImporter( @@ -102,6 +124,7 @@ async def get_memory_json_importer_v2_external( markdown_processor, file_service, project_name=project_config.name, + read_cache=read_cache, ) diff --git a/src/basic_memory/deps/read_cache.py b/src/basic_memory/deps/read_cache.py new file mode 100644 index 000000000..ec81b341e --- /dev/null +++ b/src/basic_memory/deps/read_cache.py @@ -0,0 +1,47 @@ +"""Optional semantic read-cache dependency.""" + +from typing import Annotated + +from fastapi import Depends, Request +from pydantic import BaseModel + +from basic_memory.read_cache import ModelReadCache, ReadCache +from basic_memory.read_cache.policy import ( + READ_CACHE_MAX_PAYLOAD_BYTES, + READ_CACHE_TTL_SECONDS, +) + + +def get_read_cache(request: Request) -> ReadCache | None: + """Return the optional host-injected cache backend.""" + try: + container = request.app.state.container + except AttributeError: + pass + else: + return container.read_cache + + # Deferred import avoids api.app -> routers -> deps circular initialization. + from basic_memory.api.container import resolve_container + + return resolve_container().read_cache + + +ReadCacheDep = Annotated[ReadCache | None, Depends(get_read_cache)] + + +def create_model_read_cache[ModelT: BaseModel]( + read_cache: ReadCache | None, + model_type: type[ModelT], + *, + max_payload_bytes: int = READ_CACHE_MAX_PAYLOAD_BYTES, +) -> ModelReadCache[ModelT] | None: + """Bind one response model to the host cache and Basic Memory's read policy.""" + if read_cache is None: + return None + return ModelReadCache( + backend=read_cache, + model_type=model_type, + ttl_seconds=READ_CACHE_TTL_SECONDS, + max_payload_bytes=max_payload_bytes, + ) diff --git a/src/basic_memory/deps/services.py b/src/basic_memory/deps/services.py index d5996b103..1cf33e030 100644 --- a/src/basic_memory/deps/services.py +++ b/src/basic_memory/deps/services.py @@ -14,7 +14,7 @@ from pathlib import Path from typing import Annotated -from fastapi import Depends +from fastapi import Depends, Path as FastAPIPath from loguru import logger from basic_memory.deps.config import AppConfigDep @@ -23,6 +23,7 @@ ProjectConfigV2ExternalDep, ProjectRepositoryDep, ) +from basic_memory.deps.read_cache import ReadCacheDep from basic_memory.deps.repositories import ( EntityRepositoryV2ExternalDep, ObservationRepositoryV2ExternalDep, @@ -47,6 +48,7 @@ from basic_memory.index.local_project import ( LocalProjectIndexCommand, LocalProjectIndexRunner, + LocalProjectIndexRuntimeFactory, ) from basic_memory.index.project_indexing import ( ProjectIndexCommand, @@ -313,6 +315,7 @@ async def get_note_content_mutation_service( file_indexer: IndexFileExecutorV2ExternalDep, session_maker: SessionMakerDep, app_config: AppConfigDep, + read_cache: ReadCacheDep, ) -> NoteContentMutationService: """Create the local accepted-note mutation facade for API routes.""" accepted_note_repositories = AcceptedNoteRepositories( @@ -347,6 +350,7 @@ async def get_note_content_mutation_service( file_indexer=file_indexer, session_maker=session_maker, ), + read_cache=read_cache, ) @@ -361,11 +365,13 @@ async def get_note_content_mutation_service( async def get_project_index_runner( project_repository: ProjectRepositoryDep, session_maker: SessionMakerDep, + read_cache: ReadCacheDep, ) -> LocalProjectIndexRunner: """Create the local project-index runner used by API routes and tasks.""" return LocalProjectIndexRunner( project_repository=project_repository, session_maker=session_maker, + runtime_factory=LocalProjectIndexRuntimeFactory(read_cache=read_cache), ) @@ -411,22 +417,32 @@ async def get_project_index_scheduler( async def get_search_reindex_scheduler( + project_external_id: Annotated[ + str, FastAPIPath(alias="project_id", description="Project external UUID") + ], search_service: SearchServiceV2ExternalDep, app_config: AppConfigDep, + read_cache: ReadCacheDep, ) -> SearchReindexScheduler: return LocalSearchReindexScheduler( search_service=search_service, + project_external_id=project_external_id, + read_cache=read_cache, test_mode=app_config.is_test_env, ) async def get_relation_resolution_scheduler( + project_external_id: Annotated[ + str, FastAPIPath(alias="project_id", description="Project external UUID") + ], session_maker: SessionMakerDep, entity_repository: EntityRepositoryV2ExternalDep, relation_repository: RelationRepositoryV2ExternalDep, link_resolver: LinkResolverV2ExternalDep, search_service: SearchServiceV2ExternalDep, app_config: AppConfigDep, + read_cache: ReadCacheDep, ) -> RelationResolutionScheduler: # Build the project-scoped resolution runtime. It owns its own sessions via # session_maker, so it is safe to run from a detached background task. @@ -443,6 +459,8 @@ async def get_relation_resolution_scheduler( ) return LocalRelationResolutionScheduler( relation_runtime=runtime, + project_external_id=project_external_id, + read_cache=read_cache, test_mode=app_config.is_test_env, ) @@ -474,11 +492,15 @@ async def get_relation_resolution_scheduler( async def get_note_content_materialization_provider( + project_external_id: Annotated[ + str, FastAPIPath(alias="project_id", description="Project external UUID") + ], file_service: FileServiceV2ExternalDep, file_indexer: IndexFileExecutorV2ExternalDep, session_maker: SessionMakerDep, app_config: AppConfigDep, relation_resolution_scheduler: RelationResolutionSchedulerDep, + read_cache: ReadCacheDep, ) -> LocalNoteContentMaterializationProvider: """Create the local materializer for accepted-note route writes. @@ -489,6 +511,8 @@ async def get_note_content_materialization_provider( return LocalNoteContentMaterializationProvider( session_maker=session_maker, file_service=file_service, + project_external_id=project_external_id, + read_cache=read_cache, file_indexer=file_indexer, test_mode=app_config.is_test_env, materialization_workers=app_config.materialization_workers, diff --git a/src/basic_memory/importers/__init__.py b/src/basic_memory/importers/__init__.py index 3336cce02..ba7876afc 100644 --- a/src/basic_memory/importers/__init__.py +++ b/src/basic_memory/importers/__init__.py @@ -1,6 +1,6 @@ """Import services for Basic Memory.""" -from basic_memory.importers.base import Importer +from basic_memory.importers.base import Importer, ImportReadCache from basic_memory.importers.chatgpt_importer import ChatGPTImporter from basic_memory.importers.claude_conversations_importer import ( ClaudeConversationsImporter, @@ -22,6 +22,7 @@ __all__ = [ "Importer", + "ImportReadCache", "ChatGPTImporter", "ClaudeConversationsImporter", "ClaudeProjectsImporter", diff --git a/src/basic_memory/importers/base.py b/src/basic_memory/importers/base.py index d3ba81d25..280fb82f0 100644 --- a/src/basic_memory/importers/base.py +++ b/src/basic_memory/importers/base.py @@ -2,11 +2,14 @@ import logging from abc import abstractmethod +from contextlib import nullcontext +from dataclasses import dataclass from pathlib import Path from typing import TYPE_CHECKING, Any, Optional, TypeVar from basic_memory.markdown.markdown_processor import MarkdownProcessor from basic_memory.markdown.schemas import EntityMarkdown +from basic_memory.read_cache import ReadCacheInvalidator, invalidate_cache from basic_memory.schemas.importer import ImportResult from basic_memory.utils import build_canonical_permalink, generate_permalink @@ -18,6 +21,14 @@ T = TypeVar("T", bound=ImportResult) +@dataclass(frozen=True, slots=True) +class ImportReadCache: + """Bind an importer's file writes to one project cache generation.""" + + backend: ReadCacheInvalidator + project_id: str + + class Importer[T: ImportResult]: """Base class for all import services. @@ -31,6 +42,8 @@ def __init__( markdown_processor: MarkdownProcessor, file_service: "FileService", project_name: Optional[str] = None, + *, + read_cache: ImportReadCache | None = None, ): """Initialize the import service. @@ -44,6 +57,7 @@ def __init__( self.file_service = file_service self.project_name = project_name self.project_permalink = generate_permalink(project_name) if project_name else None + self.read_cache = read_cache @abstractmethod async def import_data(self, source_data, destination_folder: str, **kwargs: Any) -> T: @@ -74,8 +88,19 @@ async def write_entity(self, entity: EntityMarkdown, file_path: str | Path) -> s Checksum of written file. """ content = self.markdown_processor.to_markdown_string(entity) - # FileService.write_file handles directory creation and returns checksum - return await self.file_service.write_file(file_path, content) + + # Trigger: one imported file can become visible before the remaining batch completes. + # Why: cached file-first resources must not retain the overwritten bytes until the + # import's final failure-safe invalidation. + # Outcome: advance the project generation after every attempted file write. + invalidation_scope = ( + invalidate_cache(self.read_cache.backend, self.read_cache.project_id) + if self.read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + # FileService.write_file handles directory creation and returns checksum + return await self.file_service.write_file(file_path, content) def canonical_permalink(self, path: str) -> str: """Build a canonical permalink for imported content.""" diff --git a/src/basic_memory/index/local_moves.py b/src/basic_memory/index/local_moves.py index 2f71275e8..da6c332ba 100644 --- a/src/basic_memory/index/local_moves.py +++ b/src/basic_memory/index/local_moves.py @@ -3,6 +3,7 @@ from __future__ import annotations from collections.abc import Mapping, Sequence +from contextlib import nullcontext from dataclasses import dataclass from pathlib import Path from typing import override, Protocol @@ -35,6 +36,7 @@ STORAGE_OBJECT_CREATED_EVENTS, STORAGE_OBJECT_DELETED_EVENT, ) +from basic_memory.read_cache import ReadCache, invalidate_cache from basic_memory.services import FileService @@ -171,6 +173,8 @@ class LocalWatchMoveProcessor: entity_repository: LocalMoveEntityRepository maintenance_runner: ProjectIndexMaintenanceRunner moved_entity_search_refresher: ProjectIndexMovedEntitySearchRefresher + project_external_id: str + read_cache: ReadCache | None batch_size: int = 100 async def process_moves( @@ -182,15 +186,26 @@ async def process_moves( removed_event_indexes: set[int] = set() if moved_files: - move_run = await self.maintenance_runner.run_move_batches( - moved_files=moved_files, - batch_size=self.batch_size, + # Move pairs bypass ordinary file/delete completion callbacks. + # Invalidate after maintenance and search refresh so cached paths, + # permalinks, and relations cannot retain the pre-move state. + invalidation_scope = ( + invalidate_cache(self.read_cache, self.project_external_id) + if self.read_cache is not None + else nullcontext() ) - refresh_entity_ids = move_run.moved_entity_ids | move_run.relation_cleanup_entity_ids - if refresh_entity_ids: - await self.moved_entity_search_refresher.refresh_moved_entities( - sorted(refresh_entity_ids) + async with invalidation_scope: + move_run = await self.maintenance_runner.run_move_batches( + moved_files=moved_files, + batch_size=self.batch_size, ) + refresh_entity_ids = ( + move_run.moved_entity_ids | move_run.relation_cleanup_entity_ids + ) + if refresh_entity_ids: + await self.moved_entity_search_refresher.refresh_moved_entities( + sorted(refresh_entity_ids) + ) moved_old_paths = set(moved_files) - set(move_run.missing_paths) moved_new_paths = {moved_files[old_path] for old_path in moved_old_paths} diff --git a/src/basic_memory/index/local_project.py b/src/basic_memory/index/local_project.py index fb3a0b41f..1c56aa3b4 100644 --- a/src/basic_memory/index/local_project.py +++ b/src/basic_memory/index/local_project.py @@ -5,6 +5,7 @@ import asyncio import os from collections.abc import Mapping, Sequence +from contextlib import nullcontext from dataclasses import dataclass from pathlib import Path from typing import Any, override, Protocol @@ -65,6 +66,7 @@ run_project_index_coordinator, ) from basic_memory.indexing.project_index_maintenance import ( + InvalidatingProjectIndexBatchStore, ProjectIndexDeletePathVerifier, ProjectIndexMaintenanceRunner, ProjectIndexMovedEntitySearchRefresher, @@ -79,6 +81,11 @@ resolve_project_index_completion_relations, ) from basic_memory.models import Entity, Project +from basic_memory.read_cache import ( + ReadCache, + ReadCacheInvalidator, + invalidate_cache, +) from basic_memory.repository import NoteContentRepository from basic_memory.runtime.jobs import ( RuntimeIndexFileBatchJobRequest, @@ -459,6 +466,7 @@ class LocalProjectIndexRuntime: embedding_vector_sync: EmbeddingBatchVectorSync | None = None batch_size: int = 100 coordinator_job_id: RuntimeJobId | None = None + read_cache: ReadCache | None = None LocalProjectIndexObservation = ProjectIndexObservation @@ -544,6 +552,7 @@ class LocalProjectIndexBatchEnqueuer(ProjectIndexBatchEnqueuer): reader: IndexFileBatchReader[IndexInputFile] indexer: IndexFileBatchIndexer[IndexInputFile] content_classifier: IndexFileBatchContentClassifier + read_cache: ReadCacheInvalidator | None = None read_max_concurrent: int = 8 index_max_concurrent: int = 8 @@ -552,15 +561,24 @@ async def enqueue_index_file_batch( self, request: RuntimeIndexFileBatchJobRequest, ) -> IndexFileBatchJobResult: - return await run_index_file_batch( - request, - checker=self.checker, - reader=self.reader, - indexer=self.indexer, - content_classifier=self.content_classifier, - read_max_concurrent=self.read_max_concurrent, - index_max_concurrent=self.index_max_concurrent, + invalidation_scope = ( + invalidate_cache( + self.read_cache, + request.project.project_external_id, + ) + if self.read_cache is not None + else nullcontext() ) + async with invalidation_scope: + return await run_index_file_batch( + request, + checker=self.checker, + reader=self.reader, + indexer=self.indexer, + content_classifier=self.content_classifier, + read_max_concurrent=self.read_max_concurrent, + index_max_concurrent=self.index_max_concurrent, + ) @dataclass(frozen=True, slots=True) @@ -573,6 +591,7 @@ class LocalProjectIndexRuntimeFactory: batch_size: int = 100 read_max_concurrent: int = 8 index_max_concurrent: int = 8 + read_cache: ReadCache | None = None async def dependencies_for_project(self, project: Project) -> LocalIndexProjectDependencies: return await self.dependency_provider.dependencies_for_project(project) @@ -580,6 +599,8 @@ async def dependencies_for_project(self, project: Project) -> LocalIndexProjectD def runtime_from_dependencies( self, dependencies: LocalIndexProjectDependencies, + *, + project_external_id: str, ) -> LocalProjectIndexRuntime: metadata_source = LocalStorageFileMetadataSource(dependencies.file_service) checker = FileIndexChecker( @@ -613,6 +634,16 @@ def runtime_from_dependencies( # concurrent creation that must be checksum-verified before deletion. verify_replaced_move_targets=True, ) + active_maintenance_store = ( + InvalidatingProjectIndexBatchStore( + move_store=maintenance_store, + delete_store=maintenance_store, + read_cache=self.read_cache, + project_external_id=project_external_id, + ) + if self.read_cache is not None + else maintenance_store + ) return LocalProjectIndexRuntime( observed_file_source=LocalProjectIndexObservedFileSource( dependencies.file_service, @@ -626,8 +657,8 @@ def runtime_from_dependencies( entity_repository=dependencies.entity_repository, ), maintenance_runner=StoreProjectIndexMaintenanceRunner( - move_store=maintenance_store, - delete_store=maintenance_store, + move_store=active_maintenance_store, + delete_store=active_maintenance_store, ), moved_entity_search_refresher=RepositoryProjectIndexMovedEntitySearchRefresher( session_maker=dependencies.session_maker, @@ -639,6 +670,7 @@ def runtime_from_dependencies( reader=LocalIndexFileBatchReader(dependencies.file_service), indexer=dependencies.file_batch_indexer, content_classifier=dependencies.file_service, + read_cache=self.read_cache, read_max_concurrent=self.read_max_concurrent, index_max_concurrent=self.index_max_concurrent, ), @@ -652,10 +684,14 @@ def runtime_from_dependencies( ), embedding_vector_sync=local_project_embedding_vector_sync(dependencies), batch_size=self.batch_size, + read_cache=self.read_cache, ) async def runtime_for_project(self, project: Project) -> LocalProjectIndexRuntime: - return self.runtime_from_dependencies(await self.dependencies_for_project(project)) + return self.runtime_from_dependencies( + await self.dependencies_for_project(project), + project_external_id=str(project.external_id), + ) async def run_local_project_index_for_project( @@ -699,7 +735,10 @@ async def _get_project(self, project_id: int) -> Project: async def observe_project(self, project_id: int) -> LocalProjectIndexObservation: project = await self._get_project(project_id) dependencies = await self.runtime_factory.dependencies_for_project(project) - runtime = self.runtime_factory.runtime_from_dependencies(dependencies) + runtime = self.runtime_factory.runtime_from_dependencies( + dependencies, + project_external_id=str(project.external_id), + ) observed_files = await runtime.observed_file_source.list_observed_index_files() return LocalProjectIndexObservation(observed_files=observed_files) @@ -757,25 +796,49 @@ async def run_local_project_index( runtime: LocalProjectIndexRuntime, ) -> ProjectIndexCoordinatorResult: """Run project-wide local indexing through the storage-neutral coordinator.""" - result = await run_project_index_coordinator( - request, - coordinator_job_id=runtime.coordinator_job_id, - observed_file_source=runtime.observed_file_source, - change_detector=runtime.change_detector, - maintenance_runner=runtime.maintenance_runner, - moved_entity_search_refresher=runtime.moved_entity_search_refresher, - workflow_starter=runtime.workflow_starter, - batch_enqueuer=runtime.batch_enqueuer, - fanout_failure_recorder=runtime.fanout_failure_recorder, - batch_size=runtime.batch_size, - embedding_vector_sync=runtime.embedding_vector_sync, + # The coordinator commits moves, deletes, and file batches incrementally. + # Invalidate even when a later batch or vector sync raises so already + # published changes cannot remain behind the previous generation. + invalidation_scope = ( + invalidate_cache( + runtime.read_cache, + request.project.project_external_id, + ) + if runtime.read_cache is not None + else nullcontext() ) + async with invalidation_scope: + result = await run_project_index_coordinator( + request, + coordinator_job_id=runtime.coordinator_job_id, + observed_file_source=runtime.observed_file_source, + change_detector=runtime.change_detector, + maintenance_runner=runtime.maintenance_runner, + moved_entity_search_refresher=runtime.moved_entity_search_refresher, + workflow_starter=runtime.workflow_starter, + batch_enqueuer=runtime.batch_enqueuer, + fanout_failure_recorder=runtime.fanout_failure_recorder, + batch_size=runtime.batch_size, + embedding_vector_sync=runtime.embedding_vector_sync, + ) if runtime.completion_relation_runtime is not None: - await resolve_project_index_completion_relations( - ProjectIndexRelationResolutionContext( - project_id=request.project.project_id, - project_path=request.project.project_path, - ), - runtime.completion_relation_runtime, + # Relation repair can mutate cached entity responses after the first + # invalidation. Clear any value filled during that window, including + # when repair commits partial progress before raising. + relation_invalidation_scope = ( + invalidate_cache( + runtime.read_cache, + request.project.project_external_id, + ) + if runtime.read_cache is not None + else nullcontext() ) + async with relation_invalidation_scope: + await resolve_project_index_completion_relations( + ProjectIndexRelationResolutionContext( + project_id=request.project.project_id, + project_path=request.project.project_path, + ), + runtime.completion_relation_runtime, + ) return result diff --git a/src/basic_memory/index/local_runtime.py b/src/basic_memory/index/local_runtime.py index 4a2dc8c96..16d443eee 100644 --- a/src/basic_memory/index/local_runtime.py +++ b/src/basic_memory/index/local_runtime.py @@ -2,6 +2,7 @@ from __future__ import annotations +from contextlib import nullcontext from dataclasses import dataclass from loguru import logger @@ -27,7 +28,12 @@ StorageEventOperationProcessorFactory, StorageEventProjectResolver, ) -from basic_memory.indexing.external_file_delete_runner import ExternalFileDeleteResult +from basic_memory.indexing.external_file_delete_runner import ( + ExternalFileDeleteEntities, + ExternalFileDeleteResult, + InvalidatingExternalFileDeleteEntities, + RepositoryExternalFileDeleteEntities, +) from basic_memory.indexing.file_index_checking import ( FileIndexChecker, RepositoryIndexedFileChecksumSource, @@ -37,10 +43,13 @@ from basic_memory.repository.note_file_vacate_repository import NoteFileVacateRepository from basic_memory.indexing.index_file_runner import ( IndexFileObjectMetadata, + IndexFileExecutor, + InvalidatingIndexFileExecutor, RepositoryCurrentMaterializedNoteSource, ) from basic_memory.indexing.models import IndexFileJobResult, IndexFileJobStatus from basic_memory.indexing.project_index_maintenance import ( + InvalidatingProjectIndexBatchStore, ProjectIndexMovedEntitySearchRefresher, RepositoryProjectIndexMaintenanceStore, RepositoryProjectIndexMovedEntitySearchRefresher, @@ -53,10 +62,12 @@ plan_index_file_relation_resolution, resolve_project_relations, ) -from basic_memory.indexing.external_file_delete_runner import ( - RepositoryExternalFileDeleteEntities, -) from basic_memory.models import Entity, Project +from basic_memory.read_cache import ( + ReadCache, + finish_project_read_cache_invalidation, + invalidate_cache, +) from basic_memory.repository import NoteContentRepository from basic_memory.runtime.projects import ProjectRuntimeReference from basic_memory.runtime.storage import ( @@ -64,6 +75,7 @@ RuntimeFileChecksum, RuntimeFilePath, RuntimeStorageEventOperation, + RuntimeStorageEventOperationKind, ) from basic_memory.services import FileService from basic_memory.services.exceptions import FileOperationError @@ -144,6 +156,7 @@ class LocalInlineStorageEventResultRecorder: relation_cleanup_search_refresher: ProjectIndexMovedEntitySearchRefresher relation_runtime: RelationResolutionRuntime index_embeddings: bool + read_cache: ReadCache | None = None async def index_file_completed( self, @@ -158,6 +171,12 @@ async def index_file_completed( entity_id=result.entity_id, ) + if result.status == IndexFileJobStatus.processed and self.read_cache is not None: + await finish_project_read_cache_invalidation( + self.read_cache, + self.project.project_external_id, + ) + # --- Relation repair --- # Back-resolve forward references now that this file is indexed. relation_request = plan_index_file_relation_resolution( @@ -168,15 +187,27 @@ async def index_file_completed( ) ) if relation_request is not None: - relation_result = await resolve_project_relations(self.relation_runtime) - logger.info( - "Local event-index relation repair completed", - project_id=relation_request.project_id, - project_path=relation_request.project_path, - resolved=relation_result.resolved, - remaining=relation_result.remaining, - passes=relation_result.passes, + # Relation repair changes cached entity payloads after indexing. + # A second generation bump closes the fill window opened by the + # first post-index invalidation, even after partial failure. + invalidation_scope = ( + invalidate_cache( + self.read_cache, + self.project.project_external_id, + ) + if self.read_cache is not None + else nullcontext() ) + async with invalidation_scope: + relation_result = await resolve_project_relations(self.relation_runtime) + logger.info( + "Local event-index relation repair completed", + project_id=relation_request.project_id, + project_path=relation_request.project_path, + resolved=relation_result.resolved, + remaining=relation_result.remaining, + passes=relation_result.passes, + ) # --- Semantic embedding --- # Trigger: a file was (re)indexed and semantic embeddings are enabled. @@ -209,12 +240,31 @@ async def delete_file_completed( ) if not result.entity_deleted: return - if not isinstance(result.deleted_entity, Entity): - raise RuntimeError("Local external file delete returned an incomplete entity result") - await self.search_service.handle_delete(result.deleted_entity) - await self.relation_cleanup_search_refresher.refresh_moved_entities( - tuple(sorted(result.relation_cleanup_entity_ids)), + if self.read_cache is not None: + await finish_project_read_cache_invalidation( + self.read_cache, + self.project.project_external_id, + ) + # Cleanup may rewrite relations on surviving entities. Invalidate + # values filled after the delete became visible, including partial + # cleanup progress followed by an error. + invalidation_scope = ( + invalidate_cache( + self.read_cache, + self.project.project_external_id, + ) + if self.read_cache is not None + else nullcontext() ) + async with invalidation_scope: + if not isinstance(result.deleted_entity, Entity): + raise RuntimeError( + "Local external file delete returned an incomplete entity result" + ) + await self.search_service.handle_delete(result.deleted_entity) + await self.relation_cleanup_search_refresher.refresh_moved_entities( + tuple(sorted(result.relation_cleanup_entity_ids)), + ) async def skip_event(self, operation: RuntimeStorageEventOperation) -> None: logger.debug( @@ -233,6 +283,17 @@ async def event_failed( file_path=operation.relative_path, error=str(exc), ) + if ( + operation.kind == RuntimeStorageEventOperationKind.index_file + and self.read_cache is not None + ): + # LocalMarkdownFileIndexer commits the entity before all search and + # reconciliation follow-ups complete. A watcher failure can therefore + # publish partial state even though the success callback never runs. + await finish_project_read_cache_invalidation( + self.read_cache, + self.project.project_external_id, + ) @dataclass(frozen=True, slots=True) @@ -264,6 +325,7 @@ class LocalWatchEventIndexRuntimeFactory: # let runtime construction opt in via semantic_search_enabled (#1016). index_embeddings: bool = False move_batch_size: int = 100 + read_cache: ReadCache | None = None async def runtime_for_project(self, project: Project) -> StorageEventIndexRuntime: dependencies = await self.dependency_provider.dependencies_for_project(project) @@ -294,15 +356,41 @@ async def runtime_for_project(self, project: Project) -> StorageEventIndexRuntim file_service=dependencies.file_service, ), ) + maintenance_batch_store = ( + InvalidatingProjectIndexBatchStore( + move_store=maintenance_store, + delete_store=maintenance_store, + read_cache=self.read_cache, + project_external_id=project_ref.project_external_id, + ) + if self.read_cache is not None + else maintenance_store + ) maintenance_runner = StoreProjectIndexMaintenanceRunner( - move_store=maintenance_store, - delete_store=maintenance_store, + move_store=maintenance_batch_store, + delete_store=maintenance_batch_store, ) moved_entity_search_refresher = RepositoryProjectIndexMovedEntitySearchRefresher( session_maker=dependencies.session_maker, entity_repository=dependencies.entity_repository, entity_indexer=dependencies.search_service, ) + file_indexer: IndexFileExecutor = dependencies.file_indexer + delete_entities: ExternalFileDeleteEntities = RepositoryExternalFileDeleteEntities( + session_maker=dependencies.session_maker, + entity_repository=dependencies.entity_repository, + ) + if self.read_cache is not None: + file_indexer = InvalidatingIndexFileExecutor( + executor=file_indexer, + read_cache=self.read_cache, + project_external_id=project_ref.project_external_id, + ) + delete_entities = InvalidatingExternalFileDeleteEntities( + entities=delete_entities, + read_cache=self.read_cache, + project_external_id=project_ref.project_external_id, + ) inline_runtime = InlineStorageEventIndexRuntime( project=project_ref, checker=checker, @@ -311,11 +399,8 @@ async def runtime_for_project(self, project: Project) -> StorageEventIndexRuntim session_maker=dependencies.session_maker, entity_repository=dependencies.entity_repository, ), - file_indexer=dependencies.file_indexer, - delete_entities=RepositoryExternalFileDeleteEntities( - session_maker=dependencies.session_maker, - entity_repository=dependencies.entity_repository, - ), + file_indexer=file_indexer, + delete_entities=delete_entities, delete_objects=LocalExternalFileDeleteObjects(dependencies.file_service), result_recorder=LocalInlineStorageEventResultRecorder( project=project_ref, @@ -332,6 +417,7 @@ async def runtime_for_project(self, project: Project) -> StorageEventIndexRuntim entity_indexer=dependencies.search_service, ), index_embeddings=self.index_embeddings, + read_cache=self.read_cache, ), index_embeddings=self.index_embeddings, ) @@ -349,6 +435,8 @@ async def runtime_for_project(self, project: Project) -> StorageEventIndexRuntim entity_repository=dependencies.entity_repository, maintenance_runner=maintenance_runner, moved_entity_search_refresher=moved_entity_search_refresher, + project_external_id=project_ref.project_external_id, + read_cache=self.read_cache, batch_size=self.move_batch_size, ), ) diff --git a/src/basic_memory/index/local_schedulers.py b/src/basic_memory/index/local_schedulers.py index 99dd6ca4c..f71dda676 100644 --- a/src/basic_memory/index/local_schedulers.py +++ b/src/basic_memory/index/local_schedulers.py @@ -9,6 +9,7 @@ """ import asyncio +from contextlib import nullcontext from dataclasses import dataclass from typing import Any, Coroutine @@ -20,6 +21,7 @@ RelationResolutionRuntime, resolve_project_relations, ) +from basic_memory.read_cache import ReadCacheInvalidator, invalidate_cache from basic_memory.runtime.vector_sync import EntityVectorSync # --- Background Task Machinery --- @@ -154,15 +156,29 @@ async def _run_project_index(self, project_id: int, *, force_full: bool) -> None @dataclass(frozen=True, slots=True) class LocalSearchReindexScheduler: search_service: SearchReindexService + project_external_id: str + read_cache: ReadCacheInvalidator | None test_mode: bool def schedule_search_reindex(self, *, project_id: int) -> None: _ = project_id _schedule_background_coroutine( - self.search_service.reindex_all(), + self._run_search_reindex(), test_mode=self.test_mode, ) + async def _run_search_reindex(self) -> None: + # A rebuild publishes search rows incrementally after dropping the old + # index. Invalidate after success or partial failure so fuzzy resolutions + # cached during that window cannot survive the rebuild. + invalidation_scope = ( + invalidate_cache(self.read_cache, self.project_external_id) + if self.read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + await self.search_service.reindex_all() + # Process-lifetime coalescing state: project ids with a relation-resolution # pass already pending or in flight. A burst of writes collapses to a single @@ -192,6 +208,8 @@ class LocalRelationResolutionScheduler: """ relation_runtime: RelationResolutionRuntime + project_external_id: str + read_cache: ReadCacheInvalidator | None test_mode: bool debounce_seconds: float = 0.5 @@ -220,12 +238,22 @@ async def _resolve_after_debounce(self, project_id: int) -> None: # Writes up to here are covered by the scan we are about to run, so only # writes that land DURING the scan should force a re-run. _dirty_relation_resolution.discard(project_id) - await resolve_project_relations(self.relation_runtime) + # Relation resolution commits entity changes after the index pass. + # A second bump closes the window in which an intermediate entity + # response could have populated the current generation. + invalidation_scope = ( + invalidate_cache(self.read_cache, self.project_external_id) + if self.read_cache is not None + else nullcontext() + ) + async with invalidation_scope: + await resolve_project_relations(self.relation_runtime) finally: rerun = project_id in _dirty_relation_resolution _dirty_relation_resolution.discard(project_id) _pending_relation_resolution.discard(project_id) - # Re-arm outside the in-flight window (pending now cleared) so a write that - # raced the scan gets its own pass. Bounded to one extra pass per burst. - if rerun: - self.schedule_relation_resolution(project_id=project_id) + # Re-arm after clearing the in-flight marker so a write that raced the + # scan gets its own pass. Keep this in the cleanup path so a cache + # failure cannot drop the required relation rerun. + if rerun: + self.schedule_relation_resolution(project_id=project_id) diff --git a/src/basic_memory/index/note_content_materialization.py b/src/basic_memory/index/note_content_materialization.py index 172558963..8775781c2 100644 --- a/src/basic_memory/index/note_content_materialization.py +++ b/src/basic_memory/index/note_content_materialization.py @@ -4,7 +4,7 @@ import asyncio from collections.abc import Coroutine, Mapping -from contextlib import suppress +from contextlib import nullcontext, suppress from dataclasses import dataclass, replace from typing import Any @@ -51,6 +51,7 @@ NoteFileVacateRepository, RecoverableVacate, ) +from basic_memory.read_cache import ReadCacheInvalidator, invalidate_cache from basic_memory.schemas.response import ObservationResponse, RelationResponse from basic_memory.services.file_service import FileService @@ -183,6 +184,14 @@ async def drain_pending_materializations() -> None: RECOVERY_NOTE_ACTOR_NAME = "startup-recovery" +@dataclass(frozen=True, slots=True) +class MaterializationRecoverySummary: + """State-publication summary for one startup recovery sweep.""" + + attempted: int = 0 + written: int = 0 + + async def run_recovery_materialization( request: RuntimeNoteMaterializationJobRequest, *, @@ -214,13 +223,13 @@ async def recover_stuck_materializations( session_maker: async_sessionmaker[AsyncSession], file_service: FileService, project_id: int, -) -> int: +) -> MaterializationRecoverySummary: """Re-drive every note materialization stuck in writing/pending/failed for a project. Meant to run once per project at startup, before serving. Non-fatal per row: a single row that raises is logged and skipped so one poisoned note cannot - block startup recovery for the rest of the project. Returns the number of rows - that reached a written file state. + block startup recovery for the rest of the project. Every attempt can publish + terminal status, so the summary distinguishes attempted rows from writes. """ async with db.scoped_session(session_maker) as session: stuck_rows = await NoteContentRepository(project_id=project_id).find_stuck_materializations( @@ -228,14 +237,14 @@ async def recover_stuck_materializations( ) if not stuck_rows: - return 0 + return MaterializationRecoverySummary() logger.info( "Recovering stuck note materializations", project_id=project_id, stuck_count=len(stuck_rows), ) - recovered = 0 + written = 0 for row in stuck_rows: # Rebuild the queue request from the row's own accepted db_version/db_checksum # so the preflight guard matches the current accepted state; if a newer write @@ -266,8 +275,11 @@ async def recover_stuck_materializations( ) continue if result.status is RuntimeNoteMaterializationStatus.written: - recovered += 1 - return recovered + written += 1 + return MaterializationRecoverySummary( + attempted=len(stuck_rows), + written=written, + ) async def _recover_deleted_destination_vacate( @@ -571,6 +583,8 @@ class LocalNoteContentMaterializationProvider: session_maker: async_sessionmaker[AsyncSession] file_service: FileService + project_external_id: str + read_cache: ReadCacheInvalidator | None file_indexer: IndexFileExecutor | None = None test_mode: bool = False materialization_workers: int = 4 @@ -626,60 +640,77 @@ async def _materialize_write_now( ) -> RuntimeAcceptedNoteChange[RuntimeNoteContentResponsePayload]: if accepted.materialization is None: # pragma: no cover - guarded by caller return accepted - storage = LocalNoteContentStorage(self.file_service) - cleanup_enqueuer = InlineNoteFileDeleteEnqueuer( - storage, - vacate_clearer=RepositoryMoveVacateClearer(session_maker=self.session_maker), + # The accepted-write invalidation runs before deferred materialization. + # The outer boundary retires reads filled during later indexing. + final_invalidation_scope = ( + invalidate_cache(self.read_cache, self.project_external_id) + if self.read_cache is not None + else nullcontext() ) - result = await run_note_materialization( - plan_note_materialization_job_request(accepted.materialization), - preflight=RepositoryNoteMaterializationPreflight( - session_maker=self.session_maker, - ), - writer=ContentStoreNoteMaterializationFileWriter(storage), - publisher=RepositoryNoteMaterializationPublisher( - session_maker=self.session_maker, - ), - status_publisher=RepositoryNoteMaterializationStatusPublisher( - session_maker=self.session_maker, - ), - cleanup_enqueuer=cleanup_enqueuer, - ) - if result.status is not RuntimeNoteMaterializationStatus.written: - return replace( - accepted, - payload=note_content_payload_with_materialization_result( - accepted.payload, - result, - ), + async with final_invalidation_scope: + storage = LocalNoteContentStorage(self.file_service) + cleanup_enqueuer = InlineNoteFileDeleteEnqueuer( + storage, + vacate_clearer=RepositoryMoveVacateClearer(session_maker=self.session_maker), ) - - file_path = note_content_payload_file_path(accepted.payload) - if file_path is not None and self.file_indexer is not None: - await self.file_indexer.index_file( - file_path, - source="note-content-materialization", + # Materialization commits terminal status before indexing begins. Retire + # pending/writing reads at that commit instead of holding them for the + # potentially slow index operation. + publication_invalidation_scope = ( + invalidate_cache(self.read_cache, self.project_external_id) + if self.read_cache is not None + else nullcontext() ) - # The deferred index has now inserted this note's entity/relation rows, - # so back-resolve inbound forward references. The router schedules an - # eager pass right after enqueue, but under load that pass can scan - # before this index lands; scheduling here (coalesced/re-armed by the - # resolution scheduler) guarantees a pass runs after indexing (#1002). - if self.relation_resolution_scheduler is not None: - self.relation_resolution_scheduler.schedule_relation_resolution( - project_id=accepted.materialization.project_id, + async with publication_invalidation_scope: + result = await run_note_materialization( + plan_note_materialization_job_request(accepted.materialization), + preflight=RepositoryNoteMaterializationPreflight( + session_maker=self.session_maker, + ), + writer=ContentStoreNoteMaterializationFileWriter(storage), + publisher=RepositoryNoteMaterializationPublisher( + session_maker=self.session_maker, + ), + status_publisher=RepositoryNoteMaterializationStatusPublisher( + session_maker=self.session_maker, + ), + cleanup_enqueuer=cleanup_enqueuer, ) - return replace( - accepted, - payload=await load_indexed_note_content_response_payload( - session_maker=self.session_maker, - project_id=accepted.materialization.project_id, - entity_id=accepted.materialization.entity_id, - fallback_source=accepted.materialization.source - or "note-content-materialization", - ), - ) - return accepted + if result.status is not RuntimeNoteMaterializationStatus.written: + return replace( + accepted, + payload=note_content_payload_with_materialization_result( + accepted.payload, + result, + ), + ) + + file_path = note_content_payload_file_path(accepted.payload) + if file_path is not None and self.file_indexer is not None: + await self.file_indexer.index_file( + file_path, + source="note-content-materialization", + ) + # The deferred index has now inserted this note's entity/relation rows, + # so back-resolve inbound forward references. The router schedules an + # eager pass right after enqueue, but under load that pass can scan + # before this index lands; scheduling here (coalesced/re-armed by the + # resolution scheduler) guarantees a pass runs after indexing (#1002). + if self.relation_resolution_scheduler is not None: + self.relation_resolution_scheduler.schedule_relation_resolution( + project_id=accepted.materialization.project_id, + ) + return replace( + accepted, + payload=await load_indexed_note_content_response_payload( + session_maker=self.session_maker, + project_id=accepted.materialization.project_id, + entity_id=accepted.materialization.entity_id, + fallback_source=accepted.materialization.source + or "note-content-materialization", + ), + ) + return accepted async def materialize_delete_change( self, diff --git a/src/basic_memory/index/watch_coordinator.py b/src/basic_memory/index/watch_coordinator.py index eba94eea3..8f3d0a07c 100644 --- a/src/basic_memory/index/watch_coordinator.py +++ b/src/basic_memory/index/watch_coordinator.py @@ -9,6 +9,7 @@ from loguru import logger from basic_memory.config import BasicMemoryConfig +from basic_memory.read_cache import ReadCache class WatchStatus(Enum): @@ -30,6 +31,7 @@ class WatchCoordinator: should_watch: bool = True skip_reason: str | None = None quiet: bool = True + read_cache: ReadCache | None = None _status: WatchStatus = field(default=WatchStatus.NOT_STARTED, init=False) _watch_task: asyncio.Task[None] | None = field(default=None, init=False) @@ -72,6 +74,7 @@ async def _watch_runner() -> None: # pragma: no cover self.config, quiet=self.quiet, recovery_complete=recovery_complete, + read_cache=self.read_cache, ) except asyncio.CancelledError: logger.debug("Local event-index watcher cancelled") diff --git a/src/basic_memory/indexing/external_file_delete_runner.py b/src/basic_memory/indexing/external_file_delete_runner.py index e88d51f54..4e37a97bd 100644 --- a/src/basic_memory/indexing/external_file_delete_runner.py +++ b/src/basic_memory/indexing/external_file_delete_runner.py @@ -10,6 +10,7 @@ from basic_memory import db from basic_memory.models import Relation +from basic_memory.read_cache import ReadCacheInvalidator, invalidate_cache from basic_memory.runtime.cleanup import RuntimeExternalFileDeletePlan from basic_memory.runtime.note_content import ( RuntimeDeletedNoteEntityDeleteSource, @@ -126,6 +127,37 @@ async def delete_entity_if_file_path_matches( ) +@dataclass(frozen=True, slots=True) +class InvalidatingExternalFileDeleteEntities(ExternalFileDeleteEntities): + """Advance cached reads after a repository-backed delete attempt exits.""" + + entities: ExternalFileDeleteEntities + read_cache: ReadCacheInvalidator + project_external_id: str + + @override + async def find_entity_by_file_path( + self, + file_path: RuntimeFilePath, + ) -> RuntimeDeletedNoteEntityDeleteSource | None: + return await self.entities.find_entity_by_file_path(file_path) + + @override + async def delete_entity_if_file_path_matches( + self, + *, + entity_id: RuntimeEntityId, + file_path: RuntimeFilePath, + ) -> ExternalFileDeleteEntityDeleteResult: + # The repository transaction can commit while its context manager exits. + # Keeping that exit inside the scope makes cancellation wait for invalidation. + async with invalidate_cache(self.read_cache, self.project_external_id): + return await self.entities.delete_entity_if_file_path_matches( + entity_id=entity_id, + file_path=file_path, + ) + + class ExternalFileDeleteObjects(Protocol): """Storage capability required to detect stale delete notifications.""" diff --git a/src/basic_memory/indexing/index_file_runner.py b/src/basic_memory/indexing/index_file_runner.py index aab43b1db..59f33f222 100644 --- a/src/basic_memory/indexing/index_file_runner.py +++ b/src/basic_memory/indexing/index_file_runner.py @@ -4,7 +4,7 @@ from collections.abc import Sequence from dataclasses import dataclass, field -from typing import Protocol +from typing import Protocol, override from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker @@ -25,6 +25,7 @@ plan_current_materialized_note_result, plan_indexed_file_live_update_metadata, ) +from basic_memory.read_cache import ReadCacheInvalidator, invalidate_cache from basic_memory.runtime.jobs import RuntimeStorageFileIndexMode from basic_memory.runtime.note_object_metadata import RuntimeNoteObjectMetadataMap from basic_memory.runtime.storage import RuntimeFileChecksum, RuntimeFilePath @@ -74,6 +75,27 @@ async def index_file( ) -> FileIndexResult: ... +@dataclass(frozen=True, slots=True) +class InvalidatingIndexFileExecutor(IndexFileExecutor): + """Advance cached reads after a transaction-bearing file index attempt exits.""" + + executor: IndexFileExecutor + read_cache: ReadCacheInvalidator + project_external_id: str + + @override + async def index_file( + self, + file_path: RuntimeFilePath, + *, + source: str, + ) -> FileIndexResult: + # File indexing can commit entity state while its session context exits. + # Keep that exit inside the scope so cancellation waits for invalidation. + async with invalidate_cache(self.read_cache, self.project_external_id): + return await self.executor.index_file(file_path, source=source) + + class CurrentMaterializedNoteEntityRepository(Protocol): """Repository capability needed to load the current materialized note entity.""" diff --git a/src/basic_memory/indexing/project_index_maintenance.py b/src/basic_memory/indexing/project_index_maintenance.py index 57f84e451..e5c811cdb 100644 --- a/src/basic_memory/indexing/project_index_maintenance.py +++ b/src/basic_memory/indexing/project_index_maintenance.py @@ -16,7 +16,8 @@ ProjectIndexExternalVectorCleaner, delete_project_index_vector_rows, ) -from basic_memory.runtime.storage import ProjectId +from basic_memory.read_cache import ReadCacheInvalidator, invalidate_cache +from basic_memory.runtime.storage import ProjectExternalId, ProjectId class ProjectIndexMaintenanceRunner(Protocol): @@ -37,6 +38,24 @@ async def run_delete_batches( ) -> ProjectIndexDeleteRun: ... +class ProjectIndexMoveBatchStore(Protocol): + """Capability that commits one project-index move batch.""" + + async def apply_project_index_move_batch( + self, + move_batch: ProjectIndexMoveBatch, + ) -> ProjectIndexMoveBatchResult: ... + + +class ProjectIndexDeleteBatchStore(Protocol): + """Capability that commits one project-index delete batch.""" + + async def apply_project_index_delete_batch( + self, + delete_batch: ProjectIndexDeleteBatch, + ) -> ProjectIndexDeleteBatchResult: ... + + class ProjectIndexMovedEntityRepository(Protocol): """Repository capability for loading moved entities after path maintenance.""" @@ -882,12 +901,41 @@ async def apply_project_index_delete_batch( ) +@dataclass(frozen=True, slots=True) +class InvalidatingProjectIndexBatchStore( + ProjectIndexMoveBatchStore, + ProjectIndexDeleteBatchStore, +): + """Invalidate semantic reads after each durable move or delete batch.""" + + move_store: ProjectIndexMoveBatchStore + delete_store: ProjectIndexDeleteBatchStore + read_cache: ReadCacheInvalidator + project_external_id: ProjectExternalId + + @override + async def apply_project_index_move_batch( + self, + move_batch: ProjectIndexMoveBatch, + ) -> ProjectIndexMoveBatchResult: + async with invalidate_cache(self.read_cache, self.project_external_id): + return await self.move_store.apply_project_index_move_batch(move_batch) + + @override + async def apply_project_index_delete_batch( + self, + delete_batch: ProjectIndexDeleteBatch, + ) -> ProjectIndexDeleteBatchResult: + async with invalidate_cache(self.read_cache, self.project_external_id): + return await self.delete_store.apply_project_index_delete_batch(delete_batch) + + @dataclass(frozen=True, slots=True) class StoreProjectIndexMaintenanceRunner(ProjectIndexMaintenanceRunner): """Run project-index maintenance through explicit move/delete batch stores.""" - move_store: RepositoryProjectIndexMaintenanceStore - delete_store: RepositoryProjectIndexMaintenanceStore + move_store: ProjectIndexMoveBatchStore + delete_store: ProjectIndexDeleteBatchStore @override async def run_move_batches( @@ -1011,7 +1059,7 @@ async def run_project_index_move_batches( *, moved_files: Mapping[str, str], batch_size: int, - move_store: RepositoryProjectIndexMaintenanceStore, + move_store: ProjectIndexMoveBatchStore, ) -> ProjectIndexMoveRun: """Apply project-index move maintenance through a storage adapter.""" move_plan = build_project_index_move_batch_plan( @@ -1064,7 +1112,7 @@ async def run_project_index_delete_batches( *, deleted_paths: Sequence[str], batch_size: int, - delete_store: RepositoryProjectIndexMaintenanceStore, + delete_store: ProjectIndexDeleteBatchStore, ) -> ProjectIndexDeleteRun: """Apply project-index delete maintenance through a storage adapter.""" delete_plan = build_project_index_delete_batch_plan( diff --git a/src/basic_memory/read_cache/__init__.py b/src/basic_memory/read_cache/__init__.py new file mode 100644 index 000000000..ff47c8521 --- /dev/null +++ b/src/basic_memory/read_cache/__init__.py @@ -0,0 +1,38 @@ +"""Optional semantic read caching for Basic Memory.""" + +from basic_memory.read_cache.contract import ( + ReadCache, + ReadCacheDataError, + ReadCacheInvalidator, + ReadCacheInvalidationStatus, + ReadCacheKey, + ReadCacheLookup, + ReadCacheOperation, + ReadCacheStoreStatus, + ReadCacheUnavailable, +) +from basic_memory.read_cache.invalidation import ( + finish_project_read_cache_invalidation, + invalidate_cache, + invalidate_project_read_cache, +) +from basic_memory.read_cache.keys import read_cache_request_digest +from basic_memory.read_cache.read_through import ModelReadCache, ReadCacheScope + +__all__ = [ + "ModelReadCache", + "ReadCache", + "ReadCacheDataError", + "ReadCacheInvalidator", + "ReadCacheInvalidationStatus", + "ReadCacheKey", + "ReadCacheLookup", + "ReadCacheOperation", + "ReadCacheScope", + "ReadCacheStoreStatus", + "ReadCacheUnavailable", + "finish_project_read_cache_invalidation", + "invalidate_cache", + "invalidate_project_read_cache", + "read_cache_request_digest", +] diff --git a/src/basic_memory/read_cache/contract.py b/src/basic_memory/read_cache/contract.py new file mode 100644 index 000000000..1dc99dad4 --- /dev/null +++ b/src/basic_memory/read_cache/contract.py @@ -0,0 +1,114 @@ +"""Portable contract for best-effort semantic read caching.""" + +from dataclasses import dataclass +from enum import StrEnum +from typing import Protocol +from uuid import UUID + + +class ReadCacheOperation(StrEnum): + """Read operations supported by the initial cache rollout.""" + + directory_list = "directory_list" + directory_structure = "directory_structure" + directory_tree = "directory_tree" + entity = "entity" + resolve = "resolve" + resource = "resource" + + +class ReadCacheStoreStatus(StrEnum): + """Outcome of one best-effort cache store.""" + + stored = "stored" + superseded = "superseded" + + +class ReadCacheInvalidationStatus(StrEnum): + """Outcome of one project-generation invalidation attempt.""" + + invalidated = "invalidated" + unavailable = "unavailable" + + +def canonical_read_cache_project_id(project_id: str) -> str: + """Return one canonical UUID spelling for project cache scope.""" + if not project_id: + raise ValueError("read-cache project_id must not be empty") + try: + return str(UUID(project_id)) + except ValueError as error: + raise ValueError("read-cache project_id must be a valid UUID") from error + + +@dataclass(frozen=True, slots=True) +class ReadCacheKey: + """Project-scoped identity for one canonical read request.""" + + project_id: str + operation: ReadCacheOperation + request_digest: str + + def __post_init__(self) -> None: + object.__setattr__( + self, + "project_id", + canonical_read_cache_project_id(self.project_id), + ) + if len(self.request_digest) != 64: + raise ValueError("read-cache request_digest must be a SHA-256 hex digest") + try: + decoded_digest = bytes.fromhex(self.request_digest) + except ValueError as error: + raise ValueError("read-cache request_digest must be a SHA-256 hex digest") from error + if len(decoded_digest) != 32: + raise ValueError("read-cache request_digest must be a SHA-256 hex digest") + object.__setattr__(self, "request_digest", self.request_digest.lower()) + + +@dataclass(frozen=True, slots=True) +class ReadCacheLookup: + """Cache lookup result plus the generation observed by that read.""" + + generation: str + payload: bytes | None = None + + def __post_init__(self) -> None: + if not self.generation: + raise ValueError("read-cache lookup generation must not be empty") + + @property + def is_hit(self) -> bool: + return self.payload is not None + + +class ReadCacheUnavailable(RuntimeError): + """The cache backend could not complete an optional operation.""" + + +class ReadCacheDataError(RuntimeError): + """A cached value violated the Basic Memory cache encoding contract.""" + + +class ReadCacheInvalidator(Protocol): + """Capability for making one project's cached reads unreachable.""" + + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + """Make every existing cached value for one project unreachable.""" + + +class ReadCache(ReadCacheInvalidator, Protocol): + """Best-effort read-through backend with project-generation invalidation.""" + + async def lookup(self, key: ReadCacheKey) -> ReadCacheLookup: + """Return a cached payload and the generation observed by this lookup.""" + + async def store( + self, + key: ReadCacheKey, + lookup: ReadCacheLookup, + payload: bytes, + *, + ttl_seconds: int, + ) -> ReadCacheStoreStatus: + """Store a payload under the generation observed by ``lookup``.""" diff --git a/src/basic_memory/read_cache/invalidation.py b/src/basic_memory/read_cache/invalidation.py new file mode 100644 index 000000000..0bd445fe2 --- /dev/null +++ b/src/basic_memory/read_cache/invalidation.py @@ -0,0 +1,94 @@ +"""Best-effort project invalidation shared by mutation and indexing runtimes.""" + +import asyncio +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager + +from loguru import logger + +import logfire +from basic_memory.read_cache.contract import ( + ReadCacheInvalidator, + ReadCacheInvalidationStatus, + ReadCacheUnavailable, +) + + +def _record_invalidation_event(event: ReadCacheInvalidationStatus) -> None: + logfire.metric_counter("basic_memory_read_cache_events_total").add( + 1, + attributes={ + "operation": "project", + "event": event.value, + }, + ) + + +async def invalidate_project_read_cache( + cache: ReadCacheInvalidator, + project_id: str, +) -> ReadCacheInvalidationStatus: + """Invalidate one project without failing an already-committed mutation.""" + with logfire.span("read_cache.invalidate_project") as span: + try: + status = await cache.invalidate_project(project_id) + except ReadCacheUnavailable as error: + # Trigger: an authoritative mutation committed while Redis was unavailable. + # Why: failing the request cannot roll the mutation back and would invite + # duplicate retries; the 60-second TTL already bounds stale exposure. + # Outcome: surface prominent telemetry and let the committed write succeed. + status = ReadCacheInvalidationStatus.unavailable + logger.error( + "Read cache project invalidation unavailable; cached values may remain " + "reachable until TTL expiry", + error=str(error), + ) + + _record_invalidation_event(status) + span.set_attribute("cache.outcome", status.value) + return status + + +async def finish_project_read_cache_invalidation( + cache: ReadCacheInvalidator, + project_id: str, +) -> ReadCacheInvalidationStatus: + """Finish invalidation before propagating caller cancellation.""" + invalidation = asyncio.create_task(invalidate_project_read_cache(cache, project_id)) + try: + return await asyncio.shield(invalidation) + except asyncio.CancelledError as cancellation: + # Trigger: cancellation landed after authoritative state may have committed. + # Why: abandoning this Redis write can leave that state hidden behind the + # previous generation until its TTL expires. + # Outcome: finish invalidation, then preserve the caller's cancellation. + cleanup_error: BaseException | None = None + while not invalidation.done(): + try: + await asyncio.shield(invalidation) + except asyncio.CancelledError: + continue + except BaseException as error: + cleanup_error = error + if cleanup_error is None: + if invalidation.cancelled(): + cleanup_error = asyncio.CancelledError("read-cache invalidation task was cancelled") + else: + cleanup_error = invalidation.exception() + if cleanup_error is not None: + cancellation.add_note( + f"Read-cache invalidation failed during cancellation: {cleanup_error!r}" + ) + raise + + +@asynccontextmanager +async def invalidate_cache( + cache: ReadCacheInvalidator, + project_id: str, +) -> AsyncIterator[None]: + """Invalidate one project's read cache when the enclosed operation exits.""" + try: + yield + finally: + await finish_project_read_cache_invalidation(cache, project_id) diff --git a/src/basic_memory/read_cache/keys.py b/src/basic_memory/read_cache/keys.py new file mode 100644 index 000000000..cd4c1f21f --- /dev/null +++ b/src/basic_memory/read_cache/keys.py @@ -0,0 +1,90 @@ +"""Canonical request digests and Redis key construction.""" + +from dataclasses import dataclass +from hashlib import sha256 + +from basic_memory.read_cache.contract import ( + ReadCacheKey, + canonical_read_cache_project_id, +) + +DEFAULT_READ_CACHE_PREFIX = "bm:read:v1" + + +def read_cache_request_digest(*parts: str) -> str: + """Hash length-delimited request parts into one stable cache identity.""" + digest = sha256() + for part in parts: + encoded = part.encode("utf-8") + digest.update(len(encoded).to_bytes(8, byteorder="big")) + digest.update(encoded) + return digest.hexdigest() + + +@dataclass(frozen=True, slots=True) +class RedisReadCacheKeys: + """Redis keys for one project generation and canonical request.""" + + generation_key: str + data_key: str + + +def _redis_read_cache_cluster_scope(*, namespace: str, project_id: str) -> str: + namespace = namespace.strip() + if not namespace: + raise ValueError("read-cache namespace must not be empty") + + scope_digest = read_cache_request_digest( + namespace, + canonical_read_cache_project_id(project_id), + ) + return f"{{{scope_digest}}}" + + +def _redis_read_cache_key_base( + *, + prefix: str, + namespace: str, + project_id: str, +) -> str: + if not prefix or any(character.isspace() for character in prefix): + raise ValueError("read-cache prefix must be non-empty and contain no whitespace") + + cluster_scope = _redis_read_cache_cluster_scope( + namespace=namespace, + project_id=project_id, + ) + return f"{prefix}:{cluster_scope}" + + +def redis_read_cache_generation_key( + *, + prefix: str, + namespace: str, + project_id: str, +) -> str: + """Build the generation key shared by every cached read in one project.""" + key_base = _redis_read_cache_key_base( + prefix=prefix, + namespace=namespace, + project_id=project_id, + ) + return f"{key_base}:generation" + + +def redis_read_cache_keys( + *, + prefix: str, + namespace: str, + key: ReadCacheKey, +) -> RedisReadCacheKeys: + """Build versioned Redis keys without exposing namespace values.""" + key_base = _redis_read_cache_key_base( + prefix=prefix, + namespace=namespace, + project_id=key.project_id, + ) + return RedisReadCacheKeys( + generation_key=f"{key_base}:generation", + data_key=f"{key_base}:{key.operation.value}:{key.request_digest}", + ) diff --git a/src/basic_memory/read_cache/policy.py b/src/basic_memory/read_cache/policy.py new file mode 100644 index 000000000..b269120cf --- /dev/null +++ b/src/basic_memory/read_cache/policy.py @@ -0,0 +1,5 @@ +"""Initial semantic read-cache policy.""" + +READ_CACHE_TTL_SECONDS = 60 +READ_CACHE_MAX_PAYLOAD_BYTES = 1024 * 1024 +DIRECTORY_READ_CACHE_MAX_PAYLOAD_BYTES = 2 * 1024 * 1024 diff --git a/src/basic_memory/read_cache/read_through.py b/src/basic_memory/read_cache/read_through.py new file mode 100644 index 000000000..719ae24d5 --- /dev/null +++ b/src/basic_memory/read_cache/read_through.py @@ -0,0 +1,137 @@ +"""Typed read-through behavior shared by cacheable API boundaries.""" + +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from dataclasses import dataclass + +import logfire +from pydantic import BaseModel + +from basic_memory.read_cache.contract import ( + ReadCache, + ReadCacheInvalidationStatus, + ReadCacheKey, + ReadCacheUnavailable, +) + + +def _record_event(key: ReadCacheKey, event: str) -> None: + logfire.metric_counter("basic_memory_read_cache_events_total").add( + 1, + attributes={ + "operation": key.operation.value, + "event": event, + }, + ) + + +@dataclass(slots=True) +class ReadCacheScope[ModelT: BaseModel]: + """Mutable state exchanged with one configured read-cache scope.""" + + value: ModelT | None = None + cacheable: bool = True + + def require_value(self) -> ModelT: + """Return the authoritative value supplied by the route.""" + if self.value is None: + raise RuntimeError("read-through cache scope exited without a result") + return self.value + + +@dataclass(frozen=True, slots=True) +class ModelReadCache[ModelT: BaseModel]: + """Typed facade that binds one cache backend to a response model and policy.""" + + backend: ReadCache + model_type: type[ModelT] + ttl_seconds: int + max_payload_bytes: int + + def __post_init__(self) -> None: + if self.ttl_seconds <= 0: + raise ValueError("read-cache ttl_seconds must be positive") + if self.max_payload_bytes <= 0: + raise ValueError("read-cache max_payload_bytes must be positive") + + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + """Delegate invalidation without exposing the backend to API routes.""" + return await self.backend.invalidate_project(project_id) + + @asynccontextmanager + async def read( + self, + *, + key: ReadCacheKey, + ) -> AsyncIterator[ReadCacheScope[ModelT]]: + """Yield a cached model or store the authoritative value supplied by the route.""" + with logfire.span( + "read_cache.read_through", + operation=key.operation.value, + ) as span: + try: + lookup = await self.backend.lookup(key) + except ReadCacheUnavailable: + # Trigger: Redis is unreachable or timed out. + # Why: the database or storage path remains authoritative. + # Outcome: return fresh data without attempting another cache operation. + _record_event(key, "bypass") + span.set_attribute("cache.outcome", "bypass") + result = ReadCacheScope[ModelT]() + yield result + result.require_value() + return + + if lookup.payload is not None: + _record_event(key, "hit") + span.set_attributes( + { + "cache.outcome": "hit", + "cache.payload_bytes": len(lookup.payload), + } + ) + yield ReadCacheScope( + value=self.model_type.model_validate_json(lookup.payload), + cacheable=False, + ) + return + + _record_event(key, "miss") + result = ReadCacheScope[ModelT]() + yield result + value = result.require_value() + if not result.cacheable: + _record_event(key, "ineligible") + span.set_attribute("cache.outcome", "ineligible") + return + + payload = value.model_dump_json().encode("utf-8") + if len(payload) > self.max_payload_bytes: + _record_event(key, "oversize") + span.set_attributes( + { + "cache.outcome": "oversize", + "cache.payload_bytes": len(payload), + } + ) + return + + try: + store_status = await self.backend.store( + key, + lookup, + payload, + ttl_seconds=self.ttl_seconds, + ) + except ReadCacheUnavailable: + _record_event(key, "store_unavailable") + span.set_attribute("cache.outcome", "store_unavailable") + return + + _record_event(key, store_status.value) + span.set_attributes( + { + "cache.outcome": store_status.value, + "cache.payload_bytes": len(payload), + } + ) diff --git a/src/basic_memory/read_cache/redis.py b/src/basic_memory/read_cache/redis.py new file mode 100644 index 000000000..b40dd6ca4 --- /dev/null +++ b/src/basic_memory/read_cache/redis.py @@ -0,0 +1,210 @@ +"""Redis implementation of Basic Memory semantic read caching. + +This module is imported only when the optional ``redis`` dependency is installed. +The caller owns the Redis client lifecycle. +""" + +from uuid import uuid4 + +from redis.asyncio import Redis +from redis.exceptions import ClusterError as RedisClusterError +from redis.exceptions import ConnectionError as RedisConnectionError +from redis.exceptions import InvalidResponse as RedisInvalidResponse +from redis.exceptions import ResponseError as RedisResponseError +from redis.exceptions import TimeoutError as RedisTimeoutError + +from basic_memory.read_cache.contract import ( + ReadCacheDataError, + ReadCacheInvalidationStatus, + ReadCacheKey, + ReadCacheLookup, + ReadCacheStoreStatus, + ReadCacheUnavailable, +) +from basic_memory.read_cache.keys import ( + DEFAULT_READ_CACHE_PREFIX, + RedisReadCacheKeys, + redis_read_cache_generation_key, + redis_read_cache_keys, +) + +_ENVELOPE_SEPARATOR = b"\n" +_DEFAULT_GENERATION_TTL_SECONDS = 60 +_REDIS_OPERATIONAL_ERRORS = ( + RedisClusterError, + RedisConnectionError, + RedisInvalidResponse, + RedisResponseError, + RedisTimeoutError, +) +_LOOKUP_SCRIPT = """ +local generation = redis.call("GET", KEYS[1]) +if generation then + if redis.call("TTL", KEYS[1]) == -1 then + redis.call("EXPIRE", KEYS[1], ARGV[2]) + end +else + redis.call("SET", KEYS[1], ARGV[1], "EX", ARGV[2]) +end +return redis.call("MGET", KEYS[1], KEYS[2]) +""" +_STORE_IF_CURRENT_SCRIPT = """ +if redis.call("GET", KEYS[1]) ~= ARGV[1] then + return 0 +end +redis.call("SET", KEYS[2], ARGV[2], "EX", ARGV[3]) +local response_ttl_ms = tonumber(ARGV[3]) * 1000 +if redis.call("PTTL", KEYS[1]) < response_ttl_ms then + redis.call("PEXPIRE", KEYS[1], response_ttl_ms) +end +return 1 +""" + + +def create_redis_read_cache_client( + url: str, + *, + max_connections: int = 20, + socket_timeout: float = 0.1, +) -> Redis: + """Create an async Redis client whose lifecycle remains caller-owned.""" + return Redis.from_url( + url, + decode_responses=False, + max_connections=max_connections, + socket_connect_timeout=socket_timeout, + socket_timeout=socket_timeout, + ) + + +def _required_bytes(value: object, *, field: str) -> bytes: + if isinstance(value, bytes): + return value + if isinstance(value, str): + return value.encode("utf-8") + raise ReadCacheDataError(f"Redis returned an invalid {field} value") + + +def _decode_generation(value: object) -> tuple[bytes, str]: + encoded = _required_bytes(value, field="generation") + try: + generation = encoded.decode("ascii") + decoded = bytes.fromhex(generation) + except (UnicodeDecodeError, ValueError) as error: + raise ReadCacheDataError("Redis returned an invalid generation token") from error + if len(decoded) != 16: + raise ReadCacheDataError("Redis returned an invalid generation token") + return encoded, generation + + +def _store_status(value: object) -> ReadCacheStoreStatus: + if value == 1: + return ReadCacheStoreStatus.stored + if value == 0: + return ReadCacheStoreStatus.superseded + raise ReadCacheDataError("Redis returned an invalid cache store result") + + +class RedisReadCache: + """Namespace-bound Redis cache with race-safe generation invalidation. + + A managed host may reuse one client and Redis deployment across tenants. + It must construct this lightweight adapter from trusted tenant/workspace + context so the mandatory namespace cannot be supplied by an API caller. + """ + + def __init__( + self, + *, + client: Redis, + namespace: str, + prefix: str = DEFAULT_READ_CACHE_PREFIX, + generation_ttl_seconds: int = _DEFAULT_GENERATION_TTL_SECONDS, + ) -> None: + namespace = namespace.strip() + if not namespace: + raise ValueError("read-cache namespace must not be empty") + if generation_ttl_seconds <= 0: + raise ValueError("read-cache generation_ttl_seconds must be positive") + self._client = client + self._namespace = namespace + self._prefix = prefix + self._generation_ttl_seconds = generation_ttl_seconds + + def _keys(self, key: ReadCacheKey) -> RedisReadCacheKeys: + return redis_read_cache_keys( + prefix=self._prefix, + namespace=self._namespace, + key=key, + ) + + async def lookup(self, key: ReadCacheKey) -> ReadCacheLookup: + keys = self._keys(key) + try: + generation_value, cached_value = await self._client.eval( + _LOOKUP_SCRIPT, + 2, + keys.generation_key, + keys.data_key, + uuid4().hex.encode("ascii"), + self._generation_ttl_seconds, + ) + except _REDIS_OPERATIONAL_ERRORS as error: + raise ReadCacheUnavailable("Redis cache lookup failed") from error + + generation, generation_text = _decode_generation(generation_value) + if cached_value is None: + return ReadCacheLookup(generation=generation_text) + + encoded = _required_bytes(cached_value, field="cached payload") + cached_generation, separator, payload = encoded.partition(_ENVELOPE_SEPARATOR) + if not separator or not cached_generation: + raise ReadCacheDataError("Redis cached payload has an invalid generation envelope") + if cached_generation != generation: + return ReadCacheLookup(generation=generation_text) + return ReadCacheLookup(generation=generation_text, payload=payload) + + async def store( + self, + key: ReadCacheKey, + lookup: ReadCacheLookup, + payload: bytes, + *, + ttl_seconds: int, + ) -> ReadCacheStoreStatus: + if ttl_seconds <= 0: + raise ValueError("read-cache ttl_seconds must be positive") + + keys = self._keys(key) + generation, _ = _decode_generation(lookup.generation) + encoded = generation + _ENVELOPE_SEPARATOR + payload + try: + stored = await self._client.eval( + _STORE_IF_CURRENT_SCRIPT, + 2, + keys.generation_key, + keys.data_key, + generation, + encoded, + ttl_seconds, + ) + except _REDIS_OPERATIONAL_ERRORS as error: + raise ReadCacheUnavailable("Redis cache store failed") from error + + return _store_status(stored) + + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + generation_key = redis_read_cache_generation_key( + prefix=self._prefix, + namespace=self._namespace, + project_id=project_id, + ) + try: + await self._client.set( + generation_key, + uuid4().hex.encode("ascii"), + ex=self._generation_ttl_seconds, + ) + except _REDIS_OPERATIONAL_ERRORS as error: + raise ReadCacheUnavailable("Redis project invalidation failed") from error + return ReadCacheInvalidationStatus.invalidated diff --git a/src/basic_memory/services/directory_deletes.py b/src/basic_memory/services/directory_deletes.py index 75a476a97..6cc676df8 100644 --- a/src/basic_memory/services/directory_deletes.py +++ b/src/basic_memory/services/directory_deletes.py @@ -19,6 +19,10 @@ finish_directory_delete_acceptance, normalize_directory_delete_path, ) +from basic_memory.read_cache import ( + ReadCacheInvalidator, + finish_project_read_cache_invalidation, +) class DirectoryDeleteServiceError(Exception): @@ -57,6 +61,7 @@ async def delete_directory( *, project_external_id: str, directory: str, + read_cache: ReadCacheInvalidator | None = None, ) -> DirectoryDeleteAcceptedResult: """Delete directory entities immediately and queue file cleanup in the background. @@ -67,6 +72,7 @@ async def delete_directory( project_external_id=project_external_id, directory=directory, ) + delete_may_have_committed = False try: # scoped_session enables `PRAGMA foreign_keys=ON` for SQLite; this bulk # delete issues a Core DELETE on entity and relies on ON DELETE CASCADE @@ -78,27 +84,46 @@ async def delete_directory( request=request, store=self.runtime.store, ) + delete_may_have_committed = bool(accepted.files) except DirectoryDeleteRejected as error: raise directory_delete_service_error_from_rejection(error.rejection) from error + finally: + if delete_may_have_committed and read_cache is not None: + # Acceptance commits entity and search deletion before storage cleanup. + # Finish the generation bump even when cancellation interrupts the + # transaction exit, then preserve that cancellation. + await finish_project_read_cache_invalidation( + read_cache, + project_external_id, + ) - result = await finish_directory_delete_acceptance( - request=request, - accepted=accepted, - enqueuer=self.runtime.file_delete_enqueuer, - ) - - # Trigger: notes outside the deleted directory linked into it. - # Why: the delete cascaded their relation rows away, but those sources own - # matching search_index relation rows that now dangle; without a reindex - # they linger until an unrelated rebuild. - # Outcome: reindex each surviving source inline when the runtime provides a - # refresher (local); queued runtimes consume the ids from the result. - if accepted.relation_cleanup_entity_ids and self.runtime.relation_cleanup_refresher: - await self.runtime.relation_cleanup_refresher.refresh_relation_sources( - sorted(accepted.relation_cleanup_entity_ids) + try: + result = await finish_directory_delete_acceptance( + request=request, + accepted=accepted, + enqueuer=self.runtime.file_delete_enqueuer, ) - return result + # Trigger: notes outside the deleted directory linked into it. + # Why: the delete cascaded their relation rows away, but those sources own + # matching search_index relation rows that now dangle; without a reindex + # they linger until an unrelated rebuild. + # Outcome: reindex each surviving source inline when the runtime provides a + # refresher (local); queued runtimes consume the ids from the result. + if accepted.relation_cleanup_entity_ids and self.runtime.relation_cleanup_refresher: + await self.runtime.relation_cleanup_refresher.refresh_relation_sources( + sorted(accepted.relation_cleanup_entity_ids) + ) + + return result + finally: + if accepted.files and read_cache is not None: + # Cleanup and relation refresh can publish additional state or fail + # after partial progress. Close the fill window in either case. + await finish_project_read_cache_invalidation( + read_cache, + project_external_id, + ) @staticmethod def normalize_directory_path(directory: str) -> str: diff --git a/src/basic_memory/services/entity_service.py b/src/basic_memory/services/entity_service.py index 610d4647a..72f74ffd9 100644 --- a/src/basic_memory/services/entity_service.py +++ b/src/basic_memory/services/entity_service.py @@ -1,6 +1,7 @@ """Service for managing entities in the database.""" from collections.abc import Callable +from contextlib import nullcontext from dataclasses import dataclass from pathlib import Path from typing import Any, List, Optional, Sequence, Tuple, Union @@ -22,6 +23,7 @@ from basic_memory.models.knowledge import Entity from basic_memory.repository import ObservationRepository, RelationRepository from basic_memory.repository.entity_repository import EntityRepository +from basic_memory.read_cache import ReadCache, invalidate_cache from basic_memory.runtime.note_move import normalize_note_move_destination_path from basic_memory.schemas import Entity as EntitySchema from basic_memory.schemas.base import Permalink @@ -1129,6 +1131,9 @@ async def move_directory( destination_directory: str, project_config: ProjectConfig, app_config: BasicMemoryConfig, + *, + project_external_id: str, + read_cache: ReadCache | None, ) -> DirectoryMoveResult: """Move all entities in a directory to a new location. @@ -1141,6 +1146,8 @@ async def move_directory( destination_directory: Destination directory path relative to project root project_config: Project configuration for file operations app_config: App configuration for permalink update settings + project_external_id: Canonical project UUID used for cache invalidation + read_cache: Namespace-bound semantic read cache Returns: DirectoryMoveResult with counts and details of moved files @@ -1177,34 +1184,46 @@ async def move_directory( # Process each entity for entity in entities: - try: - # Calculate new path by replacing source prefix with destination - old_path = entity.file_path - # Replace only the first occurrence of the source directory prefix - if old_path.startswith(f"{source_directory}/"): - new_path = old_path.replace( - f"{source_directory}/", f"{destination_directory}/", 1 + # Calculate new path by replacing source prefix with destination + old_path = entity.file_path + # Replace only the first occurrence of the source directory prefix + if old_path.startswith(f"{source_directory}/"): + new_path = old_path.replace(f"{source_directory}/", f"{destination_directory}/", 1) + else: # pragma: no cover + # Entity is directly in the source directory (shouldn't happen with prefix match) + new_path = f"{destination_directory}/{old_path}" + + # Trigger: one file move can publish filesystem or database state before returning. + # Why: every move publishes independently and cached reads must not retain + # an earlier file's state while the remaining directory batch runs. + # Outcome: finish one generation bump before reporting the result or cancellation. + invalidation_scope = ( + invalidate_cache(read_cache, project_external_id) + if read_cache is not None + else nullcontext() + ) + move_error: Exception | None = None + async with invalidation_scope: + try: + # Move the individual entity + await self.move_entity( + identifier=entity.file_path, + destination_path=new_path, + project_config=project_config, + app_config=app_config, ) - else: # pragma: no cover - # Entity is directly in the source directory (shouldn't happen with prefix match) - new_path = f"{destination_directory}/{old_path}" - - # Move the individual entity - await self.move_entity( - identifier=entity.file_path, - destination_path=new_path, - project_config=project_config, - app_config=app_config, - ) - - moved_files.append(new_path) - successful_moves += 1 - logger.debug(f"Moved entity: {old_path} -> {new_path}") + except Exception as error: # pragma: no cover + move_error = error - except Exception as e: # pragma: no cover + if move_error is not None: # pragma: no cover failed_moves += 1 - errors.append(DirectoryMoveError(path=entity.file_path, error=str(e))) - logger.error(f"Failed to move entity {entity.file_path}: {e}") + errors.append(DirectoryMoveError(path=entity.file_path, error=str(move_error))) + logger.error(f"Failed to move entity {entity.file_path}: {move_error}") + continue + + moved_files.append(new_path) + successful_moves += 1 + logger.debug(f"Moved entity: {old_path} -> {new_path}") logger.info( f"Directory move complete: {successful_moves} succeeded, {failed_moves} failed " diff --git a/src/basic_memory/services/initialization.py b/src/basic_memory/services/initialization.py index 7ce0d7815..c25965874 100644 --- a/src/basic_memory/services/initialization.py +++ b/src/basic_memory/services/initialization.py @@ -6,6 +6,7 @@ import asyncio import os +from contextlib import nullcontext from pathlib import Path from typing import TYPE_CHECKING @@ -23,6 +24,7 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker from basic_memory.index.local_project import LocalProjectIndexRuntimeProvider + from basic_memory.read_cache import ReadCache, ReadCacheInvalidator async def run_initial_project_index( @@ -50,6 +52,8 @@ async def run_initial_project_index( async def recover_project_materializations( project: Project, session_maker: "async_sessionmaker[AsyncSession]", + *, + read_cache: "ReadCacheInvalidator | None" = None, ) -> None: """Re-drive note materialization and move cleanup lost across a process exit. @@ -65,31 +69,63 @@ async def recover_project_materializations( recover_move_vacates, recover_stuck_materializations, ) + from basic_memory.read_cache import invalidate_cache from basic_memory.services.file_service import FileService - try: - # FileService needs only base_path to write the accepted markdown bytes; - # the markdown_processor/app_config are unused on the materialization path. - file_service = FileService(Path(project.path)) - recovered = await recover_stuck_materializations( - session_maker=session_maker, - file_service=file_service, - project_id=project.id, - ) - recovered_vacates = await recover_move_vacates( - session_maker=session_maker, - file_service=file_service, - project_id=project.id, + # FileService needs only base_path to write the accepted markdown bytes; + # the markdown_processor/app_config are unused on the materialization path. + file_service = FileService(Path(project.path)) + # Recovery reports whether it published state only after its transaction-bearing + # phase exits. Scope the whole phase so cancellation cannot land in that window; + # a harmless generation bump after a no-op recovery is the correctness tradeoff. + materialization_scope = ( + invalidate_cache(read_cache, str(project.external_id)) + if read_cache is not None + else nullcontext() + ) + async with materialization_scope: + try: + materialization_recovery = await recover_stuck_materializations( + session_maker=session_maker, + file_service=file_service, + project_id=project.id, + ) + except Exception as e: # pragma: no cover - defensive startup guard + logger.error(f"Error recovering stuck materializations for project {project.name}: {e}") + return + + if materialization_recovery.attempted: + logger.info( + "Recovered note materialization state on startup", + project=project.name, + attempted_materializations=materialization_recovery.attempted, + recovered_materializations=materialization_recovery.written, ) - if recovered or recovered_vacates: - logger.info( - "Recovered note materialization state on startup", - project=project.name, - recovered_materializations=recovered, - recovered_move_vacates=recovered_vacates, + + vacate_scope = ( + invalidate_cache(read_cache, str(project.external_id)) + if read_cache is not None + else nullcontext() + ) + async with vacate_scope: + try: + recovered_vacates = await recover_move_vacates( + session_maker=session_maker, + file_service=file_service, + project_id=project.id, ) - except Exception as e: # pragma: no cover - defensive startup guard - logger.error(f"Error recovering stuck materializations for project {project.name}: {e}") + except Exception as e: # pragma: no cover - defensive startup guard + logger.error(f"Error recovering move vacates for project {project.name}: {e}") + return + + if not recovered_vacates: + return + + logger.info( + "Recovered note move-vacate state on startup", + project=project.name, + recovered_move_vacates=recovered_vacates, + ) async def initialize_database(app_config: BasicMemoryConfig) -> None: @@ -151,6 +187,7 @@ async def initialize_file_indexing( quiet: bool = True, *, recovery_complete: asyncio.Event | None = None, + read_cache: "ReadCache | None" = None, ) -> None: """Initialize file indexing services. @@ -160,6 +197,7 @@ async def initialize_file_indexing( app_config: The Basic Memory project configuration quiet: Whether to suppress Rich console output (True for MCP, False for CLI watch) recovery_complete: Optional startup barrier set after durable recovery finishes. + read_cache: Optional host-injected semantic read cache. Returns: The watch service task that's monitoring file changes @@ -190,11 +228,13 @@ async def initialize_file_indexing( # running multiple `basic-memory mcp --project X` processes does not produce # duplicate watchers fighting over the same files. constrained_project = os.environ.get("BASIC_MEMORY_MCP_PROJECT") - event_index_runtime_factory = LocalWatchEventIndexRuntimeFactory( index_embeddings=app_config.semantic_search_enabled, + read_cache=read_cache, + ) + project_index_runtime_factory = LocalProjectIndexRuntimeFactory( + read_cache=read_cache, ) - project_index_runtime_factory = LocalProjectIndexRuntimeFactory() # Initialize watch service watch_service = WatchService( @@ -227,7 +267,11 @@ async def initialize_file_indexing( # cleanup whose in-process enqueue was lost. Runs synchronously so the files # converge before the initial project index scans them. for project in active_projects: - await recover_project_materializations(project, session_maker) + await recover_project_materializations( + project, + session_maker, + read_cache=read_cache, + ) # Trigger: the API/MCP lifespan is waiting for durable startup recovery. # Why: serving accepted writes while recovery holds cached vacate work can diff --git a/src/basic_memory/services/note_content_reads.py b/src/basic_memory/services/note_content_reads.py index c409b66f6..9ab50ff6e 100644 --- a/src/basic_memory/services/note_content_reads.py +++ b/src/basic_memory/services/note_content_reads.py @@ -2,6 +2,8 @@ from __future__ import annotations +from contextlib import nullcontext + from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker from basic_memory import db @@ -15,6 +17,10 @@ run_note_content_read_repair_with_default_reconciler, ) from basic_memory.models import Entity, NoteContent, Project +from basic_memory.read_cache import ( + ReadCacheInvalidator, + invalidate_cache, +) from basic_memory.runtime.note_content import ( RuntimeNoteContentResource, RuntimeNoteContentResponsePayload, @@ -91,6 +97,7 @@ async def get_note_entity_payload_with_read_repair( entity_external_id: str, session: AsyncSession | None = None, source: str = "read_repair", + read_cache: ReadCacheInvalidator | None = None, ) -> RuntimeNoteContentResponsePayload | None: """Return entity payload, repairing missing note_content when a reader exists.""" payload = await self.get_note_entity_payload( @@ -105,6 +112,7 @@ async def get_note_entity_payload_with_read_repair( project_external_id=project_external_id, entity_external_id=entity_external_id, source=source, + read_cache=read_cache, ) if not repaired: return None @@ -147,6 +155,7 @@ async def get_note_resource_with_read_repair( entity_external_id: str, session: AsyncSession | None = None, source: str = "read_repair", + read_cache: ReadCacheInvalidator | None = None, ) -> RuntimeNoteContentResource | None: """Return markdown resource, repairing missing note_content when possible.""" resource = await self.get_note_resource( @@ -161,6 +170,7 @@ async def get_note_resource_with_read_repair( project_external_id=project_external_id, entity_external_id=entity_external_id, source=source, + read_cache=read_cache, ) if not repaired: return None @@ -177,6 +187,7 @@ async def reconcile_note_content_from_file( project_external_id: str, entity_external_id: str, source: str, + read_cache: ReadCacheInvalidator | None = None, ) -> bool: """Repair a missing note_content row from the runtime's canonical file source.""" async with db.scoped_session(self.session_maker) as session: @@ -193,10 +204,19 @@ async def reconcile_note_content_from_file( if self.read_repair_file_reader is None: raise RuntimeError("note-content read repair requires a file reader") - repair_run = await run_note_content_read_repair_with_default_reconciler( - repair_preflight, - session_maker=self.session_maker, - file_reader=self.read_repair_file_reader, - source=source, + # The repair runner owns the transaction that may create note_content. + # Keep invalidation around that whole await so cancellation during commit + # exit still advances the cache generation before it propagates. + repair_scope = ( + invalidate_cache(read_cache, project_external_id) + if read_cache is not None + else nullcontext() ) + async with repair_scope: + repair_run = await run_note_content_read_repair_with_default_reconciler( + repair_preflight, + session_maker=self.session_maker, + file_reader=self.read_repair_file_reader, + source=source, + ) return repair_run.repaired diff --git a/src/basic_memory/services/note_content_writes.py b/src/basic_memory/services/note_content_writes.py index e6c5e0ae1..ba5822e89 100644 --- a/src/basic_memory/services/note_content_writes.py +++ b/src/basic_memory/services/note_content_writes.py @@ -30,6 +30,10 @@ RuntimeAcceptedNoteChange, RuntimeNoteContentResponsePayload, ) +from basic_memory.read_cache import ( + ReadCacheInvalidator, + finish_project_read_cache_invalidation, +) from basic_memory.schemas.base import Entity as EntitySchema from basic_memory.schemas.request import EditEntityRequest @@ -138,11 +142,50 @@ def __init__( mutation_dependencies: AcceptedNoteMutationDependencies, content_freshener: NoteContentMutationFreshener | None = None, actor_resolver: NoteContentMutationActorResolver | None = None, + read_cache: ReadCacheInvalidator | None = None, ) -> None: self.session_maker = session_maker self.mutation_dependencies = mutation_dependencies self.content_freshener = content_freshener self.actor_resolver = actor_resolver + self.read_cache = read_cache + + @asynccontextmanager + async def _mutation_cache_scope( + self, + project_external_id: str, + *, + invalidate_on_rejection: bool = False, + ) -> AsyncIterator[None]: + """Invalidate after a mutation can publish authoritative state.""" + read_cache = self.read_cache + if read_cache is None: + yield + return + + try: + yield + except AcceptedNoteMutationRejected: + if invalidate_on_rejection: + await finish_project_read_cache_invalidation( + read_cache, + project_external_id, + ) + raise + except BaseException: + # Transaction exit can raise after its commit reached the database. + # Invalidation is safe after an earlier rollback and required after + # any commit whose response was interrupted. + await finish_project_read_cache_invalidation( + read_cache, + project_external_id, + ) + raise + else: + await finish_project_read_cache_invalidation( + read_cache, + project_external_id, + ) def _resolve_actor( self, @@ -171,14 +214,27 @@ async def freshen_existing_note_content( *, project_external_id: str, entity_external_id: str, - ) -> None: - """Let the runtime converge observed file state before an existing-note mutation.""" + ) -> bool: + """Converge observed file state and report whether it may have published state.""" if self.content_freshener is None: - return - await self.content_freshener.freshen_note_content( - project_external_id=project_external_id, - entity_external_id=entity_external_id, - ) + return False + + try: + await self.content_freshener.freshen_note_content( + project_external_id=project_external_id, + entity_external_id=entity_external_id, + ) + except BaseException: + # Freshening can commit entity and note-content state before a later + # indexing follow-up raises. Invalidate before propagating so those + # partial publications cannot retain the previous cache generation. + if self.read_cache is not None: + await finish_project_read_cache_invalidation( + self.read_cache, + project_external_id, + ) + raise + return True async def create_note( self, @@ -199,21 +255,23 @@ async def create_note( actor_name=actor_name, ) try: - async with accepted_note_transaction(self.session_maker) as session: - return await run_accepted_note_create( - session, - request=AcceptedNoteCreateMutation( - project_external_id=project_external_id, - data=data, - actor=accepted_note_mutation_actor( - user_profile_id=actor_context.user_profile_id, - actor_kind=actor_context.actor_kind, - actor_name=actor_context.actor_name, + async with self._mutation_cache_scope(project_external_id): + async with accepted_note_transaction(self.session_maker) as session: + accepted = await run_accepted_note_create( + session, + request=AcceptedNoteCreateMutation( + project_external_id=project_external_id, + data=data, + actor=accepted_note_mutation_actor( + user_profile_id=actor_context.user_profile_id, + actor_kind=actor_context.actor_kind, + actor_name=actor_context.actor_name, + ), + source=actor_context.source, ), - source=actor_context.source, - ), - dependencies=self.mutation_dependencies, - ) + dependencies=self.mutation_dependencies, + ) + return accepted except AcceptedNoteMutationRejected as error: raise note_content_mutation_error_from_rejection(error.rejection) from error @@ -245,30 +303,36 @@ async def update_note( actor_kind=actor_kind, actor_name=actor_name, ) + freshening_may_have_published = False try: - await self.freshen_existing_note_content( + freshening_may_have_published = await self.freshen_existing_note_content( project_external_id=project_external_id, entity_external_id=entity_external_id, ) - async with accepted_note_transaction(self.session_maker) as session: - return await run_accepted_note_update( - session, - request=AcceptedNoteUpdateMutation( - project_external_id=project_external_id, - entity_external_id=entity_external_id, - data=data, - actor=accepted_note_mutation_actor( - user_profile_id=actor_context.user_profile_id, - actor_kind=actor_context.actor_kind, - actor_name=actor_context.actor_name, + async with self._mutation_cache_scope( + project_external_id, + invalidate_on_rejection=freshening_may_have_published, + ): + async with accepted_note_transaction(self.session_maker) as session: + accepted = await run_accepted_note_update( + session, + request=AcceptedNoteUpdateMutation( + project_external_id=project_external_id, + entity_external_id=entity_external_id, + data=data, + actor=accepted_note_mutation_actor( + user_profile_id=actor_context.user_profile_id, + actor_kind=actor_context.actor_kind, + actor_name=actor_context.actor_name, + ), + source=actor_context.source, + base_checksum=base_checksum, ), - source=actor_context.source, - base_checksum=base_checksum, - ), - dependencies=self.mutation_dependencies, - ) + dependencies=self.mutation_dependencies, + ) except AcceptedNoteMutationRejected as error: raise note_content_mutation_error_from_rejection(error.rejection) from error + return accepted async def edit_note( self, @@ -289,29 +353,35 @@ async def edit_note( actor_kind=actor_kind, actor_name=actor_name, ) + freshening_may_have_published = False try: - await self.freshen_existing_note_content( + freshening_may_have_published = await self.freshen_existing_note_content( project_external_id=project_external_id, entity_external_id=entity_external_id, ) - async with accepted_note_transaction(self.session_maker) as session: - return await run_accepted_note_edit( - session, - request=AcceptedNoteEditMutation( - project_external_id=project_external_id, - entity_external_id=entity_external_id, - data=data, - actor=accepted_note_mutation_actor( - user_profile_id=actor_context.user_profile_id, - actor_kind=actor_context.actor_kind, - actor_name=actor_context.actor_name, + async with self._mutation_cache_scope( + project_external_id, + invalidate_on_rejection=freshening_may_have_published, + ): + async with accepted_note_transaction(self.session_maker) as session: + accepted = await run_accepted_note_edit( + session, + request=AcceptedNoteEditMutation( + project_external_id=project_external_id, + entity_external_id=entity_external_id, + data=data, + actor=accepted_note_mutation_actor( + user_profile_id=actor_context.user_profile_id, + actor_kind=actor_context.actor_kind, + actor_name=actor_context.actor_name, + ), + source=actor_context.source, ), - source=actor_context.source, - ), - dependencies=self.mutation_dependencies, - ) + dependencies=self.mutation_dependencies, + ) except AcceptedNoteMutationRejected as error: raise note_content_mutation_error_from_rejection(error.rejection) from error + return accepted async def move_note( self, @@ -332,29 +402,35 @@ async def move_note( actor_kind=actor_kind, actor_name=actor_name, ) + freshening_may_have_published = False try: - await self.freshen_existing_note_content( + freshening_may_have_published = await self.freshen_existing_note_content( project_external_id=project_external_id, entity_external_id=entity_external_id, ) - async with accepted_note_transaction(self.session_maker) as session: - return await run_accepted_note_move( - session, - request=AcceptedNoteMoveMutation( - project_external_id=project_external_id, - entity_external_id=entity_external_id, - destination_path=destination_path, - actor=accepted_note_mutation_actor( - user_profile_id=actor_context.user_profile_id, - actor_kind=actor_context.actor_kind, - actor_name=actor_context.actor_name, + async with self._mutation_cache_scope( + project_external_id, + invalidate_on_rejection=freshening_may_have_published, + ): + async with accepted_note_transaction(self.session_maker) as session: + accepted = await run_accepted_note_move( + session, + request=AcceptedNoteMoveMutation( + project_external_id=project_external_id, + entity_external_id=entity_external_id, + destination_path=destination_path, + actor=accepted_note_mutation_actor( + user_profile_id=actor_context.user_profile_id, + actor_kind=actor_context.actor_kind, + actor_name=actor_context.actor_name, + ), + source=actor_context.source, ), - source=actor_context.source, - ), - dependencies=self.mutation_dependencies, - ) + dependencies=self.mutation_dependencies, + ) except AcceptedNoteMutationRejected as error: raise note_content_mutation_error_from_rejection(error.rejection) from error + return accepted async def delete_note( self, @@ -363,19 +439,25 @@ async def delete_note( entity_external_id: str, ) -> AcceptedNoteChange: """DELETE the DB note and return the runtime follow-up change.""" + freshening_may_have_published = False try: - await self.freshen_existing_note_content( + freshening_may_have_published = await self.freshen_existing_note_content( project_external_id=project_external_id, entity_external_id=entity_external_id, ) - async with accepted_note_transaction(self.session_maker) as session: - return await run_accepted_note_delete( - session, - request=AcceptedNoteDeleteMutation( - project_external_id=project_external_id, - entity_external_id=entity_external_id, - ), - dependencies=self.mutation_dependencies, - ) + async with self._mutation_cache_scope( + project_external_id, + invalidate_on_rejection=freshening_may_have_published, + ): + async with accepted_note_transaction(self.session_maker) as session: + accepted = await run_accepted_note_delete( + session, + request=AcceptedNoteDeleteMutation( + project_external_id=project_external_id, + entity_external_id=entity_external_id, + ), + dependencies=self.mutation_dependencies, + ) except AcceptedNoteMutationRejected as error: raise note_content_mutation_error_from_rejection(error.rejection) from error + return accepted diff --git a/test-int/read_cache/conftest.py b/test-int/read_cache/conftest.py new file mode 100644 index 000000000..505855715 --- /dev/null +++ b/test-int/read_cache/conftest.py @@ -0,0 +1,106 @@ +"""Real Redis fixtures for read-cache integration tests.""" + +from __future__ import annotations + +import os +import shutil +import time +from collections.abc import AsyncGenerator, Generator +from dataclasses import dataclass +from uuid import uuid4 + +import pytest +import pytest_asyncio +from redis import Redis as SyncRedis +from redis.asyncio import Redis +from redis.exceptions import RedisError +from testcontainers.core.container import DockerContainer + +from basic_memory.read_cache.redis import RedisReadCache, create_redis_read_cache_client + + +def _wait_for_redis(url: str) -> None: + client = SyncRedis.from_url( + url, + socket_connect_timeout=0.1, + socket_timeout=0.1, + ) + deadline = time.monotonic() + 10 + last_error: RedisError | None = None + try: + while time.monotonic() < deadline: + try: + client.ping() + return + except RedisError as error: + last_error = error + time.sleep(0.05) + finally: + client.close() + + raise RuntimeError("Redis test server did not become ready") from last_error + + +@pytest.fixture(scope="session") +def redis_url() -> Generator[str]: + """Use a configured Redis server or start a real Redis 8 testcontainer.""" + configured_url = os.environ.get("BASIC_MEMORY_TEST_REDIS_URL") + if configured_url: + _wait_for_redis(configured_url) + yield configured_url + return + + # GitHub's Windows runners cannot start the Linux Redis image through testcontainers. + # A configured URL still runs the same real-server suite on Windows when one is available. + if os.name == "nt": + pytest.skip("Set BASIC_MEMORY_TEST_REDIS_URL to run real Redis tests on Windows") + + if shutil.which("docker") is None: + pytest.skip("Docker is required for real Redis integration tests") + + image = os.environ.get("BASIC_MEMORY_TEST_REDIS_IMAGE", "redis:8.8-alpine") + with DockerContainer(image).with_exposed_ports(6379) as container: + host = container.get_container_host_ip() + port = container.get_exposed_port(6379) + url = f"redis://{host}:{port}/0" + _wait_for_redis(url) + yield url + + +@dataclass(frozen=True, slots=True) +class RedisCacheHarness: + """One isolated Redis prefix and the client that owns it.""" + + cache: RedisReadCache + client: Redis + namespace: str + prefix: str + + +@pytest_asyncio.fixture +async def redis_cache(redis_url: str) -> AsyncGenerator[RedisCacheHarness]: + """Yield an isolated cache over the suite's real Redis server.""" + prefix = f"bm:test:read:{uuid4().hex}" + namespace = f"tenant-{uuid4().hex}" + # Container scheduling can make the first async handshake exceed the + # production cache's intentionally tight timeout. Behavioral timeout tests + # construct their own 50ms clients, so keep this shared harness stable. + client = create_redis_read_cache_client(redis_url, socket_timeout=1.0) + cache = RedisReadCache( + client=client, + namespace=namespace, + prefix=prefix, + ) + try: + await client.ping() + yield RedisCacheHarness( + cache=cache, + client=client, + namespace=namespace, + prefix=prefix, + ) + finally: + keys = [key async for key in client.scan_iter(match=f"{prefix}:*")] + if keys: + await client.delete(*keys) + await client.aclose() diff --git a/test-int/read_cache/test_api_read_cache.py b/test-int/read_cache/test_api_read_cache.py new file mode 100644 index 000000000..1bd89d030 --- /dev/null +++ b/test-int/read_cache/test_api_read_cache.py @@ -0,0 +1,1070 @@ +"""Full-stack API coverage against the real Redis read cache.""" + +from __future__ import annotations + +import asyncio +import json +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from datetime import datetime, timezone +from pathlib import Path +from typing import Literal, Protocol, override +from uuid import uuid4 + +import pytest +from fastapi import FastAPI +from httpx import ASGITransport, AsyncClient +from redis.asyncio import Redis +from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker + +import basic_memory.indexing.note_content_reconciler as note_content_reconciler +import basic_memory.services.note_content_writes as note_content_writes +from basic_memory import db +from basic_memory.deps import ( + get_chatgpt_importer_v2_external, + get_index_file_executor_v2_external, + get_note_content_query_service, + get_read_cache, +) +from basic_memory.indexing.note_content_read_repair_runner import ( + NoteContentReadRepairFile, + NoteContentReadRepairTarget, +) +from basic_memory.indexing.models import FileIndexResult +from basic_memory.index.note_content_materialization import drain_pending_materializations +from basic_memory.models import Project +from basic_memory.models.knowledge import Entity +from basic_memory.read_cache import ( + ReadCacheInvalidator, + ReadCacheInvalidationStatus, + ReadCacheKey, + ReadCacheOperation, + read_cache_request_digest, +) +from basic_memory.read_cache.keys import ( + redis_read_cache_generation_key, + redis_read_cache_keys, +) +from basic_memory.read_cache.redis import RedisReadCache +from basic_memory.repository import EntityRepository +from basic_memory.repository.note_content_repository import NoteContentRepository +from basic_memory.runtime.note_content import NOTE_CONTENT_BASE_CHECKSUM_HEADER +from basic_memory.schemas.directory import DirectoryListResponse, DirectoryNode +from basic_memory.schemas.v2 import EntityResolveRequest +from basic_memory.services.note_content_reads import NoteContentQueryService +from basic_memory.workspace_context import ( + WORKSPACE_SLUG_HEADER, + WORKSPACE_TYPE_HEADER, +) + + +class RedisCacheHarness(Protocol): + """Structural type for the real Redis fixture.""" + + cache: RedisReadCache + client: Redis + namespace: str + prefix: str + + +pytestmark = pytest.mark.redis + + +class FailingIndexFileExecutor: + """Represent an indexer that fails after it may have committed entity state.""" + + async def index_file( + self, + file_path: str, + *, + source: str, + ) -> FileIndexResult: + del file_path, source + raise RuntimeError("partial direct index failure") + + +class PartiallyFailingImporter: + """Overwrite one resource before reporting an import failure.""" + + def __init__(self, target: Path) -> None: + self.target = target + + async def import_data( + self, + source_data: object, + destination_folder: str, + **kwargs: object, + ) -> None: + del source_data, destination_folder, kwargs + self.target.write_text("# Imported before failure\n", encoding="utf-8") + raise RuntimeError("partial import failure") + + +class LocalReadRepairFileReader: + """Read canonical project files through the hosted read-repair protocol.""" + + async def read_note_content_repair_file( + self, + target: NoteContentReadRepairTarget[Project, Entity], + ) -> NoteContentReadRepairFile | None: + path = Path(target.project.path) / target.entity.file_path + if not path.exists(): + return None + stat = path.stat() + return NoteContentReadRepairFile( + markdown_content=path.read_text(encoding="utf-8"), + observed_at=datetime.fromtimestamp(stat.st_mtime, timezone.utc), + ) + + +class NoAcceptedNoteContent: + """Keep this resource on the file-first path for project-root cache coverage.""" + + async def get_note_resource_with_read_repair( + self, + *, + project_external_id: str, + entity_external_id: str, + session: AsyncSession, + read_cache: ReadCacheInvalidator, + ) -> None: + del project_external_id, entity_external_id, session, read_cache + return None + + +class DestinationObservingRedisReadCache(RedisReadCache): + """Record how many target files exist at each real invalidation.""" + + def __init__( + self, + *, + client: Redis, + namespace: str, + prefix: str, + destination_paths: tuple[Path, ...], + ) -> None: + super().__init__(client=client, namespace=namespace, prefix=prefix) + self.destination_paths = destination_paths + self.destination_counts: list[int] = [] + self.project_ids: list[str] = [] + + @override + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + status = await super().invalidate_project(project_id) + self.project_ids.append(project_id) + self.destination_counts.append( + sum(destination.exists() for destination in self.destination_paths) + ) + return status + + +class BlockingInvalidationRedisReadCache(RedisReadCache): + """Hold one real invalidation so the request can be cancelled repeatedly.""" + + def __init__( + self, + *, + client: Redis, + namespace: str, + prefix: str, + ) -> None: + super().__init__(client=client, namespace=namespace, prefix=prefix) + self.invalidation_started = asyncio.Event() + self.release_invalidation = asyncio.Event() + + @override + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + self.invalidation_started.set() + await self.release_invalidation.wait() + return await super().invalidate_project(project_id) + + +def _cache_key( + *, + project_id: str, + operation: ReadCacheOperation, + request: str, + request_context: tuple[str, ...] = (), +) -> ReadCacheKey: + return ReadCacheKey( + project_id=project_id, + operation=operation, + request_digest=read_cache_request_digest(request, *request_context), + ) + + +def descendant_paths(node: DirectoryNode) -> set[str]: + """Collect one directory response's complete path set.""" + paths = {node.directory_path} + for child in node.children: + paths.update(descendant_paths(child)) + return paths + + +async def _initialized_generation( + redis_cache: RedisCacheHarness, + project_id: str, + *, + request: str, +) -> bytes | str: + await redis_cache.cache.lookup( + _cache_key( + project_id=project_id, + operation=ReadCacheOperation.entity, + request=request, + ) + ) + generation = await redis_cache.client.get( + redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=project_id, + ) + ) + assert generation is not None + return generation + + +@pytest.mark.asyncio +async def test_cancelled_committed_create_finishes_real_redis_invalidation( + app: FastAPI, + client: AsyncClient, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Cancellation after commit waits for invalidation before propagating.""" + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-create", + ) + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + app.dependency_overrides[get_read_cache] = lambda: blocking_cache + + transaction_committed = asyncio.Event() + hold_after_commit = asyncio.Event() + original_transaction = note_content_writes.accepted_note_transaction + + @asynccontextmanager + async def pause_after_commit( + session_maker: async_sessionmaker[AsyncSession], + ) -> AsyncIterator[AsyncSession]: + async with original_transaction(session_maker) as session: + yield session + transaction_committed.set() + await hold_after_commit.wait() + + monkeypatch.setattr( + note_content_writes, + "accepted_note_transaction", + pause_after_commit, + ) + title = f"Cancelled committed create {uuid4()}" + request_task = asyncio.create_task( + client.post( + f"/v2/projects/{project_external_id}/knowledge/entities", + json={ + "title": title, + "directory": "cache", + "content": f"# {title}\n", + }, + ) + ) + + async with asyncio.timeout(5): + await transaction_committed.wait() + request_task.cancel() + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + # A second cancellation must not abandon the already-running cleanup. + request_task.cancel() + await asyncio.sleep(0) + assert not request_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await request_task + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-create", + ) + assert generation_after != generation_before + + _, session_maker = engine_factory + async with db.scoped_session(session_maker) as session: + committed_entities = await EntityRepository(project_id=test_project.id).get_by_title( + session, + title, + ) + assert len(committed_entities) == 1 + + +@pytest.mark.asyncio +async def test_entity_resolve_and_markdown_reads_cache_then_freshened_write_invalidates( + app: FastAPI, + client: AsyncClient, + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Pre-write freshening invalidates even when the following write is rejected.""" + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + project_external_id = str(test_project.external_id) + project_url = f"/v2/projects/{project_external_id}" + + created_response = await client.post( + f"{project_url}/knowledge/entities", + json={ + "title": "Redis Cached Note", + "directory": "cache", + "content": "# Redis Cached Note\n\nVersion one.", + }, + ) + assert created_response.status_code == 202 + created = created_response.json() + entity_id = created["external_id"] + await drain_pending_materializations() + + entity_response = await client.get(f"{project_url}/knowledge/entities/{entity_id}") + resolve_request = EntityResolveRequest(identifier=created["permalink"]) + resolve_response = await client.post( + f"{project_url}/knowledge/resolve", + json=resolve_request.model_dump(mode="json"), + ) + workspace_resolve_response = await client.post( + f"{project_url}/knowledge/resolve", + headers={ + WORKSPACE_SLUG_HEADER: "team-paul", + WORKSPACE_TYPE_HEADER: "organization", + }, + json=resolve_request.model_dump(mode="json"), + ) + resource_response = await client.get(f"{project_url}/resource/{entity_id}") + + assert entity_response.status_code == 200 + assert resolve_response.status_code == 200 + assert workspace_resolve_response.status_code == 200 + assert resource_response.status_code == 200 + assert "Version one." in resource_response.text + + keys = ( + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.entity, + request=entity_id, + ), + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.resolve, + request=resolve_request.model_dump_json(), + request_context=("", ""), + ), + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.resolve, + request=resolve_request.model_dump_json(), + request_context=("team-paul", "organization"), + ), + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.resource, + request=entity_id, + ), + ) + for key in keys: + redis_keys = redis_read_cache_keys( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + key=key, + ) + assert await redis_cache.client.exists(redis_keys.data_key) == 1 + + generation_key = redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=project_external_id, + ) + populated_generation = await redis_cache.client.get(generation_key) + assert populated_generation is not None + + disk_path = Path(test_project.path) / created["file_path"] + disk_path.write_text( + "# Redis Cached Note\n\nExternally edited before rejected update.", + encoding="utf-8", + ) + rejected_response = await client.put( + f"{project_url}/knowledge/entities/{entity_id}", + headers={NOTE_CONTENT_BASE_CHECKSUM_HEADER: "stale-checksum"}, + json={ + "title": "Redis Cached Note", + "directory": "cache", + "content": "# Redis Cached Note\n\nRejected replacement.", + }, + ) + assert rejected_response.status_code == 409 + freshened_generation = await redis_cache.client.get(generation_key) + assert freshened_generation is not None + assert freshened_generation != populated_generation + + freshened_entity = await client.get(f"{project_url}/knowledge/entities/{entity_id}") + freshened_resource = await client.get(f"{project_url}/resource/{entity_id}") + assert freshened_entity.status_code == 200 + assert "Externally edited" in freshened_entity.json()["content"] + assert freshened_resource.status_code == 200 + assert "Externally edited" in freshened_resource.text + + edited_response = await client.patch( + f"{project_url}/knowledge/entities/{entity_id}", + json={ + "operation": "append", + "content": "\n\nVersion two.", + }, + ) + assert edited_response.status_code == 202 + invalidated_generation = await redis_cache.client.get(generation_key) + assert invalidated_generation is not None + assert invalidated_generation != freshened_generation + + refreshed_entity = await client.get(f"{project_url}/knowledge/entities/{entity_id}") + refreshed_resource = await client.get(f"{project_url}/resource/{entity_id}") + assert refreshed_entity.status_code == 200 + assert "Externally edited" in refreshed_entity.json()["content"] + assert "Version two." in refreshed_entity.json()["content"] + assert refreshed_resource.status_code == 200 + assert "Externally edited" in refreshed_resource.text + assert "Version two." in refreshed_resource.text + + +@pytest.mark.asyncio +async def test_directory_reads_cache_then_project_invalidation_refreshes( + app: FastAPI, + client: AsyncClient, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Tree, structure, and paginated list reads share real project invalidation.""" + project_external_id = str(test_project.external_id) + project_url = f"/v2/projects/{project_external_id}" + repository = EntityRepository(project_id=test_project.id) + _, session_maker = engine_factory + + async with db.scoped_session(session_maker) as session: + await repository.add( + session, + Entity( + title="Existing cached directory note", + note_type="note", + content_type="text/markdown", + file_path="cache-surface/existing.md", + checksum="existing-cached-directory-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + tree_url = f"{project_url}/directory/tree" + structure_url = f"{project_url}/directory/structure" + list_url = f"{project_url}/directory/list" + list_params = { + "dir_name": "/cache-surface", + "depth": 2, + "page": 1, + "page_size": 200, + } + + first_tree_response = await client.get(tree_url) + first_structure_response = await client.get(structure_url) + first_list_response = await client.get(list_url, params=list_params) + assert first_tree_response.status_code == 200 + assert first_structure_response.status_code == 200 + assert first_list_response.status_code == 200 + + first_tree = DirectoryNode.model_validate(first_tree_response.json()) + first_structure = DirectoryNode.model_validate(first_structure_response.json()) + first_list = DirectoryListResponse.model_validate(first_list_response.json()) + assert "/cache-surface/after-cache" not in descendant_paths(first_tree) + assert "/cache-surface/after-cache" not in descendant_paths(first_structure) + assert "/cache-surface/after-cache" not in {node.directory_path for node in first_list.nodes} + + cache_keys = ( + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.directory_tree, + request="tree", + ), + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.directory_structure, + request="structure", + ), + _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.directory_list, + request="/cache-surface", + request_context=("2", "", "1", "200"), + ), + ) + for key in cache_keys: + redis_keys = redis_read_cache_keys( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + key=key, + ) + assert await redis_cache.client.exists(redis_keys.data_key) == 1 + + async with db.scoped_session(session_maker) as session: + await repository.add( + session, + Entity( + title="Added after directory cache fill", + note_type="note", + content_type="text/markdown", + file_path="cache-surface/after-cache/new.md", + checksum="added-after-directory-cache-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + cached_tree_response = await client.get(tree_url) + cached_structure_response = await client.get(structure_url) + cached_list_response = await client.get(list_url, params=list_params) + assert cached_tree_response.json() == first_tree_response.json() + assert cached_structure_response.json() == first_structure_response.json() + assert cached_list_response.json() == first_list_response.json() + + await redis_cache.cache.invalidate_project(project_external_id) + + refreshed_tree = DirectoryNode.model_validate((await client.get(tree_url)).json()) + refreshed_structure = DirectoryNode.model_validate((await client.get(structure_url)).json()) + refreshed_list = DirectoryListResponse.model_validate( + (await client.get(list_url, params=list_params)).json() + ) + + assert "/cache-surface/after-cache" in descendant_paths(refreshed_tree) + assert "/cache-surface/after-cache" in descendant_paths(refreshed_structure) + assert "/cache-surface/after-cache" in {node.directory_path for node in refreshed_list.nodes} + + +@pytest.mark.asyncio +async def test_non_markdown_resource_is_never_cached( + app: FastAPI, + client: AsyncClient, + test_project: Project, + redis_cache: RedisCacheHarness, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], +) -> None: + """Arbitrary files remain authoritative even when they fit under the payload cap.""" + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + project_external_id = str(test_project.external_id) + file_path = "cache/plain.txt" + disk_path = Path(test_project.path) / file_path + disk_path.parent.mkdir(parents=True, exist_ok=True) + disk_path.write_text("Plain text remains uncached.", encoding="utf-8") + + repository = EntityRepository(project_id=test_project.id) + _, session_maker = engine_factory + async with db.scoped_session(session_maker) as session: + entity = await repository.add( + session, + Entity( + title="plain.txt", + note_type="file", + content_type="text/plain", + file_path=file_path, + checksum="plain-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + resource_url = f"/v2/projects/{project_external_id}/resource/{entity.external_id}" + first = await client.get(resource_url) + second = await client.get(resource_url) + assert first.status_code == 200 + assert second.status_code == 200 + assert second.text == "Plain text remains uncached." + + key = _cache_key( + project_id=project_external_id, + operation=ReadCacheOperation.resource, + request=entity.external_id, + ) + redis_keys = redis_read_cache_keys( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + key=key, + ) + assert await redis_cache.client.exists(redis_keys.data_key) == 0 + + +@pytest.mark.asyncio +async def test_project_root_change_invalidates_cached_markdown_resource( + app: FastAPI, + client: AsyncClient, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, + tmp_path: Path, +) -> None: + """A stable project/entity UUID cannot retain bytes from the previous root.""" + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + app.dependency_overrides[get_note_content_query_service] = NoAcceptedNoteContent + project_external_id = str(test_project.external_id) + file_path = "cache/project-root.md" + old_disk_path = Path(test_project.path) / file_path + old_disk_path.parent.mkdir(parents=True, exist_ok=True) + old_disk_path.write_text("# Old project root\n", encoding="utf-8") + + new_root = tmp_path / "new-project-root" + new_disk_path = new_root / file_path + new_disk_path.parent.mkdir(parents=True, exist_ok=True) + new_disk_path.write_text("# New project root\n", encoding="utf-8") + + repository = EntityRepository(project_id=test_project.id) + _, session_maker = engine_factory + async with db.scoped_session(session_maker) as session: + entity = await repository.add( + session, + Entity( + title="project-root.md", + note_type="file", + content_type="text/markdown", + file_path=file_path, + checksum="project-root-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + project_url = f"/v2/projects/{project_external_id}" + resource_url = f"{project_url}/resource/{entity.external_id}" + cached_response = await client.get(resource_url) + assert cached_response.status_code == 200 + assert cached_response.text == "# Old project root\n" + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="project-root-change", + ) + + move_response = await client.patch( + project_url, + json={"path": str(new_root)}, + ) + + assert move_response.status_code == 200 + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="project-root-change", + ) + assert generation_after != generation_before + refreshed_response = await client.get(resource_url) + assert refreshed_response.status_code == 200 + assert refreshed_response.text == "# New project root\n" + + +@pytest.mark.asyncio +async def test_direct_index_failure_invalidates_real_redis( + app: FastAPI, + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """A partial direct index commit cannot retain the previous cache generation.""" + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + app.dependency_overrides[get_index_file_executor_v2_external] = FailingIndexFileExecutor + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="direct-index-failure", + ) + file_path = "cache/direct-index-failure.md" + disk_path = Path(test_project.path) / file_path + disk_path.parent.mkdir(parents=True, exist_ok=True) + disk_path.write_text("# Partial direct index\n", encoding="utf-8") + + async with AsyncClient( + transport=ASGITransport(app=app, raise_app_exceptions=False), + base_url="http://test", + ) as failure_client: + response = await failure_client.post( + f"/v2/projects/{project_external_id}/knowledge/index-file", + json={"file_path": file_path}, + ) + + assert response.status_code == 500 + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="direct-index-failure", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_partial_import_failure_invalidates_cached_resource_in_real_redis( + app: FastAPI, + client: AsyncClient, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """File writes from a failed import cannot retain cached pre-import bytes.""" + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + project_external_id = str(test_project.external_id) + file_path = "cache/partial-import.md" + disk_path = Path(test_project.path) / file_path + disk_path.parent.mkdir(parents=True, exist_ok=True) + disk_path.write_text("# Before import\n", encoding="utf-8") + + repository = EntityRepository(project_id=test_project.id) + _, session_maker = engine_factory + async with db.scoped_session(session_maker) as session: + entity = await repository.add( + session, + Entity( + title="partial-import.md", + note_type="note", + content_type="text/markdown", + file_path=file_path, + checksum="partial-import-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + resource_url = f"/v2/projects/{project_external_id}/resource/{entity.external_id}" + cached_response = await client.get(resource_url) + assert cached_response.status_code == 200 + assert cached_response.text == "# Before import\n" + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="partial-import-failure", + ) + app.dependency_overrides[get_chatgpt_importer_v2_external] = lambda: PartiallyFailingImporter( + disk_path + ) + + response = await client.post( + f"/v2/projects/{project_external_id}/import/chatgpt", + files={"file": ("conversations.json", b"[]", "application/json")}, + ) + + assert response.status_code == 500 + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="partial-import-failure", + ) + assert generation_after != generation_before + refreshed_response = await client.get(resource_url) + assert refreshed_response.status_code == 200 + assert refreshed_response.text == "# Imported before failure\n" + + +@pytest.mark.asyncio +async def test_import_invalidates_after_each_written_file_in_real_redis( + app: FastAPI, + client: AsyncClient, + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Each imported file advances Redis before the next item is processed.""" + project_external_id = str(test_project.external_id) + destination_directory = "cache/import-per-file" + destination_paths = tuple( + Path(test_project.path) / destination_directory / "note" / f"{name}.md" + for name in ("First import", "Second import") + ) + observing_cache = DestinationObservingRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + destination_paths=destination_paths, + ) + app.dependency_overrides[get_read_cache] = lambda: observing_cache + source_data = b"\n".join( + json.dumps( + { + "type": "entity", + "name": name, + "entityType": "note", + "observations": [], + } + ).encode() + for name in ("First import", "Second import") + ) + + response = await client.post( + f"/v2/projects/{project_external_id}/import/memory-json", + files={"file": ("memory.json", source_data, "application/json")}, + data={"directory": destination_directory}, + ) + + assert response.status_code == 200 + assert response.json()["entities"] == 2 + assert observing_cache.destination_counts[:2] == [1, 2] + assert observing_cache.project_ids[:2] == [project_external_id, project_external_id] + assert all(destination.exists() for destination in destination_paths) + + +@pytest.mark.asyncio +async def test_resource_read_repair_invalidates_cached_entity_in_real_redis( + app: FastAPI, + client: AsyncClient, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """A resource-triggered note_content repair supersedes cached entity metadata.""" + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + project_external_id = str(test_project.external_id) + file_path = "cache/read-repair.md" + disk_path = Path(test_project.path) / file_path + disk_path.parent.mkdir(parents=True, exist_ok=True) + disk_path.write_text("# Read repair\n\nCanonical file content.\n", encoding="utf-8") + + repository = EntityRepository(project_id=test_project.id) + _, session_maker = engine_factory + async with db.scoped_session(session_maker) as session: + entity = await repository.add( + session, + Entity( + title="Read repair", + note_type="note", + content_type="text/markdown", + file_path=file_path, + checksum="read-repair-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + project_url = f"/v2/projects/{project_external_id}" + entity_url = f"{project_url}/knowledge/entities/{entity.external_id}" + resource_url = f"{project_url}/resource/{entity.external_id}" + cached_entity = await client.get(entity_url) + assert cached_entity.status_code == 200 + assert cached_entity.json().get("db_version") is None + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="resource-read-repair", + ) + + query_service = NoteContentQueryService( + session_maker=session_maker, + read_repair_file_reader=LocalReadRepairFileReader(), + ) + app.dependency_overrides[get_note_content_query_service] = lambda: query_service + + repaired_resource = await client.get(resource_url) + assert repaired_resource.status_code == 200 + assert repaired_resource.text == "# Read repair\n\nCanonical file content.\n" + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="resource-read-repair", + ) + assert generation_after != generation_before + + refreshed_entity = await client.get(entity_url) + assert refreshed_entity.status_code == 200 + assert refreshed_entity.json()["db_version"] == 1 + assert "Canonical file content." in refreshed_entity.json()["content"] + + +@pytest.mark.asyncio +@pytest.mark.parametrize("read_method", ["entity", "resource"]) +async def test_cancelled_committed_read_repair_finishes_real_redis_invalidation( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, + read_method: Literal["entity", "resource"], + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Cancellation during read-repair transaction exit waits for invalidation.""" + project_external_id = str(test_project.external_id) + file_path = "cache/cancelled-read-repair.md" + markdown_content = "# Cancelled read repair\n\nCommitted canonical content.\n" + disk_path = Path(test_project.path) / file_path + disk_path.parent.mkdir(parents=True, exist_ok=True) + disk_path.write_text(markdown_content, encoding="utf-8") + + repository = EntityRepository(project_id=test_project.id) + _, session_maker = engine_factory + async with db.scoped_session(session_maker) as session: + entity = await repository.add( + session, + Entity( + title="Cancelled read repair", + note_type="note", + content_type="text/markdown", + file_path=file_path, + checksum="cancelled-read-repair-checksum", + created_at=datetime.now(timezone.utc), + updated_at=datetime.now(timezone.utc), + ), + ) + + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-read-repair", + ) + transaction_committed = asyncio.Event() + hold_after_commit = asyncio.Event() + original_scoped_session = note_content_reconciler.db.scoped_session + + @asynccontextmanager + async def pause_after_repair_commit( + scoped_session_maker: async_sessionmaker[AsyncSession], + session: AsyncSession | None = None, + ) -> AsyncIterator[AsyncSession]: + async with original_scoped_session(scoped_session_maker, session) as scoped_session: + yield scoped_session + if transaction_committed.is_set(): + return + + async with original_scoped_session(scoped_session_maker) as verification_session: + repaired = await NoteContentRepository(project_id=test_project.id).get_by_entity_id( + verification_session, + entity.id, + ) + if repaired is not None: + transaction_committed.set() + await hold_after_commit.wait() + + monkeypatch.setattr( + note_content_reconciler.db, + "scoped_session", + pause_after_repair_commit, + ) + + query_service = NoteContentQueryService( + session_maker=session_maker, + read_repair_file_reader=LocalReadRepairFileReader(), + ) + if read_method == "entity": + read_repair = query_service.get_note_entity_payload_with_read_repair( + project_external_id=project_external_id, + entity_external_id=entity.external_id, + read_cache=blocking_cache, + ) + else: + read_repair = query_service.get_note_resource_with_read_repair( + project_external_id=project_external_id, + entity_external_id=entity.external_id, + read_cache=blocking_cache, + ) + request_task = asyncio.create_task(read_repair) + + async with asyncio.timeout(5): + await transaction_committed.wait() + request_task.cancel() + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + # A second cancellation must not abandon the already-running generation bump. + request_task.cancel() + await asyncio.sleep(0) + assert not request_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await request_task + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-read-repair", + ) + assert generation_after != generation_before + + async with db.scoped_session(session_maker) as session: + repaired = await NoteContentRepository(project_id=test_project.id).get_by_entity_id( + session, + entity.id, + ) + assert repaired is not None + assert repaired.db_version == 1 + assert repaired.markdown_content == markdown_content + + +@pytest.mark.asyncio +async def test_directory_move_invalidates_after_each_committed_file_in_real_redis( + app: FastAPI, + client: AsyncClient, + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Each file move advances Redis before the next directory item is processed.""" + project_external_id = str(test_project.external_id) + project_url = f"/v2/projects/{project_external_id}" + created_paths: list[str] = [] + for title in ("First directory move", "Second directory move"): + response = await client.post( + f"{project_url}/knowledge/entities", + json={ + "title": title, + "directory": "move-source", + "content": f"# {title}\n", + }, + ) + assert response.status_code == 202 + created_paths.append(response.json()["file_path"]) + + destination_paths = tuple( + Path(test_project.path) / source_path.replace("move-source/", "move-destination/", 1) + for source_path in created_paths + ) + observing_cache = DestinationObservingRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + destination_paths=destination_paths, + ) + app.dependency_overrides[get_read_cache] = lambda: observing_cache + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="directory-move", + ) + + response = await client.post( + f"{project_url}/knowledge/move-directory", + json={ + "source_directory": "move-source", + "destination_directory": "move-destination", + }, + ) + + assert response.status_code == 200 + assert response.json()["successful_moves"] == 2 + assert observing_cache.destination_counts[:2] == [1, 2] + assert all(destination.exists() for destination in destination_paths) + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="directory-move", + ) + assert generation_after != generation_before diff --git a/test-int/read_cache/test_materialization_invalidation.py b/test-int/read_cache/test_materialization_invalidation.py new file mode 100644 index 000000000..ec9eb6db6 --- /dev/null +++ b/test-int/read_cache/test_materialization_invalidation.py @@ -0,0 +1,202 @@ +"""Real Redis coverage for accepted-note materialization invalidation phases.""" + +from __future__ import annotations + +import asyncio +from datetime import UTC, datetime +from hashlib import sha256 +from pathlib import Path +from typing import Protocol + +import pytest +from redis.asyncio import Redis +from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker + +from basic_memory import db +from basic_memory.index.note_content_materialization import ( + LocalNoteContentMaterializationProvider, +) +from basic_memory.indexing.models import FileIndexOperation, FileIndexResult +from basic_memory.models import Entity, Project +from basic_memory.read_cache import ( + ReadCacheKey, + ReadCacheOperation, + read_cache_request_digest, +) +from basic_memory.read_cache.keys import redis_read_cache_generation_key +from basic_memory.read_cache.redis import RedisReadCache +from basic_memory.repository import EntityRepository, NoteContentRepository +from basic_memory.repository.note_content_repository import AcceptedNoteContentWrite +from basic_memory.runtime.note_content import ( + RuntimeAcceptedNoteChange, + RuntimeAcceptedNoteResponse, + RuntimePendingNoteMaterialization, + plan_accepted_note_response, +) +from basic_memory.services.file_service import FileService + +pytestmark = pytest.mark.redis + + +class RedisCacheHarness(Protocol): + cache: RedisReadCache + client: Redis + namespace: str + prefix: str + + +class BlockingIndexFileExecutor: + """Hold indexing after materialization has published its terminal state.""" + + def __init__(self, entity: Entity) -> None: + self.entity = entity + self.started = asyncio.Event() + self.release = asyncio.Event() + + async def index_file(self, file_path: str, *, source: str) -> FileIndexResult: + del source + self.started.set() + await self.release.wait() + checksum = self.entity.checksum + if checksum is None: + raise AssertionError("seeded materialization entity must carry a checksum") + return FileIndexResult( + file_path=file_path, + entity_id=self.entity.id, + external_id=str(self.entity.external_id), + title=self.entity.title, + permalink=self.entity.permalink, + checksum=checksum, + operation=FileIndexOperation.updated, + ) + + +async def _current_generation( + redis_cache: RedisCacheHarness, + project_external_id: str, +) -> bytes | str: + generation = await redis_cache.client.get( + redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=project_external_id, + ) + ) + assert generation is not None + return generation + + +async def _initialized_generation( + redis_cache: RedisCacheHarness, + project_external_id: str, +) -> bytes | str: + await redis_cache.cache.lookup( + ReadCacheKey( + project_id=project_external_id, + operation=ReadCacheOperation.entity, + request_digest=read_cache_request_digest("materialization-publication"), + ) + ) + return await _current_generation(redis_cache, project_external_id) + + +async def _accepted_note( + session_maker: async_sessionmaker[AsyncSession], + project: Project, +) -> tuple[ + Entity, + RuntimeAcceptedNoteChange[RuntimeAcceptedNoteResponse], +]: + markdown_content = "# Terminal materialization\n" + checksum = sha256(markdown_content.encode()).hexdigest() + now = datetime.now(UTC) + entity_repository = EntityRepository(project_id=project.id) + content_repository = NoteContentRepository(project_id=project.id) + + async with db.scoped_session(session_maker) as session: + entity = await entity_repository.add( + session, + Entity( + title="Terminal Materialization", + note_type="note", + content_type="text/markdown", + file_path="notes/terminal-materialization.md", + checksum=checksum, + created_at=now, + updated_at=now, + ), + ) + await content_repository.accept_write( + session, + AcceptedNoteContentWrite( + entity_id=entity.id, + markdown_content=markdown_content, + db_version=1, + db_checksum=checksum, + last_source="api", + updated_at=now, + ), + ) + note_content = await content_repository.get_by_entity_id(session, entity.id) + assert note_content is not None + payload = plan_accepted_note_response( + entity=entity, + note_content=note_content, + fallback_source="api", + ) + + return entity, RuntimeAcceptedNoteChange( + status_code=202, + payload=payload, + materialization=RuntimePendingNoteMaterialization( + project_id=project.id, + entity_id=entity.id, + db_version=1, + db_checksum=checksum, + source="api", + ), + ) + + +@pytest.mark.asyncio +async def test_terminal_materialization_invalidates_before_index_completes( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """A synced publication advances Redis while the following index is still blocked.""" + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation(redis_cache, project_external_id) + entity, accepted = await _accepted_note(session_maker, test_project) + indexer = BlockingIndexFileExecutor(entity) + provider = LocalNoteContentMaterializationProvider( + session_maker=session_maker, + file_service=FileService(Path(test_project.path)), + project_external_id=project_external_id, + read_cache=redis_cache.cache, + file_indexer=indexer, + test_mode=True, + ) + + materialization = asyncio.create_task(provider.materialize_write_change(accepted)) + async with asyncio.timeout(5): + await indexer.started.wait() + + try: + content_repository = NoteContentRepository(project_id=test_project.id) + async with db.scoped_session(session_maker) as session: + note_content = await content_repository.get_by_entity_id(session, entity.id) + generation_after_publication = await _current_generation( + redis_cache, + project_external_id, + ) + finally: + indexer.release.set() + await materialization + + assert note_content is not None + assert note_content.file_write_status == "synced" + assert generation_after_publication != generation_before + generation_after_index = await _current_generation(redis_cache, project_external_id) + assert generation_after_index != generation_after_publication diff --git a/test-int/read_cache/test_read_cache_benchmark.py b/test-int/read_cache/test_read_cache_benchmark.py new file mode 100644 index 000000000..996ad9c35 --- /dev/null +++ b/test-int/read_cache/test_read_cache_benchmark.py @@ -0,0 +1,107 @@ +"""Small end-to-end benchmark for warmed entity reads.""" + +from __future__ import annotations + +from statistics import median +from time import perf_counter +from typing import Protocol + +import pytest +from fastapi import FastAPI +from httpx import AsyncClient +from redis.asyncio import Redis +from sqlalchemy import event +from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker + +from basic_memory.deps import get_read_cache +from basic_memory.models import Project +from basic_memory.read_cache.redis import RedisReadCache + + +class RedisCacheHarness(Protocol): + """Structural type for the real Redis fixture.""" + + cache: RedisReadCache + client: Redis + namespace: str + prefix: str + + +pytestmark = [pytest.mark.redis, pytest.mark.benchmark] + + +async def _entity_read_latencies( + client: AsyncClient, + url: str, + *, + samples: int, +) -> list[float]: + latencies: list[float] = [] + for _ in range(samples): + started = perf_counter() + response = await client.get(url) + latencies.append(perf_counter() - started) + assert response.status_code == 200 + return latencies + + +@pytest.mark.asyncio +async def test_benchmark_warmed_entity_reads_reduce_authoritative_queries( + app: FastAPI, + client: AsyncClient, + test_project: Project, + redis_cache: RedisCacheHarness, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], +) -> None: + """Report HTTP latency and prove cache hits remove authoritative SQL work.""" + project_external_id = str(test_project.external_id) + project_url = f"/v2/projects/{project_external_id}" + created_response = await client.post( + f"{project_url}/knowledge/entities", + json={ + "title": "Redis Benchmark Note", + "directory": "cache", + "content": "# Redis Benchmark Note\n\nRepeated read benchmark.", + }, + ) + assert created_response.status_code == 202 + entity_url = f"{project_url}/knowledge/entities/{created_response.json()['external_id']}" + + query_count = 0 + + def count_query(*_: object) -> None: + nonlocal query_count + query_count += 1 + + engine, _ = engine_factory + event.listen(engine.sync_engine, "before_cursor_execute", count_query) + try: + app.dependency_overrides[get_read_cache] = lambda: None + await client.get(entity_url) + query_count = 0 + authoritative_latencies = await _entity_read_latencies( + client, + entity_url, + samples=20, + ) + authoritative_queries = query_count + + app.dependency_overrides[get_read_cache] = lambda: redis_cache.cache + await client.get(entity_url) + query_count = 0 + cached_latencies = await _entity_read_latencies( + client, + entity_url, + samples=20, + ) + cached_queries = query_count + finally: + event.remove(engine.sync_engine, "before_cursor_execute", count_query) + + assert cached_queries < authoritative_queries + print( + "\nRedis entity-read benchmark: " + f"authoritative_median_ms={median(authoritative_latencies) * 1_000:.3f}, " + f"cached_median_ms={median(cached_latencies) * 1_000:.3f}, " + f"authoritative_sql={authoritative_queries}, cached_sql={cached_queries}" + ) diff --git a/test-int/read_cache/test_redis_read_cache.py b/test-int/read_cache/test_redis_read_cache.py new file mode 100644 index 000000000..0c3e6758c --- /dev/null +++ b/test-int/read_cache/test_redis_read_cache.py @@ -0,0 +1,887 @@ +"""Integration coverage for the Redis semantic read cache.""" + +from __future__ import annotations + +import asyncio +import socket +from typing import override, Protocol +from uuid import uuid4 + +import pytest +from pydantic import BaseModel, ValidationError +from redis.asyncio import Redis + +from basic_memory.read_cache import ( + ModelReadCache, + ReadCacheDataError, + ReadCacheInvalidationStatus, + ReadCacheKey, + ReadCacheLookup, + ReadCacheOperation, + ReadCacheStoreStatus, + ReadCacheUnavailable, + finish_project_read_cache_invalidation, + invalidate_cache, + invalidate_project_read_cache, + read_cache_request_digest, +) +from basic_memory.read_cache.keys import ( + redis_read_cache_generation_key, + redis_read_cache_keys, +) +from basic_memory.read_cache.redis import ( + RedisReadCache, + _required_bytes, + _store_status, + create_redis_read_cache_client, +) + + +class RedisCacheHarness(Protocol): + """Structural type for the real Redis fixture.""" + + cache: RedisReadCache + client: Redis + namespace: str + prefix: str + + +class CachedEntity(BaseModel): + """Small typed boundary value used by read-through tests.""" + + external_id: str + title: str + + +class CachedResolution(BaseModel): + """A second boundary type used to prove facade-local serialization.""" + + external_id: str + resolution_method: str + + +pytestmark = pytest.mark.redis + +PROJECT_ID = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa" +OTHER_PROJECT_ID = "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb" + + +def _key( + *, + project_id: str = PROJECT_ID, + operation: ReadCacheOperation = ReadCacheOperation.entity, + request: str = "entity-1", +) -> ReadCacheKey: + return ReadCacheKey( + project_id=project_id, + operation=operation, + request_digest=read_cache_request_digest(request), + ) + + +def _model_cache( + backend: RedisReadCache, + *, + ttl_seconds: int = 60, + max_payload_bytes: int = 1_024, +) -> ModelReadCache[CachedEntity]: + return ModelReadCache( + backend=backend, + model_type=CachedEntity, + ttl_seconds=ttl_seconds, + max_payload_bytes=max_payload_bytes, + ) + + +@pytest.mark.asyncio +async def test_round_trip_and_ttl_expiry(redis_cache: RedisCacheHarness) -> None: + key = _key() + + miss = await redis_cache.cache.lookup(key) + assert miss.generation is not None + assert not miss.is_hit + + await redis_cache.cache.store(key, miss, b"cached entity", ttl_seconds=1) + hit = await redis_cache.cache.lookup(key) + assert hit.is_hit + assert hit.payload == b"cached entity" + assert hit.generation == miss.generation + + await asyncio.sleep(1.1) + expired = await redis_cache.cache.lookup(key) + assert not expired.is_hit + assert expired.generation == miss.generation + + +@pytest.mark.asyncio +async def test_generation_metadata_expires_and_tracks_response_lifetime( + redis_cache: RedisCacheHarness, +) -> None: + """Inactive projects cannot leave permanent generation metadata in Redis.""" + cache = RedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + generation_ttl_seconds=1, + ) + long_lived_key = _key(request="long-lived-generation") + redis_keys = redis_read_cache_keys( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + key=long_lived_key, + ) + + # Existing deployments may already have persistent generation keys. A + # lookup migrates them to bounded metadata without changing the token. + legacy_generation = b"0" * 32 + await redis_cache.client.set(redis_keys.generation_key, legacy_generation) + lookup = await cache.lookup(long_lived_key) + assert lookup.generation == legacy_generation.decode("ascii") + assert await redis_cache.client.pttl(redis_keys.generation_key) > 0 + + stored = await cache.store( + long_lived_key, + lookup, + b"long-lived payload", + ttl_seconds=3, + ) + generation_ttl = await redis_cache.client.pttl(redis_keys.generation_key) + data_ttl = await redis_cache.client.pttl(redis_keys.data_key) + assert stored is ReadCacheStoreStatus.stored + assert generation_ttl >= data_ttl > 2_000 + + short_lived_key = _key(request="short-lived-generation") + short_lookup = await cache.lookup(short_lived_key) + await cache.store( + short_lived_key, + short_lookup, + b"short-lived payload", + ttl_seconds=1, + ) + assert await redis_cache.client.pttl(redis_keys.generation_key) > 2_000 + + generation_before_invalidation = await redis_cache.client.get(redis_keys.generation_key) + await cache.invalidate_project(PROJECT_ID) + generation_after_invalidation = await redis_cache.client.get(redis_keys.generation_key) + invalidated_ttl = await redis_cache.client.pttl(redis_keys.generation_key) + assert generation_before_invalidation is not None + assert isinstance(generation_after_invalidation, bytes) + assert generation_after_invalidation != generation_before_invalidation + assert 0 < invalidated_ttl <= 1_000 + + await asyncio.sleep(1.1) + assert not await redis_cache.client.exists(redis_keys.generation_key) + after_expiry = await cache.lookup(long_lived_key) + assert not after_expiry.is_hit + assert after_expiry.generation != generation_after_invalidation.decode("ascii") + + +@pytest.mark.asyncio +async def test_cache_identity_isolates_every_key_dimension( + redis_cache: RedisCacheHarness, +) -> None: + key = _key() + miss = await redis_cache.cache.lookup(key) + await redis_cache.cache.store(key, miss, b"only this key", ttl_seconds=60) + + variants = [ + _key(project_id=OTHER_PROJECT_ID), + _key(operation=ReadCacheOperation.resolve), + _key(request="entity-2"), + ] + for variant in variants: + assert not (await redis_cache.cache.lookup(variant)).is_hit + + other_namespace = RedisReadCache( + client=redis_cache.client, + namespace="another-tenant", + prefix=redis_cache.prefix, + ) + assert not (await other_namespace.lookup(key)).is_hit + await other_namespace.invalidate_project(key.project_id) + assert (await redis_cache.cache.lookup(key)).payload == b"only this key" + + +@pytest.mark.asyncio +async def test_project_invalidation_rejects_a_concurrent_stale_fill( + redis_cache: RedisCacheHarness, +) -> None: + project_key = _key() + other_project_key = _key(project_id=OTHER_PROJECT_ID) + + stale_lookup = await redis_cache.cache.lookup(project_key) + other_lookup = await redis_cache.cache.lookup(other_project_key) + other_status = await redis_cache.cache.store( + other_project_key, + other_lookup, + b"other project", + ttl_seconds=60, + ) + + invalidation_status = await redis_cache.cache.invalidate_project(project_key.project_id) + current_lookup = await redis_cache.cache.lookup(project_key) + current_status = await redis_cache.cache.store( + project_key, + current_lookup, + b"current fill", + ttl_seconds=60, + ) + stale_status = await redis_cache.cache.store( + project_key, + stale_lookup, + b"stale fill", + ttl_seconds=60, + ) + + assert other_status is ReadCacheStoreStatus.stored + assert invalidation_status is ReadCacheInvalidationStatus.invalidated + assert current_status is ReadCacheStoreStatus.stored + assert stale_status is ReadCacheStoreStatus.superseded + assert (await redis_cache.cache.lookup(project_key)).payload == b"current fill" + assert (await redis_cache.cache.lookup(other_project_key)).payload == b"other project" + + +@pytest.mark.asyncio +async def test_invalidation_context_invalidates_real_generation_on_exit( + redis_cache: RedisCacheHarness, +) -> None: + key = _key(request="context-normal-exit") + await redis_cache.cache.lookup(key) + generation_key = redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=PROJECT_ID, + ) + generation_before = await redis_cache.client.get(generation_key) + + async with invalidate_cache(redis_cache.cache, PROJECT_ID): + assert await redis_cache.client.get(generation_key) == generation_before + + generation_after = await redis_cache.client.get(generation_key) + assert generation_before is not None + assert generation_after is not None + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_invalidation_context_invalidates_real_generation_after_body_error( + redis_cache: RedisCacheHarness, +) -> None: + key = _key(request="context-error-exit") + await redis_cache.cache.lookup(key) + generation_key = redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=PROJECT_ID, + ) + generation_before = await redis_cache.client.get(generation_key) + + with pytest.raises(RuntimeError, match="operation failed"): + async with invalidate_cache(redis_cache.cache, PROJECT_ID): + raise RuntimeError("operation failed") + + generation_after = await redis_cache.client.get(generation_key) + assert generation_before is not None + assert generation_after is not None + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_cancelled_invalidation_records_real_redis_cleanup_failure( + redis_cache: RedisCacheHarness, +) -> None: + """Preserve cancellation while attaching a failure from the shielded cleanup.""" + + class FailingAfterRealInvalidation(RedisReadCache): + def __init__(self) -> None: + super().__init__( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + self.invalidation_started = asyncio.Event() + self.release_invalidation = asyncio.Event() + + @override + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + self.invalidation_started.set() + await self.release_invalidation.wait() + await super().invalidate_project(project_id) + raise ReadCacheDataError("cleanup failed after real invalidation") + + key = _key(request="cancelled-cleanup-failure") + await redis_cache.cache.lookup(key) + generation_key = redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=PROJECT_ID, + ) + generation_before = await redis_cache.client.get(generation_key) + cache = FailingAfterRealInvalidation() + invalidation = asyncio.create_task(finish_project_read_cache_invalidation(cache, PROJECT_ID)) + + await cache.invalidation_started.wait() + invalidation.cancel() + await asyncio.sleep(0) + assert not invalidation.done() + cache.release_invalidation.set() + + with pytest.raises(asyncio.CancelledError) as exc_info: + await invalidation + + generation_after = await redis_cache.client.get(generation_key) + assert generation_before is not None + assert generation_after is not None + assert generation_after != generation_before + assert any( + "cleanup failed after real invalidation" in note + for note in getattr(exc_info.value, "__notes__", ()) + ) + + +@pytest.mark.asyncio +async def test_child_cancelled_after_real_invalidation_is_recorded( + redis_cache: RedisCacheHarness, +) -> None: + """A cancelled cleanup task remains visible on the propagated cancellation.""" + + class CancellingAfterRealInvalidation(RedisReadCache): + @override + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + await super().invalidate_project(project_id) + raise asyncio.CancelledError("cleanup task cancelled") + + key = _key(request="child-cancelled-after-invalidation") + await redis_cache.cache.lookup(key) + generation_key = redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=PROJECT_ID, + ) + generation_before = await redis_cache.client.get(generation_key) + cache = CancellingAfterRealInvalidation( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + + with pytest.raises(asyncio.CancelledError) as exc_info: + await finish_project_read_cache_invalidation(cache, PROJECT_ID) + + generation_after = await redis_cache.client.get(generation_key) + assert generation_before is not None + assert generation_after is not None + assert generation_after != generation_before + assert any( + "read-cache invalidation task was cancelled" in note + for note in getattr(exc_info.value, "__notes__", ()) + ) + + +@pytest.mark.asyncio +async def test_lost_generation_key_cannot_revive_old_data( + redis_cache: RedisCacheHarness, +) -> None: + key = _key() + miss = await redis_cache.cache.lookup(key) + await redis_cache.cache.store(key, miss, b"old data", ttl_seconds=60) + redis_keys = redis_read_cache_keys( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + key=key, + ) + + await redis_cache.client.delete(redis_keys.generation_key) + after_eviction = await redis_cache.cache.lookup(key) + + assert not after_eviction.is_hit + assert after_eviction.generation != miss.generation + + +@pytest.mark.asyncio +async def test_invalidation_leaves_unrelated_redis_data_untouched( + redis_cache: RedisCacheHarness, +) -> None: + unrelated_key = f"unrelated:{uuid4().hex}" + await redis_cache.client.set(unrelated_key, b"keep") + try: + await redis_cache.cache.invalidate_project(PROJECT_ID) + assert await redis_cache.client.get(unrelated_key) == b"keep" + finally: + await redis_cache.client.delete(unrelated_key) + + +@pytest.mark.asyncio +async def test_corrupt_redis_values_fail_fast(redis_cache: RedisCacheHarness) -> None: + key = _key() + await redis_cache.cache.lookup(key) + redis_keys = redis_read_cache_keys( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + key=key, + ) + + await redis_cache.client.set(redis_keys.data_key, b"missing envelope") + with pytest.raises(ReadCacheDataError, match="invalid generation envelope"): + await redis_cache.cache.lookup(key) + + await redis_cache.client.set(redis_keys.data_key, b"\npayload") + with pytest.raises(ReadCacheDataError, match="invalid generation envelope"): + await redis_cache.cache.lookup(key) + + await redis_cache.client.set(redis_keys.generation_key, b"\xff") + with pytest.raises(ReadCacheDataError, match="invalid generation token"): + await redis_cache.cache.lookup(key) + + await redis_cache.client.set(redis_keys.generation_key, b"abcd") + with pytest.raises(ReadCacheDataError, match="invalid generation token"): + await redis_cache.cache.lookup(key) + + +@pytest.mark.asyncio +async def test_decode_responses_client_remains_compatible(redis_url: str) -> None: + prefix = f"bm:test:read:{uuid4().hex}" + client = Redis.from_url(redis_url, decode_responses=True) + cache = RedisReadCache( + client=client, + namespace="decoded-client", + prefix=prefix, + ) + key = _key() + try: + miss = await cache.lookup(key) + await cache.store(key, miss, b"text payload", ttl_seconds=60) + assert (await cache.lookup(key)).payload == b"text payload" + finally: + keys = [key async for key in client.scan_iter(match=f"{prefix}:*")] + if keys: + await client.delete(*keys) + await client.aclose() + + +@pytest.mark.asyncio +async def test_unavailable_redis_is_explicit_for_every_operation() -> None: + with socket.socket() as reserved_port: + reserved_port.bind(("127.0.0.1", 0)) + port = reserved_port.getsockname()[1] + + client = create_redis_read_cache_client( + f"redis://127.0.0.1:{port}/0", + socket_timeout=0.05, + ) + cache = RedisReadCache(client=client, namespace="unavailable") + key = _key() + try: + with pytest.raises(ReadCacheUnavailable, match="lookup failed"): + await cache.lookup(key) + with pytest.raises(ReadCacheUnavailable, match="store failed"): + await cache.store( + key, + ReadCacheLookup(generation="0" * 32), + b"payload", + ttl_seconds=60, + ) + with pytest.raises(ReadCacheUnavailable, match="invalidation failed"): + await cache.invalidate_project(key.project_id) + finally: + await client.aclose() + + +@pytest.mark.asyncio +async def test_real_redis_capacity_failures_are_cache_unavailable( + redis_cache: RedisCacheHarness, +) -> None: + """A no-eviction maxmemory error cannot fail reads or committed-write invalidation.""" + key = _key(request="capacity-limited-store") + original_maxmemory = str((await redis_cache.client.config_get("maxmemory"))["maxmemory"]) + original_policy = str( + (await redis_cache.client.config_get("maxmemory-policy"))["maxmemory-policy"] + ) + await redis_cache.cache.lookup(key) + read_cache = _model_cache(redis_cache.cache) + + try: + await redis_cache.client.config_set("maxmemory-policy", "noeviction") + await redis_cache.client.config_set("maxmemory", "1") + + async with read_cache.read(key=key) as cached: + assert cached.value is None + cached.value = CachedEntity( + external_id="entity-1", + title="Authoritative", + ) + result = cached.value + invalidation_status = await invalidate_project_read_cache( + redis_cache.cache, + PROJECT_ID, + ) + finally: + await redis_cache.client.config_set("maxmemory", original_maxmemory) + await redis_cache.client.config_set("maxmemory-policy", original_policy) + + assert result.title == "Authoritative" + assert invalidation_status is ReadCacheInvalidationStatus.unavailable + + +@pytest.mark.asyncio +async def test_typed_read_through_uses_real_cached_representation( + redis_cache: RedisCacheHarness, +) -> None: + loads = 0 + read_cache = _model_cache(redis_cache.cache) + + results: list[CachedEntity] = [] + for _ in range(2): + async with read_cache.read(key=_key()) as cached: + if cached.value is None: + loads += 1 + cached.value = CachedEntity( + external_id="entity-1", + title="First", + ) + results.append(cached.value) + + first, second = results + assert first == CachedEntity(external_id="entity-1", title="First") + assert second == first + assert loads == 1 + + +@pytest.mark.asyncio +async def test_typed_facades_share_backend_and_keep_model_types_local( + redis_cache: RedisCacheHarness, +) -> None: + entity_cache = _model_cache(redis_cache.cache) + resolution_cache = ModelReadCache( + backend=redis_cache.cache, + model_type=CachedResolution, + ttl_seconds=60, + max_payload_bytes=1_024, + ) + entity_key = _key(request="typed-entity") + resolution_key = _key( + operation=ReadCacheOperation.resolve, + request="typed-resolution", + ) + + async with entity_cache.read(key=entity_key) as cached_entity: + cached_entity.value = CachedEntity(external_id="entity-1", title="First") + async with resolution_cache.read(key=resolution_key) as cached_resolution: + cached_resolution.value = CachedResolution( + external_id="entity-1", + resolution_method="permalink", + ) + + async with entity_cache.read(key=entity_key) as cached_entity: + assert cached_entity.value == CachedEntity(external_id="entity-1", title="First") + async with resolution_cache.read(key=resolution_key) as cached_resolution: + assert cached_resolution.value == CachedResolution( + external_id="entity-1", + resolution_method="permalink", + ) + + assert entity_cache.backend is redis_cache.cache + assert resolution_cache.backend is redis_cache.cache + + +@pytest.mark.asyncio +async def test_typed_read_through_does_not_cache_oversize_models( + redis_cache: RedisCacheHarness, +) -> None: + loads = 0 + read_cache = _model_cache(redis_cache.cache, max_payload_bytes=1) + + for _ in range(2): + async with read_cache.read(key=_key()) as cached: + assert cached.value is None + loads += 1 + cached.value = CachedEntity( + external_id="entity-1", + title="Too large", + ) + + assert loads == 2 + + +@pytest.mark.asyncio +async def test_typed_read_through_does_not_cache_ineligible_models( + redis_cache: RedisCacheHarness, +) -> None: + loads = 0 + read_cache = _model_cache(redis_cache.cache) + + for _ in range(2): + async with read_cache.read( + key=_key(operation=ReadCacheOperation.resolve), + ) as cached: + assert cached.value is None + loads += 1 + cached.cacheable = False + cached.value = CachedEntity( + external_id="cross-project", + title="Other tenant", + ) + + assert loads == 2 + + +@pytest.mark.asyncio +async def test_typed_read_through_does_not_store_after_body_error( + redis_cache: RedisCacheHarness, +) -> None: + key = _key(request="failed-authoritative-read") + read_cache = _model_cache(redis_cache.cache) + + with pytest.raises(RuntimeError, match="authoritative read failed"): + async with read_cache.read(key=key) as cached: + cached.value = CachedEntity( + external_id="entity-1", + title="Must not be stored", + ) + raise RuntimeError("authoritative read failed") + + assert not (await redis_cache.cache.lookup(key)).is_hit + + +@pytest.mark.asyncio +async def test_typed_read_through_rejects_invalid_cached_models( + redis_cache: RedisCacheHarness, +) -> None: + key = _key() + miss = await redis_cache.cache.lookup(key) + await redis_cache.cache.store(key, miss, b'{"wrong":"shape"}', ttl_seconds=60) + read_cache = _model_cache(redis_cache.cache) + + with pytest.raises(ValidationError): + async with read_cache.read(key=key): + raise AssertionError("invalid cache data must not enter the read scope") + + +@pytest.mark.asyncio +async def test_typed_read_through_bypasses_unavailable_real_redis() -> None: + with socket.socket() as reserved_port: + reserved_port.bind(("127.0.0.1", 0)) + port = reserved_port.getsockname()[1] + + client = create_redis_read_cache_client( + f"redis://127.0.0.1:{port}/0", + socket_timeout=0.05, + ) + cache = RedisReadCache(client=client, namespace="unavailable") + read_cache = _model_cache(cache) + + try: + async with read_cache.read(key=_key()) as cached: + assert cached.value is None + cached.value = CachedEntity( + external_id="entity-1", + title="Authoritative", + ) + result = cached.value + finally: + await client.aclose() + + assert result.title == "Authoritative" + + +@pytest.mark.asyncio +async def test_invalidation_helper_preserves_committed_write_when_redis_is_unavailable() -> None: + with socket.socket() as reserved_port: + reserved_port.bind(("127.0.0.1", 0)) + port = reserved_port.getsockname()[1] + + client = create_redis_read_cache_client( + f"redis://127.0.0.1:{port}/0", + socket_timeout=0.05, + ) + cache = RedisReadCache(client=client, namespace="unavailable") + try: + status = await invalidate_project_read_cache(cache, PROJECT_ID) + finally: + await client.aclose() + + assert status is ReadCacheInvalidationStatus.unavailable + + +@pytest.mark.asyncio +async def test_typed_read_through_returns_data_when_real_redis_store_times_out( + redis_cache: RedisCacheHarness, + redis_url: str, +) -> None: + prefix = f"bm:test:read:{uuid4().hex}" + client = create_redis_read_cache_client(redis_url, socket_timeout=0.05) + cache = RedisReadCache( + client=client, + namespace="paused-store", + prefix=prefix, + ) + read_cache = _model_cache(cache) + + try: + async with read_cache.read(key=_key()) as cached: + assert cached.value is None + await redis_cache.client.execute_command("CLIENT", "PAUSE", 200, "WRITE") + cached.value = CachedEntity( + external_id="entity-1", + title="Authoritative", + ) + result = cached.value + assert result.title == "Authoritative" + finally: + await asyncio.sleep(0.25) + keys = [key async for key in client.scan_iter(match=f"{prefix}:*")] + if keys: + await client.delete(*keys) + await client.aclose() + + +def test_typed_read_through_validates_policy_before_loading( + redis_cache: RedisCacheHarness, +) -> None: + with pytest.raises(ValueError, match="ttl_seconds"): + _model_cache( + redis_cache.cache, + ttl_seconds=0, + max_payload_bytes=1, + ) + with pytest.raises(ValueError, match="max_payload_bytes"): + _model_cache( + redis_cache.cache, + ttl_seconds=1, + max_payload_bytes=0, + ) + + +@pytest.mark.asyncio +async def test_typed_read_through_requires_an_authoritative_result( + redis_cache: RedisCacheHarness, +) -> None: + read_cache = _model_cache(redis_cache.cache) + with pytest.raises(RuntimeError, match="exited without a result"): + async with read_cache.read(key=_key(request="missing-authoritative-result")): + pass + + +def test_key_validation_and_canonicalization() -> None: + assert read_cache_request_digest("ab", "c") != read_cache_request_digest("a", "bc") + assert read_cache_request_digest("same") == read_cache_request_digest("same") + + key = _key() + assert _key(project_id=PROJECT_ID.upper()).project_id == PROJECT_ID + generation_key = redis_read_cache_generation_key( + prefix="bm:read:v1", + namespace="tenant", + project_id=key.project_id, + ) + redis_keys = redis_read_cache_keys( + prefix="bm:read:v1", + namespace="tenant", + key=key, + ) + assert redis_keys.generation_key == generation_key + assert f"{{{read_cache_request_digest('tenant', key.project_id)}}}" in redis_keys.data_key + assert generation_key == redis_read_cache_generation_key( + prefix="bm:read:v1", + namespace="tenant", + project_id=PROJECT_ID.upper(), + ) + assert generation_key == redis_read_cache_generation_key( + prefix="bm:read:v1", + namespace=" tenant ", + project_id=PROJECT_ID, + ) + + with pytest.raises(ValueError, match="project_id"): + _key(project_id="") + with pytest.raises(ValueError, match="valid UUID"): + _key(project_id="not-a-uuid") + with pytest.raises(ValueError, match="SHA-256"): + ReadCacheKey( + project_id=PROJECT_ID, + operation=ReadCacheOperation.entity, + request_digest="short", + ) + with pytest.raises(ValueError, match="SHA-256"): + ReadCacheKey( + project_id=PROJECT_ID, + operation=ReadCacheOperation.entity, + request_digest="z" * 64, + ) + with pytest.raises(ValueError, match="SHA-256"): + ReadCacheKey( + project_id=PROJECT_ID, + operation=ReadCacheOperation.entity, + request_digest=("0" * 62) + " ", + ) + uppercase_digest = read_cache_request_digest("uppercase").upper() + assert ( + ReadCacheKey( + project_id=PROJECT_ID, + operation=ReadCacheOperation.entity, + request_digest=uppercase_digest, + ).request_digest + == uppercase_digest.lower() + ) + with pytest.raises(ValueError, match="generation"): + ReadCacheLookup(generation="", payload=b"orphaned") + with pytest.raises(ValueError, match="prefix"): + redis_read_cache_generation_key( + prefix="", + namespace="tenant", + project_id=PROJECT_ID, + ) + with pytest.raises(ValueError, match="prefix"): + redis_read_cache_generation_key( + prefix="bad prefix", + namespace="tenant", + project_id=PROJECT_ID, + ) + with pytest.raises(ValueError, match="namespace"): + redis_read_cache_generation_key( + prefix="bm:read:v1", + namespace="", + project_id=PROJECT_ID, + ) + with pytest.raises(ValueError, match="project_id"): + redis_read_cache_generation_key(prefix="bm:read:v1", namespace="tenant", project_id="") + + +@pytest.mark.asyncio +async def test_invalid_store_inputs_fail_before_redis( + redis_cache: RedisCacheHarness, +) -> None: + key = _key() + + with pytest.raises(ValueError, match="positive"): + await redis_cache.cache.store( + key, + ReadCacheLookup(generation="0" * 32), + b"payload", + ttl_seconds=0, + ) + with pytest.raises(ReadCacheDataError, match="generation token"): + await redis_cache.cache.store( + key, + ReadCacheLookup(generation="invalid"), + b"payload", + ttl_seconds=60, + ) + with pytest.raises(ValueError, match="namespace"): + RedisReadCache(client=redis_cache.client, namespace="") + with pytest.raises(ValueError, match="namespace"): + RedisReadCache(client=redis_cache.client, namespace=" ") + with pytest.raises(ValueError, match="generation_ttl_seconds"): + RedisReadCache( + client=redis_cache.client, + namespace="tenant", + generation_ttl_seconds=0, + ) + with pytest.raises(ValueError, match="project_id"): + await redis_cache.cache.invalidate_project("") + + +def test_required_bytes_rejects_non_string_values() -> None: + with pytest.raises(ReadCacheDataError, match="invalid test value"): + _required_bytes(1, field="test") + with pytest.raises(ReadCacheDataError, match="invalid cache store result"): + _store_status(2) diff --git a/test-int/read_cache/test_runtime_invalidation.py b/test-int/read_cache/test_runtime_invalidation.py new file mode 100644 index 000000000..14fefdd35 --- /dev/null +++ b/test-int/read_cache/test_runtime_invalidation.py @@ -0,0 +1,1578 @@ +"""Real Redis coverage for non-request cache invalidation boundaries.""" + +from __future__ import annotations + +import asyncio +from collections.abc import AsyncIterator, Mapping, Sequence +from contextlib import asynccontextmanager +from datetime import UTC, datetime +from hashlib import sha256 +from pathlib import Path +from typing import Literal, Protocol, cast, override + +import pytest +from redis.asyncio import Redis +from sqlalchemy.ext.asyncio import AsyncEngine, AsyncSession, async_sessionmaker + +import basic_memory.indexing.external_file_delete_runner as external_file_delete_runner +import basic_memory.indexing.index_file_runner as index_file_runner +import basic_memory.services.directory_deletes as directory_deletes +import basic_memory.services.entity_service as entity_service_module +from basic_memory import db +from basic_memory.config import ProjectConfig, BasicMemoryConfig +from basic_memory.index import note_content_materialization +from basic_memory.index.local_moves import ( + LocalMoveEntityRepository, + LocalWatchMoveProcessor, +) +from basic_memory.index.local_dependencies import LocalIndexSearchService +from basic_memory.index.local_schedulers import ( + LocalSearchReindexScheduler, + drain_background_tasks, +) +from basic_memory.index.local_project import ( + LocalProjectIndexBatchEnqueuer, + LocalProjectIndexRuntime, + run_local_project_index, +) +from basic_memory.index.local_runtime import LocalInlineStorageEventResultRecorder +from basic_memory.indexing.change_planning import ChangeReport +from basic_memory.indexing.directory_delete_runner import ( + DirectoryDeleteRuntime, + RepositoryDirectoryDeleteAcceptanceStore, +) +from basic_memory.indexing.file_batch_runner import ( + IndexFileBatchChecker, + IndexFileBatchContentClassifier, + IndexFileBatchIndexer, + IndexFileBatchReader, +) +from basic_memory.indexing.external_file_delete_runner import ( + ExternalFileDeleteResult, + InvalidatingExternalFileDeleteEntities, + RepositoryExternalFileDeleteEntities, +) +from basic_memory.indexing.models import IndexFileJobResult, IndexFileJobStatus, IndexInputFile +from basic_memory.indexing.models import FileIndexResult +from basic_memory.indexing.index_file_runner import InvalidatingIndexFileExecutor +from basic_memory.indexing.project_index_maintenance import ( + InvalidatingProjectIndexBatchStore, + ProjectIndexDeleteBatch, + ProjectIndexDeleteBatchResult, + ProjectIndexDeleteRun, + ProjectIndexMoveBatch, + ProjectIndexMoveBatchResult, + ProjectIndexMoveRun, + ProjectIndexMovedEntitySearchRefresher, + StoreProjectIndexMaintenanceRunner, +) +from basic_memory.indexing.relation_resolution import RelationResolutionRuntime +from basic_memory.markdown import EntityParser +from basic_memory.markdown.markdown_processor import MarkdownProcessor +from basic_memory.models import Project +from basic_memory.models.knowledge import Entity +from basic_memory.read_cache import ( + ReadCacheInvalidationStatus, + ReadCacheKey, + ReadCacheOperation, + read_cache_request_digest, +) +from basic_memory.read_cache.keys import redis_read_cache_generation_key +from basic_memory.read_cache.redis import RedisReadCache +from basic_memory.repository import ( + EntityRepository, + ObservationRepository, + RelationRepository, +) +from basic_memory.repository.note_content_repository import ( + AcceptedNoteContentWrite, + NoteContentRepository, +) +from basic_memory.repository.note_file_vacate_repository import NoteFileVacateRepository +from basic_memory.runtime.cleanup import ( + RuntimeExternalFileDeletePlan, + RuntimeFileDeleteResult, + RuntimeNoteFileDeleteJobRequest, +) +from basic_memory.runtime.jobs import ( + RuntimeIndexFileBatchJobRequest, + RuntimeObservedIndexFile, + RuntimeProjectIndexJobRequest, +) +from basic_memory.runtime.projects import ProjectRuntimeReference +from basic_memory.runtime.storage import ( + STORAGE_OBJECT_DELETED_EVENT, + StorageEventPayload, + StorageObjectIdentity, + StorageObjectVersion, + RuntimeStorageEventOperation, + RuntimeStorageEventOperationKind, +) +from basic_memory.schemas import Entity as EntitySchema +from basic_memory.services.directory_deletes import DirectoryDeleteService +from basic_memory.services.entity_service import EntityService +from basic_memory.services.file_service import FileService +from basic_memory.services.initialization import recover_project_materializations +from basic_memory.services.link_resolver import LinkResolver +from basic_memory.services.search_service import SearchService + +pytestmark = pytest.mark.redis + + +class RedisCacheHarness(Protocol): + cache: RedisReadCache + client: Redis + namespace: str + prefix: str + + +async def _current_generation( + redis_cache: RedisCacheHarness, + project_external_id: str, +) -> bytes | str: + generation = await redis_cache.client.get( + redis_read_cache_generation_key( + prefix=redis_cache.prefix, + namespace=redis_cache.namespace, + project_id=project_external_id, + ) + ) + assert generation is not None + return generation + + +class DetectedMoveProcessor(LocalWatchMoveProcessor): + """Exercise move completion without coupling the test to detection I/O.""" + + @override + async def detect_moves( + self, + events: Sequence[StorageEventPayload], + ) -> tuple[dict[str, str], set[int]]: + del events + return {"notes/old.md": "notes/new.md"}, {0, 1} + + @override + async def detect_transient_missing_events( + self, + events: Sequence[StorageEventPayload], + *, + exclude_indexes: set[int], + ) -> set[int]: + del events, exclude_indexes + return set() + + @override + async def detect_missing_entity_delete_events( + self, + events: Sequence[StorageEventPayload], + *, + exclude_indexes: set[int], + ) -> set[int]: + del events, exclude_indexes + return set() + + +class DetectedMoveBatchProcessor(DetectedMoveProcessor): + """Exercise multiple durable watcher move batches without detection I/O.""" + + @override + async def detect_moves( + self, + events: Sequence[StorageEventPayload], + ) -> tuple[dict[str, str], set[int]]: + del events + return { + "notes/old-1.md": "notes/new-1.md", + "notes/old-2.md": "notes/new-2.md", + }, {0, 1, 2, 3} + + +class RecordingMoveMaintenance: + def __init__(self) -> None: + self.calls: list[tuple[dict[str, str], int]] = [] + + async def run_move_batches( + self, + *, + moved_files: Mapping[str, str], + batch_size: int, + ) -> ProjectIndexMoveRun: + self.calls.append((dict(moved_files), batch_size)) + return ProjectIndexMoveRun( + total_moves=1, + total_updated_files=1, + records=(), + moved_entity_ids=frozenset({17}), + ) + + async def run_delete_batches( + self, + *, + deleted_paths: Sequence[str], + batch_size: int, + ) -> ProjectIndexDeleteRun: + del deleted_paths, batch_size + raise AssertionError("watcher move completion must not run delete maintenance") + + +class RecordingMovedEntitySearchRefresher: + def __init__(self) -> None: + self.calls: list[list[int]] = [] + + async def refresh_moved_entities(self, entity_ids: Sequence[int]) -> None: + self.calls.append(list(entity_ids)) + + +class EmptyObservedFileSource: + async def list_observed_index_files(self) -> tuple[RuntimeObservedIndexFile, ...]: + return () + + +class DeletedFileChangeDetector: + async def detect_all_changes( + self, + storage_files: Mapping[str, RuntimeObservedIndexFile], + ) -> ChangeReport: + del storage_files + return ChangeReport(deleted_files=["notes/stale.md"]) + + +class FailingDeleteMaintenance: + async def run_move_batches( + self, + *, + moved_files: Mapping[str, str], + batch_size: int, + ) -> ProjectIndexMoveRun: + del moved_files, batch_size + return ProjectIndexMoveRun( + total_moves=0, + total_updated_files=0, + records=(), + ) + + async def run_delete_batches( + self, + *, + deleted_paths: Sequence[str], + batch_size: int, + ) -> ProjectIndexDeleteRun: + del deleted_paths, batch_size + raise RuntimeError("partial project index failure") + + +class UnusedBatchEnqueuer: + async def enqueue_index_file_batch( + self, + request: RuntimeIndexFileBatchJobRequest, + ) -> None: + del request + raise AssertionError("failing maintenance must stop before file batches") + + +class GenerationObservingProjectIndexBatchStore: + """Record the real Redis generation seen before each durable batch.""" + + def __init__( + self, + redis_cache: RedisCacheHarness, + project_external_id: str, + ) -> None: + self.redis_cache = redis_cache + self.project_external_id = project_external_id + self.move_generations: list[bytes | str] = [] + self.delete_generations: list[bytes | str] = [] + + async def apply_project_index_move_batch( + self, + move_batch: ProjectIndexMoveBatch, + ) -> ProjectIndexMoveBatchResult: + self.move_generations.append( + await _current_generation(self.redis_cache, self.project_external_id) + ) + return ProjectIndexMoveBatchResult(updated_files=len(move_batch.targets)) + + async def apply_project_index_delete_batch( + self, + delete_batch: ProjectIndexDeleteBatch, + ) -> ProjectIndexDeleteBatchResult: + self.delete_generations.append( + await _current_generation(self.redis_cache, self.project_external_id) + ) + return ProjectIndexDeleteBatchResult(deleted_entities=len(delete_batch.paths)) + + +class GenerationObservingDirectoryDeleteEnqueuer: + def __init__( + self, + redis_cache: RedisCacheHarness, + project_external_id: str, + ) -> None: + self.redis_cache = redis_cache + self.project_external_id = project_external_id + self.observed_generations: list[bytes | str] = [] + + async def enqueue_directory_file_delete( + self, + request: RuntimeNoteFileDeleteJobRequest, + ) -> RuntimeFileDeleteResult: + generation = await self.redis_cache.client.get( + redis_read_cache_generation_key( + prefix=self.redis_cache.prefix, + namespace=self.redis_cache.namespace, + project_id=self.project_external_id, + ) + ) + assert generation is not None + self.observed_generations.append(generation) + return RuntimeFileDeleteResult.already_absent( + entity_id=request.entity_id, + file_path=request.file_path, + ) + + +class BlockingInvalidationRedisReadCache(RedisReadCache): + """Hold one real invalidation so cancellation can repeat during cleanup.""" + + def __init__( + self, + *, + client: Redis, + namespace: str, + prefix: str, + block_on_call: int = 1, + ) -> None: + super().__init__(client=client, namespace=namespace, prefix=prefix) + self.block_on_call = block_on_call + self.invalidation_calls = 0 + self.invalidation_started = asyncio.Event() + self.release_invalidation = asyncio.Event() + + @override + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + self.invalidation_calls += 1 + if self.invalidation_calls == self.block_on_call: + self.invalidation_started.set() + await self.release_invalidation.wait() + return await super().invalidate_project(project_id) + + +class PartiallyFailingSearchReindexService: + """Observe the generation inside a rebuild that fails after publishing work.""" + + def __init__( + self, + redis_cache: RedisCacheHarness, + project_external_id: str, + ) -> None: + self.redis_cache = redis_cache + self.project_external_id = project_external_id + self.generation_during_reindex: bytes | str | None = None + + async def reindex_all(self) -> None: + self.generation_during_reindex = await _current_generation( + self.redis_cache, + self.project_external_id, + ) + raise RuntimeError("partial search reindex failure") + + +class CommittingIndexFileExecutor: + """Publish one real entity row before the caller cancels transaction exit.""" + + def __init__( + self, + session_maker: async_sessionmaker[AsyncSession], + entity_repository: EntityRepository, + ) -> None: + self.session_maker = session_maker + self.entity_repository = entity_repository + self.entity_id: int | None = None + + async def index_file(self, file_path: str, *, source: str) -> FileIndexResult: + del source + async with db.scoped_session(self.session_maker) as session: + entity = await self.entity_repository.add( + session, + Entity( + title="Cancelled Watcher Index Commit", + note_type="note", + content_type="text/markdown", + file_path=file_path, + checksum="cancelled-watcher-index-commit-checksum", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ), + ) + self.entity_id = entity.id + raise AssertionError("test transaction-exit pause should be cancelled") + + +async def _initialized_generation( + redis_cache: RedisCacheHarness, + project_external_id: str, + *, + request: str, +) -> bytes | str: + await redis_cache.cache.lookup( + ReadCacheKey( + project_id=project_external_id, + operation=ReadCacheOperation.entity, + request_digest=read_cache_request_digest(request), + ) + ) + return await _current_generation(redis_cache, project_external_id) + + +async def _seed_recovery_note( + session_maker: async_sessionmaker[AsyncSession], + project: Project, + *, + title: str, + file_path: str, + markdown_content: str, +) -> Entity: + entity_repository = EntityRepository(project_id=project.id) + content_repository = NoteContentRepository(project_id=project.id) + async with db.scoped_session(session_maker) as session: + entity = await entity_repository.add( + session, + Entity( + title=title, + note_type="note", + content_type="text/markdown", + file_path=file_path, + checksum="entity-checksum-1", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ), + ) + await content_repository.accept_write( + session, + AcceptedNoteContentWrite( + entity_id=entity.id, + markdown_content=markdown_content, + db_version=1, + db_checksum="db-checksum-1", + last_source="api", + updated_at=datetime.now(UTC), + ), + ) + row = await content_repository.select_by_id(session, entity.id) + assert row is not None + row.file_write_status = "writing" + await session.flush() + return entity + + +def _move_event(event_name: str, path: str) -> StorageEventPayload: + return StorageEventPayload( + event_name=event_name, + event_time="2026-07-29T00:00:00Z", + object_version=StorageObjectVersion( + identity=StorageObjectIdentity( + bucket_name="local-filesystem", + key=f"project/{path}", + ), + etag="move-etag", + ), + ) + + +def _index_operation(path: str) -> RuntimeStorageEventOperation: + return RuntimeStorageEventOperation( + kind=RuntimeStorageEventOperationKind.index_file, + storage_event=_move_event("OBJECT_CREATED_PUT", path), + relative_path=path, + ) + + +@pytest.mark.asyncio +async def test_watcher_move_completion_invalidates_real_redis( + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="watcher-move", + ) + maintenance = RecordingMoveMaintenance() + search_refresher = RecordingMovedEntitySearchRefresher() + processor = DetectedMoveProcessor( + session_maker=cast(async_sessionmaker[AsyncSession], object()), + file_service=cast(FileService, object()), + entity_repository=cast(LocalMoveEntityRepository, object()), + maintenance_runner=maintenance, + moved_entity_search_refresher=search_refresher, + project_external_id=project_external_id, + read_cache=redis_cache.cache, + ) + + result = await processor.process_moves( + ( + _move_event(STORAGE_OBJECT_DELETED_EVENT, "notes/old.md"), + _move_event("OBJECT_CREATED_PUT", "notes/new.md"), + ) + ) + + assert result.remaining_events == () + assert result.processed_moves == 1 + assert maintenance.calls == [({"notes/old.md": "notes/new.md"}, 100)] + assert search_refresher.calls == [[17]] + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="watcher-move", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_watcher_move_batches_invalidate_real_redis_before_next_batch( + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Every committed watcher move batch advances Redis before the next batch.""" + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="watcher-move-batches", + ) + observed_store = GenerationObservingProjectIndexBatchStore( + redis_cache, + project_external_id, + ) + invalidating_store = InvalidatingProjectIndexBatchStore( + move_store=observed_store, + delete_store=observed_store, + read_cache=redis_cache.cache, + project_external_id=project_external_id, + ) + processor = DetectedMoveBatchProcessor( + session_maker=cast(async_sessionmaker[AsyncSession], object()), + file_service=cast(FileService, object()), + entity_repository=cast(LocalMoveEntityRepository, object()), + maintenance_runner=StoreProjectIndexMaintenanceRunner( + move_store=invalidating_store, + delete_store=invalidating_store, + ), + moved_entity_search_refresher=RecordingMovedEntitySearchRefresher(), + project_external_id=project_external_id, + read_cache=redis_cache.cache, + batch_size=1, + ) + + result = await processor.process_moves( + ( + _move_event(STORAGE_OBJECT_DELETED_EVENT, "notes/old-1.md"), + _move_event("OBJECT_CREATED_PUT", "notes/new-1.md"), + _move_event(STORAGE_OBJECT_DELETED_EVENT, "notes/old-2.md"), + _move_event("OBJECT_CREATED_PUT", "notes/new-2.md"), + ) + ) + + assert result.remaining_events == () + assert result.processed_moves == 2 + assert observed_store.move_generations[0] == generation_before + assert observed_store.move_generations[1] != observed_store.move_generations[0] + generation_after = await _current_generation(redis_cache, project_external_id) + assert generation_after != observed_store.move_generations[1] + + +@pytest.mark.asyncio +async def test_watcher_index_failure_invalidates_real_redis( + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="watcher-index-failure", + ) + recorder = LocalInlineStorageEventResultRecorder( + project=ProjectRuntimeReference.from_project(test_project), + search_service=cast(LocalIndexSearchService, object()), + relation_cleanup_search_refresher=cast( + ProjectIndexMovedEntitySearchRefresher, + object(), + ), + relation_runtime=cast(RelationResolutionRuntime, object()), + index_embeddings=False, + read_cache=redis_cache.cache, + ) + + await recorder.event_failed( + _index_operation("notes/partial-index.md"), + RuntimeError("partial watcher index failure"), + ) + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="watcher-index-failure", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_partial_search_reindex_invalidates_real_redis( + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """A partial search rebuild cannot retain resolutions filled during the run.""" + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="partial-search-reindex", + ) + search_service = PartiallyFailingSearchReindexService( + redis_cache, + project_external_id, + ) + scheduler = LocalSearchReindexScheduler( + search_service=search_service, + project_external_id=project_external_id, + read_cache=redis_cache.cache, + test_mode=False, + ) + + scheduler.schedule_search_reindex(project_id=test_project.id) + await drain_background_tasks() + + assert search_service.generation_during_reindex == generation_before + generation_after = await _current_generation(redis_cache, project_external_id) + assert generation_after != generation_before + + +@pytest.mark.asyncio +@pytest.mark.parametrize("completion", ["index", "delete"]) +async def test_cancelled_watcher_completion_finishes_real_redis_invalidation( + test_project: Project, + redis_cache: RedisCacheHarness, + completion: Literal["index", "delete"], +) -> None: + """Cancellation after a durable watcher event waits for invalidation.""" + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request=f"cancelled-watcher-{completion}", + ) + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + recorder = LocalInlineStorageEventResultRecorder( + project=ProjectRuntimeReference.from_project(test_project), + search_service=cast(LocalIndexSearchService, object()), + relation_cleanup_search_refresher=cast( + ProjectIndexMovedEntitySearchRefresher, + object(), + ), + relation_runtime=cast(RelationResolutionRuntime, object()), + index_embeddings=False, + read_cache=blocking_cache, + ) + + if completion == "index": + completion_call = recorder.index_file_completed( + _index_operation("notes/cancelled-index.md"), + IndexFileJobResult( + status=IndexFileJobStatus.processed, + reason="file indexed", + entity_id=42, + ), + ) + else: + deleted_entity = Entity( + id=42, + project_id=test_project.id, + external_id="cancelled-watcher-delete", + title="Cancelled watcher delete", + permalink="notes/cancelled-watcher-delete", + note_type="note", + content_type="text/markdown", + file_path="notes/cancelled-delete.md", + checksum="cancelled-watcher-delete-checksum", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ) + completion_call = recorder.delete_file_completed( + RuntimeStorageEventOperation( + kind=RuntimeStorageEventOperationKind.delete_file, + storage_event=_move_event( + STORAGE_OBJECT_DELETED_EVENT, + deleted_entity.file_path, + ), + relative_path=deleted_entity.file_path, + ), + ExternalFileDeleteResult( + plan=RuntimeExternalFileDeletePlan.from_existing_entity( + deleted_entity, + file_path=deleted_entity.file_path, + object_exists=False, + ), + entity_deleted=True, + deleted_entity=deleted_entity, + ), + ) + + completion_task = asyncio.create_task(completion_call) + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + completion_task.cancel() + await asyncio.sleep(0) + completion_task.cancel() + await asyncio.sleep(0) + assert not completion_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await completion_task + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request=f"cancelled-watcher-{completion}", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_cancelled_watcher_delete_commit_finishes_real_redis_invalidation( + monkeypatch: pytest.MonkeyPatch, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Cancellation during watcher delete transaction exit cannot skip invalidation.""" + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-watcher-delete-commit", + ) + entity_repository = EntityRepository(project_id=test_project.id) + file_path = "notes/cancelled-watcher-delete-commit.md" + async with db.scoped_session(session_maker) as session: + entity = await entity_repository.add( + session, + Entity( + title="Cancelled Watcher Delete Commit", + note_type="note", + content_type="text/markdown", + file_path=file_path, + checksum="cancelled-watcher-delete-commit-checksum", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ), + ) + + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + delete_entities = InvalidatingExternalFileDeleteEntities( + entities=RepositoryExternalFileDeleteEntities( + session_maker=session_maker, + entity_repository=entity_repository, + ), + read_cache=blocking_cache, + project_external_id=project_external_id, + ) + found = await delete_entities.find_entity_by_file_path(file_path) + assert found is not None + assert found.id == entity.id + + transaction_committed = asyncio.Event() + hold_after_commit = asyncio.Event() + original_scoped_session = external_file_delete_runner.db.scoped_session + + @asynccontextmanager + async def pause_after_delete_commit( + scoped_session_maker: async_sessionmaker[AsyncSession], + ) -> AsyncIterator[AsyncSession]: + async with original_scoped_session(scoped_session_maker) as session: + yield session + transaction_committed.set() + await hold_after_commit.wait() + + monkeypatch.setattr( + external_file_delete_runner.db, + "scoped_session", + pause_after_delete_commit, + ) + delete_task = asyncio.create_task( + delete_entities.delete_entity_if_file_path_matches( + entity_id=entity.id, + file_path=file_path, + ) + ) + + async with asyncio.timeout(5): + await transaction_committed.wait() + async with original_scoped_session(session_maker) as session: + assert await entity_repository.get_by_id(session, entity.id) is None + + delete_task.cancel() + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + # Repeated cancellation must not abandon the in-flight Redis generation bump. + delete_task.cancel() + await asyncio.sleep(0) + assert not delete_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await delete_task + + generation_after = await _current_generation(redis_cache, project_external_id) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_cancelled_watcher_index_commit_finishes_real_redis_invalidation( + monkeypatch: pytest.MonkeyPatch, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Cancellation during watcher index transaction exit cannot skip invalidation.""" + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-watcher-index-commit", + ) + entity_repository = EntityRepository(project_id=test_project.id) + executor = CommittingIndexFileExecutor(session_maker, entity_repository) + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + invalidating_executor = InvalidatingIndexFileExecutor( + executor=executor, + read_cache=blocking_cache, + project_external_id=project_external_id, + ) + + transaction_committed = asyncio.Event() + hold_after_commit = asyncio.Event() + original_scoped_session = index_file_runner.db.scoped_session + + @asynccontextmanager + async def pause_after_index_commit( + scoped_session_maker: async_sessionmaker[AsyncSession], + ) -> AsyncIterator[AsyncSession]: + async with original_scoped_session(scoped_session_maker) as session: + yield session + transaction_committed.set() + await hold_after_commit.wait() + + monkeypatch.setattr( + index_file_runner.db, + "scoped_session", + pause_after_index_commit, + ) + file_path = "notes/cancelled-watcher-index-commit.md" + index_task = asyncio.create_task( + invalidating_executor.index_file(file_path, source="s3_webhook") + ) + + async with asyncio.timeout(5): + await transaction_committed.wait() + assert executor.entity_id is not None + async with original_scoped_session(session_maker) as session: + indexed_entity = await entity_repository.get_by_id(session, executor.entity_id) + assert indexed_entity is not None + assert indexed_entity.file_path == file_path + + index_task.cancel() + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + index_task.cancel() + await asyncio.sleep(0) + assert not index_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await index_task + + generation_after = await _current_generation(redis_cache, project_external_id) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_startup_materialization_recovery_invalidates_real_redis( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="startup-recovery", + ) + entity = await _seed_recovery_note( + session_maker, + test_project, + title="Recovered Note", + file_path="notes/recovered.md", + markdown_content="# Recovered with cache invalidation\n", + ) + + await recover_project_materializations( + test_project, + session_maker, + read_cache=redis_cache.cache, + ) + + written = Path(test_project.path) / entity.file_path + assert written.read_text(encoding="utf-8") == "# Recovered with cache invalidation\n" + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="startup-recovery", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_cancelled_startup_materialization_recovery_finishes_real_redis_invalidation( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Cancellation after recovery commits waits for the Redis generation bump.""" + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-startup-materialization-recovery", + ) + entity = await _seed_recovery_note( + session_maker, + test_project, + title="Cancelled Startup Recovery", + file_path="notes/cancelled-startup-recovery.md", + markdown_content="# Recovered before cancellation\n", + ) + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + + recovery_task = asyncio.create_task( + recover_project_materializations( + test_project, + session_maker, + read_cache=blocking_cache, + ) + ) + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + written = Path(test_project.path) / entity.file_path + assert written.read_text(encoding="utf-8") == "# Recovered before cancellation\n" + recovery_task.cancel() + await asyncio.sleep(0) + recovery_task.cancel() + await asyncio.sleep(0) + assert not recovery_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await recovery_task + + content_repository = NoteContentRepository(project_id=test_project.id) + async with db.scoped_session(session_maker) as session: + row = await content_repository.get_by_entity_id(session, entity.id) + assert row is not None + assert row.file_write_status == "synced" + generation_after = await _current_generation(redis_cache, project_external_id) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_cancelled_startup_move_vacate_recovery_finishes_real_redis_invalidation( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Cancellation after vacate cleanup commits waits for its own generation bump.""" + _, session_maker = engine_factory + markdown_content = "# Moved before cancellation\n" + entity = await _seed_recovery_note( + session_maker, + test_project, + title="Cancelled Startup Vacate Recovery", + file_path="notes/cancelled-startup-vacate-recovery.md", + markdown_content=markdown_content, + ) + await recover_project_materializations( + test_project, + session_maker, + read_cache=None, + ) + + source_relative = "old/cancelled-startup-vacate-recovery.md" + source = Path(test_project.path) / source_relative + source.parent.mkdir(parents=True, exist_ok=True) + source.write_text(markdown_content, encoding="utf-8") + source_checksum = sha256(markdown_content.encode()).hexdigest() + vacate_repository = NoteFileVacateRepository(project_id=test_project.id) + async with db.scoped_session(session_maker) as session: + await vacate_repository.record_vacate( + session, + entity_id=entity.id, + file_path=source_relative, + file_checksum=source_checksum, + ) + + project_external_id = str(test_project.external_id) + await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-startup-vacate-recovery", + ) + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + block_on_call=2, + ) + recovery_task = asyncio.create_task( + recover_project_materializations( + test_project, + session_maker, + read_cache=blocking_cache, + ) + ) + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + generation_before_vacate_invalidation = await _current_generation( + redis_cache, + project_external_id, + ) + assert not source.exists() + async with db.scoped_session(session_maker) as session: + assert await vacate_repository.load_vacate_markers(session, [source_relative]) == {} + + recovery_task.cancel() + await asyncio.sleep(0) + recovery_task.cancel() + await asyncio.sleep(0) + assert not recovery_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await recovery_task + + generation_after = await _current_generation(redis_cache, project_external_id) + assert generation_after != generation_before_vacate_invalidation + + +@pytest.mark.asyncio +async def test_startup_recovery_invalidates_before_later_vacate_failure_in_real_redis( + monkeypatch: pytest.MonkeyPatch, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """A later recovery phase failure cannot retain phase-one published state.""" + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="startup-recovery-later-failure", + ) + entity = await _seed_recovery_note( + session_maker, + test_project, + title="Recovered Before Vacate Failure", + file_path="notes/recovered-before-vacate-failure.md", + markdown_content="# Recovered before later failure\n", + ) + + async def fail_move_vacate_recovery( + *, + session_maker: async_sessionmaker[AsyncSession], + file_service: FileService, + project_id: int, + ) -> int: + del session_maker, file_service, project_id + raise RuntimeError("move-vacate setup failure") + + monkeypatch.setattr( + note_content_materialization, + "recover_move_vacates", + fail_move_vacate_recovery, + ) + + await recover_project_materializations( + test_project, + session_maker, + read_cache=redis_cache.cache, + ) + + written = Path(test_project.path) / entity.file_path + assert written.read_text(encoding="utf-8") == "# Recovered before later failure\n" + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="startup-recovery-later-failure", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_startup_recovery_conflict_invalidates_published_failure_in_real_redis( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="startup-recovery-conflict", + ) + entity = await _seed_recovery_note( + session_maker, + test_project, + title="Conflicted Recovery", + file_path="notes/conflicted-recovery.md", + markdown_content="# Accepted recovery content\n", + ) + target = Path(test_project.path) / entity.file_path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text("# External edit\n", encoding="utf-8") + + await recover_project_materializations( + test_project, + session_maker, + read_cache=redis_cache.cache, + ) + + content_repository = NoteContentRepository(project_id=test_project.id) + async with db.scoped_session(session_maker) as session: + row = await content_repository.get_by_entity_id(session, entity.id) + assert row is not None + assert row.file_write_status == "external_change_detected" + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="startup-recovery-conflict", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_project_index_batches_invalidate_real_redis( + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Every inline move, delete, and file batch advances the generation.""" + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="project-index-batches", + ) + observed_store = GenerationObservingProjectIndexBatchStore( + redis_cache, + project_external_id, + ) + invalidating_store = InvalidatingProjectIndexBatchStore( + move_store=observed_store, + delete_store=observed_store, + read_cache=redis_cache.cache, + project_external_id=project_external_id, + ) + maintenance = StoreProjectIndexMaintenanceRunner( + move_store=invalidating_store, + delete_store=invalidating_store, + ) + + await maintenance.run_move_batches( + moved_files={ + "notes/old-1.md": "notes/new-1.md", + "notes/old-2.md": "notes/new-2.md", + }, + batch_size=1, + ) + assert observed_store.move_generations[0] == generation_before + assert observed_store.move_generations[1] != observed_store.move_generations[0] + generation_after_moves = await _current_generation(redis_cache, project_external_id) + assert generation_after_moves != observed_store.move_generations[1] + + await maintenance.run_delete_batches( + deleted_paths=("notes/deleted-1.md", "notes/deleted-2.md"), + batch_size=1, + ) + assert observed_store.delete_generations[0] == generation_after_moves + assert observed_store.delete_generations[1] != observed_store.delete_generations[0] + generation_after_deletes = await _current_generation(redis_cache, project_external_id) + assert generation_after_deletes != observed_store.delete_generations[1] + + batch_enqueuer = LocalProjectIndexBatchEnqueuer( + checker=cast(IndexFileBatchChecker, object()), + reader=cast(IndexFileBatchReader[IndexInputFile], object()), + indexer=cast(IndexFileBatchIndexer[IndexInputFile], object()), + content_classifier=cast(IndexFileBatchContentClassifier, object()), + read_cache=redis_cache.cache, + ) + project = ProjectRuntimeReference.from_project(test_project) + await batch_enqueuer.enqueue_index_file_batch( + RuntimeIndexFileBatchJobRequest( + project=project, + batch_index=0, + batch_count=2, + ) + ) + generation_after_first_file_batch = await _current_generation( + redis_cache, + project_external_id, + ) + assert generation_after_first_file_batch != generation_after_deletes + + await batch_enqueuer.enqueue_index_file_batch( + RuntimeIndexFileBatchJobRequest( + project=project, + batch_index=1, + batch_count=2, + ) + ) + generation_after_second_file_batch = await _current_generation( + redis_cache, + project_external_id, + ) + assert generation_after_second_file_batch != generation_after_first_file_batch + + +@pytest.mark.asyncio +async def test_project_index_failure_invalidates_real_redis( + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="project-index-failure", + ) + + with pytest.raises(RuntimeError, match="partial project index failure"): + await run_local_project_index( + RuntimeProjectIndexJobRequest( + project=ProjectRuntimeReference.from_project(test_project), + embeddings=False, + ), + runtime=LocalProjectIndexRuntime( + observed_file_source=EmptyObservedFileSource(), + change_detector=DeletedFileChangeDetector(), + maintenance_runner=FailingDeleteMaintenance(), + moved_entity_search_refresher=RecordingMovedEntitySearchRefresher(), + batch_enqueuer=UnusedBatchEnqueuer(), + read_cache=redis_cache.cache, + ), + ) + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="project-index-failure", + ) + assert generation_after != generation_before + + +@pytest.mark.asyncio +async def test_directory_delete_invalidates_before_and_after_cleanup_in_real_redis( + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="directory-delete", + ) + entity_repository = EntityRepository(project_id=test_project.id) + async with db.scoped_session(session_maker) as session: + await entity_repository.add( + session, + Entity( + title="Deleted During Cleanup", + note_type="note", + content_type="text/markdown", + file_path="delete-with-cache/note.md", + checksum="directory-delete-checksum", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ), + ) + + enqueuer = GenerationObservingDirectoryDeleteEnqueuer( + redis_cache, + project_external_id, + ) + service = DirectoryDeleteService( + session_maker=session_maker, + runtime=DirectoryDeleteRuntime( + store=RepositoryDirectoryDeleteAcceptanceStore(), + file_delete_enqueuer=enqueuer, + ), + ) + + result = await service.delete_directory( + project_external_id=project_external_id, + directory="delete-with-cache", + read_cache=redis_cache.cache, + ) + + assert result.deleted_files == ("delete-with-cache/note.md",) + assert len(enqueuer.observed_generations) == 1 + generation_during_cleanup = enqueuer.observed_generations[0] + assert generation_during_cleanup != generation_before + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="directory-delete", + ) + assert generation_after != generation_during_cleanup + + +@pytest.mark.asyncio +async def test_cancelled_committed_directory_delete_finishes_real_redis_invalidation( + monkeypatch: pytest.MonkeyPatch, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Cancellation during transaction exit cannot skip delete invalidation.""" + _, session_maker = engine_factory + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-directory-delete", + ) + entity_repository = EntityRepository(project_id=test_project.id) + title = "Cancelled Committed Directory Delete" + async with db.scoped_session(session_maker) as session: + await entity_repository.add( + session, + Entity( + title=title, + note_type="note", + content_type="text/markdown", + file_path="cancelled-delete/note.md", + checksum="cancelled-directory-delete-checksum", + created_at=datetime.now(UTC), + updated_at=datetime.now(UTC), + ), + ) + + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + transaction_committed = asyncio.Event() + hold_after_commit = asyncio.Event() + original_scoped_session = directory_deletes.db.scoped_session + + @asynccontextmanager + async def pause_after_commit( + scoped_session_maker: async_sessionmaker[AsyncSession], + ) -> AsyncIterator[AsyncSession]: + async with original_scoped_session(scoped_session_maker) as session: + yield session + transaction_committed.set() + await hold_after_commit.wait() + + monkeypatch.setattr(directory_deletes.db, "scoped_session", pause_after_commit) + service = DirectoryDeleteService( + session_maker=session_maker, + runtime=DirectoryDeleteRuntime( + store=RepositoryDirectoryDeleteAcceptanceStore(), + file_delete_enqueuer=GenerationObservingDirectoryDeleteEnqueuer( + redis_cache, + project_external_id, + ), + ), + ) + delete_task = asyncio.create_task( + service.delete_directory( + project_external_id=project_external_id, + directory="cancelled-delete", + read_cache=blocking_cache, + ) + ) + + async with asyncio.timeout(5): + await transaction_committed.wait() + delete_task.cancel() + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + # A second cancellation must not abandon the already-running generation bump. + delete_task.cancel() + await asyncio.sleep(0) + assert not delete_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await delete_task + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-directory-delete", + ) + assert generation_after != generation_before + + async with original_scoped_session(session_maker) as session: + deleted_entities = await entity_repository.get_by_title(session, title) + assert deleted_entities == [] + + +@pytest.mark.asyncio +async def test_cancelled_committed_directory_move_finishes_real_redis_invalidation( + monkeypatch: pytest.MonkeyPatch, + engine_factory: tuple[AsyncEngine, async_sessionmaker[AsyncSession]], + search_service: SearchService, + app_config: BasicMemoryConfig, + project_config: ProjectConfig, + test_project: Project, + redis_cache: RedisCacheHarness, +) -> None: + """Cancellation during a file-move commit cannot skip invalidation.""" + _, session_maker = engine_factory + entity_repository = EntityRepository(project_id=test_project.id) + entity_parser = EntityParser(project_config.home) + file_service = FileService(project_config.home, MarkdownProcessor(entity_parser)) + entity_service = EntityService( + entity_parser=entity_parser, + entity_repository=entity_repository, + observation_repository=ObservationRepository(project_id=test_project.id), + relation_repository=RelationRepository(project_id=test_project.id), + file_service=file_service, + link_resolver=LinkResolver( + entity_repository, + search_service, + session_maker=session_maker, + app_config=app_config, + ), + session_maker=session_maker, + search_service=search_service, + app_config=app_config, + ) + project_external_id = str(test_project.external_id) + generation_before = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-directory-move", + ) + entity = await entity_service.create_entity( + EntitySchema( + title="Cancelled Committed Directory Move", + directory="cancelled-move-source", + note_type="note", + content="Move must remain visible after cancellation.", + ) + ) + destination_path = "cancelled-move-destination/Cancelled Committed Directory Move.md" + blocking_cache = BlockingInvalidationRedisReadCache( + client=redis_cache.client, + namespace=redis_cache.namespace, + prefix=redis_cache.prefix, + ) + transaction_committed = asyncio.Event() + hold_after_commit = asyncio.Event() + original_scoped_session = entity_service_module.db.scoped_session + + @asynccontextmanager + async def pause_after_move_commit( + scoped_session_maker: async_sessionmaker[AsyncSession], + existing_session: AsyncSession | None = None, + ) -> AsyncIterator[AsyncSession]: + async with original_scoped_session(scoped_session_maker, existing_session) as session: + yield session + if transaction_committed.is_set(): + return + + async with original_scoped_session(scoped_session_maker) as session: + moved_entity = await entity_repository.get_by_id( + session, + entity.id, + load_relations=False, + ) + if moved_entity is not None and moved_entity.file_path == destination_path: + transaction_committed.set() + await hold_after_commit.wait() + + monkeypatch.setattr(entity_service_module.db, "scoped_session", pause_after_move_commit) + move_task = asyncio.create_task( + entity_service.move_directory( + source_directory="cancelled-move-source", + destination_directory="cancelled-move-destination", + project_config=project_config, + app_config=app_config.model_copy(update={"update_permalinks_on_move": False}), + project_external_id=project_external_id, + read_cache=blocking_cache, + ) + ) + + async with asyncio.timeout(5): + await transaction_committed.wait() + move_task.cancel() + async with asyncio.timeout(5): + await blocking_cache.invalidation_started.wait() + + # A second cancellation must not abandon the already-running generation bump. + move_task.cancel() + await asyncio.sleep(0) + assert not move_task.done() + + blocking_cache.release_invalidation.set() + with pytest.raises(asyncio.CancelledError): + await move_task + + generation_after = await _initialized_generation( + redis_cache, + project_external_id, + request="cancelled-committed-directory-move", + ) + assert generation_after != generation_before + + async with original_scoped_session(entity_service.session_maker) as session: + moved_entity = await entity_repository.get_by_id( + session, + entity.id, + load_relations=False, + ) + assert moved_entity is not None + assert moved_entity.file_path == destination_path diff --git a/tests/cloud/test_cloud_services.py b/tests/cloud/test_cloud_services.py index b37626cd9..4c08d8707 100644 --- a/tests/cloud/test_cloud_services.py +++ b/tests/cloud/test_cloud_services.py @@ -396,10 +396,12 @@ async def fake_reconcile_note_content_from_file( project_external_id: str, entity_external_id: str, source: str, + read_cache: object | None = None, ) -> bool: assert project_external_id == "project-123" assert entity_external_id == "note-456" assert source == "read_repair" + assert read_cache is None return True service = NoteContentQueryService( diff --git a/tests/cloud/test_note_content_materialization.py b/tests/cloud/test_note_content_materialization.py index 7ad9c5b3d..124416ada 100644 --- a/tests/cloud/test_note_content_materialization.py +++ b/tests/cloud/test_note_content_materialization.py @@ -27,6 +27,10 @@ NoteContentRepository, ) from basic_memory.repository.note_file_vacate_repository import NoteFileVacateRepository +from basic_memory.read_cache import ( + ReadCacheInvalidator, + ReadCacheInvalidationStatus, +) from basic_memory.runtime.cleanup import RuntimeNoteFileDeleteJobRequest from basic_memory.indexing.models import FileIndexOperation, FileIndexResult from basic_memory.runtime.note_content import ( @@ -42,6 +46,8 @@ ) from basic_memory.services.file_service import FileService +PROJECT_EXTERNAL_ID = "00000000-0000-0000-0000-000000000007" + class RecordingFileIndexer: def __init__(self) -> None: @@ -60,6 +66,15 @@ async def index_file(self, file_path: str, *, source: str) -> FileIndexResult: ) +class RecordingReadCache: + def __init__(self) -> None: + self.invalidated_project_ids: list[str] = [] + + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + self.invalidated_project_ids.append(project_id) + return ReadCacheInvalidationStatus.invalidated + + def accepted_materialization_change() -> RuntimeAcceptedNoteChange[ RuntimeNoteContentResponsePayload ]: @@ -102,12 +117,15 @@ def local_materialization_provider( indexer: RecordingFileIndexer, *, test_mode: bool = True, + read_cache: ReadCacheInvalidator | None = None, ) -> LocalNoteContentMaterializationProvider: # test_mode=True keeps materialization inline so these tests can assert the # result synchronously; production defers it to a background task. return LocalNoteContentMaterializationProvider( session_maker=cast(async_sessionmaker[AsyncSession], object()), file_service=cast(FileService, object()), + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=read_cache, file_indexer=indexer, test_mode=test_mode, ) @@ -277,19 +295,27 @@ async def fake_run_note_materialization( pool = note_content_materialization._MaterializationWorkerPool() monkeypatch.setattr(note_content_materialization, "_materialization_pool", pool) indexer = RecordingFileIndexer() + read_cache = RecordingReadCache() accepted = accepted_materialization_change() result = await local_materialization_provider( - indexer, test_mode=False + indexer, + test_mode=False, + read_cache=read_cache, ).materialize_write_change(accepted) # Returned immediately with the accepted DB state — no inline write yet. assert result is accepted assert requests == [] + assert read_cache.invalidated_project_ids == [] # The write happens off the accept path via the bounded pool; drain to confirm. await pool.join() assert len(requests) == 1 + assert read_cache.invalidated_project_ids == [ + PROJECT_EXTERNAL_ID, + PROJECT_EXTERNAL_ID, + ] await pool.aclose() @@ -350,9 +376,12 @@ class RecordingScheduler: def schedule_relation_resolution(self, *, project_id: int) -> None: scheduled.append(project_id) + read_cache = RecordingReadCache() provider = LocalNoteContentMaterializationProvider( session_maker=cast(async_sessionmaker[AsyncSession], object()), file_service=cast(FileService, object()), + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=read_cache, file_indexer=RecordingFileIndexer(), test_mode=True, relation_resolution_scheduler=RecordingScheduler(), @@ -362,6 +391,10 @@ def schedule_relation_resolution(self, *, project_id: int) -> None: assert accepted.materialization is not None assert scheduled == [accepted.materialization.project_id] + assert read_cache.invalidated_project_ids == [ + PROJECT_EXTERNAL_ID, + PROJECT_EXTERNAL_ID, + ] @pytest.mark.asyncio @@ -531,7 +564,8 @@ async def test_recover_stuck_materializations_writes_file_and_marks_synced( project_id=test_project.id, ) - assert recovered == 1 + assert recovered.attempted == 1 + assert recovered.written == 1 written = file_service.base_path / sample_entity.file_path assert written.read_text(encoding="utf-8") == "# Recovered\n\nThe crash left this unwritten.\n" @@ -587,14 +621,13 @@ async def test_move_vacate_recovery_waits_for_destination_then_cleans_source( ) assert source.exists() - assert ( - await recover_stuck_materializations( - session_maker=session_maker, - file_service=file_service, - project_id=test_project.id, - ) - == 1 + recovered = await recover_stuck_materializations( + session_maker=session_maker, + file_service=file_service, + project_id=test_project.id, ) + assert recovered.attempted == 1 + assert recovered.written == 1 assert ( await recover_move_vacates( session_maker=session_maker, @@ -639,7 +672,8 @@ async def test_recover_stuck_materializations_re_drives_failed_row( project_id=test_project.id, ) - assert recovered == 1 + assert recovered.attempted == 1 + assert recovered.written == 1 written = file_service.base_path / sample_entity.file_path assert written.read_text(encoding="utf-8") == "# Recovered after transient failure\n" @@ -674,7 +708,8 @@ async def test_recover_stuck_materializations_returns_zero_when_none_stuck( project_id=test_project.id, ) - assert recovered == 0 + assert recovered.attempted == 0 + assert recovered.written == 0 @pytest.mark.asyncio @@ -707,7 +742,8 @@ async def boom(*_: Any, **__: Any) -> None: project_id=test_project.id, ) - assert recovered == 0 + assert recovered.attempted == 1 + assert recovered.written == 0 @pytest.mark.asyncio @@ -739,7 +775,8 @@ async def test_recover_stuck_materializations_does_not_overwrite_unexpected_file project_id=test_project.id, ) - assert recovered == 0 + assert recovered.attempted == 1 + assert recovered.written == 0 assert target.read_text(encoding="utf-8") == "# External edit\n" repository = NoteContentRepository(project_id=test_project.id) @@ -788,7 +825,8 @@ async def test_recover_stuck_materializations_publishes_already_written_file( project_id=test_project.id, ) - assert recovered == 1 + assert recovered.attempted == 1 + assert recovered.written == 1 assert target.read_text(encoding="utf-8") == markdown_content repository = NoteContentRepository(project_id=test_project.id) diff --git a/tests/index/test_local_project_vector_cleaner_wiring.py b/tests/index/test_local_project_vector_cleaner_wiring.py index fcc7c9571..ea4ca1428 100644 --- a/tests/index/test_local_project_vector_cleaner_wiring.py +++ b/tests/index/test_local_project_vector_cleaner_wiring.py @@ -33,11 +33,12 @@ def test_full_project_runtime_forwards_external_vector_cleaner() -> None: external_vector_cleaner=cleaner, ) - runtime = LocalProjectIndexRuntimeFactory().runtime_from_dependencies(dependencies) + runtime = LocalProjectIndexRuntimeFactory().runtime_from_dependencies( + dependencies, + project_external_id="project-external-id", + ) assert isinstance(runtime.maintenance_runner, StoreProjectIndexMaintenanceRunner) - assert isinstance( - runtime.maintenance_runner.delete_store, - RepositoryProjectIndexMaintenanceStore, - ) - assert runtime.maintenance_runner.delete_store.external_vector_cleaner is cleaner + delete_store = runtime.maintenance_runner.delete_store + assert isinstance(delete_store, RepositoryProjectIndexMaintenanceStore) + assert delete_store.external_vector_cleaner is cleaner diff --git a/tests/index/test_local_schedulers.py b/tests/index/test_local_schedulers.py index 75bcd154f..c60872471 100644 --- a/tests/index/test_local_schedulers.py +++ b/tests/index/test_local_schedulers.py @@ -13,6 +13,9 @@ LocalSearchReindexScheduler, drain_background_tasks, ) +from basic_memory.read_cache import ReadCacheInvalidationStatus + +PROJECT_EXTERNAL_ID = "00000000-0000-0000-0000-000000000013" class StubProjectIndexRunner: @@ -41,6 +44,15 @@ async def reindex_all(self) -> None: self.reindexed_project = True +class RecordingReadCache: + def __init__(self) -> None: + self.invalidated_project_ids: list[str] = [] + + async def invalidate_project(self, project_id: str) -> ReadCacheInvalidationStatus: + self.invalidated_project_ids.append(project_id) + return ReadCacheInvalidationStatus.invalidated + + @pytest.mark.asyncio async def test_entity_vector_scheduler_maps_to_search_service(): """Entity vector scheduling should call the semantic vector sync method.""" @@ -213,15 +225,19 @@ async def test_project_index_scheduler_is_noop_in_test_mode(): async def test_search_reindex_scheduler_maps_to_search_service(): """Search reindex scheduling should rebuild the search index.""" search_service = StubSearchService() + read_cache = RecordingReadCache() scheduler = LocalSearchReindexScheduler( search_service=search_service, + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=read_cache, test_mode=False, ) scheduler.schedule_search_reindex(project_id=13) await asyncio.sleep(0.05) assert search_service.reindexed_project is True + assert read_cache.invalidated_project_ids == [PROJECT_EXTERNAL_ID] class StubRelationResolutionRuntime: @@ -243,9 +259,12 @@ async def test_relation_resolution_scheduler_runs_project_resolution(): _pending_relation_resolution.clear() runtime = StubRelationResolutionRuntime() + read_cache = RecordingReadCache() scheduler = LocalRelationResolutionScheduler( relation_runtime=runtime, + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=read_cache, test_mode=False, debounce_seconds=0.0, ) @@ -253,6 +272,7 @@ async def test_relation_resolution_scheduler_runs_project_resolution(): await asyncio.sleep(0.05) assert runtime.resolve_calls == 1 + assert read_cache.invalidated_project_ids == [PROJECT_EXTERNAL_ID] # The pending marker is cleared after the pass so later writes can schedule. assert 13 not in _pending_relation_resolution @@ -267,6 +287,8 @@ async def test_relation_resolution_scheduler_coalesces_a_burst(): scheduler = LocalRelationResolutionScheduler( relation_runtime=runtime, + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=None, test_mode=False, debounce_seconds=0.02, ) @@ -289,6 +311,7 @@ async def test_relation_resolution_scheduler_reruns_for_write_during_pass(): _pending_relation_resolution.clear() _dirty_relation_resolution.clear() + read_cache = RecordingReadCache() class WriteDuringScanRuntime: def __init__(self) -> None: @@ -309,6 +332,8 @@ async def resolve_relations(self, entity_id: int | None = None) -> set[int]: runtime = WriteDuringScanRuntime() scheduler = LocalRelationResolutionScheduler( relation_runtime=runtime, + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=read_cache, test_mode=False, debounce_seconds=0.0, ) @@ -318,6 +343,10 @@ async def resolve_relations(self, entity_id: int | None = None) -> set[int]: await asyncio.sleep(0.05) assert runtime.resolve_calls == 2 + assert read_cache.invalidated_project_ids == [ + PROJECT_EXTERNAL_ID, + PROJECT_EXTERNAL_ID, + ] assert 21 not in _pending_relation_resolution assert 21 not in _dirty_relation_resolution @@ -369,6 +398,8 @@ async def resolve_relations(self, entity_id: int | None = None) -> set[int]: runtime = WriteDuringScanRuntime() scheduler = LocalRelationResolutionScheduler( relation_runtime=runtime, + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=None, test_mode=False, debounce_seconds=0.0, ) @@ -388,14 +419,18 @@ async def test_relation_resolution_scheduler_is_noop_in_test_mode(): _pending_relation_resolution.clear() runtime = StubRelationResolutionRuntime() + read_cache = RecordingReadCache() scheduler = LocalRelationResolutionScheduler( relation_runtime=runtime, + project_external_id=PROJECT_EXTERNAL_ID, + read_cache=read_cache, test_mode=True, ) scheduler.schedule_relation_resolution(project_id=13) await asyncio.sleep(0.05) assert runtime.resolve_calls == 0 + assert read_cache.invalidated_project_ids == [] # Test mode must not leak a pending marker (it never runs the clearer). assert 13 not in _pending_relation_resolution diff --git a/tests/index/test_watch_coordinator.py b/tests/index/test_watch_coordinator.py index d9ecfc3be..00f1bf194 100644 --- a/tests/index/test_watch_coordinator.py +++ b/tests/index/test_watch_coordinator.py @@ -24,8 +24,9 @@ async def initialize_after_recovery( quiet: bool = True, *, recovery_complete: asyncio.Event | None = None, + read_cache: object | None = None, ) -> None: - del config, quiet + del config, quiet, read_cache await allow_recovery_to_finish.wait() assert recovery_complete is not None recovery_complete.set() @@ -62,8 +63,9 @@ async def fail_before_recovery( quiet: bool = True, *, recovery_complete: asyncio.Event | None = None, + read_cache: object | None = None, ) -> None: - del config, quiet, recovery_complete + del config, quiet, recovery_complete, read_cache raise RuntimeError("recovery boom") monkeypatch.setattr( diff --git a/tests/services/test_initialization.py b/tests/services/test_initialization.py index 6ad901862..55707c052 100644 --- a/tests/services/test_initialization.py +++ b/tests/services/test_initialization.py @@ -315,6 +315,9 @@ def capture_task(coro): return original_create_task(coro) class RecordingProjectIndexRuntimeFactory: + def __init__(self, *, read_cache: object) -> None: + self.read_cache = read_cache + async def runtime_for_project(self, project): # noqa: ANN001 return f"runtime:{project.name}" diff --git a/tests/test_deps.py b/tests/test_deps.py index 252da8512..cb106be13 100644 --- a/tests/test_deps.py +++ b/tests/test_deps.py @@ -5,7 +5,7 @@ from basic_memory.api import container as container_module from basic_memory.api.container import ApiContainer, resolve_container -from basic_memory.deps import get_app_config, validate_project_external_id +from basic_memory.deps import get_app_config, get_read_cache, validate_project_external_id from basic_memory.models.project import Project from basic_memory.repository.project_repository import ProjectRepository from basic_memory.runtime.mode import resolve_runtime_mode @@ -44,6 +44,29 @@ def test_resolve_container_prefers_installed_container(app_config, monkeypatch): assert resolve_container() is installed +def test_get_read_cache_reads_lifespan_container(app_config): + """API requests preserve the container's absent cache backend.""" + app = FastAPI() + app.state.container = ApiContainer( + config=app_config, + mode=resolve_runtime_mode(is_test_env=True), + ) + + assert get_read_cache(_request_for(app)) is None + + +def test_get_read_cache_falls_back_to_composition_root(app_config, monkeypatch): + """Off-lifespan requests preserve the composition root's absent cache backend.""" + app = FastAPI() + installed = ApiContainer( + config=app_config, + mode=resolve_runtime_mode(is_test_env=True), + ) + monkeypatch.setattr(container_module, "_container", installed) + + assert get_read_cache(_request_for(app)) is None + + @pytest.mark.asyncio async def test_validate_project_external_id_success( project_repository: ProjectRepository, test_project: Project, session_maker diff --git a/uv.lock b/uv.lock index 6940a9a2d..df31cd178 100644 --- a/uv.lock +++ b/uv.lock @@ -332,6 +332,9 @@ milvus = [ { name = "pymilvus" }, { name = "pymilvus", extra = ["milvus-lite"], marker = "sys_platform != 'win32'" }, ] +redis = [ + { name = "redis" }, +] [package.dev-dependencies] dev = [ @@ -351,6 +354,7 @@ dev = [ { name = "pytest-testmon" }, { name = "pytest-timeout" }, { name = "pytest-xdist" }, + { name = "redis" }, { name = "ruff" }, { name = "testcontainers" }, { name = "ty" }, @@ -395,6 +399,7 @@ requires-dist = [ { name = "python-dotenv", specifier = ">=1.1.0" }, { name = "python-frontmatter", specifier = ">=1.1.0" }, { name = "pyyaml", specifier = ">=6.0.1" }, + { name = "redis", marker = "extra == 'redis'", specifier = ">=8.0.0,<9" }, { name = "rich", specifier = ">=13.9.4" }, { name = "sniffio", specifier = ">=1.3.1" }, { name = "sqlalchemy", specifier = ">=2.0.0" }, @@ -404,7 +409,7 @@ requires-dist = [ { name = "uvloop", marker = "sys_platform != 'win32'", specifier = ">=0.21.0" }, { name = "watchfiles", specifier = ">=1.0.4" }, ] -provides-extras = ["milvus"] +provides-extras = ["milvus", "redis"] [package.metadata.requires-dev] dev = [ @@ -424,6 +429,7 @@ dev = [ { name = "pytest-testmon", specifier = ">=2.2.0" }, { name = "pytest-timeout", specifier = ">=2.4.0" }, { name = "pytest-xdist", specifier = ">=3.0.0" }, + { name = "redis", specifier = ">=8.0.0,<9" }, { name = "ruff", specifier = ">=0.16.0" }, { name = "testcontainers", extras = ["postgres"], specifier = ">=4.0.0" }, { name = "ty", specifier = ">=0.0.64" }, @@ -3556,6 +3562,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/c0/28/26534bed77109632a956977f60d8519049f545abc39215d086e33a61f1f2/pyyaml_ft-8.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:de04cfe9439565e32f178106c51dd6ca61afaa2907d143835d501d84703d3793", size = 171579, upload-time = "2025-06-10T15:32:14.34Z" }, ] +[[package]] +name = "redis" +version = "8.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cc/c3/928b290c2c0ca99ab96eea5b4ff8f30be8112b075301a7d3ba214a3c8c12/redis-8.0.1.tar.gz", hash = "sha256:afc5a7a2f5a084f5b1880dec548dd45be17db7e43c82a30d84f952aefb05cfb0", size = 5114170, upload-time = "2026-06-23T14:52:37.728Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/0a/c2345ebf1ebe70840ce3f6c6ee612f8fa749cfbd1b03069c53bf0c62aaad/redis-8.0.1-py3-none-any.whl", hash = "sha256:47daa35a058c23468d6437f17a8c76882cb316b838ef763036af99b96cedd743", size = 502406, upload-time = "2026-06-23T14:52:36.137Z" }, +] + [[package]] name = "referencing" version = "0.37.0"