Repository navigation
refactor(rag): refactor embedding flow - #2975
Conversation
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| return source | ||
| .replace(/^(?:import|export)\s[^\n]*$/gm, "") // ESM statements | ||
| .replace(/```(\w*)\n([\s\S]*?)```/g, (_match, lang: string, code: string) => | ||
| condenseCodeBlock(lang, code) | ||
| ) | ||
| .replace(/!\[([^\]]*)\]\([^)]*\)/g, "$1") // images -> alt text | ||
| .replace(/\[([^\]]*)\]\([^)]*\)/g, "$1") // links -> label | ||
| .replace(/<\/?[A-Za-z][^>\n]*>/g, "") // JSX / HTML tags, keep children |
| if (FENCE_REGEX.test(line)) { | ||
| insideCodeFence = !insideCodeFence; | ||
| } | ||
| const heading = insideCodeFence ? null : HEADING_REGEX.exec(line); |
📝 WalkthroughWalkthroughThe pull request replaces Algolia and legacy feed embeddings with resource chunk indexing, BM25/semantic/hybrid search, centralized embedding providers, token-aware document processing, and direct translation content storage. ChangesResource storage and indexing
Embedding and retrieval
API and application integration
Estimated code review effort: 5 (Critical) | ~120 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 15
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (9)
packages/api/feeds/search.ts-59-70 (1)
59-70: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDeduplicate feeds when several translations of one feed match.
searchResourcesaggregates per source (feed translation), so two translations of the same feed produce two hits.resolveFeedRefsmaps both to the samefeedId, and theflatMapkeeps both. Iflocaleis undefined, callers get duplicate feeds and the effective result count drops belowlimit.getRelatedFeedsinpackages/db/src/libs/feeds/search.tsalready drops other translations of the same feed, so the behavior is inconsistent.apps/service/src/services/agent-content.port.tscalls this service with an optional locale, so the path is reachable.🔧 Proposed fix to keep the best hit per feed
return { mode, - items: items.flatMap((item) => { - const ref = refs.get(item.sourceId); - return ref ? [{ ...item, ...ref }] : []; - }), + items: (() => { + const seenFeeds = new Set<number>(); + return items.flatMap((item) => { + const ref = refs.get(item.sourceId); + if (!ref || seenFeeds.has(ref.feedId)) { + return []; + } + seenFeeds.add(ref.feedId); + return [{ ...item, ...ref }]; + }); + })(), };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/api/feeds/search.ts` around lines 59 - 70, Update searchResources to deduplicate resolved results by feedId when multiple source translations map to the same feed, keeping only the best-ranked hit and preserving the requested limit. Apply this especially when locale is undefined, while retaining distinct feeds and the existing item enrichment behavior.apps/service/src/services/agent-content.port.ts-82-92 (1)
82-92: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winPass
input.limittosearchFeedsService.
searchFeedsServicedefaultslimitto 5. Ifinput.limitis greater than 5, the laterslice(0, input.limit)cannot restore the missing results, so the agent receives fewer hits than it requested.🔧 Proposed fix
const result = await searchFeedsService({ db, keyword: input.keyword, // `keyword` is in-database BM25; `semantic` fuses dense and lexical, // because a single document vector alone under-recalls exact terms // (package names, CLI flags, error messages) model: input.mode === "keyword" ? "bm25" : "hybrid", locale: input.locale, + limit: input.limit, });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/src/services/agent-content.port.ts` around lines 82 - 92, Pass input.limit as the limit option in the searchFeedsService call within the agent content search flow, while preserving the existing result slicing and mapping behavior.apps/www/src/app/[locale]/(blog)/[type]/[slug]/llm.md/route.ts-23-23 (1)
23-23: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winReturn
404when localized Markdown content is missing.
feed.translations[0]?.contentcan benullorundefined. This returns a successful empty Markdown response. Validate the translation content before constructing the response.Proposed fix
+ const content = feed.translations[0]?.content; + if (!content) { + notFound(); + } - return new Response(feed.translations[0]?.content, { + return new Response(content, {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/www/src/app/`[locale]/(blog)/[type]/[slug]/llm.md/route.ts at line 23, Validate feed.translations[0]?.content before constructing the Response in the route handler, and return a 404 response when the localized Markdown content is null or undefined. Preserve the existing successful response for available content.packages/api/package.json-124-143 (1)
124-143: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMigrate the legacy Trigger consumers before removing the subpath export.
legacy/triggerdepends on@chia/apiand includes both task files that import@chia/api/services/feeds. Update these imports before release. No source imports reference@chia/api/services/validatorsor@chia/api/algolia.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/api/package.json` around lines 124 - 143, The legacy Trigger task files still import the soon-to-be-removed `@chia/api/services/feeds` subpath. Update both task-file imports under legacy/trigger to use the supported feed exports (such as the exposed feeds/validator or feeds/write entry points), then remove the obsolete subpath export only after those consumers no longer depend on it.packages/db/src/libs/feeds/index.ts-466-473 (1)
466-473: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winA feed with no translations now resolves to
null.The query uses
innerJoinonfeedTranslations, so a feed without any translation row produces zero rows and the function returnsnull.feedIndexingWorkflowthen reports"Feed not found", which is misleading and hides a real feed.If a translation-less feed is possible, use
leftJoinand filter out null translations, or fetch the feed and its translations separately.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/feeds/index.ts` around lines 466 - 473, The feed lookup query in the function containing firstRow currently uses innerJoin, causing feeds without translations to return null. Replace it with a leftJoin and filter out null translation records while retaining the feed row, so feedIndexingWorkflow does not report an existing translation-less feed as missing.packages/db/src/schemas/contents.schema.ts-133-135 (1)
133-135: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winSynchronize feed-translation visibility mirrors.
upsertFeedTranslationdoes not copy the current feed flags, so new translations default tofalse.updateFeed,softDeleteFeed, andrestoreFeedalso leave existing mirrors stale. UpdatefeedTranslations.publishedandfeedTranslations.deletedin these paths, or remove the mirror columns from visibility queries. The schema comment must identifyfeedsas the indexing visibility source.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/schemas/contents.schema.ts` around lines 133 - 135, Synchronize the visibility mirrors in upsertFeedTranslation, updateFeed, softDeleteFeed, and restoreFeed with the corresponding feeds.published and feeds.deleted values, including both new and existing translations. Update the schema comment for published to identify feeds as the indexing visibility source, while preserving the existing mirror columns and visibility-query behavior.packages/agent-writing/src/tools/retrieval.tool.ts-147-159 (1)
147-159: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winSignal dropped locales instead of returning empty content.
buildDocumentContextdrops a document when the shared budget is exhausted and reports it indroppedSlugs. Here every input carries the samepost.slug, sodroppedSlugscannot identify which locale was dropped, and the code discards it. The dropped locale then returnscontent: ""anddetail: "outline". The model reads that as an empty post.Mark the omission explicitly.
🔧 Proposed change
const translations = post.translations.map((translation) => { const document = byLocale.get(translation.locale); + if (!document) { + return { + ...translation, + content: "", + detail: "omitted" as const, + omittedReason: "context budget exhausted", + tokenCount: 0, + anchors: [], + }; + } return { ...translation, - content: document?.text ?? "", - detail: document?.detail ?? "outline", - tokenCount: document?.tokenCount ?? 0, - anchors: document?.anchors.map((anchor) => anchor.anchor) ?? [], + content: document.text, + detail: document.detail, + tokenCount: document.tokenCount, + anchors: document.anchors.map((anchor) => anchor.anchor), }; });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/agent-writing/src/tools/retrieval.tool.ts` around lines 147 - 159, Update the translation mapping in buildDocumentContext’s caller to detect when a locale’s document was omitted by the shared budget and mark that translation explicitly as dropped, rather than returning empty content with outline detail. Use the available dropped-slugs information and locale-specific context to preserve normal document fields for retained locales.pnpm-workspace.yaml-169-172 (1)
169-172: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winAlign the
drizzle-ormoverride with the package manifest.
@paradedb/drizzle-paradedb@0.4.0declaresdrizzle-ormas a direct dependency at1.0.0-rc.2, not as a peer dependency. Confirm that1.0.0-rc.4is compatible before overriding the declared version.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pnpm-workspace.yaml` around lines 169 - 172, Update the drizzle-orm override under the `@paradedb/drizzle-paradedb` selector to match its declared direct dependency version 1.0.0-rc.2, unless compatibility with 1.0.0-rc.4 has been explicitly confirmed.packages/ai/src/embeddings/openai.ts-13-17 (1)
13-17: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winPreserve the
TextEmbeddingModelexport from@chia/ai/embeddings/utils. External consumers of this public subpath will fail because the type now exists only inopenai.ts. Add a compatibility re-export.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/src/embeddings/openai.ts` around lines 13 - 17, Restore the TextEmbeddingModel export from the embeddings/utils public subpath by adding a compatibility re-export there, while keeping the existing definition in openai.ts as the source of truth. Ensure external imports from `@chia/ai/embeddings/utils` continue resolving to the same type.
🧹 Nitpick comments (25)
packages/api/feeds/search.ts (1)
169-181: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winInclude the embedding provider id in the related-feeds cache key.
The query result depends on
resolveEmbeddingProvider().id, but the key contains only locale, slug, and limit. IfEMBEDDING_PROVIDERchanges, the cache serves results computed with the previous model for up to six hours.♻️ Proposed cache key change
- const cacheKey = `feeds:related:${locale}:${slug}:${limit}`; + const model = resolveEmbeddingProvider().id; + const cacheKey = `feeds:related:${model}:${locale}:${slug}:${limit}`; const cached = await kv.get<RelatedFeedItems>(cacheKey); if (cached) { return cached; } const items = await getRelatedFeeds(db, { slug, locale, limit, - model: resolveEmbeddingProvider().id, + model, });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/api/feeds/search.ts` around lines 169 - 181, Update the cache-key construction in the related-feeds flow to include the embedding provider id from resolveEmbeddingProvider().id, ensuring reads and writes use the same provider-specific key. Keep the existing locale, slug, and limit components unchanged.apps/service/src/steps/feed-indexing.step.ts (1)
47-53: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueDeduplicate the per-locale tag names.
allTagNamesremoves duplicates with aSet, buttagNamesByLocaledoes not. If two tag translations in one locale share a name, the document card repeats that name. Apply the same deduplication to the grouped values.♻️ Proposed fix
- const tagNamesByLocale = new Map<string, string[]>(); + const tagNamesByLocale = new Map<string, Set<string>>(); for (const tag of feed.tags) { - const names = tagNamesByLocale.get(tag.locale) ?? []; - names.push(tag.name); + const names = tagNamesByLocale.get(tag.locale) ?? new Set<string>(); + names.add(tag.name); tagNamesByLocale.set(tag.locale, names); } const allTagNames = [...new Set(feed.tags.map((tag) => tag.name))];Then read it as:
- tags: tagNamesByLocale.get(translation.locale) ?? allTagNames, + tags: [...(tagNamesByLocale.get(translation.locale) ?? allTagNames)],🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/src/steps/feed-indexing.step.ts` around lines 47 - 53, Deduplicate each locale’s tag names while building tagNamesByLocale, so duplicate tag.name values within the same tag.locale are stored only once. Preserve allTagNames behavior and the existing locale grouping.apps/service/src/workflows/feed-indexing.workflow.ts (1)
15-18: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value
settledStatusputs the raw rejection text into the workflow result.
String(result.reason)is embedded in the returned value, which the workflow engine persists as durable run state. Embedding-provider errors can echo request content. Consider mapping to a short code plus the error name, and logging the full reason separately.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/src/workflows/feed-indexing.workflow.ts` around lines 15 - 18, Update settledStatus to return only a short failure code plus the rejection error name, rather than embedding String(result.reason) in BranchStatus. Log the full rejection reason separately at the failure point while preserving the existing "ok" result for fulfilled promises.apps/service/src/workflows/feed-removal.workflow.ts (1)
7-9: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low valueConstrain the ID schema to integers.
z.number()accepts1.5and-0.5.translationIDsmaps tosourceId, which is an integer column. A non-integer value passes validation and fails at the database instead of at the boundary. The same gap exists inapps/service/src/workflows/resource-index.workflow.ts.🛡️ Proposed fix
const requestSchema = z.object({ - translationIDs: z.array(z.number()), + translationIDs: z.array(z.number().int().positive()), });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/src/workflows/feed-removal.workflow.ts` around lines 7 - 9, Constrain the translationIDs elements in requestSchema to integers and apply the same validation in the corresponding schema in resource-index.workflow.ts. Preserve the existing array validation while rejecting fractional IDs before they reach sourceId/database operations.apps/service/src/services/feed-indexing.service.ts (1)
13-17: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUpdate the table names in the doc comment.
This PR replaces
feed_embeddingwith the resource chunk and embedding tables. The comment still namesfeed_search_documentandfeed_embeddingas the rows that cascade fromfeed_translation. Name the resource chunk table instead so the rationale stays verifiable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/src/services/feed-indexing.service.ts` around lines 13 - 17, Update the doc comment above the feed indexing logic to replace the obsolete feed_embedding table name with the current resource chunk table name, while preserving the existing feed_search_document reference and explanation of cascade and soft-delete behavior.apps/service/src/workflows/resource-index.workflow.ts (1)
6-9: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winValidate
sourceTypeagainst the registered adapters.
z.string()accepts any value. The value flows togetResourceAdapterinpackages/api/resources/registry.ts, which performs an unguarded property lookup. Constrain the schema with the exportedresourceTypesso the boundary rejects unknown types before the lookup runs. Also constrainsourceIdto an integer.🛡️ Proposed fix
+import { resourceTypes } from "`@chia/api/resources/registry`"; + export const resourceIndexRequestSchema = z.object({ - sourceType: z.string(), - sourceId: z.number(), + sourceType: z.enum(resourceTypes as [string, ...string[]]), + sourceId: z.number().int().positive(), });
z.enumrequires a non-empty tuple type. If the cast is unwanted, usez.string().refine((value) => resourceTypes.includes(value)).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/src/workflows/resource-index.workflow.ts` around lines 6 - 9, Update resourceIndexRequestSchema to validate sourceType against the exported resourceTypes collection before getResourceAdapter is called, using an approach compatible with the collection’s type (such as a refinement if needed), and require sourceId to be an integer rather than any number.packages/contents/src/types.ts (1)
72-73: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider declaring the property optional.
content: string | null | undefinedis a required key that acceptsundefined. Callers must still writecontent: undefined.content?: string | nullexpresses the same contract and removes that requirement.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/contents/src/types.ts` around lines 72 - 73, Update the content property declaration in the relevant type to be optional, using an optional string-or-null type instead of a required property that explicitly accepts undefined. Preserve support for string and null values while allowing callers to omit content entirely.packages/api/resources/registry.ts (1)
15-15: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider freezing the exported
resourceTypesarray.
resourceTypesis a mutable array exported from a module. A consumer can push or splice it and change behavior for every other consumer.Object.freeze(Object.keys(registry))with areadonly string[]type removes that risk.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/api/resources/registry.ts` at line 15, Freeze the exported resourceTypes array created from registry, and expose it with a readonly string[] type so consumers cannot mutate the shared list. Preserve the existing registry-derived values and export name.apps/service/__tests__/setup.ts (1)
85-93: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueRemove stale Algolia variables from
apps/service/.env.example. The mocks match all runtime exports and imports. The environment schema has no Algolia entries, but the example file still listsALGOLIA_APPLICATION_ID,ALGOLIA_API_KEY, andALGOLIA_FEEDS_INDEX_NAME.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/service/__tests__/setup.ts` around lines 85 - 93, Remove the stale ALGOLIA_APPLICATION_ID, ALGOLIA_API_KEY, and ALGOLIA_FEEDS_INDEX_NAME entries from the environment example configuration. Leave the existing search mocks and runtime exports unchanged.packages/db/src/libs/resources/chunk.ts (4)
87-99: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe unchanged branch does not refresh
updatedAt.The changed branch on line 119 sets
updatedAt. The visibility-only branch does not. That is defensible, but it makesupdatedAtmean "content changed" in one branch and nothing in the other. Please make the intent explicit with a short comment or set the column in both branches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/chunk.ts` around lines 87 - 99, Clarify the intended updatedAt semantics in the unchanged branch of the chunk update flow around the previous?.contentHash check: either add a brief comment explaining why visibility-only updates leave updatedAt unchanged, or update the .set call to refresh updatedAt consistently with the changed-content branch.
116-116: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThe comment describes a cascade that does not happen.
The foreign key on
resource_embedding.chunk_idcascades onDELETEof the chunk row only. AnUPDATEofcontentdoes not remove the vector. The explicitDELETEon lines 121-123 is what clears it. Please correct the comment so a later reader does not remove that delete.📝 Proposed wording
- // content changed — the update cascades the old vector away + // content changed — the vector must be dropped explicitly; an + // UPDATE does not trigger the FK cascade🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/chunk.ts` at line 116, Correct the comment near the explicit vector deletion in the chunk content update flow: state that updating content does not cascade-delete the old vector and that the explicit DELETE is required to clear it. Keep the deletion logic unchanged.
191-197: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winThe backlog query has no
ORDER BY, so repeated runs can revisit the same chunks.
limitbounds the result, but Postgres does not guarantee a stable row order withoutORDER BY. A worker that calls this in a loop can receive an overlapping set and starve some chunks. Order bychunks.id.♻️ Proposed change
.where( and( isNull(embeddings.chunkId), dto.ref ? sourceFilter(dto.ref) : undefined ) ) + .orderBy(chunks.id) .limit(dto.limit ?? 200);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/chunk.ts` around lines 191 - 197, Update the backlog query in the chunk resource to order results by chunks.id before applying the limit, preserving the existing filters and limit behavior.
84-139: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftThe replace loop issues one round trip per chunk and scans a list per orphan check.
Two points:
- Every chunk runs its own
UPDATEorINSERTinside the transaction. A document with 100 sections costs more than 100 sequential round trips while holding the transaction open.keptKeys.includes(...)on line 134 makes the orphan scan O(n·m). ASetmakes it O(n).A single
INSERT ... ON CONFLICT (source_type, source_id, kind, chunk_index) DO UPDATEcovers the insert and the changed-content branch in one statement. The unique indexresource_chunk_source_kind_index_idxalready supports that conflict target.♻️ Minimum change for the orphan scan
- const keptKeys = dto.chunks.map((chunk) => - key(chunk.kind, chunk.chunkIndex) - ); + const keptKeys = new Set( + dto.chunks.map((chunk) => key(chunk.kind, chunk.chunkIndex)) + ); const orphans = existing - .filter((row) => !keptKeys.includes(key(row.kind, row.chunkIndex))) + .filter((row) => !keptKeys.has(key(row.kind, row.chunkIndex))) .map((row) => row.id);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/chunk.ts` around lines 84 - 139, Refactor the chunk replacement loop to batch all chunk inserts and updates with a single transaction statement using INSERT ... ON CONFLICT on (source_type, source_id, kind, chunk_index), updating changed content and mirrored visibility while preserving unchanged handling and embedding cleanup. In the orphan cleanup after this batch, replace the keptKeys array and includes lookup with a Set so membership checks are O(1), while retaining the existing orphan deletion behavior.packages/db/src/libs/resources/search.ts (3)
329-333: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe
innerJoinon line 329 has no join predicate againstsource.
eq(chunks.kind, "card")is a filter, not a join condition. The statement relies onsourceholding at most one row so the cross product stays bounded. That holds today because of thelimit(1)on line 316, but the intent is not obvious.Move
eq(chunks.kind, "card")into thewhereclause and use an explicitcrossJoin, or add a comment that states the single-row assumption.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/search.ts` around lines 329 - 333, Update the query around the joins in the search flow to make the missing predicate against source explicit: move eq(chunks.kind, "card") into the where clause and replace the predicate-less innerJoin(chunks, ...) with an explicit crossJoin, preserving the existing limit(1) single-row assumption and other join conditions.
319-346: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift
findSimilarResourcescannot use the HNSW index.The right-hand operand of
<=>issource.embedding, a column of a CTE rather than a parameter. Postgres does not use a vector index when both sides are columns, so this runs a sequential scan over every card embedding and computes the distance for each row.Fetch the source vector in a separate query, then pass it as a bound parameter to the similarity query. That keeps the index usable and also removes the duplicated similarity expression on lines 319 and 342.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/search.ts` around lines 319 - 346, Update findSimilarResources to fetch the source embedding in a separate query, then pass that vector as a bound parameter to the similarity query so the HNSW index remains usable. Remove the source CTE and use one shared similarity expression for both selection and threshold filtering, while preserving the existing joins, exclusions, ordering, and limit.
251-281: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
aggregateChunkHitsdepends on the caller sorting the hits.Line 267 states that hits arrive in score order. Nothing in the signature enforces it. If an unsorted array is passed,
bestChunkand the top-N mean are both wrong, and the failure is silent.Sort each bucket by score before slicing.
♻️ Proposed change
return [...byResource.values()] .map((bucket) => { - // hits arrive in score order, so the head is already the best - const top = bucket.slice(0, topN); + const ordered = [...bucket].sort((a, b) => b.score - a.score); + const top = ordered.slice(0, topN); const score = top.reduce((sum, hit) => sum + hit.score, 0) / top.length; - const best = bucket[0]!; + const best = ordered[0]!;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/resources/search.ts` around lines 251 - 281, Update aggregateChunkHits to sort each resource bucket by descending hit.score before selecting top and bestChunk. Use the sorted bucket for both the top-N mean and bestChunk, while preserving matchedChunks and the existing resource-level sorting and limit behavior.packages/db/src/schemas/resources.schema.ts (2)
163-167: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffThe HNSW index does not include
model, so multi-model recall degrades.
searchChunksSemanticandfindSimilarResourcesboth filter onembeddings.model. The HNSW index covers every row of every model. Postgres retrieves the approximate top-K across all models and then discards rows of other models, so the effective result count drops as soon as a second model exists.If more than one model will ever be stored, index per model or add a partial index.
♻️ Example: partial index per active model
index("resource_embedding_model_idx").on(table.model), index("resource_embedding_hnsw_idx").using( "hnsw", table.embedding.op("vector_cosine_ops") - ), + ).where(sql`model = 'text-embedding-3-small'`),🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/schemas/resources.schema.ts` around lines 163 - 167, Update the resource_embedding_hnsw_idx definition in the schema to account for the model filter used by searchChunksSemantic and findSimilarResources. Scope the HNSW index per stored embedding model, or otherwise add the required model-aware partial/index strategy, so approximate nearest-neighbor retrieval does not rank unrelated models together.
33-37: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffThese casts hide a duplicated
drizzle-orminstall.The comment explains the symptom. The root cause is that
@paradedb/drizzle-paradedbresolves a second copy ofdrizzle-orm. A pnpmoverridesorpeerDependencyRulesentry inpnpm-workspace.yamldeduplicates the instance and removes the need forunknowncasts. The casts silently accept any value today, so a wrong column would compile.This is optional if the duplicate resolution is intentional.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/schemas/resources.schema.ts` around lines 33 - 37, Deduplicate the drizzle-orm dependency used by `@paradedb/drizzle-paradedb` via the workspace pnpm configuration, using an appropriate overrides or peerDependencyRules entry. Then remove the unknown-based casts in ParadedbIndexArgs, pdbKeyField, pdbField, and pdbTokenized, relying on the unified drizzle-orm types while preserving their existing argument typing.packages/db/src/libs/feeds/index.ts (2)
477-491: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueConfirm the tag query cost per indexing run.
getFeedForIndexingnow runs a second query for tag names and returns one row per(tag, locale)pair.loadFeedForIndexingStepgroups them again in memory. This is correct, but the indexing workflow calls it per feed. Consider whether a batched variant is needed once a full reindex runs over every feed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/feeds/index.ts` around lines 477 - 491, Review the getFeedForIndexing and loadFeedForIndexingStep flow to confirm the added tagTranslations query is acceptable when invoked once per feed. If full reindexing makes this N+1 cost excessive, introduce a batched feed-loading path that retrieves tag data for multiple feeds while preserving the existing per-feed tag and translation output; otherwise document or retain the current behavior without unrelated changes.
234-240: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTwo raw subqueries hard-code the
chia_table prefix. BothhasEmbeddingexpressions writechia_resource_chunkandchia_resource_embeddingas string literals. The prefix is produced by the custompgTablecreator, so a prefix change breaks both queries at runtime with no compile-time signal.
packages/db/src/libs/feeds/index.ts#L234-L240: interpolate theresourceChunksandresourceEmbeddingsschema objects into theextras.hasEmbeddingtemplate instead of the literal names.packages/db/src/libs/feeds/index.ts#L341-L345: apply the same substitution in thegetFeedDetailsselect, and extract the shared expression into one helper so the two sites cannot diverge.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/src/libs/feeds/index.ts` around lines 234 - 240, In packages/db/src/libs/feeds/index.ts at lines 234-240 and 341-345, extract the shared hasEmbedding SQL expression into one helper and update both extras.hasEmbedding and the getFeedDetails select to interpolate the resourceChunks and resourceEmbeddings schema objects instead of hard-coded chia_resource_chunk and chia_resource_embedding names, keeping both query sites consistent.packages/db/.drizzle/migrations/20260811165041_resource_chunks/migration.sql (1)
38-50: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low valueConfirm the deploy order for the destructive step.
The backfill and the
DROP TABLEstatements run in the same migration. Any running instance of the old code that still readschia_contentorchia_feed_embeddingfails immediately after the drop. There is also no path back once the tables are gone.If a rolling deploy is possible, split the drop into a follow-up migration that runs after the new code is live. If the deploy is a single cutover, this is fine as written.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/db/.drizzle/migrations/20260811165041_resource_chunks/migration.sql` around lines 38 - 50, Separate the destructive DROP TABLE statements from the backfill migration when rolling deployments are supported: keep the data migration for chia_content and chia_feed_embedding, then add a follow-up migration to drop them after all instances run the new code. If deployment is guaranteed to be a single cutover, document or otherwise preserve the current ordering.Source: Linters/SAST tools
packages/ai/__tests__/embeddings-document-card.spec.mts (2)
97-99: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe comment states the wrong count.
The comment says 500 copies. The code repeats 5000 times.
🔧 Proposed change
- // 500 copies of the body under the same outline: the card must not grow + // 5000 copies of the body under the same outline: the card must not grow🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/__tests__/embeddings-document-card.spec.mts` around lines 97 - 99, The test comment above the long article fixture incorrectly says 500 copies; update it to accurately describe the 5000 repetitions used by the repeat call, without changing the test logic.
160-168: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a case with no sentence boundary.
Every oversized fixture uses
。, sosplitOversizedalways finds a split point. A paragraph with no sentence punctuation takes the path that emits an over-target chunk, described in the comment onpackages/ai/src/embeddings/chunking.tslines 42-54. Add a fixture such as one long CJK run without punctuation to cover it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/__tests__/embeddings-document-card.spec.mts` around lines 160 - 168, Add a test case alongside the existing oversized-section test using a long CJK paragraph with no sentence punctuation, then pass it through chunkMarkdown and assert the resulting chunks follow the intended oversized-chunk behavior. Ensure the fixture does not contain punctuation that lets splitOversized find a sentence boundary.packages/ai/__tests__/embeddings-context.spec.mts (1)
151-161: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winThe test does not assert the behavior in its name.
The name states that dropped documents are reported. The body only checks
totalTokens.droppedSlugsis the only signal for exhaustion, and no test covers it.🔧 Proposed change
expect(result.totalTokens).toBeLessThanOrEqual(900); + expect(result.droppedSlugs.length).toBeGreaterThan(0); + expect(result.documents.map((document) => document.slug)).not.toEqual([ + "a", + "b", + "c", + ]);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/__tests__/embeddings-context.spec.mts` around lines 151 - 161, Update the test named “reports documents dropped once the budget is exhausted” to assert the droppedSlugs result, verifying that the documents excluded after the budget is exhausted are reported. Keep the existing totalTokens budget assertion and use the expected slug values from the supplied documents.packages/ai/src/embeddings/provider.ts (1)
82-92: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUpdate the storage location named in the comment.
The comment says the key is stored in
feed_translation.index_key. This PR moves chunk and embedding storage to the resource chunk tables. Name the column that actually holds the value, so the invalidation contract stays traceable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai/src/embeddings/provider.ts` around lines 82 - 92, Update the documentation comment above buildIndexKey to name the resource chunk table column that stores the generated key, replacing the outdated feed_translation.index_key reference. Keep the description of the key composition and invalidation behavior unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 00a6f783-5b21-4a86-8737-bd8a54f731e3
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (76)
apps/dash/src/app/(workspace)/feed/(edit)/edit/[id]/page.tsxapps/dash/src/components/feed/meta-chip.tsxapps/dash/src/components/feed/search-feed.tsxapps/service/__tests__/__mocks__/db.mock.tsapps/service/__tests__/feeds-search-auth.controller.test.tsapps/service/__tests__/feeds.controller.test.tsapps/service/__tests__/feeds.service.test.tsapps/service/__tests__/setup.tsapps/service/src/env.tsapps/service/src/services/agent-content.port.tsapps/service/src/services/feed-indexing.service.tsapps/service/src/steps/algolia-search.step.tsapps/service/src/steps/feed-embeddings.step.tsapps/service/src/steps/feed-indexing.step.tsapps/service/src/steps/resource-index.step.tsapps/service/src/workflows/algolia-search.workflow.tsapps/service/src/workflows/feed-indexing.workflow.tsapps/service/src/workflows/feed-removal.workflow.tsapps/service/src/workflows/resource-index.workflow.tsapps/www/src/app/[locale]/(blog)/[type]/[slug]/llm.md/route.tsapps/www/src/app/[locale]/(blog)/[type]/[slug]/page.tsxpackages/agent-writing/package.jsonpackages/agent-writing/src/ports.tspackages/agent-writing/src/tools/retrieval.tool.tspackages/agent-writing/src/types.tspackages/ai/__tests__/embeddings-context.spec.mtspackages/ai/__tests__/embeddings-document-card.spec.mtspackages/ai/__tests__/embeddings-tokenizer.spec.mtspackages/ai/package.jsonpackages/ai/src/embeddings/chunking.tspackages/ai/src/embeddings/context.tspackages/ai/src/embeddings/markdown.tspackages/ai/src/embeddings/ollama.tspackages/ai/src/embeddings/openai.tspackages/ai/src/embeddings/provider.tspackages/ai/src/embeddings/tokenizer.tspackages/ai/src/embeddings/utils.tspackages/api/__tests__/router-import.test.tspackages/api/algolia/client.tspackages/api/algolia/env.tspackages/api/feeds/search.tspackages/api/feeds/validator.tspackages/api/feeds/write.tspackages/api/orpc/contracts/content.contract.tspackages/api/orpc/contracts/feeds.contract.tspackages/api/orpc/routes/content.route.tspackages/api/orpc/routes/feeds.route.tspackages/api/package.jsonpackages/api/resources/feed-translation.resource.tspackages/api/resources/registry.tspackages/api/resources/search.tspackages/api/resources/types.tspackages/api/services/utils.tspackages/api/services/validators.tspackages/api/turbo.jsonpackages/contents/src/services.tspackages/contents/src/types.tspackages/db/.drizzle/migrations/20260811165041_resource_chunks/migration.sqlpackages/db/.drizzle/migrations/20260811165041_resource_chunks/snapshot.jsonpackages/db/package.jsonpackages/db/src/libs/feeds/embedding.tspackages/db/src/libs/feeds/index.tspackages/db/src/libs/feeds/search.tspackages/db/src/libs/resources/chunk.tspackages/db/src/libs/resources/search.tspackages/db/src/libs/validator/feeds.tspackages/db/src/schemas/contents.schema.tspackages/db/src/schemas/index.tspackages/db/src/schemas/relations.tspackages/db/src/schemas/resources.schema.tspackages/db/vitest.config.tsplans/.gitkeepplans/rag-long-document-retrieval-plan.mdplans/rag-optimization-plan.mdplans/service-transport-unification-plan.mdpnpm-workspace.yaml
💤 Files with no reviewable changes (12)
- packages/api/services/utils.ts
- apps/service/src/env.ts
- plans/service-transport-unification-plan.md
- packages/api/algolia/env.ts
- apps/service/src/workflows/algolia-search.workflow.ts
- packages/api/services/validators.ts
- apps/service/src/steps/algolia-search.step.ts
- packages/db/src/libs/feeds/embedding.ts
- plans/rag-optimization-plan.md
- plans/rag-long-document-retrieval-plan.md
- packages/api/algolia/client.ts
- apps/service/src/steps/feed-embeddings.step.ts
| const [row] = await db | ||
| .update(feedTranslations) | ||
| .set({ | ||
| content: dto.content, | ||
| source: dto.source, | ||
| unstableSerializedSource: dto.unstableSerializedSource, | ||
| updatedAt: new Date(), | ||
| }) | ||
| .onConflictDoUpdate({ | ||
| target: contents.feedTranslationId, | ||
| set: { | ||
| content: dto.content, | ||
| source: dto.source, | ||
| unstableSerializedSource: dto.unstableSerializedSource, | ||
| updatedAt: new Date(), | ||
| }, | ||
| }) | ||
| .returning(); | ||
|
|
||
| return content; | ||
| .where(eq(feedTranslations.id, dto.feedTranslationId)) | ||
| .returning({ | ||
| feedTranslationId: feedTranslations.id, | ||
| content: feedTranslations.content, | ||
| source: feedTranslations.source, | ||
| unstableSerializedSource: feedTranslations.unstableSerializedSource, | ||
| }); | ||
|
|
||
| return row; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
upsertContent now returns undefined when the translation does not exist.
The previous implementation inserted into contents with a conflict clause, so it always produced a row. The UPDATE matches nothing when dto.feedTranslationId is unknown, and row is undefined. The function name still says "upsert", so callers can dereference the result.
Either throw on an empty result or change the name and the return type to make the miss explicit.
🐛 Proposed fix
.returning({
feedTranslationId: feedTranslations.id,
content: feedTranslations.content,
source: feedTranslations.source,
unstableSerializedSource: feedTranslations.unstableSerializedSource,
});
+ if (!row) {
+ throw new Error(
+ `Feed translation ${dto.feedTranslationId} not found`
+ );
+ }
return row;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const [row] = await db | |
| .update(feedTranslations) | |
| .set({ | |
| content: dto.content, | |
| source: dto.source, | |
| unstableSerializedSource: dto.unstableSerializedSource, | |
| updatedAt: new Date(), | |
| }) | |
| .onConflictDoUpdate({ | |
| target: contents.feedTranslationId, | |
| set: { | |
| content: dto.content, | |
| source: dto.source, | |
| unstableSerializedSource: dto.unstableSerializedSource, | |
| updatedAt: new Date(), | |
| }, | |
| }) | |
| .returning(); | |
| return content; | |
| .where(eq(feedTranslations.id, dto.feedTranslationId)) | |
| .returning({ | |
| feedTranslationId: feedTranslations.id, | |
| content: feedTranslations.content, | |
| source: feedTranslations.source, | |
| unstableSerializedSource: feedTranslations.unstableSerializedSource, | |
| }); | |
| return row; | |
| const [row] = await db | |
| .update(feedTranslations) | |
| .set({ | |
| content: dto.content, | |
| source: dto.source, | |
| unstableSerializedSource: dto.unstableSerializedSource, | |
| updatedAt: new Date(), | |
| }) | |
| .where(eq(feedTranslations.id, dto.feedTranslationId)) | |
| .returning({ | |
| feedTranslationId: feedTranslations.id, | |
| content: feedTranslations.content, | |
| source: feedTranslations.source, | |
| unstableSerializedSource: feedTranslations.unstableSerializedSource, | |
| }); | |
| if (!row) { | |
| throw new Error( | |
| `Feed translation ${dto.feedTranslationId} not found` | |
| ); | |
| } | |
| return row; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/db/src/libs/feeds/index.ts` around lines 743 - 759, Update
upsertContent so an update that matches no feed translation cannot return
undefined: detect an empty returning result and throw an appropriate error
before returning row. Preserve the existing returned translation fields for
successful updates and keep the upsertContent contract safe for callers.
| export const pruneStaleEmbeddings = withDTO( | ||
| async (db, dto: { model: string; indexVersion: string }) => { | ||
| const deleted = await db | ||
| .delete(embeddings) | ||
| .where( | ||
| sql`(${embeddings.model}, ${embeddings.indexVersion}) <> (${dto.model}, ${dto.indexVersion})` | ||
| ) | ||
| .returning({ chunkId: embeddings.chunkId }); | ||
| return { deletedCount: deleted.length }; | ||
| } | ||
| ); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
pruneStaleEmbeddings deletes every vector that does not match the single supplied pair.
The statement has no resource scope and no guard. If two models are in use, or if a model migration is partly complete, this call removes all vectors of the other model in one statement. Semantic search then returns nothing until a full reindex finishes.
Add an explicit allow-list of active (model, indexVersion) pairs, and require that the new pair already covers the chunk before the old one is dropped.
🛡️ Proposed change
export const pruneStaleEmbeddings = withDTO(
- async (db, dto: { model: string; indexVersion: string }) => {
+ async (
+ db,
+ dto: { active: { model: string; indexVersion: string }[]; ref?: ResourceRef }
+ ) => {
+ if (dto.active.length === 0) {
+ return { deletedCount: 0 };
+ }
const deleted = await db
.delete(embeddings)
.where(
- sql`(${embeddings.model}, ${embeddings.indexVersion}) <> (${dto.model}, ${dto.indexVersion})`
+ sql`(${embeddings.model}, ${embeddings.indexVersion}) not in ${dto.active.map((pair) => [pair.model, pair.indexVersion])}`
)
.returning({ chunkId: embeddings.chunkId });
return { deletedCount: deleted.length };
}
);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/db/src/libs/resources/chunk.ts` around lines 239 - 249, Update
pruneStaleEmbeddings to accept an explicit allow-list of active
model/indexVersion pairs, rather than deleting every embedding that differs from
one supplied pair. Scope deletion to embeddings belonging to the relevant
resource and only remove an old embedding when the new active pair already
covers the same chunk. Preserve the deletedCount result while ensuring active
pairs and uncovered chunks are never deleted.
| const query = dto.query.trim(); | ||
| if (!query && dto.embedding.length === 0) { | ||
| return []; | ||
| } | ||
|
|
||
| const limit = dto.limit ?? 20; | ||
| const candidateLimit = dto.candidateLimit ?? Math.max(limit * 3, 40); | ||
| const scope = scopeFilter(dto); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
The guard passes when only one input is empty.
return [] requires both the query and the embedding to be empty. Two reachable states get through:
queryis empty andembeddingis populated.lexicalMatch("")buildspdb.matchAny(content, '')andpdb.phrase(..., ''). ParadeDB rejects an empty phrase query.embeddingis empty andqueryis populated.cosineDistance(embeddings.embedding, [])produces'[]'::vector, which fails the 1536-dimension check.
searchResources in packages/api/resources/search.ts calls this with whatever embedQuery returns, so a provider failure that yields an empty array reaches the second state.
Handle each side independently.
🐛 Proposed fix
const query = dto.query.trim();
- if (!query && dto.embedding.length === 0) {
+ const hasLexical = query.length > 0;
+ const hasSemantic = dto.embedding.length > 0;
+ if (!hasLexical && !hasSemantic) {
return [];
}
+ if (!hasSemantic) {
+ return await searchChunksLexical(db, { ...dto, query, limit: dto.limit });
+ }
+ if (!hasLexical) {
+ return await searchChunksSemantic(db, {
+ ...dto,
+ embedding: dto.embedding,
+ model: dto.model,
+ limit: dto.limit,
+ });
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const query = dto.query.trim(); | |
| if (!query && dto.embedding.length === 0) { | |
| return []; | |
| } | |
| const limit = dto.limit ?? 20; | |
| const candidateLimit = dto.candidateLimit ?? Math.max(limit * 3, 40); | |
| const scope = scopeFilter(dto); | |
| const query = dto.query.trim(); | |
| const hasLexical = query.length > 0; | |
| const hasSemantic = dto.embedding.length > 0; | |
| if (!hasLexical && !hasSemantic) { | |
| return []; | |
| } | |
| if (!hasSemantic) { | |
| return await searchChunksLexical(db, { ...dto, query, limit: dto.limit }); | |
| } | |
| if (!hasLexical) { | |
| return await searchChunksSemantic(db, { | |
| ...dto, | |
| embedding: dto.embedding, | |
| model: dto.model, | |
| limit: dto.limit, | |
| }); | |
| } | |
| const limit = dto.limit ?? 20; | |
| const candidateLimit = dto.candidateLimit ?? Math.max(limit * 3, 40); | |
| const scope = scopeFilter(dto); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/db/src/libs/resources/search.ts` around lines 171 - 178, Update the
guard in searchResources so it returns an empty result whenever either the
trimmed query is empty or dto.embedding has no values, preventing lexical
searches with an empty phrase and vector searches with an empty embedding.
Preserve the existing limit, candidateLimit, and scope setup for requests where
both inputs are present.
- Complete pending embedding batches reliably - Deduplicate feed search and related-feed results
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai/src/embeddings/chunking.ts`:
- Around line 94-112: Update splitOversized’s buffer accounting to include both
unit.joiner and the "\n\n" separator between buffered sections when calculating
the pending token total before the capacity check. After flush(), recalculate
the pending addition with zero separator cost, then update bufferTokens using
the complete addition so maxTokens and targetTokens cannot be exceeded.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a9a65833-c9dd-464b-96bd-d8b3d35553f2
📒 Files selected for processing (16)
apps/service/src/services/agent-content.port.tsapps/service/src/steps/resource-index.step.tsapps/service/src/workflows/feed-indexing.workflow.tsapps/service/src/workflows/resource-index.workflow.tspackages/ai/src/embeddings/chunking.tspackages/ai/src/embeddings/context.tspackages/ai/src/embeddings/markdown.tspackages/ai/src/embeddings/ollama.tspackages/ai/src/embeddings/provider.tspackages/ai/src/embeddings/utils.tspackages/api/feeds/search.tspackages/api/resources/feed-translation.resource.tspackages/api/resources/registry.tspackages/db/src/libs/feeds/search.tspackages/db/src/libs/resources/chunk.tspackages/db/src/schemas/resources.schema.ts
💤 Files with no reviewable changes (1)
- packages/db/src/libs/resources/chunk.ts
🚧 Files skipped from review as they are similar to previous changes (10)
- apps/service/src/workflows/feed-indexing.workflow.ts
- packages/db/src/libs/feeds/search.ts
- packages/ai/src/embeddings/provider.ts
- packages/ai/src/embeddings/context.ts
- apps/service/src/workflows/resource-index.workflow.ts
- packages/api/resources/feed-translation.resource.ts
- packages/ai/src/embeddings/markdown.ts
- apps/service/src/services/agent-content.port.ts
- apps/service/src/steps/resource-index.step.ts
- packages/api/feeds/search.ts
| for (const unit of toUnits(text, maxTokens, encoding)) { | ||
| const unitTokens = countEmbeddingTokens(unit.text, encoding); | ||
|
|
||
| if (buffer && bufferTokens + unitTokens > maxTokens) { | ||
| flush(); | ||
| } | ||
| const heading = insideCodeFence ? null : /^(#{1,6})\s+(.+)$/.exec(line); | ||
| if (heading?.[1] && heading[2]) { | ||
|
|
||
| if (unitTokens > maxTokens) { | ||
| flush(); | ||
| const level = heading[1].length; | ||
| while ( | ||
| headingStack.length > 0 && | ||
| headingStack[headingStack.length - 1]!.level >= level | ||
| ) { | ||
| headingStack.pop(); | ||
| for (const slice of sliceToBudget(unit.text, maxTokens)) { | ||
| if (slice.trim()) { | ||
| pieces.push(slice.trim()); | ||
| } | ||
| } | ||
| headingStack.push({ level, title: heading[2].trim() }); | ||
| continue; | ||
| } | ||
| buffer.push(line); | ||
|
|
||
| buffer += (buffer ? unit.joiner : "") + unit.text; | ||
| bufferTokens += unitTokens; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Count joiners in both token budgets.
bufferTokens excludes unit.joiner in splitOversized. It also excludes the "\n\n" separator between buffered sections. A chunk can exceed maxTokens or targetTokens even when the tracked total fits.
Count the complete pending addition before the capacity check. After a flush, recalculate the separator cost as zero.
Proposed fix
for (const unit of toUnits(text, maxTokens, encoding)) {
- const unitTokens = countEmbeddingTokens(unit.text, encoding);
+ const addition = (buffer ? unit.joiner : "") + unit.text;
+ const additionTokens = countEmbeddingTokens(addition, encoding);
- if (buffer && bufferTokens + unitTokens > maxTokens) {
+ if (buffer && bufferTokens + additionTokens > maxTokens) {
flush();
}
- buffer += (buffer ? unit.joiner : "") + unit.text;
- bufferTokens += unitTokens;
+ buffer += (buffer ? unit.joiner : "") + unit.text;
+ bufferTokens += countEmbeddingTokens(
+ buffer ? unit.joiner + unit.text : unit.text,
+ encoding
+ );
}Also applies to: 184-188
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai/src/embeddings/chunking.ts` around lines 94 - 112, Update
splitOversized’s buffer accounting to include both unit.joiner and the "\n\n"
separator between buffered sections when calculating the pending token total
before the capacity check. After flush(), recalculate the pending addition with
zero separator cost, then update bufferTokens using the complete addition so
maxTokens and targetTokens cannot be exceeded.
- Cover chunk indexing, hybrid retrieval, and embedding maintenance
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/rag-architecture.md`:
- Line 45: Add language identifiers to every fenced code block in
docs/rag-architecture.md to satisfy markdownlint MD040; use text for tree
diagrams and pseudocode, while preserving specific identifiers such as mermaid
where applicable.
- Around line 81-85: The RAG schema and search flow must prevent mixed embedding
versions during reindexing. Use the existing index_version field to track the
active version, require semantic and hybrid search APIs to accept and filter by
that active version alongside model, and ensure reindexing and rollback update
the active version consistently.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 53120bc5-4a72-4a97-b06f-a00180007f8c
📒 Files selected for processing (2)
apps/service/src/steps/resource-index.step.tsdocs/rag-architecture.md
🚧 Files skipped from review as they are similar to previous changes (1)
- apps/service/src/steps/resource-index.step.ts
| 向量獨立成表,不長在 `feed_translation` 上——換模型、加模型、重新索引都是資料操作,不需要 schema migration。 | ||
| ### 2.1 `chia_resource_chunk` | ||
|
|
||
| ``` |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add language identifiers to the fenced blocks.
markdownlint MD040 reports these fences without a language. Use text for trees and pseudocode, and keep specific identifiers such as mermaid where applicable.
Also applies to: 77-77, 121-121, 147-147, 160-160, 201-201, 235-235, 264-264
🧰 Tools
🪛 markdownlint-cli2 (0.23.2)
[warning] 45-45: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/rag-architecture.md` at line 45, Add language identifiers to every
fenced code block in docs/rag-architecture.md to satisfy markdownlint MD040; use
text for tree diagrams and pseudocode, while preserving specific identifiers
such as mermaid where applicable.
Source: Linters/SAST tools
| ├── index_version text | ||
| ├── embedding vector(1536) | ||
| └── created_at / updated_at | ||
| PRIMARY KEY (chunk_id, model) | ||
| + HNSW index(vector_cosine_ops) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 8 \
'resource_embedding|resourceEmbeddings|index_version|indexVersion|EMBEDDING_INDEX_VERSION|saveChunkEmbeddings|embedPendingChunks' \
packages/db apps/service packages/aiRepository: Chia1104/chia1104.dev
Length of output: 29653
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
files = {
"docs/rag-architecture.md": [(75, 90), (150, 175), (270, 305)],
"packages/db/src/schemas/resources.schema.ts": [(151, 171)],
"packages/db/src/libs/resources/chunk.ts": [(162, 198), (201, 234)],
"packages/db/src/libs/resources/search.ts": [(1, 180)],
"apps/service/src/steps/resource-index.step.ts": [(61, 125)],
}
for name, ranges in files.items():
path = Path(name)
print(f"\n### {name}")
lines = path.read_text().splitlines()
for start, end in ranges:
print(f"-- lines {start}-{end} --")
for number in range(start, min(end, len(lines)) + 1):
print(f"{number}: {lines[number - 1]}")
PY
python3 - <<'PY'
from pathlib import Path
schema = Path("packages/db/src/schemas/resources.schema.ts").read_text()
chunk = Path("packages/db/src/libs/resources/chunk.ts").read_text()
search = Path("packages/db/src/libs/resources/search.ts").read_text()
checks = {
"schema primary key excludes indexVersion": "primaryKey({ columns: [table.chunkId, table.model] })" in schema
and "primaryKey({ columns: [table.chunkId, table.model, table.indexVersion] })" not in schema,
"upsert conflict target excludes indexVersion": "target: [embeddings.chunkId, embeddings.model]" in chunk,
"upsert overwrites indexVersion": "indexVersion: dto.indexVersion" in chunk
and "indexVersion: dto.indexVersion" in chunk[chunk.find("onConflictDoUpdate"):],
"search source references embeddings": "const embeddings = schema.resourceEmbeddings" in search,
"search filters by model": "eq(embeddings.model" in search,
"search filters by indexVersion": "indexVersion" in search[search.find("const embeddings"):],
}
for label, result in checks.items():
print(f"{label}: {result}")
PYRepository: Chia1104/chia1104.dev
Length of output: 14801
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 6 \
'searchChunksSemantic|searchChunksHybrid|normalizeQueryForEmbedding|buildIndexKey|EMBEDDING_INDEX_VERSION|resource_embedding_model_idx|resource_embedding_hnsw_idx' \
apps packages docs/rag-architecture.md
python3 - <<'PY'
from pathlib import Path
path = Path("packages/db/src/libs/resources/search.ts")
lines = path.read_text().splitlines()
for start, end in [(176, 290)]:
print(f"-- {path}:{start}-{end} --")
for n in range(start, min(end, len(lines)) + 1):
print(f"{n}: {lines[n-1]}")
PYRepository: Chia1104/chia1104.dev
Length of output: 18359
Do not serve mixed embedding versions during reindexing.
The (chunk_id, model) key and upsert overwrite vectors in place, while semantic and hybrid searches filter only by model. A version bump therefore mixes old vectors with new query embeddings until reindexing completes. Add an active index_version filter and pass it through the search APIs, or define a migration strategy that guarantees version-consistent results and rollback.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/rag-architecture.md` around lines 81 - 85, The RAG schema and search
flow must prevent mixed embedding versions during reindexing. Use the existing
index_version field to track the active version, require semantic and hybrid
search APIs to accept and filter by that active version alongside model, and
ensure reindexing and rollback update the active version consistently.
Summary by CodeRabbit
New Features
Bug Fixes
close #2974