Skip to content

refactor(rag): refactor embedding flow - #2975

Merged
Chia1104 merged 4 commits into
developfrom
refactor/rag
Aug 12, 2026
Merged

Chia1104 merged 4 commits into
developfrom
refactor/rag

Conversation

@Chia1104

@Chia1104 Chia1104 commented Aug 11, 2026 •

Copy link
Copy Markdown
Owner

Summary by CodeRabbit

  • New Features

    • Added keyword, semantic, and hybrid search modes.
    • Improved results with cleaner highlights, relevant snippets, summaries, and consistent formatting.
    • Added related-feed recommendations.
    • Improved long-document retrieval with shared context limits, matched sections, outlines, and heading links.
    • Added resource indexing for more accurate, up-to-date search.
  • Bug Fixes

    • Fixed translation content handling across editing, publishing, and rendering.
    • Excluded unpublished or deleted content from search results.
    • Prevented empty search results from triggering unnecessary lookups.

close #2974

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
chia1104 Ready Ready Preview Aug 12, 2026 7:25am

Comment on lines +30 to +37
return source
.replace(/^(?:import|export)\s[^\n]*$/gm, "") // ESM statements
.replace(/```(\w*)\n([\s\S]*?)```/g, (_match, lang: string, code: string) =>
condenseCodeBlock(lang, code)
)
.replace(/!\[([^\]]*)\]\([^)]*\)/g, "$1") // images -> alt text
.replace(/\[([^\]]*)\]\([^)]*\)/g, "$1") // links -> label
.replace(/<\/?[A-Za-z][^>\n]*>/g, "") // JSX / HTML tags, keep children
if (FENCE_REGEX.test(line)) {
insideCodeFence = !insideCodeFence;
}
const heading = insideCodeFence ? null : HEADING_REGEX.exec(line);
@coderabbitai

coderabbitai Bot commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The pull request replaces Algolia and legacy feed embeddings with resource chunk indexing, BM25/semantic/hybrid search, centralized embedding providers, token-aware document processing, and direct translation content storage.

Changes

Resource storage and indexing

Layer / File(s) Summary
Resource persistence and search primitives
packages/db/..., packages/db/.drizzle/..., pnpm-workspace.yaml
Adds resource chunk and embedding tables, chunk persistence, visibility synchronization, BM25 search, semantic search, hybrid ranking, and resource similarity search.
Feed indexing workflows
apps/service/src/steps/..., apps/service/src/workflows/..., apps/service/src/services/feed-indexing.service.ts
Indexes feed translations as resources, embeds pending chunks, and removes chunks for deleted translations.

Embedding and retrieval

Layer / File(s) Summary
Embedding providers and token guards
packages/ai/src/embeddings/...
Adds provider resolution, token-aware input guarding, bounded OpenAI batching, Ollama preprocessing, and a shared embedding index key.
Markdown chunking and context assembly
packages/ai/src/embeddings/chunking.ts, packages/ai/src/embeddings/markdown.ts, packages/ai/src/embeddings/context.ts, packages/ai/__tests__/*
Adds heading-aware chunking, document-card construction, heading anchors, shared context budgets, and coverage for multilingual and budget-limited inputs.
Agent retrieval integration
packages/agent-writing/src/..., packages/agent-writing/package.json
Uses keyword search mode and builds shared token-bounded context across retrieved translations.

API and application integration

Layer / File(s) Summary
Resource and feed search API
packages/api/resources/*, packages/api/feeds/*, packages/api/orpc/..., packages/api/package.json
Adds resource adapters and search orchestration, maps feed translation hits to feed results, validates search modes, updates authorization, and removes Algolia exports.
Service consumers and tests
apps/service/src/services/agent-content.port.ts, apps/service/__tests__/*, apps/service/src/env.ts
Uses BM25 or hybrid search, normalizes snippets, updates mocks and authorization tests, and removes Algolia environment configuration.
Flat translation content consumers
apps/dash/..., apps/www/..., packages/contents/...
Reads translation content, source, and serialized source directly from translation fields. The dashboard renders normalized search hits with hybrid, BM25, and semantic modes.

Estimated code review effort: 5 (Critical) | ~120 minutes

Suggested labels: area:service

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately identifies the main RAG embedding-flow refactor, although it does not mention the broader resource search and chunking changes.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch refactor/rag

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 15

Note

Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.

🟡 Minor comments (9)
packages/api/feeds/search.ts-59-70 (1)

59-70: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Deduplicate feeds when several translations of one feed match.

searchResources aggregates per source (feed translation), so two translations of the same feed produce two hits. resolveFeedRefs maps both to the same feedId, and the flatMap keeps both. If locale is undefined, callers get duplicate feeds and the effective result count drops below limit. getRelatedFeeds in packages/db/src/libs/feeds/search.ts already drops other translations of the same feed, so the behavior is inconsistent. apps/service/src/services/agent-content.port.ts calls this service with an optional locale, so the path is reachable.

🔧 Proposed fix to keep the best hit per feed
   return {
     mode,
-    items: items.flatMap((item) => {
-      const ref = refs.get(item.sourceId);
-      return ref ? [{ ...item, ...ref }] : [];
-    }),
+    items: (() => {
+      const seenFeeds = new Set<number>();
+      return items.flatMap((item) => {
+        const ref = refs.get(item.sourceId);
+        if (!ref || seenFeeds.has(ref.feedId)) {
+          return [];
+        }
+        seenFeeds.add(ref.feedId);
+        return [{ ...item, ...ref }];
+      });
+    })(),
   };
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/api/feeds/search.ts` around lines 59 - 70, Update searchResources to
deduplicate resolved results by feedId when multiple source translations map to
the same feed, keeping only the best-ranked hit and preserving the requested
limit. Apply this especially when locale is undefined, while retaining distinct
feeds and the existing item enrichment behavior.
apps/service/src/services/agent-content.port.ts-82-92 (1)

82-92: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Pass input.limit to searchFeedsService.

searchFeedsService defaults limit to 5. If input.limit is greater than 5, the later slice(0, input.limit) cannot restore the missing results, so the agent receives fewer hits than it requested.

🔧 Proposed fix
       const result = await searchFeedsService({
         db,
         keyword: input.keyword,
         // `keyword` is in-database BM25; `semantic` fuses dense and lexical,
         // because a single document vector alone under-recalls exact terms
         // (package names, CLI flags, error messages)
         model: input.mode === "keyword" ? "bm25" : "hybrid",
         locale: input.locale,
+        limit: input.limit,
       });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/src/services/agent-content.port.ts` around lines 82 - 92, Pass
input.limit as the limit option in the searchFeedsService call within the agent
content search flow, while preserving the existing result slicing and mapping
behavior.
apps/www/src/app/[locale]/(blog)/[type]/[slug]/llm.md/route.ts-23-23 (1)

23-23: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Return 404 when localized Markdown content is missing.

feed.translations[0]?.content can be null or undefined. This returns a successful empty Markdown response. Validate the translation content before constructing the response.

Proposed fix
+  const content = feed.translations[0]?.content;
+  if (!content) {
+    notFound();
+  }
-  return new Response(feed.translations[0]?.content, {
+  return new Response(content, {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/www/src/app/`[locale]/(blog)/[type]/[slug]/llm.md/route.ts at line 23,
Validate feed.translations[0]?.content before constructing the Response in the
route handler, and return a 404 response when the localized Markdown content is
null or undefined. Preserve the existing successful response for available
content.
packages/api/package.json-124-143 (1)

124-143: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Migrate the legacy Trigger consumers before removing the subpath export.

legacy/trigger depends on @chia/api and includes both task files that import @chia/api/services/feeds. Update these imports before release. No source imports reference @chia/api/services/validators or @chia/api/algolia.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/api/package.json` around lines 124 - 143, The legacy Trigger task
files still import the soon-to-be-removed `@chia/api/services/feeds` subpath.
Update both task-file imports under legacy/trigger to use the supported feed
exports (such as the exposed feeds/validator or feeds/write entry points), then
remove the obsolete subpath export only after those consumers no longer depend
on it.
packages/db/src/libs/feeds/index.ts-466-473 (1)

466-473: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

A feed with no translations now resolves to null.

The query uses innerJoin on feedTranslations, so a feed without any translation row produces zero rows and the function returns null. feedIndexingWorkflow then reports "Feed not found", which is misleading and hides a real feed.

If a translation-less feed is possible, use leftJoin and filter out null translations, or fetch the feed and its translations separately.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/feeds/index.ts` around lines 466 - 473, The feed lookup
query in the function containing firstRow currently uses innerJoin, causing
feeds without translations to return null. Replace it with a leftJoin and filter
out null translation records while retaining the feed row, so
feedIndexingWorkflow does not report an existing translation-less feed as
missing.
packages/db/src/schemas/contents.schema.ts-133-135 (1)

133-135: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Synchronize feed-translation visibility mirrors.

upsertFeedTranslation does not copy the current feed flags, so new translations default to false. updateFeed, softDeleteFeed, and restoreFeed also leave existing mirrors stale. Update feedTranslations.published and feedTranslations.deleted in these paths, or remove the mirror columns from visibility queries. The schema comment must identify feeds as the indexing visibility source.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/schemas/contents.schema.ts` around lines 133 - 135,
Synchronize the visibility mirrors in upsertFeedTranslation, updateFeed,
softDeleteFeed, and restoreFeed with the corresponding feeds.published and
feeds.deleted values, including both new and existing translations. Update the
schema comment for published to identify feeds as the indexing visibility
source, while preserving the existing mirror columns and visibility-query
behavior.
packages/agent-writing/src/tools/retrieval.tool.ts-147-159 (1)

147-159: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Signal dropped locales instead of returning empty content.

buildDocumentContext drops a document when the shared budget is exhausted and reports it in droppedSlugs. Here every input carries the same post.slug, so droppedSlugs cannot identify which locale was dropped, and the code discards it. The dropped locale then returns content: "" and detail: "outline". The model reads that as an empty post.

Mark the omission explicitly.

🔧 Proposed change
     const translations = post.translations.map((translation) => {
       const document = byLocale.get(translation.locale);
+      if (!document) {
+        return {
+          ...translation,
+          content: "",
+          detail: "omitted" as const,
+          omittedReason: "context budget exhausted",
+          tokenCount: 0,
+          anchors: [],
+        };
+      }
       return {
         ...translation,
-        content: document?.text ?? "",
-        detail: document?.detail ?? "outline",
-        tokenCount: document?.tokenCount ?? 0,
-        anchors: document?.anchors.map((anchor) => anchor.anchor) ?? [],
+        content: document.text,
+        detail: document.detail,
+        tokenCount: document.tokenCount,
+        anchors: document.anchors.map((anchor) => anchor.anchor),
       };
     });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/agent-writing/src/tools/retrieval.tool.ts` around lines 147 - 159,
Update the translation mapping in buildDocumentContext’s caller to detect when a
locale’s document was omitted by the shared budget and mark that translation
explicitly as dropped, rather than returning empty content with outline detail.
Use the available dropped-slugs information and locale-specific context to
preserve normal document fields for retained locales.
pnpm-workspace.yaml-169-172 (1)

169-172: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Align the drizzle-orm override with the package manifest.

@paradedb/drizzle-paradedb@0.4.0 declares drizzle-orm as a direct dependency at 1.0.0-rc.2, not as a peer dependency. Confirm that 1.0.0-rc.4 is compatible before overriding the declared version.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pnpm-workspace.yaml` around lines 169 - 172, Update the drizzle-orm override
under the `@paradedb/drizzle-paradedb` selector to match its declared direct
dependency version 1.0.0-rc.2, unless compatibility with 1.0.0-rc.4 has been
explicitly confirmed.
packages/ai/src/embeddings/openai.ts-13-17 (1)

13-17: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Preserve the TextEmbeddingModel export from @chia/ai/embeddings/utils. External consumers of this public subpath will fail because the type now exists only in openai.ts. Add a compatibility re-export.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/src/embeddings/openai.ts` around lines 13 - 17, Restore the
TextEmbeddingModel export from the embeddings/utils public subpath by adding a
compatibility re-export there, while keeping the existing definition in
openai.ts as the source of truth. Ensure external imports from
`@chia/ai/embeddings/utils` continue resolving to the same type.
🧹 Nitpick comments (25)
packages/api/feeds/search.ts (1)

169-181: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Include the embedding provider id in the related-feeds cache key.

The query result depends on resolveEmbeddingProvider().id, but the key contains only locale, slug, and limit. If EMBEDDING_PROVIDER changes, the cache serves results computed with the previous model for up to six hours.

♻️ Proposed cache key change
-  const cacheKey = `feeds:related:${locale}:${slug}:${limit}`;
+  const model = resolveEmbeddingProvider().id;
+  const cacheKey = `feeds:related:${model}:${locale}:${slug}:${limit}`;
   const cached = await kv.get<RelatedFeedItems>(cacheKey);
   if (cached) {
     return cached;
   }
 
   const items = await getRelatedFeeds(db, {
     slug,
     locale,
     limit,
-    model: resolveEmbeddingProvider().id,
+    model,
   });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/api/feeds/search.ts` around lines 169 - 181, Update the cache-key
construction in the related-feeds flow to include the embedding provider id from
resolveEmbeddingProvider().id, ensuring reads and writes use the same
provider-specific key. Keep the existing locale, slug, and limit components
unchanged.
apps/service/src/steps/feed-indexing.step.ts (1)

47-53: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Deduplicate the per-locale tag names.

allTagNames removes duplicates with a Set, but tagNamesByLocale does not. If two tag translations in one locale share a name, the document card repeats that name. Apply the same deduplication to the grouped values.

♻️ Proposed fix
-  const tagNamesByLocale = new Map<string, string[]>();
+  const tagNamesByLocale = new Map<string, Set<string>>();
   for (const tag of feed.tags) {
-    const names = tagNamesByLocale.get(tag.locale) ?? [];
-    names.push(tag.name);
+    const names = tagNamesByLocale.get(tag.locale) ?? new Set<string>();
+    names.add(tag.name);
     tagNamesByLocale.set(tag.locale, names);
   }
   const allTagNames = [...new Set(feed.tags.map((tag) => tag.name))];

Then read it as:

-      tags: tagNamesByLocale.get(translation.locale) ?? allTagNames,
+      tags: [...(tagNamesByLocale.get(translation.locale) ?? allTagNames)],
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/src/steps/feed-indexing.step.ts` around lines 47 - 53,
Deduplicate each locale’s tag names while building tagNamesByLocale, so
duplicate tag.name values within the same tag.locale are stored only once.
Preserve allTagNames behavior and the existing locale grouping.
apps/service/src/workflows/feed-indexing.workflow.ts (1)

15-18: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

settledStatus puts the raw rejection text into the workflow result.

String(result.reason) is embedded in the returned value, which the workflow engine persists as durable run state. Embedding-provider errors can echo request content. Consider mapping to a short code plus the error name, and logging the full reason separately.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/src/workflows/feed-indexing.workflow.ts` around lines 15 - 18,
Update settledStatus to return only a short failure code plus the rejection
error name, rather than embedding String(result.reason) in BranchStatus. Log the
full rejection reason separately at the failure point while preserving the
existing "ok" result for fulfilled promises.
apps/service/src/workflows/feed-removal.workflow.ts (1)

7-9: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low value

Constrain the ID schema to integers.

z.number() accepts 1.5 and -0.5. translationIDs maps to sourceId, which is an integer column. A non-integer value passes validation and fails at the database instead of at the boundary. The same gap exists in apps/service/src/workflows/resource-index.workflow.ts.

🛡️ Proposed fix
 const requestSchema = z.object({
-  translationIDs: z.array(z.number()),
+  translationIDs: z.array(z.number().int().positive()),
 });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/src/workflows/feed-removal.workflow.ts` around lines 7 - 9,
Constrain the translationIDs elements in requestSchema to integers and apply the
same validation in the corresponding schema in resource-index.workflow.ts.
Preserve the existing array validation while rejecting fractional IDs before
they reach sourceId/database operations.
apps/service/src/services/feed-indexing.service.ts (1)

13-17: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Update the table names in the doc comment.

This PR replaces feed_embedding with the resource chunk and embedding tables. The comment still names feed_search_document and feed_embedding as the rows that cascade from feed_translation. Name the resource chunk table instead so the rationale stays verifiable.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/src/services/feed-indexing.service.ts` around lines 13 - 17,
Update the doc comment above the feed indexing logic to replace the obsolete
feed_embedding table name with the current resource chunk table name, while
preserving the existing feed_search_document reference and explanation of
cascade and soft-delete behavior.
apps/service/src/workflows/resource-index.workflow.ts (1)

6-9: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Validate sourceType against the registered adapters.

z.string() accepts any value. The value flows to getResourceAdapter in packages/api/resources/registry.ts, which performs an unguarded property lookup. Constrain the schema with the exported resourceTypes so the boundary rejects unknown types before the lookup runs. Also constrain sourceId to an integer.

🛡️ Proposed fix
+import { resourceTypes } from "`@chia/api/resources/registry`";
+
 export const resourceIndexRequestSchema = z.object({
-  sourceType: z.string(),
-  sourceId: z.number(),
+  sourceType: z.enum(resourceTypes as [string, ...string[]]),
+  sourceId: z.number().int().positive(),
 });

z.enum requires a non-empty tuple type. If the cast is unwanted, use z.string().refine((value) => resourceTypes.includes(value)).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/src/workflows/resource-index.workflow.ts` around lines 6 - 9,
Update resourceIndexRequestSchema to validate sourceType against the exported
resourceTypes collection before getResourceAdapter is called, using an approach
compatible with the collection’s type (such as a refinement if needed), and
require sourceId to be an integer rather than any number.
packages/contents/src/types.ts (1)

72-73: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider declaring the property optional.

content: string | null | undefined is a required key that accepts undefined. Callers must still write content: undefined. content?: string | null expresses the same contract and removes that requirement.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/contents/src/types.ts` around lines 72 - 73, Update the content
property declaration in the relevant type to be optional, using an optional
string-or-null type instead of a required property that explicitly accepts
undefined. Preserve support for string and null values while allowing callers to
omit content entirely.
packages/api/resources/registry.ts (1)

15-15: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider freezing the exported resourceTypes array.

resourceTypes is a mutable array exported from a module. A consumer can push or splice it and change behavior for every other consumer. Object.freeze(Object.keys(registry)) with a readonly string[] type removes that risk.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/api/resources/registry.ts` at line 15, Freeze the exported
resourceTypes array created from registry, and expose it with a readonly
string[] type so consumers cannot mutate the shared list. Preserve the existing
registry-derived values and export name.
apps/service/__tests__/setup.ts (1)

85-93: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low value

Remove stale Algolia variables from apps/service/.env.example. The mocks match all runtime exports and imports. The environment schema has no Algolia entries, but the example file still lists ALGOLIA_APPLICATION_ID, ALGOLIA_API_KEY, and ALGOLIA_FEEDS_INDEX_NAME.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/service/__tests__/setup.ts` around lines 85 - 93, Remove the stale
ALGOLIA_APPLICATION_ID, ALGOLIA_API_KEY, and ALGOLIA_FEEDS_INDEX_NAME entries
from the environment example configuration. Leave the existing search mocks and
runtime exports unchanged.
packages/db/src/libs/resources/chunk.ts (4)

87-99: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The unchanged branch does not refresh updatedAt.

The changed branch on line 119 sets updatedAt. The visibility-only branch does not. That is defensible, but it makes updatedAt mean "content changed" in one branch and nothing in the other. Please make the intent explicit with a short comment or set the column in both branches.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/chunk.ts` around lines 87 - 99, Clarify the
intended updatedAt semantics in the unchanged branch of the chunk update flow
around the previous?.contentHash check: either add a brief comment explaining
why visibility-only updates leave updatedAt unchanged, or update the .set call
to refresh updatedAt consistently with the changed-content branch.

116-116: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The comment describes a cascade that does not happen.

The foreign key on resource_embedding.chunk_id cascades on DELETE of the chunk row only. An UPDATE of content does not remove the vector. The explicit DELETE on lines 121-123 is what clears it. Please correct the comment so a later reader does not remove that delete.

📝 Proposed wording
-          // content changed — the update cascades the old vector away
+          // content changed — the vector must be dropped explicitly; an
+          // UPDATE does not trigger the FK cascade
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/chunk.ts` at line 116, Correct the comment
near the explicit vector deletion in the chunk content update flow: state that
updating content does not cascade-delete the old vector and that the explicit
DELETE is required to clear it. Keep the deletion logic unchanged.

191-197: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

The backlog query has no ORDER BY, so repeated runs can revisit the same chunks.

limit bounds the result, but Postgres does not guarantee a stable row order without ORDER BY. A worker that calls this in a loop can receive an overlapping set and starve some chunks. Order by chunks.id.

♻️ Proposed change
       .where(
         and(
           isNull(embeddings.chunkId),
           dto.ref ? sourceFilter(dto.ref) : undefined
         )
       )
+      .orderBy(chunks.id)
       .limit(dto.limit ?? 200);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/chunk.ts` around lines 191 - 197, Update the
backlog query in the chunk resource to order results by chunks.id before
applying the limit, preserving the existing filters and limit behavior.

84-139: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

The replace loop issues one round trip per chunk and scans a list per orphan check.

Two points:

  1. Every chunk runs its own UPDATE or INSERT inside the transaction. A document with 100 sections costs more than 100 sequential round trips while holding the transaction open.
  2. keptKeys.includes(...) on line 134 makes the orphan scan O(n·m). A Set makes it O(n).

A single INSERT ... ON CONFLICT (source_type, source_id, kind, chunk_index) DO UPDATE covers the insert and the changed-content branch in one statement. The unique index resource_chunk_source_kind_index_idx already supports that conflict target.

♻️ Minimum change for the orphan scan
-      const keptKeys = dto.chunks.map((chunk) =>
-        key(chunk.kind, chunk.chunkIndex)
-      );
+      const keptKeys = new Set(
+        dto.chunks.map((chunk) => key(chunk.kind, chunk.chunkIndex))
+      );
       const orphans = existing
-        .filter((row) => !keptKeys.includes(key(row.kind, row.chunkIndex)))
+        .filter((row) => !keptKeys.has(key(row.kind, row.chunkIndex)))
         .map((row) => row.id);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/chunk.ts` around lines 84 - 139, Refactor the
chunk replacement loop to batch all chunk inserts and updates with a single
transaction statement using INSERT ... ON CONFLICT on (source_type, source_id,
kind, chunk_index), updating changed content and mirrored visibility while
preserving unchanged handling and embedding cleanup. In the orphan cleanup after
this batch, replace the keptKeys array and includes lookup with a Set so
membership checks are O(1), while retaining the existing orphan deletion
behavior.
packages/db/src/libs/resources/search.ts (3)

329-333: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The innerJoin on line 329 has no join predicate against source.

eq(chunks.kind, "card") is a filter, not a join condition. The statement relies on source holding at most one row so the cross product stays bounded. That holds today because of the limit(1) on line 316, but the intent is not obvious.

Move eq(chunks.kind, "card") into the where clause and use an explicit crossJoin, or add a comment that states the single-row assumption.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/search.ts` around lines 329 - 333, Update the
query around the joins in the search flow to make the missing predicate against
source explicit: move eq(chunks.kind, "card") into the where clause and replace
the predicate-less innerJoin(chunks, ...) with an explicit crossJoin, preserving
the existing limit(1) single-row assumption and other join conditions.

319-346: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

findSimilarResources cannot use the HNSW index.

The right-hand operand of <=> is source.embedding, a column of a CTE rather than a parameter. Postgres does not use a vector index when both sides are columns, so this runs a sequential scan over every card embedding and computes the distance for each row.

Fetch the source vector in a separate query, then pass it as a bound parameter to the similarity query. That keeps the index usable and also removes the duplicated similarity expression on lines 319 and 342.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/search.ts` around lines 319 - 346, Update
findSimilarResources to fetch the source embedding in a separate query, then
pass that vector as a bound parameter to the similarity query so the HNSW index
remains usable. Remove the source CTE and use one shared similarity expression
for both selection and threshold filtering, while preserving the existing joins,
exclusions, ordering, and limit.

251-281: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

aggregateChunkHits depends on the caller sorting the hits.

Line 267 states that hits arrive in score order. Nothing in the signature enforces it. If an unsorted array is passed, bestChunk and the top-N mean are both wrong, and the failure is silent.

Sort each bucket by score before slicing.

♻️ Proposed change
   return [...byResource.values()]
     .map((bucket) => {
-      // hits arrive in score order, so the head is already the best
-      const top = bucket.slice(0, topN);
+      const ordered = [...bucket].sort((a, b) => b.score - a.score);
+      const top = ordered.slice(0, topN);
       const score = top.reduce((sum, hit) => sum + hit.score, 0) / top.length;
-      const best = bucket[0]!;
+      const best = ordered[0]!;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/search.ts` around lines 251 - 281, Update
aggregateChunkHits to sort each resource bucket by descending hit.score before
selecting top and bestChunk. Use the sorted bucket for both the top-N mean and
bestChunk, while preserving matchedChunks and the existing resource-level
sorting and limit behavior.
packages/db/src/schemas/resources.schema.ts (2)

163-167: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

The HNSW index does not include model, so multi-model recall degrades.

searchChunksSemantic and findSimilarResources both filter on embeddings.model. The HNSW index covers every row of every model. Postgres retrieves the approximate top-K across all models and then discards rows of other models, so the effective result count drops as soon as a second model exists.

If more than one model will ever be stored, index per model or add a partial index.

♻️ Example: partial index per active model
     index("resource_embedding_model_idx").on(table.model),
     index("resource_embedding_hnsw_idx").using(
       "hnsw",
       table.embedding.op("vector_cosine_ops")
-    ),
+    ).where(sql`model = 'text-embedding-3-small'`),
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/schemas/resources.schema.ts` around lines 163 - 167, Update
the resource_embedding_hnsw_idx definition in the schema to account for the
model filter used by searchChunksSemantic and findSimilarResources. Scope the
HNSW index per stored embedding model, or otherwise add the required model-aware
partial/index strategy, so approximate nearest-neighbor retrieval does not rank
unrelated models together.

33-37: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff

These casts hide a duplicated drizzle-orm install.

The comment explains the symptom. The root cause is that @paradedb/drizzle-paradedb resolves a second copy of drizzle-orm. A pnpm overrides or peerDependencyRules entry in pnpm-workspace.yaml deduplicates the instance and removes the need for unknown casts. The casts silently accept any value today, so a wrong column would compile.

This is optional if the duplicate resolution is intentional.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/schemas/resources.schema.ts` around lines 33 - 37,
Deduplicate the drizzle-orm dependency used by `@paradedb/drizzle-paradedb` via
the workspace pnpm configuration, using an appropriate overrides or
peerDependencyRules entry. Then remove the unknown-based casts in
ParadedbIndexArgs, pdbKeyField, pdbField, and pdbTokenized, relying on the
unified drizzle-orm types while preserving their existing argument typing.
packages/db/src/libs/feeds/index.ts (2)

477-491: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Confirm the tag query cost per indexing run.

getFeedForIndexing now runs a second query for tag names and returns one row per (tag, locale) pair. loadFeedForIndexingStep groups them again in memory. This is correct, but the indexing workflow calls it per feed. Consider whether a batched variant is needed once a full reindex runs over every feed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/feeds/index.ts` around lines 477 - 491, Review the
getFeedForIndexing and loadFeedForIndexingStep flow to confirm the added
tagTranslations query is acceptable when invoked once per feed. If full
reindexing makes this N+1 cost excessive, introduce a batched feed-loading path
that retrieves tag data for multiple feeds while preserving the existing
per-feed tag and translation output; otherwise document or retain the current
behavior without unrelated changes.

234-240: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Two raw subqueries hard-code the chia_ table prefix. Both hasEmbedding expressions write chia_resource_chunk and chia_resource_embedding as string literals. The prefix is produced by the custom pgTable creator, so a prefix change breaks both queries at runtime with no compile-time signal.

  • packages/db/src/libs/feeds/index.ts#L234-L240: interpolate the resourceChunks and resourceEmbeddings schema objects into the extras.hasEmbedding template instead of the literal names.
  • packages/db/src/libs/feeds/index.ts#L341-L345: apply the same substitution in the getFeedDetails select, and extract the shared expression into one helper so the two sites cannot diverge.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/feeds/index.ts` around lines 234 - 240, In
packages/db/src/libs/feeds/index.ts at lines 234-240 and 341-345, extract the
shared hasEmbedding SQL expression into one helper and update both
extras.hasEmbedding and the getFeedDetails select to interpolate the
resourceChunks and resourceEmbeddings schema objects instead of hard-coded
chia_resource_chunk and chia_resource_embedding names, keeping both query sites
consistent.
packages/db/.drizzle/migrations/20260811165041_resource_chunks/migration.sql (1)

38-50: 🗄️ Data Integrity & Integration | 🔵 Trivial | 💤 Low value

Confirm the deploy order for the destructive step.

The backfill and the DROP TABLE statements run in the same migration. Any running instance of the old code that still reads chia_content or chia_feed_embedding fails immediately after the drop. There is also no path back once the tables are gone.

If a rolling deploy is possible, split the drop into a follow-up migration that runs after the new code is live. If the deploy is a single cutover, this is fine as written.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/.drizzle/migrations/20260811165041_resource_chunks/migration.sql`
around lines 38 - 50, Separate the destructive DROP TABLE statements from the
backfill migration when rolling deployments are supported: keep the data
migration for chia_content and chia_feed_embedding, then add a follow-up
migration to drop them after all instances run the new code. If deployment is
guaranteed to be a single cutover, document or otherwise preserve the current
ordering.

Source: Linters/SAST tools

packages/ai/__tests__/embeddings-document-card.spec.mts (2)

97-99: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The comment states the wrong count.

The comment says 500 copies. The code repeats 5000 times.

🔧 Proposed change
-    // 500 copies of the body under the same outline: the card must not grow
+    // 5000 copies of the body under the same outline: the card must not grow
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/__tests__/embeddings-document-card.spec.mts` around lines 97 -
99, The test comment above the long article fixture incorrectly says 500 copies;
update it to accurately describe the 5000 repetitions used by the repeat call,
without changing the test logic.

160-168: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a case with no sentence boundary.

Every oversized fixture uses 。, so splitOversized always finds a split point. A paragraph with no sentence punctuation takes the path that emits an over-target chunk, described in the comment on packages/ai/src/embeddings/chunking.ts lines 42-54. Add a fixture such as one long CJK run without punctuation to cover it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/__tests__/embeddings-document-card.spec.mts` around lines 160 -
168, Add a test case alongside the existing oversized-section test using a long
CJK paragraph with no sentence punctuation, then pass it through chunkMarkdown
and assert the resulting chunks follow the intended oversized-chunk behavior.
Ensure the fixture does not contain punctuation that lets splitOversized find a
sentence boundary.
packages/ai/__tests__/embeddings-context.spec.mts (1)

151-161: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The test does not assert the behavior in its name.

The name states that dropped documents are reported. The body only checks totalTokens. droppedSlugs is the only signal for exhaustion, and no test covers it.

🔧 Proposed change
     expect(result.totalTokens).toBeLessThanOrEqual(900);
+    expect(result.droppedSlugs.length).toBeGreaterThan(0);
+    expect(result.documents.map((document) => document.slug)).not.toEqual([
+      "a",
+      "b",
+      "c",
+    ]);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/__tests__/embeddings-context.spec.mts` around lines 151 - 161,
Update the test named “reports documents dropped once the budget is exhausted”
to assert the droppedSlugs result, verifying that the documents excluded after
the budget is exhausted are reported. Keep the existing totalTokens budget
assertion and use the expected slug values from the supplied documents.
packages/ai/src/embeddings/provider.ts (1)

82-92: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Update the storage location named in the comment.

The comment says the key is stored in feed_translation.index_key. This PR moves chunk and embedding storage to the resource chunk tables. Name the column that actually holds the value, so the invalidation contract stays traceable.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/src/embeddings/provider.ts` around lines 82 - 92, Update the
documentation comment above buildIndexKey to name the resource chunk table
column that stores the generated key, replacing the outdated
feed_translation.index_key reference. Keep the description of the key
composition and invalidation behavior unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 00a6f783-5b21-4a86-8737-bd8a54f731e3

📥 Commits

Reviewing files that changed from the base of the PR and between 0306216 and 6284dd1.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (76)
  • apps/dash/src/app/(workspace)/feed/(edit)/edit/[id]/page.tsx
  • apps/dash/src/components/feed/meta-chip.tsx
  • apps/dash/src/components/feed/search-feed.tsx
  • apps/service/__tests__/__mocks__/db.mock.ts
  • apps/service/__tests__/feeds-search-auth.controller.test.ts
  • apps/service/__tests__/feeds.controller.test.ts
  • apps/service/__tests__/feeds.service.test.ts
  • apps/service/__tests__/setup.ts
  • apps/service/src/env.ts
  • apps/service/src/services/agent-content.port.ts
  • apps/service/src/services/feed-indexing.service.ts
  • apps/service/src/steps/algolia-search.step.ts
  • apps/service/src/steps/feed-embeddings.step.ts
  • apps/service/src/steps/feed-indexing.step.ts
  • apps/service/src/steps/resource-index.step.ts
  • apps/service/src/workflows/algolia-search.workflow.ts
  • apps/service/src/workflows/feed-indexing.workflow.ts
  • apps/service/src/workflows/feed-removal.workflow.ts
  • apps/service/src/workflows/resource-index.workflow.ts
  • apps/www/src/app/[locale]/(blog)/[type]/[slug]/llm.md/route.ts
  • apps/www/src/app/[locale]/(blog)/[type]/[slug]/page.tsx
  • packages/agent-writing/package.json
  • packages/agent-writing/src/ports.ts
  • packages/agent-writing/src/tools/retrieval.tool.ts
  • packages/agent-writing/src/types.ts
  • packages/ai/__tests__/embeddings-context.spec.mts
  • packages/ai/__tests__/embeddings-document-card.spec.mts
  • packages/ai/__tests__/embeddings-tokenizer.spec.mts
  • packages/ai/package.json
  • packages/ai/src/embeddings/chunking.ts
  • packages/ai/src/embeddings/context.ts
  • packages/ai/src/embeddings/markdown.ts
  • packages/ai/src/embeddings/ollama.ts
  • packages/ai/src/embeddings/openai.ts
  • packages/ai/src/embeddings/provider.ts
  • packages/ai/src/embeddings/tokenizer.ts
  • packages/ai/src/embeddings/utils.ts
  • packages/api/__tests__/router-import.test.ts
  • packages/api/algolia/client.ts
  • packages/api/algolia/env.ts
  • packages/api/feeds/search.ts
  • packages/api/feeds/validator.ts
  • packages/api/feeds/write.ts
  • packages/api/orpc/contracts/content.contract.ts
  • packages/api/orpc/contracts/feeds.contract.ts
  • packages/api/orpc/routes/content.route.ts
  • packages/api/orpc/routes/feeds.route.ts
  • packages/api/package.json
  • packages/api/resources/feed-translation.resource.ts
  • packages/api/resources/registry.ts
  • packages/api/resources/search.ts
  • packages/api/resources/types.ts
  • packages/api/services/utils.ts
  • packages/api/services/validators.ts
  • packages/api/turbo.json
  • packages/contents/src/services.ts
  • packages/contents/src/types.ts
  • packages/db/.drizzle/migrations/20260811165041_resource_chunks/migration.sql
  • packages/db/.drizzle/migrations/20260811165041_resource_chunks/snapshot.json
  • packages/db/package.json
  • packages/db/src/libs/feeds/embedding.ts
  • packages/db/src/libs/feeds/index.ts
  • packages/db/src/libs/feeds/search.ts
  • packages/db/src/libs/resources/chunk.ts
  • packages/db/src/libs/resources/search.ts
  • packages/db/src/libs/validator/feeds.ts
  • packages/db/src/schemas/contents.schema.ts
  • packages/db/src/schemas/index.ts
  • packages/db/src/schemas/relations.ts
  • packages/db/src/schemas/resources.schema.ts
  • packages/db/vitest.config.ts
  • plans/.gitkeep
  • plans/rag-long-document-retrieval-plan.md
  • plans/rag-optimization-plan.md
  • plans/service-transport-unification-plan.md
  • pnpm-workspace.yaml
💤 Files with no reviewable changes (12)
  • packages/api/services/utils.ts
  • apps/service/src/env.ts
  • plans/service-transport-unification-plan.md
  • packages/api/algolia/env.ts
  • apps/service/src/workflows/algolia-search.workflow.ts
  • packages/api/services/validators.ts
  • apps/service/src/steps/algolia-search.step.ts
  • packages/db/src/libs/feeds/embedding.ts
  • plans/rag-optimization-plan.md
  • plans/rag-long-document-retrieval-plan.md
  • packages/api/algolia/client.ts
  • apps/service/src/steps/feed-embeddings.step.ts

Comment thread packages/ai/src/embeddings/chunking.ts Resolved
Comment thread packages/ai/src/embeddings/context.ts Resolved
Comment thread packages/ai/src/embeddings/ollama.ts Resolved
Comment on lines +743 to +759
const [row] = await db
.update(feedTranslations)
.set({
content: dto.content,
source: dto.source,
unstableSerializedSource: dto.unstableSerializedSource,
updatedAt: new Date(),
})
.onConflictDoUpdate({
target: contents.feedTranslationId,
set: {
content: dto.content,
source: dto.source,
unstableSerializedSource: dto.unstableSerializedSource,
updatedAt: new Date(),
},
})
.returning();

return content;
.where(eq(feedTranslations.id, dto.feedTranslationId))
.returning({
feedTranslationId: feedTranslations.id,
content: feedTranslations.content,
source: feedTranslations.source,
unstableSerializedSource: feedTranslations.unstableSerializedSource,
});

return row;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

upsertContent now returns undefined when the translation does not exist.

The previous implementation inserted into contents with a conflict clause, so it always produced a row. The UPDATE matches nothing when dto.feedTranslationId is unknown, and row is undefined. The function name still says "upsert", so callers can dereference the result.

Either throw on an empty result or change the name and the return type to make the miss explicit.

🐛 Proposed fix
       .returning({
         feedTranslationId: feedTranslations.id,
         content: feedTranslations.content,
         source: feedTranslations.source,
         unstableSerializedSource: feedTranslations.unstableSerializedSource,
       });
 
+    if (!row) {
+      throw new Error(
+        `Feed translation ${dto.feedTranslationId} not found`
+      );
+    }
     return row;
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const [row] = await db
.update(feedTranslations)
.set({
content: dto.content,
source: dto.source,
unstableSerializedSource: dto.unstableSerializedSource,
updatedAt: new Date(),
})
.onConflictDoUpdate({
target: contents.feedTranslationId,
set: {
content: dto.content,
source: dto.source,
unstableSerializedSource: dto.unstableSerializedSource,
updatedAt: new Date(),
},
})
.returning();
return content;
.where(eq(feedTranslations.id, dto.feedTranslationId))
.returning({
feedTranslationId: feedTranslations.id,
content: feedTranslations.content,
source: feedTranslations.source,
unstableSerializedSource: feedTranslations.unstableSerializedSource,
});
return row;
const [row] = await db
.update(feedTranslations)
.set({
content: dto.content,
source: dto.source,
unstableSerializedSource: dto.unstableSerializedSource,
updatedAt: new Date(),
})
.where(eq(feedTranslations.id, dto.feedTranslationId))
.returning({
feedTranslationId: feedTranslations.id,
content: feedTranslations.content,
source: feedTranslations.source,
unstableSerializedSource: feedTranslations.unstableSerializedSource,
});
if (!row) {
throw new Error(
`Feed translation ${dto.feedTranslationId} not found`
);
}
return row;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/feeds/index.ts` around lines 743 - 759, Update
upsertContent so an update that matches no feed translation cannot return
undefined: detect an empty returning result and throw an appropriate error
before returning row. Preserve the existing returned translation fields for
successful updates and keep the upsertContent contract safe for callers.

Comment thread packages/db/src/libs/feeds/search.ts Resolved
Comment thread packages/db/src/libs/resources/chunk.ts Outdated
Comment on lines +239 to +249
export const pruneStaleEmbeddings = withDTO(
async (db, dto: { model: string; indexVersion: string }) => {
const deleted = await db
.delete(embeddings)
.where(
sql`(${embeddings.model}, ${embeddings.indexVersion}) <> (${dto.model}, ${dto.indexVersion})`
)
.returning({ chunkId: embeddings.chunkId });
return { deletedCount: deleted.length };
}
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

pruneStaleEmbeddings deletes every vector that does not match the single supplied pair.

The statement has no resource scope and no guard. If two models are in use, or if a model migration is partly complete, this call removes all vectors of the other model in one statement. Semantic search then returns nothing until a full reindex finishes.

Add an explicit allow-list of active (model, indexVersion) pairs, and require that the new pair already covers the chunk before the old one is dropped.

🛡️ Proposed change
 export const pruneStaleEmbeddings = withDTO(
-  async (db, dto: { model: string; indexVersion: string }) => {
+  async (
+    db,
+    dto: { active: { model: string; indexVersion: string }[]; ref?: ResourceRef }
+  ) => {
+    if (dto.active.length === 0) {
+      return { deletedCount: 0 };
+    }
     const deleted = await db
       .delete(embeddings)
       .where(
-        sql`(${embeddings.model}, ${embeddings.indexVersion}) <> (${dto.model}, ${dto.indexVersion})`
+        sql`(${embeddings.model}, ${embeddings.indexVersion}) not in ${dto.active.map((pair) => [pair.model, pair.indexVersion])}`
       )
       .returning({ chunkId: embeddings.chunkId });
     return { deletedCount: deleted.length };
   }
 );
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/chunk.ts` around lines 239 - 249, Update
pruneStaleEmbeddings to accept an explicit allow-list of active
model/indexVersion pairs, rather than deleting every embedding that differs from
one supplied pair. Scope deletion to embeddings belonging to the relevant
resource and only remove an old embedding when the new active pair already
covers the same chunk. Preserve the deletedCount result while ensuring active
pairs and uncovered chunks are never deleted.

Comment on lines +171 to +178
const query = dto.query.trim();
if (!query && dto.embedding.length === 0) {
return [];
}

const limit = dto.limit ?? 20;
const candidateLimit = dto.candidateLimit ?? Math.max(limit * 3, 40);
const scope = scopeFilter(dto);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

The guard passes when only one input is empty.

return [] requires both the query and the embedding to be empty. Two reachable states get through:

  • query is empty and embedding is populated. lexicalMatch("") builds pdb.matchAny(content, '') and pdb.phrase(..., ''). ParadeDB rejects an empty phrase query.
  • embedding is empty and query is populated. cosineDistance(embeddings.embedding, []) produces '[]'::vector, which fails the 1536-dimension check.

searchResources in packages/api/resources/search.ts calls this with whatever embedQuery returns, so a provider failure that yields an empty array reaches the second state.

Handle each side independently.

🐛 Proposed fix
     const query = dto.query.trim();
-    if (!query && dto.embedding.length === 0) {
+    const hasLexical = query.length > 0;
+    const hasSemantic = dto.embedding.length > 0;
+    if (!hasLexical && !hasSemantic) {
       return [];
     }
+    if (!hasSemantic) {
+      return await searchChunksLexical(db, { ...dto, query, limit: dto.limit });
+    }
+    if (!hasLexical) {
+      return await searchChunksSemantic(db, {
+        ...dto,
+        embedding: dto.embedding,
+        model: dto.model,
+        limit: dto.limit,
+      });
+    }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const query = dto.query.trim();
if (!query && dto.embedding.length === 0) {
return [];
}
const limit = dto.limit ?? 20;
const candidateLimit = dto.candidateLimit ?? Math.max(limit * 3, 40);
const scope = scopeFilter(dto);
const query = dto.query.trim();
const hasLexical = query.length > 0;
const hasSemantic = dto.embedding.length > 0;
if (!hasLexical && !hasSemantic) {
return [];
}
if (!hasSemantic) {
return await searchChunksLexical(db, { ...dto, query, limit: dto.limit });
}
if (!hasLexical) {
return await searchChunksSemantic(db, {
...dto,
embedding: dto.embedding,
model: dto.model,
limit: dto.limit,
});
}
const limit = dto.limit ?? 20;
const candidateLimit = dto.candidateLimit ?? Math.max(limit * 3, 40);
const scope = scopeFilter(dto);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/db/src/libs/resources/search.ts` around lines 171 - 178, Update the
guard in searchResources so it returns an empty result whenever either the
trimmed query is empty or dto.embedding has no values, preventing lexical
searches with an empty phrase and vector searches with an empty embedding.
Preserve the existing limit, candidateLimit, and scope setup for requests where
both inputs are present.

- Complete pending embedding batches reliably
- Deduplicate feed search and related-feed results

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/ai/src/embeddings/chunking.ts`:
- Around line 94-112: Update splitOversized’s buffer accounting to include both
unit.joiner and the "\n\n" separator between buffered sections when calculating
the pending token total before the capacity check. After flush(), recalculate
the pending addition with zero separator cost, then update bufferTokens using
the complete addition so maxTokens and targetTokens cannot be exceeded.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: a9a65833-c9dd-464b-96bd-d8b3d35553f2

📥 Commits

Reviewing files that changed from the base of the PR and between 6284dd1 and feafabf.

📒 Files selected for processing (16)
  • apps/service/src/services/agent-content.port.ts
  • apps/service/src/steps/resource-index.step.ts
  • apps/service/src/workflows/feed-indexing.workflow.ts
  • apps/service/src/workflows/resource-index.workflow.ts
  • packages/ai/src/embeddings/chunking.ts
  • packages/ai/src/embeddings/context.ts
  • packages/ai/src/embeddings/markdown.ts
  • packages/ai/src/embeddings/ollama.ts
  • packages/ai/src/embeddings/provider.ts
  • packages/ai/src/embeddings/utils.ts
  • packages/api/feeds/search.ts
  • packages/api/resources/feed-translation.resource.ts
  • packages/api/resources/registry.ts
  • packages/db/src/libs/feeds/search.ts
  • packages/db/src/libs/resources/chunk.ts
  • packages/db/src/schemas/resources.schema.ts
💤 Files with no reviewable changes (1)
  • packages/db/src/libs/resources/chunk.ts
🚧 Files skipped from review as they are similar to previous changes (10)
  • apps/service/src/workflows/feed-indexing.workflow.ts
  • packages/db/src/libs/feeds/search.ts
  • packages/ai/src/embeddings/provider.ts
  • packages/ai/src/embeddings/context.ts
  • apps/service/src/workflows/resource-index.workflow.ts
  • packages/api/resources/feed-translation.resource.ts
  • packages/ai/src/embeddings/markdown.ts
  • apps/service/src/services/agent-content.port.ts
  • apps/service/src/steps/resource-index.step.ts
  • packages/api/feeds/search.ts

Comment on lines +94 to +112
for (const unit of toUnits(text, maxTokens, encoding)) {
const unitTokens = countEmbeddingTokens(unit.text, encoding);

if (buffer && bufferTokens + unitTokens > maxTokens) {
flush();
}
const heading = insideCodeFence ? null : /^(#{1,6})\s+(.+)$/.exec(line);
if (heading?.[1] && heading[2]) {

if (unitTokens > maxTokens) {
flush();
const level = heading[1].length;
while (
headingStack.length > 0 &&
headingStack[headingStack.length - 1]!.level >= level
) {
headingStack.pop();
for (const slice of sliceToBudget(unit.text, maxTokens)) {
if (slice.trim()) {
pieces.push(slice.trim());
}
}
headingStack.push({ level, title: heading[2].trim() });
continue;
}
buffer.push(line);

buffer += (buffer ? unit.joiner : "") + unit.text;
bufferTokens += unitTokens;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Count joiners in both token budgets.

bufferTokens excludes unit.joiner in splitOversized. It also excludes the "\n\n" separator between buffered sections. A chunk can exceed maxTokens or targetTokens even when the tracked total fits.

Count the complete pending addition before the capacity check. After a flush, recalculate the separator cost as zero.

Proposed fix
   for (const unit of toUnits(text, maxTokens, encoding)) {
-    const unitTokens = countEmbeddingTokens(unit.text, encoding);
+    const addition = (buffer ? unit.joiner : "") + unit.text;
+    const additionTokens = countEmbeddingTokens(addition, encoding);

-    if (buffer && bufferTokens + unitTokens > maxTokens) {
+    if (buffer && bufferTokens + additionTokens > maxTokens) {
       flush();
     }

-    buffer += (buffer ? unit.joiner : "") + unit.text;
-    bufferTokens += unitTokens;
+    buffer += (buffer ? unit.joiner : "") + unit.text;
+    bufferTokens += countEmbeddingTokens(
+      buffer ? unit.joiner + unit.text : unit.text,
+      encoding
+    );
   }

Also applies to: 184-188

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/ai/src/embeddings/chunking.ts` around lines 94 - 112, Update
splitOversized’s buffer accounting to include both unit.joiner and the "\n\n"
separator between buffered sections when calculating the pending token total
before the capacity check. After flush(), recalculate the pending addition with
zero separator cost, then update bufferTokens using the complete addition so
maxTokens and targetTokens cannot be exceeded.

- Cover chunk indexing, hybrid retrieval, and embedding maintenance

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/rag-architecture.md`:
- Line 45: Add language identifiers to every fenced code block in
docs/rag-architecture.md to satisfy markdownlint MD040; use text for tree
diagrams and pseudocode, while preserving specific identifiers such as mermaid
where applicable.
- Around line 81-85: The RAG schema and search flow must prevent mixed embedding
versions during reindexing. Use the existing index_version field to track the
active version, require semantic and hybrid search APIs to accept and filter by
that active version alongside model, and ensure reindexing and rollback update
the active version consistently.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 53120bc5-4a72-4a97-b06f-a00180007f8c

📥 Commits

Reviewing files that changed from the base of the PR and between feafabf and 00fef6a.

📒 Files selected for processing (2)
  • apps/service/src/steps/resource-index.step.ts
  • docs/rag-architecture.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • apps/service/src/steps/resource-index.step.ts

Comment thread docs/rag-architecture.md
向量獨立成表,不長在 `feed_translation` 上——換模型、加模型、重新索引都是資料操作,不需要 schema migration。
### 2.1 `chia_resource_chunk`

```

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add language identifiers to the fenced blocks.

markdownlint MD040 reports these fences without a language. Use text for trees and pseudocode, and keep specific identifiers such as mermaid where applicable.

Also applies to: 77-77, 121-121, 147-147, 160-160, 201-201, 235-235, 264-264

🧰 Tools
🪛 markdownlint-cli2 (0.23.2)

[warning] 45-45: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/rag-architecture.md` at line 45, Add language identifiers to every
fenced code block in docs/rag-architecture.md to satisfy markdownlint MD040; use
text for tree diagrams and pseudocode, while preserving specific identifiers
such as mermaid where applicable.

Source: Linters/SAST tools

Comment thread docs/rag-architecture.md
Comment on lines +81 to +85
├── index_version text
├── embedding vector(1536)
└── created_at / updated_at
PRIMARY KEY (chunk_id, model)
+ HNSW index(vector_cosine_ops)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 8 \
  'resource_embedding|resourceEmbeddings|index_version|indexVersion|EMBEDDING_INDEX_VERSION|saveChunkEmbeddings|embedPendingChunks' \
  packages/db apps/service packages/ai

Repository: Chia1104/chia1104.dev

Length of output: 29653


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path

files = {
    "docs/rag-architecture.md": [(75, 90), (150, 175), (270, 305)],
    "packages/db/src/schemas/resources.schema.ts": [(151, 171)],
    "packages/db/src/libs/resources/chunk.ts": [(162, 198), (201, 234)],
    "packages/db/src/libs/resources/search.ts": [(1, 180)],
    "apps/service/src/steps/resource-index.step.ts": [(61, 125)],
}

for name, ranges in files.items():
    path = Path(name)
    print(f"\n### {name}")
    lines = path.read_text().splitlines()
    for start, end in ranges:
        print(f"-- lines {start}-{end} --")
        for number in range(start, min(end, len(lines)) + 1):
            print(f"{number}: {lines[number - 1]}")
PY

python3 - <<'PY'
from pathlib import Path

schema = Path("packages/db/src/schemas/resources.schema.ts").read_text()
chunk = Path("packages/db/src/libs/resources/chunk.ts").read_text()
search = Path("packages/db/src/libs/resources/search.ts").read_text()

checks = {
    "schema primary key excludes indexVersion": "primaryKey({ columns: [table.chunkId, table.model] })" in schema
    and "primaryKey({ columns: [table.chunkId, table.model, table.indexVersion] })" not in schema,
    "upsert conflict target excludes indexVersion": "target: [embeddings.chunkId, embeddings.model]" in chunk,
    "upsert overwrites indexVersion": "indexVersion: dto.indexVersion" in chunk
    and "indexVersion: dto.indexVersion" in chunk[chunk.find("onConflictDoUpdate"):],
    "search source references embeddings": "const embeddings = schema.resourceEmbeddings" in search,
    "search filters by model": "eq(embeddings.model" in search,
    "search filters by indexVersion": "indexVersion" in search[search.find("const embeddings"):],
}
for label, result in checks.items():
    print(f"{label}: {result}")
PY

Repository: Chia1104/chia1104.dev

Length of output: 14801


🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C 6 \
  'searchChunksSemantic|searchChunksHybrid|normalizeQueryForEmbedding|buildIndexKey|EMBEDDING_INDEX_VERSION|resource_embedding_model_idx|resource_embedding_hnsw_idx' \
  apps packages docs/rag-architecture.md

python3 - <<'PY'
from pathlib import Path

path = Path("packages/db/src/libs/resources/search.ts")
lines = path.read_text().splitlines()
for start, end in [(176, 290)]:
    print(f"-- {path}:{start}-{end} --")
    for n in range(start, min(end, len(lines)) + 1):
        print(f"{n}: {lines[n-1]}")
PY

Repository: Chia1104/chia1104.dev

Length of output: 18359


Do not serve mixed embedding versions during reindexing.

The (chunk_id, model) key and upsert overwrite vectors in place, while semantic and hybrid searches filter only by model. A version bump therefore mixes old vectors with new query embeddings until reindexing completes. Add an active index_version filter and pass it through the search APIs, or define a migration strategy that guarantees version-consistent results and rollback.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/rag-architecture.md` around lines 81 - 85, The RAG schema and search
flow must prevent mixed embedding versions during reindexing. Use the existing
index_version field to track the active version, require semantic and hybrid
search APIs to accept and filter by that active version alongside model, and
ensure reindexing and rollback update the active version consistently.

@Chia1104
Chia1104 merged commit 5389a8a into develop Aug 12, 2026
20 of 21 checks passed
@Chia1104
Chia1104 deleted the refactor/rag branch August 12, 2026 07:29

This branch was successfully deployed

1 active deployment
Preview — 00fef6af Deployed Aug 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Switch to ParadeDB

2 participants