feat(apify): one consolidated new-posts digest per scrape batch#760
Conversation
… posts first The Instagram scrape alert fired whenever a scrape returned posts, with no comparison against posts already stored — every scrape re-announced the profile's recent feed as new (observed: 6 of 7 alerts in one day announced posts up to 10 days old). New reusable filterNewPostUrls diffs candidate URLs against posts BEFORE upsert; the alert is gated on a non-empty result. Persistence unchanged (recoupable/chat#1855, PR #3 of 6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A roster scrape starts one Apify run per platform, each completing independently — extending per-platform alerts would mean 4+ emails per scrape. This registers every run under a batch_id at scrape start (apify_scraper_runs, columns from recoupable/database#41), records each webhook completion with its genuinely-new post URLs, and when the batch's last run completes sends ONE digest (per-platform sections, BCC-only) via the new digest module. Platforms with nothing new are omitted; a batch with nothing new sends nothing. Instagram's solo alert is suppressed for batch runs; legacy/non-batch runs keep today's behavior (recoupable/chat#1855, PR #4 of 6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
Warning Review limit reached
Next review available in: 27 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (6)
📝 WalkthroughWalkthroughThis PR adds batch-based digest emailing for Apify social scrapes. It introduces a new ChangesScrape digest batching feature
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Apify
participant apifyWebhookHandler
participant PlatformHandler as Instagram/TikTok/Twitter Handler
participant updateApifyScraperRun
participant maybeSendScrapeDigest
participant selectApifyScraperRuns
participant getScrapeDigestRecipients
participant sendScrapeDigestEmail
Apify->>apifyWebhookHandler: webhook payload (resource.id)
apifyWebhookHandler->>PlatformHandler: process scrape results
PlatformHandler-->>apifyWebhookHandler: newPostUrls, posts, social
apifyWebhookHandler->>updateApifyScraperRun: update run (newPostUrls, completed_at)
updateApifyScraperRun-->>apifyWebhookHandler: run with batch_id
apifyWebhookHandler->>maybeSendScrapeDigest: maybeSendScrapeDigest(batch_id)
maybeSendScrapeDigest->>selectApifyScraperRuns: fetch runs for batch
selectApifyScraperRuns-->>maybeSendScrapeDigest: runs list
maybeSendScrapeDigest->>getScrapeDigestRecipients: resolve socialIds to emails
getScrapeDigestRecipients-->>maybeSendScrapeDigest: recipient emails
maybeSendScrapeDigest->>sendScrapeDigestEmail: send consolidated digest
sendScrapeDigestEmail-->>maybeSendScrapeDigest: send result
apifyWebhookHandler-->>Apify: 200 OK
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
6 issues found across 21 files
Confidence score: 3/5
lib/supabase/apify_scraper_runs/selectApifyScraperRunsByBatch.tsreturns[]on query failure, somaybeSendScrapeDigestcan treat DB errors as “no runs” and silently skip a digest; users may miss notifications with no clear signal. Return/propagate an error (or distinguish error vs empty result) before merging.lib/apify/digest/maybeSendScrapeDigest.tscan send the same consolidated digest multiple times if a webhook is replayed after all runs are complete, which can spam recipients and create inconsistent state. Add an idempotent, atomic send guard keyed by batch before merging.lib/apify/digest/sendScrapeDigestEmail.tsinterpolates raw URLs and platform names into HTML, so malformed or hostile values could produce unsafe or misleading email content. Escape/sanitize URL/text fields before building the email body.- New batch/digest paths in
lib/artist/__tests__/postArtistSocialsScrapeHandler.test.tsandlib/apify/twitter/__tests__/handleTwitterProfileScraperResults.test.tsare not asserted, increasing regression risk in batch registration andnewPostUrlsdigest inputs. Add focused assertions for batchId/social mapping/run insertion andfilterNewPostUrlsintegration before merging.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/twitter/__tests__/handleTwitterProfileScraperResults.test.ts">
<violation number="1" location="lib/apify/twitter/__tests__/handleTwitterProfileScraperResults.test.ts:5">
P2: The handler now calls `filterNewPostUrls` and includes `newPostUrls` in its return value (drives the digest email), but the tests don't verify that integration. Consider adding an assertion on the `filterNewPostUrls` call args and on `newPostUrls` in the return value so the mock isn't purely ornamental — the digest behavior depends on correct new-post tracking.</violation>
</file>
<file name="lib/apify/digest/getScrapeDigestRecipients.ts">
<violation number="1" location="lib/apify/digest/getScrapeDigestRecipients.ts:16">
P2: Digest recipients can be silently dropped for a social watched by more than 10,000 account links because this query reads only the first page. Consider paginating or moving this lookup into a DB-side joined query rather than encoding a larger fixed cap.
(Based on your team's feedback about DB-side pagination over raised Supabase limits.) [FEEDBACK_USED]</violation>
</file>
<file name="lib/apify/digest/sendScrapeDigestEmail.ts">
<violation number="1" location="lib/apify/digest/sendScrapeDigestEmail.ts:26">
P2: Post URLs and platform names are interpolated into the email HTML body without any escaping. Both the href attribute and visible text content for each link use the raw URL directly, and the platform name goes into an `<h3>` element. A URL containing a `"` would break the href attribute; a URL or platform with `<` or `>` could inject arbitrary HTML into the email. Consider encoding the URL for the href attribute (e.g., wrapping with a helper that escapes `&`, `"`, `<`, `>`) and optionally encoding visible text too, as a defense-in-depth measure.</violation>
</file>
<file name="lib/artist/__tests__/postArtistSocialsScrapeHandler.test.ts">
<violation number="1" location="lib/artist/__tests__/postArtistSocialsScrapeHandler.test.ts:12">
P1: The batch registration logic added to `postArtistSocialsScrapeHandler` (batchId generation, socialByUrl mapping, insertApifyScraperRuns call) has no test coverage in the handler's own test file. The test file was updated with the required mock and matching return shapes, but no `it(...)` block verifies the new behavior. This leaves the digest pipeline's registration step untested, so regressions — wrong account_id, wrong platform derivation, missing social_id, or a broken batchId — would go undetected in CI. Consider adding test cases that assert on the arguments passed to `insertApifyScraperRuns`, verify the response body omits profileUrl, and confirm the call is skipped in early-return paths (auth failure, 403, 402).</violation>
</file>
<file name="lib/supabase/apify_scraper_runs/selectApifyScraperRunsByBatch.ts">
<violation number="1" location="lib/supabase/apify_scraper_runs/selectApifyScraperRunsByBatch.ts:8">
P1: Silent error swallowing in batch query can suppress the digest. When the Supabase query fails, the function returns an empty array, making it indistinguishable from an empty batch. The caller treats `runs.length === 0` as "batch not ready" and returns null without re-raising or alerting — so a transient DB error on the last sibling's completion webhook silently drops the consolidated digest. Consider either propagating the error (so the caller can retry) or adding a dedicated recovery path (e.g., a queue retry or alert) for query failures.</violation>
</file>
<file name="lib/apify/digest/maybeSendScrapeDigest.ts">
<violation number="1" location="lib/apify/digest/maybeSendScrapeDigest.ts:26">
P2: A completed batch can send the consolidated digest more than once if the webhook for any registered run is delivered or replayed after all siblings are complete. Consider making the send gate idempotent with an atomic persisted marker/claim for the batch before calling `sendScrapeDigestEmail`.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Client
participant ScrapeHandler as postArtistSocialsScrapeHandler
participant ScrapeBatch as scrapeProfileUrlBatch
participant DB as Supabase DB
participant Apify as Apify (external)
participant WebhookHandler as apifyWebhookHandler
participant ResultHandler as per‑platform result handler
participant Digest as maybeSendScrapeDigest
participant EmailService as digest email service
rect rgb(240, 240, 240)
Note over Client,ScrapeHandler: Phase 1 — Batch initialization
end
Client->>ScrapeHandler: POST /artist/socials/scrape (artist_id, socials)
ScrapeHandler->>ScrapeBatch: CHANGED: scrapeProfileUrlBatch(socials)
ScrapeBatch->>Apify: start scrape per social
Apify-->>ScrapeBatch: runId, datasetId (per social)
ScrapeBatch-->>ScrapeHandler: CHANGED: results include profileUrl
ScrapeHandler->>DB: NEW: insertApifyScraperRuns(runs with batch_id, social_id, platform)
alt registration succeeds
DB-->>ScrapeHandler: ok
else registration fails
DB-->>ScrapeHandler: error (logged, scrape continues)
end
Note over ScrapeHandler: strips profileUrl from response (contract unchanged)
ScrapeHandler-->>Client: 200 {runId, datasetId, error}[]
rect rgb(245, 245, 245)
Note over Apify,WebhookHandler: Phase 2 — Webhook processing (each run)
end
Apify->>WebhookHandler: POST /api/apify (webhook for run‑X)
alt payload has no runId
Note over WebhookHandler: skip digest bookkeeping (legacy path)
else runId present
WebhookHandler->>ResultHandler: dispatch per actor type
Note over ResultHandler: e.g., handleInstagramProfileScraperResults
alt Instagram with batch_id
ResultHandler->>ResultHandler: NEW: suppress solo email (skip sendApifyWebhookEmail)
end
ResultHandler->>DB: NEW: filterNewPostUrls (pre‑persist diff)
DB-->>ResultHandler: newPostUrls[]
ResultHandler-->>WebhookHandler: CHANGED: {posts, newPostUrls, ...}
WebhookHandler->>DB: NEW: completeApifyScraperRun(runId, newPostUrls)
DB-->>WebhookHandler: updated row (with batch_id)
WebhookHandler->>Digest: NEW: maybeSendScrapeDigest(batch_id)
Digest->>DB: NEW: selectApifyScraperRunsByBatch(batch_id)
DB-->>Digest: all runs in batch
alt any sibling incomplete
Note over Digest: wait for last run
Digest-->>WebhookHandler: null (no email)
else all completed
alt at least one platform has new posts
Digest->>EmailService: NEW: getScrapeDigestRecipients(socialIds)
EmailService->>DB: selectAccountSocials → getAccountArtistIds → selectAccountEmails
DB-->>EmailService: watcher email addresses
EmailService-->>Digest: [watcher@...]
Digest->>EmailService: NEW: sendScrapeDigestEmail(emails, sections)
Note over EmailService: BCC‑only, per‑platform sections
EmailService-->>Digest: email id
Digest-->>WebhookHandler: email id
else no new posts anywhere
Note over Digest: skip (sends nothing)
Digest-->>WebhookHandler: null
end
end
end
Note over WebhookHandler: Always return 200 (digest errors caught & logged)
WebhookHandler-->>Apify: 200 OK
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| import { ensureSocialScrapeCredits } from "@/lib/socials/ensureSocialScrapeCredits"; | ||
| import { deductSocialScrapeCredits } from "@/lib/socials/deductSocialScrapeCredits"; | ||
|
|
||
| vi.mock("@/lib/supabase/apify_scraper_runs/insertApifyScraperRuns", () => ({ |
There was a problem hiding this comment.
P1: The batch registration logic added to postArtistSocialsScrapeHandler (batchId generation, socialByUrl mapping, insertApifyScraperRuns call) has no test coverage in the handler's own test file. The test file was updated with the required mock and matching return shapes, but no it(...) block verifies the new behavior. This leaves the digest pipeline's registration step untested, so regressions — wrong account_id, wrong platform derivation, missing social_id, or a broken batchId — would go undetected in CI. Consider adding test cases that assert on the arguments passed to insertApifyScraperRuns, verify the response body omits profileUrl, and confirm the call is skipped in early-return paths (auth failure, 403, 402).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/artist/__tests__/postArtistSocialsScrapeHandler.test.ts, line 12:
<comment>The batch registration logic added to `postArtistSocialsScrapeHandler` (batchId generation, socialByUrl mapping, insertApifyScraperRuns call) has no test coverage in the handler's own test file. The test file was updated with the required mock and matching return shapes, but no `it(...)` block verifies the new behavior. This leaves the digest pipeline's registration step untested, so regressions — wrong account_id, wrong platform derivation, missing social_id, or a broken batchId — would go undetected in CI. Consider adding test cases that assert on the arguments passed to `insertApifyScraperRuns`, verify the response body omits profileUrl, and confirm the call is skipped in early-return paths (auth failure, 403, 402).</comment>
<file context>
@@ -9,6 +9,9 @@ import { checkAccountArtistAccess } from "@/lib/artists/checkAccountArtistAccess
import { ensureSocialScrapeCredits } from "@/lib/socials/ensureSocialScrapeCredits";
import { deductSocialScrapeCredits } from "@/lib/socials/deductSocialScrapeCredits";
+vi.mock("@/lib/supabase/apify_scraper_runs/insertApifyScraperRuns", () => ({
+ insertApifyScraperRuns: vi.fn(async () => ({ data: null, error: null })),
+}));
</file context>
| const emails = await getScrapeDigestRecipients( | ||
| runs.map(r => r.social_id).filter((id): id is string => Boolean(id)), | ||
| ); | ||
| return await sendScrapeDigestEmail({ emails, sections }); |
There was a problem hiding this comment.
P2: A completed batch can send the consolidated digest more than once if the webhook for any registered run is delivered or replayed after all siblings are complete. Consider making the send gate idempotent with an atomic persisted marker/claim for the batch before calling sendScrapeDigestEmail.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/digest/maybeSendScrapeDigest.ts, line 26:
<comment>A completed batch can send the consolidated digest more than once if the webhook for any registered run is delivered or replayed after all siblings are complete. Consider making the send gate idempotent with an atomic persisted marker/claim for the batch before calling `sendScrapeDigestEmail`.</comment>
<file context>
@@ -0,0 +1,27 @@
+ const emails = await getScrapeDigestRecipients(
+ runs.map(r => r.social_id).filter((id): id is string => Boolean(id)),
+ );
+ return await sendScrapeDigestEmail({ emails, sections });
+}
</file context>
| import { handleTwitterProfileScraperResults } from "@/lib/apify/twitter/handleTwitterProfileScraperResults"; | ||
| const listItems = vi.fn(); | ||
| vi.mock("@/lib/socials/filterNewPostUrls", () => ({ | ||
| filterNewPostUrls: vi.fn(async (urls: string[]) => urls), |
There was a problem hiding this comment.
P2: The handler now calls filterNewPostUrls and includes newPostUrls in its return value (drives the digest email), but the tests don't verify that integration. Consider adding an assertion on the filterNewPostUrls call args and on newPostUrls in the return value so the mock isn't purely ornamental — the digest behavior depends on correct new-post tracking.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/twitter/__tests__/handleTwitterProfileScraperResults.test.ts, line 5:
<comment>The handler now calls `filterNewPostUrls` and includes `newPostUrls` in its return value (drives the digest email), but the tests don't verify that integration. Consider adding an assertion on the `filterNewPostUrls` call args and on `newPostUrls` in the return value so the mock isn't purely ornamental — the digest behavior depends on correct new-post tracking.</comment>
<file context>
@@ -1,6 +1,9 @@
import { handleTwitterProfileScraperResults } from "@/lib/apify/twitter/handleTwitterProfileScraperResults";
const listItems = vi.fn();
+vi.mock("@/lib/socials/filterNewPostUrls", () => ({
+ filterNewPostUrls: vi.fn(async (urls: string[]) => urls),
+}));
vi.mock("@/lib/apify/client", () => ({ default: { dataset: vi.fn(() => ({ listItems })) } }));
</file context>
|
|
||
| const accountSocials = ( | ||
| await Promise.all( | ||
| uniqueSocialIds.map(socialId => selectAccountSocials({ socialId, limit: 10000 })), |
There was a problem hiding this comment.
P2: Digest recipients can be silently dropped for a social watched by more than 10,000 account links because this query reads only the first page. Consider paginating or moving this lookup into a DB-side joined query rather than encoding a larger fixed cap.
(Based on your team's feedback about DB-side pagination over raised Supabase limits.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/digest/getScrapeDigestRecipients.ts, line 16:
<comment>Digest recipients can be silently dropped for a social watched by more than 10,000 account links because this query reads only the first page. Consider paginating or moving this lookup into a DB-side joined query rather than encoding a larger fixed cap.
(Based on your team's feedback about DB-side pagination over raised Supabase limits.) </comment>
<file context>
@@ -0,0 +1,28 @@
+
+ const accountSocials = (
+ await Promise.all(
+ uniqueSocialIds.map(socialId => selectAccountSocials({ socialId, limit: 10000 })),
+ )
+ ).flat();
</file context>
| .map( | ||
| s => | ||
| `<h3 style="margin:16px 0 4px">${s.platform}</h3><ul>${s.postUrls | ||
| .map(u => `<li><a href="${u.startsWith("http") ? u : `https://${u}`}">${u}</a></li>`) |
There was a problem hiding this comment.
P2: Post URLs and platform names are interpolated into the email HTML body without any escaping. Both the href attribute and visible text content for each link use the raw URL directly, and the platform name goes into an <h3> element. A URL containing a " would break the href attribute; a URL or platform with < or > could inject arbitrary HTML into the email. Consider encoding the URL for the href attribute (e.g., wrapping with a helper that escapes &, ", <, >) and optionally encoding visible text too, as a defense-in-depth measure.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/digest/sendScrapeDigestEmail.ts, line 26:
<comment>Post URLs and platform names are interpolated into the email HTML body without any escaping. Both the href attribute and visible text content for each link use the raw URL directly, and the platform name goes into an `<h3>` element. A URL containing a `"` would break the href attribute; a URL or platform with `<` or `>` could inject arbitrary HTML into the email. Consider encoding the URL for the href attribute (e.g., wrapping with a helper that escapes `&`, `"`, `<`, `>`) and optionally encoding visible text too, as a defense-in-depth measure.</comment>
<file context>
@@ -0,0 +1,38 @@
+ .map(
+ s =>
+ `<h3 style="margin:16px 0 4px">${s.platform}</h3><ul>${s.postUrls
+ .map(u => `<li><a href="${u.startsWith("http") ? u : `https://${u}`}">${u}</a></li>`)
+ .join("")}</ul>`,
+ )
</file context>
|
Heads-up 2026-07-09: this PR's dependency changed — |
…gregation # Conflicts: # lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts # lib/apify/instagram/handleInstagramProfileScraperResults.ts
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Preview verification — 2026-07-09Branch prep first: retargeted base Preview: Method: two-platform digest batch against real Apify datasets — the existing natgeo Instagram dataset (12 posts) + a fresh
Edge documented for api#762 (not this PR's scope): re-delivering the TikTok webhook (an Apify retry) recomputes Email confirmation caveat: the digest sends aren't in the webhook response and aren't in Fixture fully cleaned (artist, 2 socials, 15 posts + social_posts + post_comments, both Verdict: the aggregation core does what it claims — suppresses solo alerts for batch runs, books completions, and assembles one digest on last-complete. Ready in merge order, with the retry double-send noted for #762. |
Full scrape-start E2E — 2026-07-09 (closes the gap in the previous verification)With a fresh access token, the previously-untested registration path was exercised live on the same preview (
@sweetmantech: Fixture fully cleaned (artist via API, 2 socials, 13 posts + social_posts + post_comments, all 4 With this, every Done-when for the aggregation PR is verified live end-to-end. The retry double-send edge noted previously stands for api#762. |
There was a problem hiding this comment.
KISS
- actual: lib/supabase/apify_scraper_runs/completeApifyScraperRun.ts
- required lib/supabase/apify_scraper_runs/updateApifyScraperRun.ts
There was a problem hiding this comment.
Renamed to updateApifyScraperRun.ts (function + all call sites/mocks) in 87599e2 — it's a plain UPDATE, so it now matches the update[TableName] convention.
There was a problem hiding this comment.
KISS
- actual: lib/supabase/apify_scraper_runs/insertApifyScraperRuns.ts
- required: lib/supabase/apify_scraper_runs/upsertApifyScraperRuns.ts
There was a problem hiding this comment.
Renamed to upsertApifyScraperRuns.ts in 87599e2 — accurate too, since it upserts on run_id with ignoreDuplicates.
There was a problem hiding this comment.
KISS
- actual: lib/supabase/apify_scraper_runs/selectApifyScraperRunsByBatch.ts
- required: lib/supabase/apify_scraper_runs/selectApifyScraperRuns.ts
There was a problem hiding this comment.
Renamed to selectApifyScraperRuns.ts in 87599e2, and the signature now takes the conventional optional filter object (selectApifyScraperRuns({ batchId })) instead of a positional batch arg. Suite green after the rename: 460 tests across lib/apify + lib/socials + lib/artist + lib/supabase; eslint clean; tsc clean on all touched files.
…ention Review feedback on #760: completeApifyScraperRun -> updateApifyScraperRun, insertApifyScraperRuns -> upsertApifyScraperRuns (it upserts on run_id), selectApifyScraperRunsByBatch -> selectApifyScraperRuns with an optional {batchId} filter object, matching the select* convention. Also repoints the stale types.ts reference from database#41 to #47. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
1 issue found across 11 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/supabase/apify_scraper_runs/selectApifyScraperRuns.ts">
<violation number="1" location="lib/supabase/apify_scraper_runs/selectApifyScraperRuns.ts:14">
P3: Digest platform sections can appear in nondeterministic order because `selectApifyScraperRuns` does not order the rows before `maybeSendScrapeDigest` maps them into email sections. Consider adding a stable `.order(...)` (for example by platform plus run_id/created_at) so the consolidated digest body is repeatable.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| }: { | ||
| batchId?: string; | ||
| } = {}): Promise<ApifyScraperRunRow[]> { | ||
| let query = supabase.from("apify_scraper_runs" as never).select("*"); |
There was a problem hiding this comment.
P3: Digest platform sections can appear in nondeterministic order because selectApifyScraperRuns does not order the rows before maybeSendScrapeDigest maps them into email sections. Consider adding a stable .order(...) (for example by platform plus run_id/created_at) so the consolidated digest body is repeatable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/supabase/apify_scraper_runs/selectApifyScraperRuns.ts, line 14:
<comment>Digest platform sections can appear in nondeterministic order because `selectApifyScraperRuns` does not order the rows before `maybeSendScrapeDigest` maps them into email sections. Consider adding a stable `.order(...)` (for example by platform plus run_id/created_at) so the consolidated digest body is repeatable.</comment>
<file context>
@@ -0,0 +1,26 @@
+}: {
+ batchId?: string;
+} = {}): Promise<ApifyScraperRunRow[]> {
+ let query = supabase.from("apify_scraper_runs" as never).select("*");
+
+ if (batchId) {
</file context>
…parse the JSONB column Regenerates database.types.ts (table landed in database#47), deletes the hand-rolled lib/supabase/apify_scraper_runs/types.ts shim, and drops every 'as never' cast — the helpers now use Tables/TablesInsert like every sibling lib. new_post_urls is the one column codegen can't narrow past Json, so a zod boundary (parseNewPostUrls) validates it at read time; malformed JSONB degrades to 'no new posts' instead of crashing the digest assembler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
Libs which do not directly query supabase should not live in the lib/supabase folder.
There was a problem hiding this comment.
Moved to lib/apify/digest/parseNewPostUrls.ts in afc0d9a — agreed, it's a pure JSONB parser with no supabase query, and its only consumer is the digest assembler next door. Suite green after the move (97 tests in lib/apify), eslint + tsc clean.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/apify/digest/maybeSendScrapeDigest.ts`:
- Around line 15-26: maybeSendScrapeDigest is vulnerable to duplicate sends
because it checks completion and sends emails non-atomically; add an idempotency
guard using the new digest_sent_at field on apify_scraper_runs so only one
handler can claim the batch. Update maybeSendScrapeDigest to atomically mark the
relevant runs as claimed before calling getScrapeDigestRecipients and
sendScrapeDigestEmail, and bail out if the claim fails or digest_sent_at is
already set.
In `@lib/apify/digest/sendScrapeDigestEmail.ts`:
- Around line 22-36: The digest email HTML in sendScrapeDigestEmail interpolates
untrusted scraped data directly into attributes and text, so escape all dynamic
values before building the markup. Update the sectionHtml mapping in
sendScrapeDigestEmail to HTML-escape s.platform, each post URL used in both the
href and visible link text, and the who value used in the subject/body so
injected quotes or tags cannot break the email template. Keep the existing
sendEmailWithResend flow and apply escaping at the point where sectionHtml and
the html string are constructed.
In `@lib/artist/postArtistSocialsScrapeHandler.ts`:
- Around line 79-93: The batch registration in postArtistSocialsScrapeHandler
still runs inside the main outer try, so a failure in upsertApifyScraperRuns can
incorrectly fail the scrape response. Move only the upsertApifyScraperRuns call
that builds the run rows from results into its own try/catch block, keep the
rest of the handler flow unchanged, and log any registration error with enough
context to identify the batchId and accountId.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 5fe8bb75-a762-43a9-ac18-642482469e85
⛔ Files ignored due to path filters (6)
lib/apify/__tests__/apifyWebhookHandler.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/digest/__tests__/maybeSendScrapeDigest.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/tiktok/__tests__/handleTiktokProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/twitter/__tests__/handleTwitterProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/artist/__tests__/postArtistSocialsScrapeHandler.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**
📒 Files selected for processing (15)
lib/apify/apifyWebhookHandler.tslib/apify/digest/getScrapeDigestRecipients.tslib/apify/digest/maybeSendScrapeDigest.tslib/apify/digest/sendScrapeDigestEmail.tslib/apify/instagram/handleInstagramProfileScraperResults.tslib/apify/scrapeProfileUrlBatch.tslib/apify/tiktok/handleTiktokProfileScraperResults.tslib/apify/twitter/handleTwitterProfileScraperResults.tslib/apify/validateApifyWebhookRequest.tslib/artist/postArtistSocialsScrapeHandler.tslib/supabase/apify_scraper_runs/selectApifyScraperRun.tslib/supabase/apify_scraper_runs/selectApifyScraperRuns.tslib/supabase/apify_scraper_runs/types.tslib/supabase/apify_scraper_runs/updateApifyScraperRun.tslib/supabase/apify_scraper_runs/upsertApifyScraperRuns.ts
| const runs = await selectApifyScraperRuns({ batchId }); | ||
| if (!runs.length || runs.some(r => !r.completed_at)) return null; | ||
|
|
||
| const sections = runs | ||
| .filter(r => (r.new_post_urls?.length ?? 0) > 0) | ||
| .map(r => ({ platform: r.platform ?? "other", postUrls: r.new_post_urls ?? [] })); | ||
| if (!sections.length) return null; | ||
|
|
||
| const emails = await getScrapeDigestRecipients( | ||
| runs.map(r => r.social_id).filter((id): id is string => Boolean(id)), | ||
| ); | ||
| return await sendScrapeDigestEmail({ emails, sections }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
TOCTOU race: concurrent webhook completions can send duplicate digest emails.
maybeSendScrapeDigest reads all batch runs, checks completion, and sends the digest in a non-atomic sequence. When two runs in the same batch complete nearly simultaneously (e.g., Instagram and TikTok webhooks arrive within milliseconds), both webhook handlers can observe all runs as completed and both call sendScrapeDigestEmail, producing duplicate emails to recipients.
The function also lacks an idempotency guard — nothing marks the batch as "digest sent" after a successful send, so a webhook retry would trigger a second digest even without concurrency.
Suggested fix: Add a digest_sent_at timestamp column to apify_scraper_runs (coordinated with database#47) and atomically claim the send right before emailing:
🔒 Proposed atomic guard
export async function maybeSendScrapeDigest(batchId: string | null | undefined) {
if (!batchId) return null;
const runs = await selectApifyScraperRuns({ batchId });
if (!runs.length || runs.some(r => !r.completed_at)) return null;
const sections = runs
.filter(r => (r.new_post_urls?.length ?? 0) > 0)
.map(r => ({ platform: r.platform ?? "other", postUrls: r.new_post_urls ?? [] }));
if (!sections.length) return null;
+ // Atomically claim the digest send — prevents duplicate emails from
+ // concurrent webhook completions (TOCTOU) and idempotency on retry.
+ const { data: claimed } = await supabase
+ .from("apify_scraper_runs" as never)
+ .update({ digest_sent_at: new Date().toISOString() } as never)
+ .eq("batch_id", batchId)
+ .is("digest_sent_at", null)
+ .select()
+ .limit(1);
+ if (!claimed || claimed.length === 0) return null; // another caller already sent
+
const emails = await getScrapeDigestRecipients(
runs.map(r => r.social_id).filter((id): id is string => Boolean(id)),
);
return await sendScrapeDigestEmail({ emails, sections });
}This requires adding digest_sent_at to the apify_scraper_runs schema in database#47. The atomic UPDATE ... WHERE digest_sent_at IS NULL ensures only one caller wins the claim.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/digest/maybeSendScrapeDigest.ts` around lines 15 - 26,
maybeSendScrapeDigest is vulnerable to duplicate sends because it checks
completion and sends emails non-atomically; add an idempotency guard using the
new digest_sent_at field on apify_scraper_runs so only one handler can claim the
batch. Update maybeSendScrapeDigest to atomically mark the relevant runs as
claimed before calling getScrapeDigestRecipients and sendScrapeDigestEmail, and
bail out if the claim fails or digest_sent_at is already set.
| const sectionHtml = sections | ||
| .map( | ||
| s => | ||
| `<h3 style="margin:16px 0 4px">${s.platform}</h3><ul>${s.postUrls | ||
| .map(u => `<li><a href="${u.startsWith("http") ? u : `https://${u}`}">${u}</a></li>`) | ||
| .join("")}</ul>`, | ||
| ) | ||
| .join(""); | ||
|
|
||
| return await sendEmailWithResend({ | ||
| from: RECOUP_FROM_EMAIL, | ||
| to: [RECOUP_FROM_EMAIL], | ||
| bcc: emails, | ||
| subject: `${who}: ${total} new post${total === 1 ? "" : "s"} found across ${sections.length} platform${sections.length === 1 ? "" : "s"}`, | ||
| html: `<p>New posts found for ${who}:</p>${sectionHtml}`, |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
HTML-escape dynamic content to prevent XSS in digest emails.
Post URLs (line 26) and artistName/who (line 36) are interpolated directly into HTML without escaping. A scraped URL containing " could break out of the href attribute, and one containing <script> could inject markup. While social platforms validate URLs, this is external data — defense in depth requires escaping.
🛡️ Proposed fix: add HTML escaping
export async function sendScrapeDigestEmail({ emails, sections, artistName }: ScrapeDigestInput) {
if (!emails.length || !sections.length) return null;
+ const escapeHtml = (s: string) =>
+ s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """).replace(/'/g, "&`#39`;");
+
const total = sections.reduce((n, s) => n + s.postUrls.length, 0);
const who = artistName || "your artist";
+ const whoEscaped = escapeHtml(who);
const sectionHtml = sections
.map(
s =>
- `<h3 style="margin:16px 0 4px">${s.platform}</h3><ul>${s.postUrls
- .map(u => `<li><a href="${u.startsWith("http") ? u : `https://${u}`}">${u}</a></li>`)
+ `<h3 style="margin:16px 0 4px">${escapeHtml(s.platform)}</h3><ul>${s.postUrls
+ .map(u => {
+ const href = u.startsWith("http") ? u : `https://${u}`;
+ return `<li><a href="${escapeHtml(href)}">${escapeHtml(u)}</a></li>`;
+ })
.join("")}</ul>`,
)
.join("");
return await sendEmailWithResend({
from: RECOUP_FROM_EMAIL,
to: [RECOUP_FROM_EMAIL],
bcc: emails,
subject: `${who}: ${total} new post${total === 1 ? "" : "s"} found across ${sections.length} platform${sections.length === 1 ? "" : "s"}`,
- html: `<p>New posts found for ${who}:</p>${sectionHtml}`,
+ html: `<p>New posts found for ${whoEscaped}:</p>${sectionHtml}`,
});
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const sectionHtml = sections | |
| .map( | |
| s => | |
| `<h3 style="margin:16px 0 4px">${s.platform}</h3><ul>${s.postUrls | |
| .map(u => `<li><a href="${u.startsWith("http") ? u : `https://${u}`}">${u}</a></li>`) | |
| .join("")}</ul>`, | |
| ) | |
| .join(""); | |
| return await sendEmailWithResend({ | |
| from: RECOUP_FROM_EMAIL, | |
| to: [RECOUP_FROM_EMAIL], | |
| bcc: emails, | |
| subject: `${who}: ${total} new post${total === 1 ? "" : "s"} found across ${sections.length} platform${sections.length === 1 ? "" : "s"}`, | |
| html: `<p>New posts found for ${who}:</p>${sectionHtml}`, | |
| const escapeHtml = (s: string) => | |
| s.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """).replace(/'/g, "&`#39`;"); | |
| const total = sections.reduce((n, s) => n + s.postUrls.length, 0); | |
| const who = artistName || "your artist"; | |
| const whoEscaped = escapeHtml(who); | |
| const sectionHtml = sections | |
| .map( | |
| s => | |
| `<h3 style="margin:16px 0 4px">${escapeHtml(s.platform)}</h3><ul>${s.postUrls | |
| .map(u => { | |
| const href = u.startsWith("http") ? u : `https://${u}`; | |
| return `<li><a href="${escapeHtml(href)}">${escapeHtml(u)}</a></li>`; | |
| }) | |
| .join("")}</ul>`, | |
| ) | |
| .join(""); | |
| return await sendEmailWithResend({ | |
| from: RECOUP_FROM_EMAIL, | |
| to: [RECOUP_FROM_EMAIL], | |
| bcc: emails, | |
| subject: `${who}: ${total} new post${total === 1 ? "" : "s"} found across ${sections.length} platform${sections.length === 1 ? "" : "s"}`, | |
| html: `<p>New posts found for ${whoEscaped}:</p>${sectionHtml}`, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/digest/sendScrapeDigestEmail.ts` around lines 22 - 36, The digest
email HTML in sendScrapeDigestEmail interpolates untrusted scraped data directly
into attributes and text, so escape all dynamic values before building the
markup. Update the sectionHtml mapping in sendScrapeDigestEmail to HTML-escape
s.platform, each post URL used in both the href and visible link text, and the
who value used in the subject/body so injected quotes or tags cannot break the
email template. Keep the existing sendEmailWithResend flow and apply escaping at
the point where sectionHtml and the html string are constructed.
| const batchId = crypto.randomUUID(); | ||
| const socialByUrl = new Map( | ||
| socials.map(s => [s.social?.profile_url ?? "", s.social?.id ?? null]), | ||
| ); | ||
| await upsertApifyScraperRuns( | ||
| results | ||
| .filter(r => r.runId) | ||
| .map(r => ({ | ||
| run_id: r.runId as string, | ||
| account_id: authResult.accountId, | ||
| social_id: r.profileUrl ? (socialByUrl.get(r.profileUrl) ?? null) : null, | ||
| platform: r.profileUrl ? getSocialPlatformByLink(r.profileUrl).toLowerCase() : null, | ||
| batch_id: batchId, | ||
| })), | ||
| ); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify getSocialPlatformByLink return type — can it return null?
ast-grep outline lib/artists/getSocialPlatformByLink.ts --items all
# Also check its return type signature
rg -n -A5 'export.*getSocialPlatformByLink' lib/artists/getSocialPlatformByLink.tsRepository: recoupable/api
Length of output: 609
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the handler around the batch-registration block and its error handling.
wc -l lib/artist/postArtistSocialsScrapeHandler.ts
sed -n '1,220p' lib/artist/postArtistSocialsScrapeHandler.tsRepository: recoupable/api
Length of output: 4632
Isolate batch registration from the scrape response path
upsertApifyScraperRuns still sits in the handler’s outer try, so a registration failure will return a 500 and fail the scrape even though the comment says it must not. Wrap just this registration step in its own try/catch and log the error.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/artist/postArtistSocialsScrapeHandler.ts` around lines 79 - 93, The batch
registration in postArtistSocialsScrapeHandler still runs inside the main outer
try, so a failure in upsertApifyScraperRuns can incorrectly fail the scrape
response. Move only the upsertApifyScraperRuns call that builds the run rows
from results into its own try/catch block, keep the rest of the handler flow
unchanged, and log any registration error with enough context to identify the
batchId and accountId.
…pify/digest Review feedback: lib/supabase is for direct queries only; this is a pure JSONB parser consumed by the digest assembler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Final regression pass after review-feedback refactors — 2026-07-09Preview:
Behavior is identical to the pre-refactor verification — the renames, generated types, and zod parse changed no semantics. Fixture fully cleaned (0 rows remaining across posts/socials/runs/artist). Latest review comment also addressed: |
There was a problem hiding this comment.
0 issues found across 8 files (changes from recent commits).
Requires human review: Auto-approval blocked by 6 unresolved issues from previous reviews.
Re-trigger cubic
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
…orphaned pre-rename supabase files Review feedback: - Every helper gets its own lib file: escapeHtml (lib/emails — repo had none), formatUtcDateLabel, truncateText, formatCompactCount, formatPostStats, getPlatformLabel, extractInstagramPosts, extractTiktokPosts, buildPostStats, asRecord/asStringOrNull/ asNumberOrNull coercers. renderScrapeDigestHtml and extractPostsFromDatasetItems now contain only their eponymous functions. - CHAT_APP_URL / WEBSITE_URL / RECOUP_LOGO_URL move to lib/const.ts as shared constants (no existing shared home found for platform labels — getPlatformLabel is the new one). - Deletes 4 orphaned pre-rename files under lib/supabase/apify_scraper_runs (completeApifyScraperRun, insertApifyScraperRuns, selectApifyScraperRunsByBatch, types): this branch predates #760's renames, so the merge of main added the renamed set alongside the branch's inherited old set — both landed with zero conflicts and zero references to the old files. Also confirms no legacy LLM email generation remains in lib/apify (generateText usage gone with the template rewrite). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… post, no LLM body (#761) * feat(apify): notify only on genuinely new posts — diff against stored posts first The Instagram scrape alert fired whenever a scrape returned posts, with no comparison against posts already stored — every scrape re-announced the profile's recent feed as new (observed: 6 of 7 alerts in one day announced posts up to 10 days old). New reusable filterNewPostUrls diffs candidate URLs against posts BEFORE upsert; the alert is gated on a non-empty result. Persistence unchanged (recoupable/chat#1855, PR #3 of 6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(apify): one consolidated new-posts digest per scrape batch A roster scrape starts one Apify run per platform, each completing independently — extending per-platform alerts would mean 4+ emails per scrape. This registers every run under a batch_id at scrape start (apify_scraper_runs, columns from recoupable/database#41), records each webhook completion with its genuinely-new post URLs, and when the batch's last run completes sends ONE digest (per-platform sections, BCC-only) via the new digest module. Platforms with nothing new are omitted; a batch with nothing new sends nothing. Instagram's solo alert is suppressed for batch runs; legacy/non-batch runs keep today's behavior (recoupable/chat#1855, PR #4 of 6). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(apify): deterministic digest template with direct links to each new post Replaces the per-send LLM email body (nondeterministic branding, vendor jargon reaching customers, no reliable post links) with a shared deterministic renderer: stable subject/branding, one section per platform, a direct link to every genuinely-new post, chat CTA secondary. Used by both the batch digest and the legacy solo Instagram alert, which now also requires new-post URLs and is BCC-only (recoupable/chat#1855, PR #5 of 6; supersedes the interim body from PR #4 and, for this file, the standalone BCC fix in api#758 — same invariant, tests included). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(apify): guard sendScrapeDigestEmail's renderer wiring + BCC invariant The branch imported renderScrapeDigestHtml in sendScrapeDigestEmail but never called it — the digest still rendered the old inline body (found during the main-sync conflict resolution; wired in that merge commit). This test fails against the unwired version: it asserts the send payload is byte-identical to the renderer's output, plus the BCC-only invariant and the empty-input no-send. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(apify): house-style digest template with post media, captions, and dates The v1 deterministic template regressed visual quality vs the old LLM emails (bare h3/ul list). This upgrades the renderer to the DESIGN.md house style — achromatic chrome, card-per-post with 72px thumbnail, escaped caption excerpt, date, and a direct link; black CTA button — while staying deterministic and email-safe (tables + inline styles). Media plumbing: extractPostsFromDatasetItems maps platform dataset items (IG latestPosts, TikTok items) to {url, caption, thumbnailUrl, timestamp}, limited to the genuinely-new URLs. The solo alert enriches from the dataset already in memory; the digest enriches via getRunDigestSection, which re-reads each run's dataset from Apify (source of truth) and degrades to URL-only links on any failure — enrichment never blocks a send. Captions are HTML-escaped so scraped content can't inject markup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: prettier pass on the digest template files Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(apify): digest addressed by artist name + per-post engagement stats + fixed chat CTA Three feedback items on the template: - The digest header/subject said 'Your artist' — the assembler never passed a name. getRunDigestSection now also extracts the profile display name from the dataset (IG fullName, TikTok authorMeta.nickName) and maybeSendScrapeDigest addresses the email with the first platform's name. - Post cards now carry compact engagement stats (12.3K likes · 678 comments · 1.2M views · shares) mapped from platform counts (IG likesCount/commentsCount/videoViewCount, TikTok diggCount/commentCount/ playCount/shareCount); omitted entirely when the scraper returns none. - The CTA now always points at https://chat.recoupable.dev — previously it derived from the deployment base URL, which on previews is the API deployment itself. Funnel-tracking landing page tracked as follow-up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(apify): Recoup logo in the email header + artist-named roster footer Header is now a two-cell row with the brand icon top-right (hosted PNG — email clients don't render SVG) linking to recoupable.com. Footer names the artist: 'because {artist} is in your roster on Recoup', falling back to 'this artist' when no profile name resolved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(apify): strict SRP/DRY pass on the digest renderer + delete orphaned pre-rename supabase files Review feedback: - Every helper gets its own lib file: escapeHtml (lib/emails — repo had none), formatUtcDateLabel, truncateText, formatCompactCount, formatPostStats, getPlatformLabel, extractInstagramPosts, extractTiktokPosts, buildPostStats, asRecord/asStringOrNull/ asNumberOrNull coercers. renderScrapeDigestHtml and extractPostsFromDatasetItems now contain only their eponymous functions. - CHAT_APP_URL / WEBSITE_URL / RECOUP_LOGO_URL move to lib/const.ts as shared constants (no existing shared home found for platform labels — getPlatformLabel is the new one). - Deletes 4 orphaned pre-rename files under lib/supabase/apify_scraper_runs (completeApifyScraperRun, insertApifyScraperRuns, selectApifyScraperRunsByBatch, types): this branch predates #760's renames, so the merge of main added the renamed set alongside the branch's inherited old set — both landed with zero conflicts and zero references to the old files. Also confirms no legacy LLM email generation remains in lib/apify (generateText usage gone with the template rewrite). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(apify): X/Twitter digest extractor + LinkedIn platform label Review feedback: Twitter's handler already feeds the digest (filterNewPostUrls -> newPostUrls) but had no extractor, so its sections degraded to URL-only links. extractTwitterPosts maps apidojo tweet items (fullText, extendedEntities media, createdAt via toIsoDate, like/reply/view/retweet counts), registered under both 'twitter' and 'x'. Artist-name extraction gains the same aliases (author.name ?? userName). LinkedIn added to platform labels. YouTube and LinkedIn deliberately have no extractors: their results handlers persist only the social row (no posts, no newPostUrls), so they never contribute digest sections today — extractors would be dead code until post persistence ships for them (chat#1833 territory). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(apify): only persist and report the artist's own tweets — no retweets or replies Feedback from a real-account digest test: an all-retweet X section reported hundreds of likes the artist never earned (retweet items carry the ORIGINAL author's stats). isOriginalTweet keeps originals and quote tweets (the artist's own words and metrics), drops retweets and replies, applied at the persistence layer in handleTwitterProfileScraperResults so both stored posts and digest sections stay accurate. Items without flags pass (defensive default on schema drift). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(apify): fetch 10 X timeline items by default — depth 1 rarely survives the retweet filter Real-account evidence (2026-07-09): the user's timeline had 7 retweets above their 2 originals and a quote tweet; a depth-1 (or 3) fetch returned only retweets, which isOriginalTweet now drops — so no authored post could ever reach the digest. Fetch deeper, filter after. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…dit onto the shipped template, add exact non-duplication Review question (sweetman): why a rate cap instead of properly enforcing non-duplication? Both, now, as separate concerns: - Completion claim: updateApifyScraperRun updates WHERE completed_at IS NULL — a webhook retry can't re-claim a completed run, re-trigger digest assembly, or overwrite the recorded diff (the 3->0 corruption observed live during #760 testing). - Batch idempotency: maybeSendScrapeDigest checks email_send_log for an existing scrape-digest:<batchId> row before sending — one digest per batch, ever, no time-window heuristics. - The 24h per-artist cap remains as the product frequency rule from the issue spec, reading the same audit rows. Every send is logged per watched artist (rawBody scrape-digest:<batchId>). Also deletes 4 orphaned pre-rename supabase files that rode in again via the merge (same both-added mechanism as #761) and supersedes the branch's selectRecentScrapeDigestLogs with filter-object selectScrapeDigestLogs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Summary
The digest architecture for recoupable/chat#1855 (PR #4 of 6, decision: one email per scrape, all platforms): registers each platform run under a
batch_idat scrape start, records completions + genuinely-new post URLs in the webhook layer, and sends one consolidated digest when the batch's last run completes.Stacked on api#759 (newness diff — provides
filterNewPostUrls); requires database#41 (batch columns) applied before this deploys. Base is #759's branch so the diff shows only this PR's work; GitHub retargets tomainwhen #759 merges.What changed
lib/supabase/apify_scraper_runs/(new):insertApifyScraperRuns(start-time registration),completeApifyScraperRun(completion + new URLs),selectApifyScraperRunsByBatch,selectApifyScraperRun, local row type (generated types don't carry database#41's columns yet).lib/apify/digest/(new):maybeSendScrapeDigest(the brain: all-siblings-complete gate → per-platform sections → skip-if-nothing-new),getScrapeDigestRecipients(same watcher chain the solo alert used),sendScrapeDigestEmail(BCC-only, deterministic minimal body — upgraded to the house template in PR Sweetmantech/myc 3550 apiimagegenerate use credits #5).postArtistSocialsScrapeHandler: mints the batch, registers runs (with platform + social mapping via a new additiveprofileUrlonscrapeProfileUrlBatchresults); response contract unchanged (extra field stripped).apifyWebhookHandler: after each result handler, completes the run and runs the digest check; failures never fail the webhook. Schema now passes through the optionalresource.idApify already sends.newPostUrls(pre-persist diff); Instagram suppresses its solo email for batch-registered runs (legacy runs keep it).Tests (RED→GREEN)
maybeSendScrapeDigest: 4 cases — waits for incomplete siblings; one send with per-platform sections (empty platforms omitted); nothing-new sends nothing; null batch no-ops.newPostUrlsreturned.lib/apify+lib/artist+lib/socials→ 272 tests passed;tsc --noEmitat exact baseline parity (200 pre-existing, 0 new); eslint clean.Merge sequencing
database#41 → api#759 → this. E2E verification (real batch scrape → one digest) planned post-merge on the tracking issue since Apify webhooks call the deployed receiver.
🤖 Generated with Claude Code
Summary by cubic
Send one consolidated new-posts digest per scrape batch across all platforms. Implements chat#1855 by batching runs, diffing genuinely new URLs, and emailing a single BCC digest when the last run completes.
New Features
apify_scraper_runshelpers:upsertApifyScraperRuns(batch registration),updateApifyScraperRun(completion + new URLs),selectApifyScraperRuns(optional{ batchId }), andselectApifyScraperRun; helpers use generatedTables/TablesInserttypes.resource.id, records completion withupdateApifyScraperRun, and callsmaybeSendScrapeDigest; digest errors never fail the webhook.getScrapeDigestRecipients,parseNewPostUrls(safe JSONB parser),maybeSendScrapeDigest,sendScrapeDigestEmail(per‑platform sections, BCC‑only; empty platforms omitted).newPostUrls(pre‑persist diff); Instagram returnsnewPostUrlsand suppresses the solo email for batch runs (legacy runs unchanged).postArtistSocialsScrapeHandlermints abatch_id, registers runs viaupsertApifyScraperRunswith platform/social mapping (fromprofileUrlinscrapeProfileUrlBatch), and returns the same response shape (extra field stripped).Migration
apify_scraper_runshasbatch_id,completed_at, andnew_post_urlsbefore deploying;types/database.types.tsregenerated (database#47). No API response changes.Written for commit 5f205f1. Summary will update on new commits.
Summary by CodeRabbit
New Features
Bug Fixes