Skip to content

feat(workflows): report blocked and failed workflow sends - #127135

Merged
trunk-io[bot] merged 5 commits into
masterfrom
workflows-tracking/send-funnel-events
Nov 3, 2026
Merged

trunk-io[bot] merged 5 commits into
masterfrom
workflows-tracking/send-funnel-events

Conversation

@Silthus

@Silthus Silthus commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • We can see when a team's workflow goes live, and billing usage reports show its first billed send. We can't see why a live workflow never reaches that first billed action.
  • The CDP service drops those sends without an analytics event. A send can be blocked by a workflow quota or by a missing recipient address, or fail on an unverified sending domain.

Impact

Pays into Grow Workflows through activation: the share of teams whose live workflow reaches its first billed action. The "went live → first billed action" step of the funnel has no reasons today. These events name them per team, so the funnel's drop-off splits by cause in PostHog's internal project. Expected move: small, up, through the fixes the drop-off reasons point to (quota upgrades, domain verification, recipient filters).

Merge Danger

Warning

🟡 Merge risk: medium · two-way door (merge-risk rubric)

  • The hog flow quota check and the email send path, the critical CDP delivery path, each gain one report call, only on branches where the send already stops (quota block, missing recipient, unverified domain). EmailService and every CDP consumer gain a required reporter dependency, which the typecheck guards. Worst case is silent harm on that path, though no send decision changes.
  • Each report is fire-and-forget inside try/catch, so a failed team lookup or capture cannot fail or delay a send. Reporter, quota, email service, pipeline and consumer suites cover the paths, and a live run on the hog flow pipeline shows the quota reports.
  • Revert: git revert the merge commit. Events already captured stay in the internal project and need no cleanup.

Changes

Kind Files Lines
Production 11 +103 −1
Tests 8 +249 −16

The CDP service reports two team events to PostHog's own analytics:

Event Fires when Properties
workflows send blocked A workflow quota stops a run of a workflow with an email step (event, cohort or warehouse trigger), or an email step has no recipient address workflow_id, channel: email, reason: quota_limited or missing_recipient
workflows send failed An email step's sending domain is not verified workflow_id, channel: email, reason: unverified_domain
  • WorkflowsActivationReporter reports each team, event and reason at most once per hour per service instance. The funnel needs only the first occurrence.
  • A failed team lookup or capture does not use up the hour, so the next send reports again.
  • Only real workflow sends report. Hog function destinations that send email, and editor test sends, report nothing.
  • A quota block reports whenever the blocked run has an email step, whichever quota stopped it: a run blocked by the destination quota drops its email steps too. A webhook-only workflow has no message to send and reports nothing. Transactional-only email workflows skip the email quota (#123849), so the email quota never blocks them.
  • The missing-recipient report sits on master's skip in EmailService (#120078), the single place every email send checks for an empty to.
  • Nothing user-visible changes.

Recreates the send-funnel part of Silthus/posthog#357, originally #111267 (closed). Moved here from the fork PR #115352 so CI can run.

How did you test this code?

Test rationale:

  • Reporter tests catch a broken hourly dedupe, and an hour used up by a failed team lookup, a missing team or a failed capture.
  • The missing-recipient case in email.service.test.ts failed with no report before EmailService reported the skip, then passed. It and the unverified-domain case cover a workflow send, an editor test send and a hog function send.
  • The quota limiting and pipeline suites assert the blocked report for the email quota, for an email step next to a webhook on the destination quota, and for transactional emails next to a webhook on the destination quota, and its absence for a webhook-only workflow. The two destination-quota cases failed before the review fix, then passed.

Test seams: the reporter's report against captureTeamEvent, EmailService.executeSendEmail, and the hog flow quota check and pipeline.

Lane gate, green locally on 7985d45 (master merged in): pnpm install --frozen-lockfile, hogli ci:preflight --strict, Node tsc, ESLint, oxlint and Prettier on the changed files, the ingestion boundary guard, the Jest suites of every touched module (nodejs/src/cdp/consumers, templates, the email service, quota limiting, pipeline, hog flow executor, hog function handler and hog executor suites, the reporter; 90 suites, 1571 passed, 1 skipped), and cdp-api.serial.test.ts (141/141).

Proofs on 7985d45, the head after the latest master merge, with both live runs repeated there. That merge brought five master commits that touch this PR's files, none of them its report call sites: #126304 labels email queue waits that the team sending cap delayed (email.service.ts), #127164 skips parsing events a consumer has no work for (cdp-events.consumer.ts), #124635 sends hog flow logs to the logs product (cdp-services.ts), and #127579 and #126479 add tests to hog_function.test.ts. The PR's own patch against master is unchanged (same patch-id as on 30f2376 and 64a9c11). Each report still fires once where the send stops: the unverified-domain failure and the missing-recipient skip both come before the team-cap reschedule, a send held by #127579's frequency-cap hold stays in the hog function step and never reaches the email service while held, and the quota check runs only when a run is created.

  • Live run on 7985d45, and earlier on 30f2376 and 64a9c11, hog flow pipeline (cdp-processed-events consumer at HEAD, local stack, workflows created and activated through the hog_flows API, analytics captures recorded by a local listener). Two events per workflow:
    • Marketing email workflow, team over its email quota: run blocked, nothing queued, exactly one event:
      { "event": "workflows send blocked", "reason": "quota_limited", "channel": "email", "workflow_id": "<the workflow>", "groups": { "project": "<team uuid>", "organization": "<org uuid>" } }
    • Transactional-only email workflow, same team over the same quota: both runs queued, no event.
    • Marketing email followed by a webhook, a second team over its destination quota only: run blocked, nothing queued, exactly one workflows send blocked with quota_limited.
  • Live run on 7985d45, and earlier on 30f2376 and 64a9c11, email steps (events consumer, cyclotron hogflow worker and cyclotron email worker at HEAD, local PersonHog, people in the persons database). A team with an unverified and a verified email sender, two events per workflow:
    • Marketing email from the unverified sender to people with an email address: both sends fail with "The selected email integration domain is not verified", exactly one workflows send failed with unverified_domain.
    • Marketing email from the verified sender to people without an email address: both sends skip with "the recipient has no email address", exactly one workflows send blocked with missing_recipient.
    • Nothing was delivered: the local mail catcher received no message for these recipients (64a9c11 run).
    • The 30f2376 and 7985d45 reruns show the same two events, one per workflow, with both email jobs per workflow completed.
  • Prod shape on 64a9c11, carried across the two clean master merges, read-only production data: most live email workflows have a non-transactional email step and some also have a destination step, so both quota branches meet real workflows; a few live email workflows send through an unverified domain today. No team sits at a workflow quota limit in the stored usage right now, so quota reports start as teams reach a limit. Missing recipients depend on person data at send time and are left to the post-deploy watch. Every stored field the change reads has the shape the code and fixtures assume.
  • Cross-version: not owed. Nothing the PR adds crosses a process or deploy boundary: the dedupe is in process memory, the reporter is a per-process constructor dependency, and queue payloads, invocation state and stored rows keep their shape. The unverified-domain error keeps its message text.
  • Post-deploy watch: after merge.
  • CI: required checks on the head 7985d45; see this PR's checks.

Release status

  • No feature flag controls this change

Docs update

None. No existing doc covers Workflows analytics events.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: T3 Code with Claude Code, Claude Opus 5.5 (claude-opus-5-5). An earlier Codex worker (GPT-6.1 Sol) recreated the change before its session dropped.

Question Answer Who Where
Report message deliveries? No. Dropped workflows message delivered: billing usage reports already give the first send, and delivery depended on SES returning the signed tracking header. Replaced the earlier delivery, block and failure scope. 👤 Human This description
Which quota blocks report quota_limited? Every quota-blocked run that has an email step, whichever quota stopped it. Replaces "only blocks by the email quota", which left a destination-quota block of an email workflow unexplained (review round 1). 🤖 Agent This description
Report a quota block on the transactional email route? No. Since #123849, the flows this route sends are transactional-only and skip the email quota, so nothing is left to report there. 🤖 Agent This description
Where does the missing-recipient report live? In EmailService's missing-recipient skip that master added, as workflows send blocked. It replaces the earlier report in RecipientPreferencesService as workflows send failed, since master now treats a missing address as a skip. 🤖 Agent This description
Dedupe per process or per service instance? Per service instance. A funnel counts the first occurrence. 🤖 Agent This description
Report an SES-side "not verified" rejection as unverified_domain? No. The funnel step is the team's own domain setup, which the stored verified flag records, and matching SES error text is fragile. 🤖 Agent This description
Assert the reporter port or captured analytics in send-path tests? The reporter port. Each test enters through a public seam, and the reporter's own tests cover port to capture. 🤖 Agent This description

Involvement: no grilling rounds, no prototypes. Reviews on the fork PR: qa-swarm round 1 (GPT-6.1 Sol: Correctness, Risk, Design, Systems, the author's Team lens; Opus 5.5: Fresh eyes), round 2, a final read, and a full-scope GPT-6.1 Sol review after the trim. On this PR: a full-scope panel (GPT-6.1 Sol: Correctness, Risk, Systems; Opus 5.5: Fresh eyes), a GPT-6.1 Sol final read of the fix, and GPT-6.1 Sol final reads of both master merges (their four findings were all in master files the PR does not touch). Proof captured: live runs of all three reasons and prod shape. CodeRabbit is retired on this repo.

  • Skills: tdd, modernize-pr, qa-swarm, merge-risk, pr-description, writing-voice.
  • 2026-11-02: moved from the fork PR feat(workflows): report blocked and failed workflow sends #115352 to a branch in this repo, merged master (no conflicts; master did not touch nodejs/src/cdp), and fixed the review finding above. A second review suggestion, lru-cache's ttl in place of the explicit hour check, was tried and dropped: the cache reads its clock at import, so Jest's fake timers cannot pin the hour boundary.
  • 2026-11-02: merged master again (66 commits, no conflicts) and reran the lane gate and both live runs on the merged head.
  • 2026-11-03: merged master again (no conflicts), because five new master commits touch this PR's files, and reran the lane gate, both live runs and a final read on the merged head.
  • Public artifact: nothing from the session beyond this repository went into the diff or this description.

🤖 Generated with Claude Code

Silthus and others added 2 commits October 31, 2026 12:09
@trunk-io

trunk-io Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

✅ Trunk lane — non-backend lane

This PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

⚠️ Duplication (TypeScript) — 5 new duplicated blocks (worst 201 tokens)

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy Second copy Lines Tokens
nodejs/src/cdp/consumers/cdp-data-warehouse-events.consumer.ts:55 nodejs/src/cdp/consumers/cdp-internal-event.consumer.ts:60 34 201
nodejs/src/cdp/consumers/cdp-cohort-membership-trigger.consumer.ts:97 nodejs/src/cdp/consumers/cdp-data-warehouse-events.consumer.ts:68 23 128
nodejs/src/cdp/consumers/cdp-cohort-membership-trigger.consumer.ts:105 nodejs/src/cdp/consumers/cdp-person-updates-consumer.ts:41 15 85
nodejs/src/cdp/consumers/cdp-data-warehouse-events.consumer.ts:67 nodejs/src/cdp/consumers/cdp-events.consumer.ts:65 17 77
nodejs/src/cdp/consumers/cdp-data-warehouse-events.consumer.ts:67 nodejs/src/cdp/consumers/cdp-dlq-replay.consumer.ts:145 13 73
✅ Hogbox preview — ready, open the preview

▶ Open the preview

🔑 Login test@posthog.com / 12345678 (demo data)
🧩 Running this PR's backend on the PostHog :master base (frontend unchanged by this PR)
🔗 Link stable across rebuilds: a re-push swaps the box underneath, the URL stays
🔒 Access tailnet only (PostHog VPN)
🛠️ Admin inspect & debug state in hogland
💤 Idle sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

commit 7985d45 · box box-80692bcea250 · ready in 1968s (push → usable) · build log · rebuilds on every push, torn down on close

@hosthog

hosthog Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

HostHog preview — posthog-desktop-web

The previews for this PR have been torn down and no longer serve.

A run blocked by the destination quota drops its email steps too, and it
reported nothing. Report `workflows send blocked` with `quota_limited` for
every quota-blocked run that has an email step, whichever quota stopped it,
and drop the `limitedBy` discriminator that only served the old rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@parameterai

parameterai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Risk: No findings

The delta since the last review is a master merge bringing in three production changes alongside this PR's analytics reporting: the events consumer can route team lookup off a Kafka team_id header (with a body-vs-header mismatch guard that parks the message before any globals are built), an InternalLogsProducer is wired into hog-function monitoring behind a default-off config, and the email queue-wait metric gains a bounded team_capped label. All three are fail-closed or disabled by default; no send decisions change and no new attacker-controlled input reaches them.

Sentinel reviewed 7985d45 · Review settings

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team November 2, 2026 12:54
@Silthus

Silthus commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@PostHog review

@posthog

posthog Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@Silthus PostHog Review is reviewing this pull request.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 30f2376: no findings.

@Silthus Silthus added the stamphog Request AI approval (no full review) label Nov 3, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

The risky paths have current-head review assurance and owning-team authorship. No unresolved substantive concerns or other refusal grounds were found.

  • PostHog Review (Standard) reviewed the current head: no findings.
  • posthog[bot] reviewed the current head.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 104L, 11F substantive, 369L/19F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1d-complex (369L, 19F, single-area, feat)
stamphog 2.7.0 .stamphog/policy.yml @ 30f2376 · reviewed head 30f2376

@Silthus

Silthus commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/trunk merge

@trunk-io
trunk-io Bot merged commit 5fd7b11 into master Nov 3, 2026
222 checks passed
@trunk-io
trunk-io Bot deleted the workflows-tracking/send-funnel-events branch November 3, 2026 12:07
Silthus added a commit that referenced this pull request Nov 3, 2026
EmailService now takes a required WorkflowsActivationReporter (#127135).
The local SES suite built the service without it and stopped compiling
after the master merge. It passes a stub, as the email service suite does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@deployment-status-posthog

deployment-status-posthog Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-11-13 14:46 UTC Run
prod-us ✅ Deployed 2026-11-13 15:05 UTC Run
prod-eu ✅ Deployed 2026-11-13 15:07 UTC Run

This table shows the posthog release only. Hogstation shows every service that this PR reaches, Kargo services included.

trunk-io Bot pushed a commit that referenced this pull request Nov 13, 2026
…fake (#127175)

* chore(workflows): cover the email send outcome matrix

Send test email through a local SES fake, so the send outcome matrix runs
against real SDK requests without mocking SES.

* test(workflows): tighten the email outcome matrix after review

- Use the production Redis pool and drop the TestRedisV2 copy.
- Cap and pacing cases assert their own log reason and a drained bucket, so a limiter outage cannot pass for a denial.
- Rows carry their own setup and expected values; tests with two flows are split.
- Titles name the behavior only, without matrix numbers or fork references.
- Drop the harness self-test and the duplicate SES throttle row.
- Keep the handbook to how to run the suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(workflows): name the start command when the local SES fake is down

The SES suite runs in the default Jest set, but the fake sits behind the
compose `ses` profile. A run without it now fails with the command that
starts it, instead of an error that reads like a regression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(workflows): share one Valkey pool across the SES suite

RedisV2 has no close, so a pool per test left one connection per test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(workflows): start the SES fake in test:full and services:start

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(workflows): close the SES suite's redis client so jest exits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(cdp): pass the activation reporter in the local SES email tests

EmailService now takes a required WorkflowsActivationReporter (#127135).
The local SES suite built the service without it and stopped compiling
after the master merge. It passes a stub, as the email service suite does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(workflows): move the uncovered email budget cases onto the in-process SES fake

Master's email service suite now sends through the real SESv2 client to an
in-process SES fake (#132730) and already covers most of this outcome
matrix. Add the cases it lacks there: shadow mode charges the team budget
and sends past an empty one, every to, cc and bcc copy is charged and a
send with more copies than the budget takes all of it and still sends, and
a send that a team suspension, a workflow pause or the suppression list
skips spends neither the workflow pacing nor the team budget.

Drop the Docker SES fake, its CI step, compose service, package scripts
and handbook line, which only served the separate suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(workflows): pin each skip gate's own metric and the full starting budget

Assert that each skipped send records its own gate's metric with no error,
and that the shadow test starts from a full budget, so a send that stops
early for another reason cannot pass as a gate skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
hogbox-preview — 7985d451 Deployed Nov 3, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hogbox-preview Spin up a per-PR PostHog preview on a hogland hogbox stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant