feat(workflows): keep sandbox sends out of tier history and record deliveries by Silthus · Pull Request #111904 · PostHog/posthog

Silthus · 2026-10-05T12:58:38Z

Problem

A team that sends from the sandbox sender gains sending-tier history it never earned on its own domain, and nobody can see when its first sandbox email actually arrives.

Stacked on #111820. This cumulative fork PR targets master. The review delta is a66bb200f89...feat/workflows-sandbox-first-delivery.

Note

🔵 Merge risk: low (merge-risk rubric)

Every team's tier decision reads the changed history code, and the public SES webhook gains a branch. A team without sandbox sends computes the same history as before, and an unset SES_SANDBOX_CONFIGURATION_SET keeps the webhook branch off.
Both behaviors sit behind the default-off sandbox sender, have focused tests on the risky paths, and revert cleanly.

Changes

Kind	Files	Lines
Production	6	+78 −25
Tests	2	+196 −1
Docs	1	+2 −0
Total	9	+276 −26

Sandbox sends no longer count toward a team's sending tier. A team with only sandbox sends gets the same tier as a team with no sends, including inactivity decay.
The tier sweep subtracts email_sandbox_sent from email_sent per team per day, floored at zero. The email worker already pushes that companion metric beside email_sent for every live sandbox send.
Bounces and complaints from sandbox sends still count. Those metrics carry no sender, and counting them can only lower a tier.
The SES webhook now captures workflows sandbox email delivered (is_test) when a Delivery event's ses:configuration-set tag is SES_SANDBOX_CONFIGURATION_SET.
The event goes to the team named by the signed tracking code. Test sends are included. A delivery on any other configuration set, or with an unsigned code, captures nothing.
An unset SES_SANDBOX_CONFIGURATION_SET never matches, so the event stays off until the CDP API deployment carries that setting.
Mechanical: createCdpCoreServices exposes the existing SandboxEmailSender, so the email worker and the webhook build the event name in one place. Two lines in the sending tiers doc.

The sandbox funnel for product analytics, which is not activation:

flowchart LR
    A[workflows sandbox sender provisioned] --> B[workflows sandbox email sent] --> C[workflows sandbox email delivered, first time]
    B -.->|Drop-off| D[workflows sandbox email blocked, by reason]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class A phYellow;
    class B,C phBlue;
    class D phGray;

"First sandbox delivery" is its own funnel step. The activation funnel must not count it, because activation means a real message to a real recipient from the team's own domain.

How did you test this code?

recompute_email_sending_tier_for_team over real app_metrics2 rows in ClickHouse, clock frozen. Sandbox-only sends decay like no sends; own plus sandbox sends stay under the use bar; own sends over the bar still promote. A pure-function test would miss a sweep that stops fetching the companion metric.
build_sending_histories over the same rows. One day with more sandbox than own sends cannot erase another day's own sends, and bounces and complaints from sandbox sends still count. These catch a whole-window subtraction and a feedback filter.
EmailTrackingService.handleSesWebhook through the CDP API route with a posted SNS Delivery envelope, signature check stubbed and captureTeamEvent spied. A sandbox-set delivery captures one event for live and test sends; a delivery on the tracked set, or on the sandbox set with an unsigned tracking code, captures none.
Each new case failed once against a deliberate mutation of the code under test, then passed on the real code.
Existing coverage: email.service.test.ts already asserts email_sandbox_sent beside email_sent for live sandbox sends, and no metrics for test sends.

Verify with pytest products/workflows/backend/test/test_email_sending_tier.py -k TestSandboxSendsInTierHistory (6 passed) and, in nodejs, npx jest src/cdp/services/messaging/email-tracking.service.test.ts -t 'sandbox deliveries' (4 passed).

Local gate: the touched pytest files, email-tracking.service.test.ts, ses.test.ts and email.service.test.ts, repo-wide mypy, ruff, prettier, eslint, scoped security semgrep and strict preflight pass. Node-wide TypeScript reports only the unbuilt @posthog/replay-anonymizer workspace package, which this PR does not touch.

Not checked: a real SES Delivery event from the sandbox configuration set. Whether SES carries the ses:configuration-set tag and the tracking header on that event belongs to the outside-code checklist.

Release status

No feature flag controls this change
This change is behind a feature flag and is not available to users
This change makes a previously flagged feature available to everyone

Sandbox sender rows exist only behind workflows-sandbox-sender, and sandbox sending stays off until WORKFLOWS_SANDBOX_SENDER_ENABLED is on. Without sandbox sends, both changes are inert.

Automatic notifications

Publish to changelog?

Docs update

Two lines in docs/internal/workflows-email-sending-tiers.md: the tier rule, and the CDP API setting the delivered event needs.

🤖 Agent context

Written by Claude Opus 5.5 (implementation lane owner), built test-first with /tdd and /writing-tests. The red tests were committed before the implementation.
QA lane taken over by Claude Sonnet 5.5 after Opus hit rate limits. It added the tests and the outcomeProperties helper from review.
Reviewed by /qa-swarm in two rounds: six lenses on GPT-6.1 Sol and a fresh-eyes lens on Claude Sonnet 5.5. Round 2 produced no fix.
Skills: /tdd, /writing-tests, /reviewing-with-coderabbit, /writing-pr-descriptions, /merge-risk, /qa-swarm.
CodeRabbit CLI: skipped, because the CLI was signed out and no person was available to sign in. The CodeRabbit bot reviewed the PR; its one inline thread is a follow-up for the base PR.
Decision: the delivered event reuses SandboxEmailSender.capture, so all sandbox events share one name pattern and one team lookup.
Decision: the subtraction happens per day before daily_sends is built, so the use bar and the decay both see own sends only.

🤖 Generated with Claude Code

…lusions Unfinished: the red/green proof runs and the OpenAPI regeneration were interrupted by a devbox rebalance. Tests are written but not yet run green; generated OpenAPI output is not regenerated. Continue test-first. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>