[Silthus]

Warning

Merge risk: medium (qa-swarm danger rubric)

• Every sandbox send now depends on a new multi-key Valkey Lua claim, and cdp-api opens the SES Valkey pool for the first time.
• Limited by: the sandbox switch is off by default, every failure path skips the send rather than sending, and own-sender sends never reach the new code.

Problem

The sandbox sender lets a project email its own organization members before it verifies a domain, but nothing limits how much it sends per day.
The sending tiers do not apply to sandbox sends, and test sends skip workflow pacing.

Stacked on #111483. This cumulative fork PR targets master. The review delta is 754efd702a4...feat/workflows-sandbox-daily-caps.

Changes

• Every sandbox send, test sends included, now has to fit a daily cap per project and a daily cap per recipient address.
• A send over either cap is skipped, not retried. The run log says the daily limit was reached and that verifying the project's own domain lifts it.
• A per-address skip names the addresses at their cap.
• A limiter error, a missing limiter, or an empty or invalid cap setting skips every sandbox send with "could not check" in the run log.
• Cap settings are parsed strictly: 25 per day, 1.5, 0 and an empty value all block. The shared env parser would read 25 per day as 25.
• Analytics records workflows sandbox email blocked with reason cap_reached or check_failed, and a count, never the addresses.
• One Valkey Lua call claims every bucket of a send, all or nothing:
  • the project bucket takes one token per recipient copy (To, CC and BCC);
  • each distinct address (trimmed, lower-cased) takes one token from its own bucket.
• A denied claim writes nothing, so a skipped send spends no tokens.
• The claim is the last step before SES, after the final membership recheck. Suppressed, rescheduled, rejected or rechecked-out sends spend nothing.
• Own-sender sends never reach the sandbox buckets.
• Test sends run inline in cdp-api, so cdp-api now opens the same SES Valkey pool as the email workers. Without it, every sandbox test send would fail closed.
• New worker settings WORKFLOWS_SANDBOX_DAILY_TEAM_CAP and WORKFLOWS_SANDBOX_DAILY_RECIPIENT_CAP. An absent setting uses the default in nodejs/src/cdp/config.ts.
• Mechanical: a new claimAllOrNothing method on RateLimiterService, a dedicated sandbox-email limiter instance for metrics, a lookup table for the sandbox skip copy, and one docs bullet.

Warning

The CDP API deployment needs the SES_RATE_LIMITER_VALKEY_* settings the email worker already has. Without them, sandbox test sends skip with "could not check the sandbox sender's daily limit".

flowchart LR
    A[Sandbox send] --> B[Recipient and suppression gates] --> C[Footer and final recipient recheck] --> D[Claim daily caps]
    D -->|Granted| E[SES]
    D -->|Cap reached or check failed| F[Skip and explain]
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    class A,F phYellow;
    class B,C,D phBlue;
    class E phRed;

Loading

Before this PR, the same path went from the final recipient recheck straight to SES.

Note

Every bucket key shares the {teamId} hash tag, so the one Lua call stays in one Valkey cluster slot.
The per-address bucket is therefore per project and address, not global per address. A cross-project bucket would need a cross-slot call, which clustered Valkey rejects.

Review notes:

• The buckets refill continuously over 24 hours, like the sending tier buckets, so a fresh bucket allows up to about twice its cap in the first day.
• An SES throttle reschedules the send after the claim, so its retry claims again. An SES rejection after the claim also keeps its tokens. Both err toward fewer sends.

How did you test this code?

Test rationale: Four new parameterized cases in the sandbox sender block of email.service.test.ts drive EmailService.executeSendEmail with real Postgres, the real Valkey limiter, and sesV2Client.send mocked. No existing test covered sandbox caps. Each case runs for live and test sends.

• The project cap case fails if a denied multi-recipient send or an own-sender send spends tokens.
• The per-address case fails if a denied send charges the other addresses, if address case is not folded, or if only the first capped address is reported. Each mutation was checked: both rows fail.
• The repeated-address case fails if the project is charged per distinct address instead of per copy, or an address per copy.
• The fail-closed case covers an unreachable limiter, no limiter, and empty, zero, fractional, negative and trailing-text cap values.
• The existing membership recheck case now also proves a send blocked at the final recheck spends no tokens. It failed before the claim moved.
• The existing sandbox cases now run with a real cap limiter by default.

Local gate: the email service and rate limiter suites, Node lint, scoped semgrep and strict preflight pass. Node-wide TypeScript reports only the unbuilt @posthog/replay-anonymizer workspace package, which this PR does not touch.

Proof on cluster-mode Valkey

A scratch script (not committed) ran SandboxEmailSender.claimDailyCaps against the dev stack's cluster-mode Valkey (cluster_enabled:1), with a project cap of 3 and an address cap of 1:

claim 1 (3 addresses, cap 3): {"type":"granted"}
claim 2 (same addresses): {"type":"project_cap_reached"}
claim 3 (new address, team spent): {"type":"project_cap_reached"}
control, keys with two different hash tags: CROSSSLOT Keys in request don't hash to the same slot

The control shows the node rejects a cross-slot call, so the granted claim proves every key of a send shares one slot.

Not checked: a multi-node Valkey cluster, the cdp-api wiring in a running server (no test seam for server.ts), and the fork's CI, which waits for a maintainer to approve the run.

Release status

• No feature flag controls this change
• This change is behind a feature flag and is not available to users
• This change makes a previously flagged feature available to everyone

Automatic notifications

• Publish to changelog?

Docs update

docs/internal/workflows-email-sending-tiers.md names the two cap settings, the fail-closed rule, and the CDP API's need for the SES Valkey settings.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5 (tests, review, refactor); Codex, GPT-6.1 Sol (implementation against the failing tests)

• Built test-first with /tdd and /writing-tests. Tests landed red in their own commit.
• Skills: /tdd, /writing-tests, /reviewing-with-coderabbit, /writing-pr-descriptions.
• CodeRabbit CLI: skipped, because the CLI was signed out and no person was available to sign in.
• qa-swarm: four rounds of a cross-model panel. GPT-6.1 Sol ran the correctness, security, production, design, tests and adversary lanes through Codex; Claude Sonnet 5.5 ran fresh eyes. The GPT correctness lane kept ending without a verdict, so Claude Opus 5.5 ran it in round 4. Round 4 came back clean on 1bdf14051a7.
• Fixed from the panel: strict cap parsing, the claim after the final recipient recheck, the project cap wording, cdp-api's missing Valkey pool, and tighter tests.
• Rejected from the panel: claiming before workflow pacing (it would charge daily tokens on every pacing reschedule).
• Call: an absent cap setting uses the code default, and an empty or invalid one blocks. The alternative is no code default, with values set only per environment.
• Decision: the cap claim runs after suppression, pacing and the final recipient recheck, so a send that never goes out spends no tokens.
• Call: server.ts is outside the ticket's write scope, but cdp-api had to receive the Valkey pool for test sends to work.
• Decision: one token per distinct address for the per-address bucket, one token per copy for the project bucket.
• Decision: recipient addresses are SHA-256 hashed in bucket keys, so no address is stored in Valkey.

🤖 Generated with Claude Code

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: c1ce9b35-4106-4edb-8f0b-2abadd6b6125

📥 Commits

Reviewing files that changed from the base of the PR and between fce4e20 and 1bdf140.

📒 Files selected for processing (3)

• docs/internal/workflows-email-sending-tiers.md
• nodejs/src/cdp/services/messaging/email.service.test.ts
• nodejs/src/server.ts

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change adds managed sandbox email sender provisioning and API protections. The Node.js email service checks recipient membership, applies daily project and recipient caps, adds an identification footer, and routes sandbox messages through configured SES settings. Sandbox sends bypass team sending-budget claims. New tests cover provisioning, lifecycle behavior, recipient checks, send handling, and caps. Documentation records sandbox limits and rollout conditions.

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to 1bdf1

Sandbox sending remains disabled by default. Before enabling it, confirm the SES settings are complete: an incomplete configuration can spend daily cap capacity before delivery succeeds.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 1bdf1

The managed sender has strong project-ownership and recipient controls, including for test sends. The main design concern is that the advertised daily caps permit an initial burst plus continuous refill rather than enforcing a strict daily maximum. Deployment configuration and rollback behavior also need an explicit operational contract.

Retained concerns

• Low · security · inferred: The new daily caps are continuously refilled token buckets, not strict daily send ceilings. An eligible sender can exhaust a full bucket and then spend refilled tokens within the same 24 hours. For example, a recipient cap of five permits five initial claims followed by four more at 4.8-hour intervals before 24 hours elapse. Atomic claiming prevents concurrent overspending of the available balance but does not enforce the advertised daily maximum. Token-bucket guidance acknowledges burst behavior; whether that allowance is acceptable for sandbox abuse containment remains a policy distinction.

Security review details

Security Blast Radius

• inferred — The new authority exposes a shared managed SES sender to eligible projects, but recipients remain confined to verified active members of each project's organization. Both project and recipient quota keys are project-scoped, so the same address has separate allowances across projects; these are not a global address or shared-tenant ceiling.

Security Findings and Attack Paths

• inferred — An eligible project sender can exceed a literal configured daily recipient count by consuming the initial burst and later refill, without bypassing membership or atomic quota checks. This is a bounded control-semantics concern; the evidence does not demonstrate arbitrary-recipient delivery or acquisition of another project's sender authority.

Trust Boundaries and Controls

• observed — Workflow-supplied integration IDs are checked against the invocation's project. Organization identity is resolved from that project, and membership lookup errors block delivery. Managed integration mutation guards prevent project users from changing the provider identity or server-owned sending configuration through the inspected API paths.

Resilience and Maintainability Implications

• observed — The quota script checks all balances before writing any granted debits, and ordinary denial writes nothing. Keys share a team hash tag; limiter errors and invalid replies block the send. Provisioning uses an existing database uniqueness constraint on project, kind, and integration identifier to contain concurrent creation.

Hardening Proposals

• proposed — Define whether sandbox policy requires a strict calendar-day or rolling-day ceiling, or intentionally permits rate-plus-burst behavior. If a strict ceiling is required, use quota state that enforces that window rather than continuous refill.
• proposed — Make coordinated configuration and rollback requirements explicit: validate the delivery sender and SES route before quota debit, align Python provisioning settings with worker settings, and distinguish provisioning eligibility from revocation of already-provisioned senders.

🚥 Pre-merge checks | ✅ 1

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description check	✅ Passed	The description explains the problem, user-visible changes, cap behavior, testing, release status, docs, and agent involvement. It is mostly complete, but it omits the required agent definition-of-don…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

nodejs/src/cdp/services/messaging/email.service.ts-601-614 (1)

601-614: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Treat missing sandbox SES routing settings as the switch being off.

The gate checks only config.enabled. SES_SANDBOX_TENANT_NAME, SES_SANDBOX_CONFIGURATION_SET, and SES_SANDBOX_FROM_ADDRESS all default to ''. If the flag is on but one of them is unset, the send still claims daily-cap tokens. sesRouteFor (Line 1194-1199) then passes TenantName: '' and ConfigurationSetName: '', and resolveFromSender (Line 1220) falls back to email: ''.

The result is either an SES rejection after tokens were already spent, or a send with no configuration set. A send with no configuration set has no bounce or complaint event destination, so suppression and reputation tracking cannot see it. The other sandbox misconfigurations already fail closed with check_failed. Handle these settings the same way.

Proposed fix

-            if (isSandbox && !this.sandboxSender?.config.enabled) {
+            const sandboxConfig = this.sandboxSender?.config
+            const sandboxReady =
+                !!sandboxConfig?.enabled &&
+                !!sandboxConfig.tenantName &&
+                !!sandboxConfig.configurationSetName &&
+                !!sandboxConfig.fromAddress
+            if (isSandbox && !sandboxReady) {

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 08301290-a6bd-4c33-99b8-f249565ea002

📥 Commits

Reviewing files that changed from the base of the PR and between 084a506 and 5d64942.

⛔ Files ignored due to path filters (2)

• pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
• products/integrations/frontend/generated/api.ts is excluded by !**/generated/**

📒 Files selected for processing (23)

• docs/internal/workflows-email-sending-tiers.md
• nodejs/package.json
• nodejs/src/cdp/cdp-services.ts
• nodejs/src/cdp/config.ts
• nodejs/src/cdp/services/managers/organization-members.service.ts
• nodejs/src/cdp/services/messaging/email.service.test.ts
• nodejs/src/cdp/services/messaging/email.service.ts
• nodejs/src/cdp/services/messaging/sandbox-email-sender.ts
• nodejs/src/cdp/services/rate-limiter/rate-limiter.service.ts
• nodejs/src/cdp/types.ts
• nodejs/src/cdp/utils.ts
• posthog/api/integration.py
• posthog/models/integration/__init__.py
• posthog/models/integration/email.py
• posthog/settings/ses.py
• products/integrations/mcp/tools.yaml
• products/workflows/backend/facade/api.py
• products/workflows/backend/facade/contracts.py
• products/workflows/backend/presentation/views/hog_flow.py
• products/workflows/backend/services/sandbox_sender.py
• products/workflows/backend/test/test_sandbox_sender_lifecycle.py
• products/workflows/backend/tests/api/test_email_reputation.py
• products/workflows/backend/tests/api/test_sandbox_sender.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

[scheduled-actions-posthog]

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, please remove the stale label – otherwise this will be closed in another week. If you want to permanently keep it open, use the waiting label.