Repository navigation
feat(cdp): send sandbox email from the sandbox worker - #131834
Conversation
Pure modules the sandbox email worker will call before it sends: a fresh primary read of membership, organization block and tenant mirror; tenant admission with freshness bounds; the two refill daily caps; recipient suppression and opt-out checks; and the sandbox message builder. Nothing calls them yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-outs A sandbox authorization read that passes its deadline now runs no SQL on a connection handed over late and destroys a connection still busy, so a stalled primary cannot hold pool slots. An opt-out lookup that fails blocks the send for a retry instead of letting it through. The caps test shares one Redis pool per file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add EmailSuppressionService.lookupSuppression, which throws when the team list or the global hard-bounce list cannot be read. isSuppressed keeps its signature, its caching and its fail-open result. The sandbox recipient check calls the strict lookup and retries on a read failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the email_sandbox workflow action, the emailsandbox queue and its worker mode. The hogflow queue routes a sandbox step to the sandbox worker, which stores its outcome on the step and hands the job back; the hogflow queue then finishes the step and runs the next action. Sending stays off: WORKFLOWS_SANDBOX_SENDER_ENABLED defaults to false and only the literal 'true' turns it on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A step whose filter stopped matching, or that errored with on_error continue, moved on inside the sandbox worker and ran the next action there. The executor now runs one action per dequeue on the emailsandbox queue, and the sandbox worker hands any job that left the sandbox step back to the hogflow queue. Also adds email_sandbox to the integration kind type the wrong-kind test inserts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…licy' into workflows-sandbox/n3-sandbox-sender
The sandbox worker renders a sandbox email step, runs the delivery checks in order, and sends through the shared sandbox SES identity, tenant and configuration set with run tags. It retries with backoff (8 tries, 30 s to 15 min, jitter), records sent, failed and blocked as sandbox-only metrics and events, and never bills. The worker claims the account-wide SES bucket the email worker uses. Editor test sends render in cdp-api and run inline on the worker through a scoped, short-lived token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PostHog review |
|
@Silthus PostHog Review is reviewing this pull request. |
🤖 CI report✅ Trunk lane — non-backend laneThis PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes. ✅ Duplication (Python) — cleanNew Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying. ✅ Duplication (TypeScript) — cleanNew TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying. |
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 778458d: 2 Must fix, 6 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
Exempt the sandbox test route from the shared internal secret, since it verifies its own scoped token. Accept the unsaved workflow id in test sends, and report a stalled response body as pending. Bound the caps Valkey pool and each SES request with deadlines, and log SES errors before classifying them. Check stored preferences under the address the unsubscribe link carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PostHog review |
|
@Silthus PostHog Review is reviewing this pull request. |
|
Note 🤖 Claude Opus 5.5 responding on behalf of Michael Decisions made while building this PR:
|
- Check sandbox opt-outs under both the trimmed address and the rendered spelling its unsubscribe link carries, and claim the frequency cap once. - A retry after SES throttles keeps the daily cap tokens it already holds. - A render failure writes its reason to the run's logs, queued and in tests. - The provisioning read runs under the authorization read's deadline. - SES sends carry ph_sbx_parent_run for batch-triggered runs, with the tag names the sandbox feedback webhook reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PostHog review |
|
@Silthus PostHog Review is reviewing this pull request. |
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 8033197: 4 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.
This is review round 3 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
- A retry reuses the daily cap tokens only after SES throttled the send, which means SES refused it. A transient failure may follow an acceptance, so the next attempt claims again. - The held claim names a digest of the organization and stored recipient, so a re-render that changes the recipient claims that recipient's caps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A retryable refusal before SES, such as a failed membership read, now carries the cap claim a throttled attempt holds, so the next attempt does not spend the daily caps again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l alive - The editor test waits 30 seconds for the sandbox worker, longer than the worker's own reads, cap claims, and 10-second SES send, so a send SES accepted is not reported as pending. - The caps Valkey pool runs in non-fatal mode, so a stalled store retries its connections instead of leaking clients or exiting the worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PostHog review |
|
@Silthus PostHog Review is reviewing this pull request. |
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 5435c1f: 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.
This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
SES accepts only letters, digits, underscores and dashes in a tag value, and the sandbox feedback webhook attributes the same set. A step whose action id falls outside it now fails with unsupported_action_id before any check or cap claim, instead of SES rejecting it after the caps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sandbox step and test-route suites asserted on a recorder that replaced SandboxEmailEvents. They now run the real events class, team lookup and captureTeamEvent against an in-memory posthog-node client, and assert on the event the SDK receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t SES reply Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # nodejs/src/cdp/services/hog-inputs.service.ts
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 68029af: 1 Must fix, 1 Should fix. This pull request is large, so the review ran in 4 parts with less depth than usual.
This is review round 10 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
…livered address A sandbox send whose AWS credentials never resolve now settles at the SES deadline as a retry, so the rest of the sandbox batch completes. A sandbox email's unsubscribe link now names the address it is delivered to, the member's stored spelling, so an ordinary email to that address honours the unsubscribe. The address comes from the bounded authorization read; when that read is unavailable the step retries and the editor's test send reports it pending. The ordinary email path keeps its synchronous identity default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The delivered-address reader takes a deadline, so the stalled-read test sets a short one and drops its wall-clock assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
✅ Security review complete — 1 finding posted as a review, in 12 min. Add the |
The test now shows the retry through the queue, its schedule and the later send, not the step's retry counters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sandbox send now claims its slot in the account-wide SES bucket after provisioning, the organization block, recipient eligibility and the daily caps admit it, right before SES. A refused send no longer spends a slot ordinary email needs. The editor test route and the sandbox worker share the order: the sandbox queue no longer claims at dequeue, and the step claims through its own pool with the worker's claim deadline. An empty or unreadable bucket sends nothing and releases the held cap claim, as an SES throttle does, so the worker's retry reuses it. The stalled delivered-address test now asserts the unsubscribe outcome instead of the decoded identifier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send refused for an SES slot now returns its daily cap tokens to both buckets and drops the claim, instead of only clearing the spent marker. An editor test runs under a new claim id each time, so a released claim was never reused and a rate-limited test spent one of the member's daily sends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…box sends Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 94cd5d9: 1 Must fix. This pull request is large, so the review ran in 4 parts with less depth than usual.
This is review round 11 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An editor test claims its daily caps under a fresh id on every request, so releasing the claim after SES throttled or rejected it left both buckets charged and the next test could hit daily_cap_reached with nothing sent. Refund both buckets instead. An ambiguous SES failure still keeps the charge. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Not approved yet — waiting on the conditions below.
Re-add the stamphog label to request another review once you have addressed this.
stamphog won't auto-review this pull request because three gates refused it. The deny-list gate matched the auth category, which covers files like internal-api-auth.ts and jwt-utils.ts and the new sandbox authorization code. The size gate refused it because it has 2322 substantive lines against a ceiling of 1200 (6125 lines across 54 files including docs, generated files and snapshots). The tier gate classified it as T2-never.
Please ask a human reviewer to look at it. Since it sits on #131731, you could also split it into smaller pull requests, such as the auth and JWT changes separately from the sandbox email send path, so each piece is easier to review.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✗ | matches: auth |
| size | ✗ | too large for auto-review (2322L substantive in global — ceiling is 1200L; 2322L, 34F total, 6125L/54F incl. docs/generated/snapshots) |
| tier | ✗ | classified as T2-never: T2-never (6125L, 54F, single-area, feat) |
| stamphog 2.8.0 | .stamphog/policy.yml @ unknown · reviewed head 8cba180 |
There was a problem hiding this comment.
Not approved yet — waiting on the conditions below.
Re-add the stamphog label to request another review once you have addressed this.
StampHog won't auto-review this pull request because three gates refused it. The deny-list gate matched auth-related files, such as nodejs/src/common/api/middleware/internal-api-auth.ts, nodejs/src/cdp/utils/jwt-utils.ts, and sandbox-authorization.ts. The size gate counted 1464 substantive lines across 27 files, over the 1200-line ceiling. The tier gate classified the change as T2-never (4027 lines, 41 files, a feature), which is never auto-reviewed.
This decision is final, so the next step is a human review from the owning team, as the PR description already requests. Splitting the change into smaller pieces could bring the size under the ceiling, but the auth-related files would still need human review.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✗ | matches: auth |
| size | ✗ | too large for auto-review (1464L substantive in global — ceiling is 1200L; 1464L, 27F total, 4027L/41F incl. docs/generated/snapshots) |
| tier | ✗ | classified as T2-never: T2-never (4027L, 41F, single-area, feat) |
| stamphog 2.8.0 | .stamphog/policy.yml @ unknown · reviewed head 2c15289 |
|
/trunk merge |
Deploy status
This table shows the |
👀 Team review: the sandbox sender
I want the team to know what's coming and to hear if you'd do it differently.
Question: this lets a team send real email before it verifies a domain. Is this enough isolation and limiting to turn it on, or is something missing?
WORKFLOWS_SANDBOX_SENDER_ENABLED(off) stops it in a region. Aworkflows_sandboxsenderorganizationblockrow stops one organization.If you're happy with that, approve this PR and these four. They're the rest of the sandbox sender, so they need no separate review:
If not, comment here with what's missing.
Stamp request: StampHog cannot auto-review this PR (auth files, and 1464 substantive lines over its 1200-line limit). Agents and the merge gate did the bug review, so it needs a stamp from someone on team-workflows.
Note
N2 #131731 has merged, so this diff shows only N3's changes.
Problem
Replaces #128943, #128946, #129053, #131463, and the send-side parts of #129060 and #129063.
Note
🔵 Merge risk: low, 🔴 once sending is on · one-way door once sending turns on: sent messages
redis-v2.ts). Callers without a deadline, which is every existing caller, get no new timer and no new Redis call.WORKFLOWS_SANDBOX_SENDER_ENABLED, default off, is one switch for every team and stops every sandbox send in a region. Aworkflows_sandboxsenderorganizationblockrow stops one organization. The shared refactors have none: live on deploy, fix forward.Rollout
cdp_sandbox_email_send_totalby outcome. No alert pages on it yet.Changes
ph_sbx_parent_runfor batch-triggered runs.email_sandbox_*app metric and one product event. Sandbox sends never bill.sending_suspendedorsending_paused, before any cap, SES token or SES call. An editor test reads the pause from the workflow cdp-api loaded.pending. Ordinary email keeps its own read.pending.This PR builds the parts outlined in blue: the sandbox worker's send through the sandbox config set.
How did you test this code?
Test rationale: Each test drives a public seam and asserts what the caller sees: the delivered mail, the daily allowance left in real Valkey, rows in the real test Postgres, and what billing recorded. SES is a local HTTP endpoint that the real AWS SDK client calls.
sandbox-email-step.test.tsandsandbox-email-test-route.test.tsrun the real event service and team lookup, and assert the product event an in-memory posthog-node client receives. Dropping the event at the step, at the test route, or at the SDK call fails them.40 mutations of the guarded behavior (the sandbox worker's Valkey pools, refusal handling, allowance reuse, charging before SES, releasing only on a throttle or rejection, opt-out spellings, the delivered-address link and its fail-closed read, the credential stall, SES classification, the test-send bucket, its deadline and its dropped connection, the worker's SES claim deadline, the sandbox frequency cap, retry state, billing) each fail at least one test.
sandbox-email-step.test.ts: the real executor routes, sends with tags, hands back, then runs the next action. Also covers each failure,retries_exhausted, and the backoff bounds. The step runs the real frequency cap: with a stored cap of one email a week, the first send arrives and the second skips withfrequency_capped.sandbox-email-step.test.ts, unsubscribes: after a member unsubscribes from a sandbox email, the ordinary email check skips their stored spelling asopted_out. An opt-out under any of the four spellings skips the sandbox send. A delivered-address read that never answers retries with no SES call, then sends with the stored address in the link. The test gives the read a short deadline and asserts no duration; removing the deadline fails it by the Jest timeout.sandbox-email-step.test.ts, credentials: with a credential provider that never resolves, the stalled send retries at the SES deadline while the rest of its batch completes.sandbox-email.service.test.ts: every refusal stops before SES. A suppression or opt-out read stuck behind a real table lock retries. A step visit replayed after SES accepted it, or after its worker died while SES took the email, is charged again. A throttled or rejected send retries free; a transient one is charged. A cap store that stops answering after the claim sends nothing and retries.The billing test in
hogflow-executor.service.test.tsgains a sandbox step. It reads the records a local usage-ingestion gRPC endpoint accepted: only the function and own-domain email bill, and the step emits no ordinary email metric.sandbox-email-test-route.test.ts: the tester posts to the route through the production app setup, with a shared secret configured. Covers the 403 cases,pending, unsaved workflows, sending off, and a full or unreadable SES bucket. A real Valkey behind a proxy that stops answering mid-connection gets asend_rate_limitedanswer before the editor gives up. Once the store answers, the next test send is accepted on a fresh connection. A valid token on a path below the route still needs the internal secret, so the auth exemption covers only the route itself.sandbox-email-step.test.ts, SES bucket: a spent bucket retries the step and charges the daily allowance only for the send that goes out. A bucket store that stops answering retries with no SES attempt, then sends once it answers.sandbox-email-step.test.tsandsandbox-email-test-route.test.ts, suspension and pause: a suspended project or paused workflow skips with its reason, leaving the daily allowance, the SES bucket and SES untouched. With the team's email settings behind a real table lock, the step retries with no SES attempt, then sends once the lock lifts.sandbox-email-test-route.test.ts, with both daily caps set to one: after the SES bucket refuses a test, or SES throttles or rejects it, the next test sends. After SES fails in a way that may have sent it, the next test getsdaily_cap_reached. Dropping the refund, refunding only the recipient or only the project cap, and refunding a transient failure each fail it.Since the 40 above: refunding only one daily cap, moving the suspension or pause check after the caps, removing the suspension check, dropping the pause at the step, tester or route, and removing the suspension read's deadline or failing it open each fail a test.
server.serial.test.ts: a sandbox worker whose frequency-cap Valkey refuses connections stays up past the 10 errors that make an ordinary email worker exit, and starts once Valkey answers.sandbox-ses-client.test.ts: the exact tag names feat(cdp): handle sandbox ses feedback #131828 reads, with and without a parent run. A send SES never answers aborts at the deadline and retries.feat(cdp): route sandbox email steps to their own queue #131766's tests cover
on_error, the live-edit route-back and the kill switch.Route walk at 8cba180, on this tree alone: the real sandbox worker, Postgres, Valkey and a fake SES, driven through the HTTP test route the editor calls, with both caps at one. A send is accepted, then capped. After a throttle and a refused SES slot the next test sends; after a rejection too. After a transient SES error the next test is capped. The editor surface ships in U1 feat(workflows): choose the sandbox sender in email steps #131916.
Shadow-diff at 8cba180 against master 92ba810: same ordinary jobs, frequency-cap decisions and unsubscribe identifiers, and Redis v2 and frequency-cap latency inside master's own run-to-run spread. The only difference is the sandbox test path's 404 on the ordinary worker.
Prod-shape: the sandbox sender tables were read in prod-us and prod-eu at b1c4b98, after P1 (feat(workflows): store the sandbox sender kind and state #131749) deployed. No change since then touches those tables, and the delivered-address read reuses the same query.
Not checked: a live SES send, a cross-version run, cancellation while queued, and the shared bucket key, which only the code structure guarantees.
Not built: the separate 2-second connect timeout. Only the 10-second total timeout exists.
Release status
Docs update
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Agent: Claude Code, Opus 5.5 (claude-opus-5-5)
Skills:
/writing-tests,/writing-pr-descriptions,/pr-description,/merge-risk,/leaving-pr-commentsfrequency_capped, no hold.SANDBOX_SES_TAGSin the SES client under the same name, since N3 cannot import N4's branch. One should import the other once both land.message_category_idandmessage_category_type, so test sends run the category check. Accepts the unsaved workflow id.More context for AI agents
pendingmapping. Final reads led to checking opt-outs under every address spelling, the member's stored one included, and to binding a held cap claim to its recipient. The opt-out check runs after authorization against the address SES receives, which also closes feat(cdp): add sandbox email delivery policy #131731's casing gap (Michael chose to fix it here, 2026-11-10). A later PostHog Review led to keeping caps across retries, render diagnostics, and the provisioning deadline.SANDBOX_SES_TAGSonce N3 and N4 land. The 2-second connect timeout. The dev walk follows I1 and C1.internal-api-auth.tsexemptions. The sandbox exemption's mock rows moved to a real HTTP test insandbox-email-test-route.test.ts. Suppression now goes through master's fail-closedisSuppressed(fix(cdp): hold an email when the suppression list cannot be read #128878), taken on feat(cdp): add sandbox email delivery policy #131731.new) counts as unattributed.ses_throttledorses_rejected, closing the window where a crash after SES acceptance sent one email free of both caps.email-mx-validationpool, and to d16bc6c, which gives the test route the worker's non-fatal pool settings with a claim deadline that drops a stalled connection. 3aa1057 and d34dcbf merge feat(cdp): add sandbox email delivery policy #131731's sandbox read pool with a connection timeout, so a database stalled in the startup handshake frees its gate slot.recipientIdentifier, so the ordinary default stays synchronous. The worker step's time budget now counts one more 2-second authorization read. The worker heartbeats its jobs, so no job deadline caps that budget.send_rate_limitedmessage. U1's banner needs copy forsending_suspended,sending_pausedandsuspension_state_unavailable(shown as pending).SES_SANDBOX_FROM_ADDRESSconfig entry was dropped, since feat(cdp): handle sandbox ses feedback #131828 now declares it on master with the same empty default.sandbox-email.service.ts(check order),sandbox-ses-bucket.ts(slot claim),sandbox-email-step.ts(retry state),sandbox-render.ts(delivered address),recipient-preferences.service.ts(shared extraction).🤖 Generated with Claude Code