[Silthus]

Problem

• Teams without a verified sending domain can't send transactional email from code, even to their own team.
• With feat(cdp): send sandbox email from the sandbox worker #131834, a code send on the sandbox sender queues, then fails.

Changes

• A code send from the team's email_sandbox sender goes out through the sandbox path: member checks, caps, retries, no billing.
• The 202 and keyed replays carry sender_kind: "email_sandbox".
• Refusals: non-member 422; cc, bcc, reply-to, custom from or file attachments 400; blocked, paused or foreign sandbox sender 403.
• A keyed retry without a from address replays after Django resolved another sender.
• A keyed retry replays across the deploy: keys store both fingerprint shapes.
• A keyed send moved by a reschedule keeps both shapes and its sender, so a retry still replays it on either version.
• A new sandbox sender missing from the replica is read from the primary.
• Sandbox code sends carry no open or click tracking.
• A project suspension or workflow email pause set after the 202 stops the send at delivery, as for an ordinary code send.
• The message status moves to sent, or to failed with the reason.
• A rerun of a failed or canceled sandbox code send goes out on the sandbox sender again. Mail SES already accepted is not sent twice. Reruns of workflow sandbox email steps behave the same.

flowchart LR
  subgraph Before
    A1[code send] --> B1[email queue] --> C1[email service: refused]
  end
  subgraph After
    A2[code send] --> B2[email queue] --> D2[sandbox queue] --> E2[sandbox SES]
  end
  classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
  classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
  classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
  class A1,A2 phYellow;
  class B1,B2,D2 phBlue;
  class C1,E2 phRed;

Loading

This PR builds the part outlined in blue: code sends that run through the sandbox worker.

Caution

🔴 Merge risk: high · one-way door at merge: keyed code sends store both fingerprint shapes, and later versions must keep replaying them. Delivered sandbox mail stays sent.

• Keyed code sends without a from address, on any sender, in both regions: keys store both fingerprint shapes. The shadow diff and the cross-version proof show replies and replays match master's. No switch-off covers this path.
• A keyed retry after the sender changed now gets the earlier send's replay; master answered a conflict. Neither sends again, so no mail is newly lost.
• Sandbox code sends and their reruns, and reruns of workflow sandbox email steps, in both regions: a wrong sender, a lost outcome or a wrong reset sends twice or not at all. Tests cover reruns; no live run does.
• Code sends with a from address, on any sender, in both regions: the endpoint reads the sender before the replay check. A cache miss reads the primary database, and a failed read answers an error instead of the replay.
• Keyed sends moved by a reschedule, in both regions: the rewritten key keeps both shapes and the sandbox sender kind. While a pod on the previous release handles a reschedule, it keeps only the older shape: a retry still replays, but without sender_kind, or answers a conflict when Django resolved another sender. Neither sends again. Tests cover this; no live run does.
• Proved at 394a392: own-sender code sends match master's replies, jobs and deliveries (shadow diff), and keyed retries replay across versions (cross-version).
• Tier: critical: hog flow delivery and the mail sending flow in CDP.
• Switch-off: WORKFLOWS_SANDBOX_SENDER_ENABLED for the sandbox path; the fingerprint change is live on deploy, fix forward.

Rollout

• Stages: Node live in both regions before the Django change that sends sandbox sender ids.
• Watch: reserve conflict in cdp_hogflow_idempotency_operations_total; no alert pages on it.

How did you test this code?

Test rationale: sandbox-code-send.test.ts drives the endpoint and the real workers through a fake Cyclotron queue into a fake SES; each case failed first. The queue keeps each handoff, releases a job only once it is due, and follows Cyclotron's overwrite rules for reruns. One cdp-api.serial.test.ts row moved from conflict to replay.

• Proofs: live run at 2153c3a, prod shape at 77d3607. Since then only the rerun rebuild, master merges and tests changed, which neither run reaches. Cross-version and shadow diff ran again at 394a392 against master; 378a61c changes only sandbox-code-send.test.ts, so both are reused there.
• Master's attachments and scheduled sends came in through merges. A sandbox code send with files is refused, because the sandbox sender cannot attach them. Both fingerprint shapes carry the scheduled time.
• A scheduled sandbox send answers scheduled, sends nothing before its time, then delivers to the member.
• A keyed retry gets its replay before sandbox-only fields are refused, so a retry that now resolves to the sandbox sender does not hide a send already accepted on an own sender.

Proof evidence

• Live run: a member send through the sandbox path reached a local SES stand-in; a non-member got 422; a keyed retry replayed with one message sent; a send paused after its 202 sent nothing and its status reads failed, sending_paused; delivered sends read sent.
• Prod shape: every stored email sender in both regions is an own sender, and no queued job carries a sandbox sender. The split of stored idempotency keys by from shape is not measurable, a recorded gap; the cross-version run covers both shapes.
• Cross-version at 394a392, against master's merge base: keyed retries replay in both directions, and own-sender sends deliver. A sandbox send run by old workers fails without sending, so the deploy order holds.
• Shadow diff at 394a392, against master's merge base: own-sender code sends give the same replies, queued jobs and SES deliveries, with the sandbox switch off and on.

• Not checked: a live send against real SES, and a live rerun. Reruns of code sends and workflow steps run only in sandbox-code-send.test.ts, from lifecycle rows the workers wrote, stored in ClickHouse.

Release status

• No feature flag controls this change

Docs update

• None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Claude Opus 5.5

Skills: /writing-tests, /writing-pr-descriptions, /qa-swarm, /pr-description, /merge-risk

More context for AI agents

• Session: T3 Code implementation lane; Claude Code with Jest, tsc and the Fleet heavy runner.
• Decisions: the endpoint checks the sender's team and kind; the worker routes a function_email step with state.senderKind to the sandbox handler.
• Decisions: a failed membership or tenant read lets the send queue; the sandbox worker rechecks and retries.
• Decisions: the workflow email quota check still applies to sandbox sends.
• Follow-ups: retire the previous fingerprint shape. One key lifetime (24 hours) after this is live in both regions, write the current shape into fingerprint; drop the previous shape in a later deploy.

Question	Answer	Who	Where
Owed proofs	Live run, cross-version, prod shape and shadow diff	👤	comment
Schema-check CI failure	Base failure, accepted CI gap	👤	comment
Suspension and email pause	Checked at delivery, as for ordinary code sends	👤	comment
Final read 6 findings	Rejected: master's tests, outside this diff	👤	comment
Prod-shape gap	Accepted: the key split by from shape is unmeasurable	👍	comment
Rerun of a sandbox code send	Keep the sender and an accepted outcome; retry a failed one	🤖	comment
Keyed retries during the deploy	Store both fingerprint shapes	🤖	comment
Sender missing from the replica	Read the primary on a miss	🤖	comment
Engagement tracking	None on sandbox sends, documented	🤖	comment
Message status	Sent, or failed with the reason	🤖	comment

Involvement: review rounds: an eight-lens panel, a fixes round, a third round and fifteen final reads; live-run, prod-shape, cross-version and shadow-diff proof captured.

• Read closely: handler dispatch in hogflow-executor.service.ts, the queue predicate in the sandbox consumer, and the rerun rebuild in rerun-paginator.service.ts.

🤖 Generated with Claude Code

New commits were pushed — dismissing the stamphog approval from an earlier head. Re-add the trigger label to request a fresh review.

New commits were pushed — dismissing the stamphog approval from an earlier head. Re-add the trigger label to request a fresh review.

A new stamphog review started for this PR — the fresh verdict replaces this approval.

New commits were pushed — dismissing the stamphog approval from an earlier head. Re-add the trigger label to request a fresh review.

New commits were pushed — dismissing the stamphog approval from an earlier head. Re-add the trigger label to request a fresh review.

[posthog]

PostHog Review · Standard · reviewed 6adb9f2: 1 Should fix.

This is review round 4 on this pull request. When findings keep coming back in the same code, the approach is usually the problem. Before the next fix, check whether a different design would remove the whole class of findings.

[posthog]

P2 · Preserve the legacy fingerprint format in queued jobs

During a rolling deploy, an old pod rescheduling a keyed code send queued by a new pod passes this object to the old replayUntilDayAfter, which writes it verbatim as Redis fingerprint. Both versions then reject identical retries with 409 because they compare that object to strings. Keep the queued fingerprint a legacy string and store the new hash separately.

[stamphog]

Approved.

Risky delivery and replay changes have owning-team authorship, strong familiarity, and targeted regression coverage. The older review concern is fixed: queued jobs retain a legacy string fingerprint and store the newer hash separately; no substantive concerns remain.

• Author wrote 88% of the modified lines and has 34 merged PRs in these paths (familiarity STRONG).

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✓	no deny categories matched
size	✓	404L, 17F substantive, 1336L/22F incl. docs/generated/snapshots — within ceiling
tier	✓	T1-agent / T1d-complex (1336L, 22F, single-area, feat)
stamphog 2.8.0		.stamphog/policy.yml @ af5172f · reviewed head af5172f

[Silthus]

/trunk merge

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-11-13 07:07 UTC	Run
prod-us	✅ Deployed	2026-11-13 07:28 UTC	Run
prod-eu	✅ Deployed	2026-11-13 07:25 UTC	Run

_{This table shows the posthog release only. Hogstation shows every service that this PR reaches, Kargo services included.}