Repository navigation
feat(cdp): add the upload route for transactional email files - #131833
trunk-io[bot] merged 218 commits into
Conversation
The store signs upload URLs for a file's key, size and SHA-256, accepts files for a send, keeps them past the bucket's 37-day expiry with a self-copy, and loads them for the worker. Nothing calls it yet. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A use that needs a file past what a self-copy can keep is refused instead of accepted. Signing refuses credentials that would expire before the URL even after a refresh, and gives up after the store's timeout. Refs PostHog/michaelr-context#77. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each bucket call now ends when its signal aborts, even while the client still waits for credentials. The retention limit applies only when a use needs a copy, so a fresh upload keeps working through its expiry. Refs PostHog/michaelr-context#77. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs PostHog/michaelr-context#77. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send that lists files loads them from the attachment store and passes them to SES as attachments. The load and the SES call share one abort deadline. A load that fails or runs past the deadline is rescheduled up to 10 times, then fails as attachment_unavailable. Sent and failed engagement events carry the file count and bytes. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SES client resolves its credentials before it checks the abort signal, so a stalled refresh held a send with files past its deadline. The call now settles at the deadline, as the attachment store's calls do. A workflow's next email action no longer inherits the attachment load retries of the email before it. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send whose files fail to load pays the workflow and team sending limits before the load. Its retry now carries a mark that the limits are paid, so it does not spend them again, and the mark stays off the next email action. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only the team's daily budget carries across attachment load retries. The per-workflow pacing limits a rate, so it applies to every try: retries that waited out a storage outage must not send all at once when the store recovers. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- defer() hands a job back for later without touching its poison count or retry budget; an email job takes a fresh dequeue_seq, so it goes to the back of its team's turn. - A job keeps its heartbeat until its release write settles. - Stopping the job queue stops dequeuing, waits for the result writes (bounded), then closes the pool. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One 25-second deadline, counted from the start of the stop, covers the batch in progress, the result writes and closing the pool. A pool close that waits on a blocked query no longer holds the stop. - Each result write is tracked on its own, so one failed write no longer ends the drain while its siblings still write. - A finished write removes only its own claim, not a newer claim of the same job. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A send whose files fail to load now carries the paid-budget mark only when the team budget was charged on that try. A try that skipped the charge, such as one whose tier lookup failed, leaves the retry to charge it. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y budget A send with files leaves its batch once an in-flight registry admits it by a memory budget sized from the pod, a heap check and a cap. A send without room is deferred with a growing, jittered delay. Deferrals keep the poison count, spend no retry, and show on a token metric and on the lateness histogram as deferred. Ticket: PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Track each storage call from its start, so a sibling failure keeps the bytes reserved. - Release the reservation only once every call settled, with no timeout. - Write the result after at most 3 s of waiting on abandoned calls, inside the stop deadline. - Handle a failing send while the plain batch runs, and wait for sends before a plain-batch error. - Run canceled jobs with files on the plain path, so a refusing worker ends them. PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A send's promise settles once its result is written; its reservation is released in the background once every storage call settled, so a failed plain batch never waits on a stalled call. - Test the canceled send with files through the real worker: it ends canceled instead of deferring. PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…16 bodies - A deferred job heartbeats until its deferral write settles, as an admitted send does until its result write. - A tracked storage call leaves the send's set once it settles, so its buffer is not held to the end of the send. - The reservation covers email text outside Latin-1, which makes the SES request body two bytes per character. PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A send naming a missing file is refused before any present file is copied. - Each reference travels with the stored file it names, rather than a parallel array of sizes. PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailure One failed write no longer settles the batch while its sibling writes still run, so the batch keeps heartbeating those jobs until each write lands. PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-in-flight-registry
Deferral writes share the result writes' wait, so one failed deferral no longer stops the heartbeat of the others. PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…load-deadline # Conflicts: # nodejs/src/cdp/services/hogflows/hogflow-executor.service.test.ts # nodejs/src/cdp/services/hogflows/hogflow-executor.service.ts # nodejs/src/cdp/services/messaging/email.service.ts
…e-in-flight-registry
cdp-api answers each requested file present or upload, behind the CDP_EMAIL_ATTACHMENTS_TEAMS allowlist, which stays empty. The files must fit the 40 MB message estimate, and a team's upload bytes per UTC day are capped in Redis. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🤖 CI report✅ Trunk lane — non-backend laneThis PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes. ✅ Duplication (Python) — cleanNew Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying. ✅ Duplication (TypeScript) — cleanNew TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying. ✅ Comment density — 0% of added code lines are comments (0 of 453)This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out. Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See This check does not block merging. It updates on every push and clears when the share drops. |
…e-in-flight-registry The email service and email worker tests now send through master's SES fake. The shared email delivery fakes keep the bucket proxy and the upload helpers, and read deliveries from that fake. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A deferral writes the job state back without the persistence sanitizer. The test runs it with the integration ref writer on and off, and checks that the files, the queue metadata and the stored integration inputs come back as the queue stored them. Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-upload-route Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…worker's resolver Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-in-flight-registry Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-upload-route Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-upload-route Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-upload-route Refs PostHog/michaelr-context#77 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This pull request was merged into |
Deploy status
This table shows the |
Merge gate: passed.
fleet proof gatepasses on e7a3351 (live run, review at HEAD, CI green, judge fail accepted by standing rule). Layer 2 of 4 in the node-store attachment stack: #133482, this PR, #133495, then #131944. The stack merges from its top PR; #131567 is live in both regions and the env settings are merged.Problem
Note
🟡 Merge risk: medium · two-way door
field_not_acceptedwhile the allowlist stays empty.CDP_EMAIL_ATTACHMENTS_TEAMS; fix forward.Changes
POST /api/projects/{team_id}/transactional_email/attachments/uploadsanswers each filepresentoruploadwith a signed PUT.attachment_uploadkind, allowlist, 1 to 10 files, the 40 MB message estimate, then the store.upload_quota_exceeded. Present files and refused requests do not count.delivery_unavailableand logs a warning with the team and the error.cdp_email_attachments_upload_requests_totalby outcome, includingunauthorizedandnot_configured, so a token mismatch with Django shows up.How it works
--- title: Upload, then send --- sequenceDiagram participant C as caller participant A as cdp-api participant B as bucket participant W as email worker rect rgba(9, 105, 218, 0.15) Note over C,B: upload #35;131833 C->>A: sends the sha256 and size of each file A->>B: HEADs team/<id>/<sha256> A-->>C: answers present, or a PUT URL signed for that size and hash C->>B: PUTs the bytes end rect rgba(26, 127, 55, 0.15) Note over C,B: send #35;131944 C->>A: sends, naming each file by its sha256 A->>B: HEADs each file, copies any with under 7 days left A-->>C: answers 202 and queues a job without bytes end W->>B: GETs each file by sha256, at most 6 days laterAttachmentUploads.prepare, in the upload span of the flow above.sha256and size, and cdp-api answerspresentor a PUT URL signed for that size and hash.presentonly with a day and 15 minutes left, longer than the 15 minutes an upload URL lives.How did you test this code?
Release status
Docs update
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Agent: Claude Code, Claude Opus 5.5
Skills:
/writing-tests,/writing-code-comments,/writing-pr-descriptions,/stacking-prsattachment_uploaddelivery_unavailable, so the cap holdsInvolvement: 8 cross-model review rounds and 2 final reads (GPT-6.1 Sol panel, Opus fresh eyes), one full qa-swarm plus the stack-tip pass, PostHog Review on each head (no findings on the final head); live run captured. CodeRabbit is retired and skipped.
More context for AI agents
AttachmentUploads.answerandchargeDailyUploadBytesinattachment-uploads.ts.🤖 Generated with Claude Code