Skip to content

feat(cdp): reserve part of the ses bucket for a higher-priority lane - #127613

Closed
meikelmosby wants to merge 2 commits into
feat/cyclotron-transactional-table-migrationfrom
feat/ses-reserve-floor
Closed

meikelmosby wants to merge 2 commits into
feat/cyclotron-transactional-table-migrationfrom
feat/ses-reserve-floor

Conversation

@meikelmosby

@meikelmosby meikelmosby commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • Transactional email outranks marketing today only because one worker fleet claims SES capacity, then dequeues in priority order.
  • The next layer splits that fleet in two. Both would claim from @posthog/ses/global as peers, so a pod holding only marketing rows could drain it before a password reset claims.

Changes

  • A token-bucket claim can carry a reserve floor: tokens that caller may not spend.
  • A caller passing no floor always finds that much capacity waiting.
  • One bucket keeps one refill rate, so the total send rate does not change.
  • A floored caller denied behind another caller's slot still waits until its own tokens refill above the floor.
  • The queue refuses to start with a negative floor or one at capacity.
  • Inert on deploy: CDP_SES_RATE_LIMIT_TRANSACTIONAL_RESERVE defaults to 0 and no caller passes a floor.
  • Nothing user-visible changes here.

How did you test this code?

Test rationale: Four cases on RateLimiterService, which had no floor coverage: a floored caller draining the reserve, a double debit, and a denied caller waking before the floor refills, alone or behind an unfloored caller's slot.

Ran locally: the rate-limiter and job-queue Jest suites.

Release status

  • No feature flag controls this change

Docs update

  • None.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, Opus 5; restacked and finished by Claude Code, Opus 5.5

Skills invoked: /writing-tests, /writing-pr-descriptions, /stacking-prs.

More context for AI agents

The floor started clamped at zero. The third test showed that a bucket below the floor then reported a shortfall of only the tokens requested, so a denied caller booked a slot before the floor could refill. The value stays negative now; the grant clamps separately.

Layer 2 of 4, on top of the table migration in #127619.

@meikelmosby meikelmosby self-assigned this Nov 2, 2026
@trunk-io

trunk-io Bot commented Nov 2, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@parameterai

parameterai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Risk: Low · 1 low

This change adds an optional reserveFloor to the SES token-bucket limiter: a floored caller may only spend tokens above the floor, leaving that capacity for callers that pass no floor, while one shared bucket keeps a single refill rate. It is inert today — the default is 0, no caller passes a floor, and I verified the Lua math preserves the no-overshoot invariant for any non-negative floor. The main risk is that the new knob reaches the Lua script with no validation: a negative or oversized value silently breaks the rate limit or starves a lane.

Sentinel reviewed 1f61448 · Review settings

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

✅ Trunk lane — non-backend lane

This PR is assigned to the non-backend lane. It does not run backend Python tests and may merge in parallel with PRs in other lanes.

⚠️ Duplication (Python) — 2 new duplicated blocks (worst 90 tokens)

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy Second copy Lines Tokens
products/warehouse_sources/sources/folk/folk.py:144 products/warehouse_sources/sources/select_star/select_star.py:156 18 90
products/warehouse_sources/sources/ruddr/ruddr.py:127 products/warehouse_sources/sources/tempo/tempo.py:136 20 72
⚠️ Duplication (TypeScript) — 11 new duplicated blocks (worst 195 tokens)

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

First copy Second copy Lines Tokens
packages/agent/packages/agent/src/adapters/claude/claude-agent.slash-command.test.ts:49 packages/agent/packages/agent/src/adapters/claude/claude-agent.task-notification.test.ts:53 38 195
products/replay_vision/frontend/components/QuillScanList.tsx:31 products/replay_vision/frontend/components/ScanList.tsx:30 34 191
products/replay_vision/frontend/components/ChapterFilmstrip.tsx:26 products/replay_vision/frontend/components/QuillChapterFilmstrip.tsx:27 31 174
products/replay_vision/frontend/components/QuillScanList.tsx:203 products/replay_vision/frontend/components/ScanList.tsx:179 15 172
frontend/src/scenes/session-recordings/player/sidebar/PlayerSidebarOverviewDeviceTab.tsx:11 frontend/src/scenes/session-recordings/player/watch/QuillOverviewDeviceCard.tsx:11 14 144
products/replay_vision/frontend/components/QuillScanList.tsx:87 products/replay_vision/frontend/components/ScanList.tsx:81 27 138
frontend/src/scenes/session-recordings/player/sidebar/PlayerSidebarOverviewSessionTab.tsx:14 frontend/src/scenes/session-recordings/player/watch/QuillOverviewSessionCard.tsx:14 12 102
products/replay_vision/frontend/components/QuillScanList.tsx:257 products/replay_vision/frontend/components/ScanList.tsx:231 24 101
products/replay_vision/frontend/components/ObservationProgressBar.tsx:12 products/replay_vision/frontend/components/QuillObservationProgressBar.tsx:10 19 87
frontend/src/scenes/session-recordings/player/sidebar/PlayerSidebarOverviewPersonTab.tsx:15 frontend/src/scenes/session-recordings/player/watch/QuillOverviewPersonCard.tsx:16 14 84
products/replay_vision/frontend/components/ObservationProgressBar.tsx:45 products/replay_vision/frontend/components/QuillObservationProgressBar.tsx:43 13 70
🚨 Comment density — 14% of added code lines are comments (13 of 92)

This section warns when comments are more than 3% of the code lines a PR adds, and alerts above 6%. Before agent-assisted PRs, the typical share was about 2%. Only full-line comments count. Docstrings, generated files, snapshots, migrations, and workflow files are left out.

Comments that restate the code, record how the change came about, or narrate the next line add noise for the next reader. Keep the comments that explain a reason the code cannot show, and remove the rest. See .agents/skills/writing-code-comments/SKILL.md for the house rules.

Files with the most added comment lines:

File Comment lines Added lines
nodejs/src/cdp/services/rate-limiter/rate-limiter.service.ts 7 26
nodejs/src/cdp/services/job-queue/job-queue-rate-limited-postgres-v2.ts 3 11
nodejs/src/cdp/services/rate-limiter/rate-limiter.service.test.ts 3 53

This check does not block merging. It updates on every push and clears when the share drops.

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team November 2, 2026 17:20

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 1f61448: 1 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

slotAt = tonumber(rawResv) + (requested / refillPerSecond) * 1000
else
local deficit = requested - available
local deficit = requested - claimable

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Apply the floor-aware shortfall when a reservation cursor exists

With an empty bucket refilling at 10/s, an unfloored one-token reservation sets resv 100ms ahead. A subsequent five-token claimOrReserve with reserveFloor: 20 takes the existing-cursor branch and reserves 600ms, when only six tokens exist, guaranteeing another denial even without competing claims. Bound both branches by the floor-aware refill deadline.

-- Left negative on purpose when the pool is below the floor: the grant clamps
-- at zero just below, and the reservation needs the true shortfall to place the
-- slot after the floor has refilled, not just after this caller's own tokens.
local claimable = available - reserveFloor

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Unvalidated reserveFloor lets a negative value overspend the SES bucket

The new reserveFloor is never validated or clamped, so a negative value inverts its meaning: claimable = available - reserveFloor becomes available + |floor|, and the grant is debited from the true pool (tokensAfter = available - granted, rate-limiter.service.ts:116), driving the pool negative. The module's own contract — concurrent callers "never overshoot the bucket capacity" (rate-limiter.service.ts:16-17) — is broken by a single caller, and the floored lane then sends above the configured SES rate until the pool refills, risking SES throttling and reputation damage. A floor greater than capacity is the mirror bug: claimable is always negative, so that lane is granted 0 forever and its emails never send.

How:

  1. An operator sets CDP_SES_RATE_LIMIT_TRANSACTIONAL_RESERVE=-10 (sign typo, or a mistaken belief that negative means a boost).
  2. overrideWithEnv (common/config/config.ts:48-49) parseInts it to -10 with no range check, and evalClaim forwards it verbatim (String(req.reserveFloor ?? 0), rate-limiter.service.ts:360).
  3. With 5 tokens available and 15 rows ready, claimable is 15, so the caller is granted 15 and the pool becomes -10 — 10 sends above budget.

Fix: Clamp the floor at the entry point — treat any value outside [0, capacity] as 0 (or fail startup) in evalClaim/claimBatchLimit, since the config loader does not validate it.


React with 👍 if useful or 👎 if not

@meikelmosby
meikelmosby added this pull request to stack #127616 November 2, 2026 17:22
@meikelmosby
meikelmosby removed this pull request from stack #127616 November 2, 2026 17:28
@meikelmosby
meikelmosby changed the base branch from master to feat/cyclotron-transactional-table-migration November 2, 2026 17:28
@meikelmosby
meikelmosby added this pull request to stack #127620 November 2, 2026 17:28
A token-bucket claim can now carry a reserve floor: tokens the caller may
not spend, so a caller that passes no floor always finds that much waiting.
One bucket still refills at one rate, so the total send rate is unchanged.

Inert on deploy — CDP_SES_RATE_LIMIT_TRANSACTIONAL_RESERVE defaults to 0
and nothing passes a floor yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@meikelmosby
meikelmosby force-pushed the feat/ses-reserve-floor branch from 1f61448 to 2196030 Compare November 4, 2026 13:59
@meikelmosby
meikelmosby marked this pull request as draft November 4, 2026 13:59
A floored caller denied behind another caller's slot took the next slot
after it, even when its own tokens could not refill above the floor by
then, which booked it a slot that was certain to be denied again. The slot
is now never earlier than the floor-aware refill time.

A rate-limited queue refuses to start with a negative floor, which would
let its lane overspend the bucket, or with one at capacity, which would
stop the lane from sending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@meikelmosby

Copy link
Copy Markdown
Collaborator Author

Closing while the transactional email work is replanned. The branch stays for reference.

@meikelmosby meikelmosby closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant