Skip to content

perf(vcs): run VCS sync batches in the placed fetch handler - #1553

Merged
Makisuo merged 3 commits into
mainfrom
perf/vcs-sync-placed-consumer
Oct 19, 2026
Merged

Makisuo merged 3 commits into
mainfrom
perf/vcs-sync-placed-consumer

Conversation

@Makisuo

@Makisuo Makisuo commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The VCS sync queue consumer (maple-vcs-sync, VcsSyncQueue.processBatch) runs in whatever colo the queue delivers to, not beside the database and the ingest gateway. Every Postgres query and every CI telemetry POST pays a cross-continent round trip.

Production, 6 h ending 2026-10-19 09:40 UTC, deployment.environment = production:

Region processBatch total p50 / p95 Postgres spans (total, p50) Ingest POSTs (total, p50) Outbound colos (cf-ray)
EU 1,875 s (740 batches) 1,895 / 6,784 ms 713.6 s, 153 ms 690.6 s, 176 ms SEA, DFW, LAX, DEN, SJC
US 518 s (489 batches) 630 / 3,254 ms 199.5 s, 80 ms 88.7 s, 78 ms SJC, LAX, DEN, DFW

For comparison, placed invocations in the same window: maple-api EU (fetch, FRA) Postgres p50 8 to 40 ms; alerting (placed through its self binding) Postgres p50 6 ms EU / 8 ms US, ingest POST p50 16 ms EU / 10 ms US.

Cause

Worker placement (resolveWorkerPlacement, packages/infra/src/cloudflare/stage.ts:57) applies only to fetch invocations. apps/api/src/worker/consumers.ts:45 ran processBatch directly in the queue invocation, so it was never placed.

Fix

The same pattern alerting uses for its cron ticks (apps/alerting/src/tick-dispatch.ts):

  • The api Worker declares a self service binding, API_SELF: Cloudflare.Workers.Self (apps/api/src/worker.ts).
  • The consumer posts the batch (id, body, attempts per message) to POST /internal/vcs-sync/batch with a hop secret as the bearer (worker/vcs-sync-dispatch.ts). The secret is an alchemy Random, minted once, kept in state and bound to the api Worker alone, so no other Worker holding the internal service token can post jobs. The route is answered in makeFetch ahead of the route graph, like /health.
  • The placed handler checks the bearer, runs the batch over the same graph the consumer used, and answers one outcome per message: ack, or retry with an optional delay. The consumer applies them to the messages it holds.
  • processBatch is split so the per-message decision (settleMessage) is shared. Inline, each message is still acked or retried as soon as it settles. Retry, rate-limit delay and exhausted-attempt handling are unchanged.
  • The route serves only the self binding. The hop secret is the guard. As a second guard, a request that carries cf-connecting-ip gets a 404 before the bearer is checked: Cloudflare stamps that header on requests from the internet, while a binding call carries only the headers the consumer set. A named entrypoint is not used because placement does not apply to RPC methods or named entrypoints.
  • The route decodes each body against the real VcsSyncJob union, takes at most one queue batch (10 messages) and at most 10 x 128 KB plus a 16 KB envelope, read through a bounded stream. Any violation is a 400.
  • Fallbacks: no binding, no hop secret, or a 400/401/404 from the handler means nothing ran there, so the batch runs inline exactly as today. A transport failure or a 5xx retries every message with deliveries left. A message on its final delivery gets recordExhaustedFailure and an ack instead (there is no dead-letter queue), the same terminal handling as a failed final attempt inline; it is not re-run, since the placed run may have finished it. A message missing from the answer is retried. An unknown outcome is otherwise never acked.
  • CI span ids are already deterministic, so a redelivered message rewrites the same spans. Telemetry content and ci-traces/logs.ts are unchanged.
  • The route's maple-api server span is suppressed (TracerDisabledWhen), so batch time stays on maple-vcs-sync and does not skew maple-api latency. The batch span now carries maple.vcs.dispatch (placed / inline) and maple.placement (cf-placement header). The consumer records a VcsSyncQueue.dispatch span.
  • The self-binding helpers (selfFetcher, workerSecretFor, internalTokenFor) move from alerting into @maple/backend/platform/self-binding and both Workers use them.

Tests (apps/api/src/worker/vcs-sync.test.ts) drive the real consumer hand-off through a binding that answers with the real route handler and the real processPlacedBatch: per-message ack / delayed retry / exhausted ack / retry; inline fallback for a missing binding and a 401; retry on a failed placed run, and terminal record plus ack for a final delivery in it; retry for a message with no outcome; and the inline processBatch path. The route itself is tested for a full batch, a 401 for the internal service token, a 404 to a request with cf-connecting-ip even with the bearer, the inline fallback when a binding call is refused that way, and 400s for 11 messages, a body that is not a job, and an oversize body with and without content-length.

Before / After

Metric (per 6 h) Before (prod) After (expected) Change
EU outbound colo SEA / DFW / LAX / DEN FRA placed
US outbound colo SJC / LAX / DEN IAD placed
EU Postgres p50 153 ms under 15 ms about -90%
US Postgres p50 80 ms under 15 ms about -85%
EU ingest POST p50 176 ms under 30 ms about -85%
EU processBatch total 1,875 s about 600 s about -1,200 s
US processBatch total 518 s about 300 s about -200 s

Method: before numbers are production telemetry via maple query --remote --since 6h, window ending 2026-10-19 09:40 UTC. After numbers are expectations from the placed alerting Worker's latencies in the same regions, not measurements. A PR preview is US only and has no GitHub installations, so the VCS queue gets no traffic there and this path cannot be exercised end to end. The preview only confirms that the route is deployed and refuses an unauthenticated call.

Preview result (PR 1553, 2026-10-19 ~09:58 UTC): the api Worker deployed at this commit (x-maple-revision: d91f47b8a), GET /health 200, POST /internal/vcs-sync/batch with no bearer 401 and with a wrong bearer 401. The preview job as a whole failed later on an AWS VpcLimitExceeded for the ingest and electric stacks, which this PR does not touch, so the seed and smoke-test step did not run.

Merge order

Merge this before #1551. #1551 then rebases to give both VCS paths (queue consumer and the batch route) the isolate token and ingest-key caches; its body describes the resolution.

How to verify after deploy

Outbound colo by region (expect FRA for EU, IAD for US):

SELECT ResourceAttributes['maple.region'] AS region,
       splitByChar('-', SpanAttributes['http.response.header.cf-ray'])[2] AS colo,
       count() AS n
FROM traces
WHERE $__orgFilter AND $__timeFilter(Timestamp)
  AND ServiceName = 'maple-vcs-sync' AND SpanKind = 'Client'
  AND SpanAttributes['http.response.header.cf-ray'] != ''
  AND ResourceAttributes['deployment.environment'] = 'production'
GROUP BY region, colo ORDER BY n DESC

Postgres and ingest latency (expect SELECT vcs_installations p50 under 15 ms, ingest POST p50 under 30 ms):

SELECT ResourceAttributes['maple.region'] AS region, SpanName,
       round(quantile(0.5)(Duration) / 1e6, 1) AS p50_ms, count() AS n
FROM traces
WHERE $__orgFilter AND $__timeFilter(Timestamp)
  AND ServiceName = 'maple-vcs-sync'
  AND (SpanName = 'SELECT vcs_installations' OR (SpanKind = 'Client' AND SpanAttributes['peer.service'] = 'ingest'))
  AND ResourceAttributes['deployment.environment'] = 'production'
GROUP BY region, SpanName

Batch time and how batches ran (compare with 1,875 s EU / 518 s US per 6 h):

SELECT ResourceAttributes['maple.region'] AS region,
       SpanAttributes['maple.vcs.dispatch'] AS dispatch,
       count() AS batches, round(sum(Duration) / 1e9, 1) AS total_s
FROM traces
WHERE $__orgFilter AND $__timeFilter(Timestamp)
  AND ServiceName = 'maple-vcs-sync' AND SpanName = 'VcsSyncQueue.processBatch'
  AND ResourceAttributes['deployment.environment'] = 'production'
GROUP BY region, dispatch

Any inline rows after deploy mean the hop was refused; VcsSyncQueue.dispatch spans carry maple.vcs.dispatch.reason. A steady refused-404 reason would mean binding calls arrive with cf-connecting-ip, which the docs do not suggest; the batches still run, inline.

Summary by CodeRabbit

  • New Features
    • VCS sync batches can now be processed through a dedicated batch route, with results applied to individual messages.
  • Bug Fixes
    • VCS sync processing now falls back to inline handling when batch dispatch is unavailable.
    • Failed or incomplete batch runs retry messages that lack a result, while terminal failures are recorded before acknowledgment.
    • Batch requests are checked for authorization and size limits, and invalid or oversized requests are rejected.

@Makisuo Makisuo added the preview Deploy a full PR preview stack (Cloudflare + AWS ingest); removing it tears it down label Oct 19, 2026
@maple-review-bot

maple-review-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 7/10 · likely safe to merge
The hand-off is contained and read end to end, but the failed-hop branch retries messages that exhausted their attempts, and no test covers it.
quality 90/100 · 1 warning · tests partial · risk medium · 3/3 new units observable

The consumer hands each VCS sync batch to the Worker's own placed fetch handler, shortening its Postgres and ingest hops. Safe to merge, but a hop that fails re-retries messages that already exhausted their attempts. The open CiTraceIngestError 403s from the CI ingest POST are untouched, so unfixed.

  • handOffVcsSyncBatch posts the batch to /internal/vcs-sync/batch over API_SELF
  • serveVcsSyncBatch runs it in the placed handler and answers per-message outcomes
  • A failed hop retries every message; a refusal falls back to an inline run
  • settleMessage holds the shared per-message ack/retry/exhaust decision

Before merge

  • Manual · Confirm INTERNAL_SERVICE_TOKEN is set on the prd maple-api Worker: without it every batch falls back inline with no-token.

Findings

🟠 Warning · F1 · Retry-all after a failed placed hop drops the exhausted message's terminal status

correctness · apps/api/src/worker/vcs-sync.ts:45-47

When the hop fails after the batch may have started, every message goes through applyOutcome(message, undefined), which is an unconditional message.retry(). A message at attempts > VCS_SYNC_MAX_RETRIES in that batch is therefore redelivered instead of being acked after recordExhaustedFailure, so its terminal status is never written and the queue drops it (vcs-sync has no DLQ, resources/queues.ts:6) — the stuck backfill the inline path's isFinalAttempt branch (vcs-sync-runtime.ts:122) exists to prevent. The new test pins retry-all as correct, so nothing catches it.

Retry only messages with `attempts <= VCS_SYNC_MAX_RETRIES` and settle the exhausted ones through a path that can reach `VcsSyncService.recordExhaustedFailure` (e.g. return `true` so the inline run handles the batch), and cover a final-attempt message inside a failed hop in `vcs-sync.test.ts`.
🤖 Prompt to fix this finding with an AI agent
Findings from an automated review of commit d91f47b8a652617b2aed194e869cdc76ea138b30. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F1 · Warning · correctness · apps/api/src/worker/vcs-sync.ts:45-47
Retry-all after a failed placed hop drops the exhausted message's terminal status
When the hop fails after the batch may have started, every message goes through `applyOutcome(message, undefined)`, which is an unconditional `message.retry()`. A message at `attempts > VCS_SYNC_MAX_RETRIES` in that batch is therefore redelivered instead of being `ack`ed after `recordExhaustedFailure`, so its terminal status is never written and the queue drops it (`vcs-sync` has no DLQ, `resources/queues.ts:6`) — the stuck backfill the inline path's `isFinalAttempt` branch (`vcs-sync-runtime.ts:122`) exists to prevent. The new test pins retry-all as correct, so nothing catches it.
Suggested fix: Retry only messages with `attempts <= VCS_SYNC_MAX_RETRIES` and settle the exhausted ones through a path that can reach `VcsSyncService.recordExhaustedFailure` (e.g. return `true` so the inline run handles the batch), and cover a final-attempt message inside a failed hop in `vcs-sync.test.ts`.

Production impact

Open errors in the changed files
Issue Service Occurrences File
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 819 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 733 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 341 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 307 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 295 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 232 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 227 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 207 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 201 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 157 apps/api/src/vcs-sync-runtime.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
apps/api/src/vcs-sync-runtime.ts 24k VcsSyncQueue.processMessage 19k/day, 0.0% err
VcsSyncQueue.processBatch 5.2k/day, 0.0% err
apps/api/src/worker/http.ts 934 POST 779/day, 4.2% err
POST 155/day, 0.5% err
apps/api/src/http/api-observability.ts 6 GET /health 6/day, 0.0% err
apps/api/src/worker.ts 6 GET /health 6/day, 0.0% err
Telemetry this change adds and removes (3)
  • ➕ attribute maple.vcs.dispatch · apps/api/src/worker/vcs-sync.ts:33
  • ➕ attribute maple.vcs.dispatch.reason · apps/api/src/worker/vcs-sync.ts:39
  • ➕ span name VcsSyncQueue.dispatch · apps/api/src/worker/vcs-sync.ts:52
Ingest cost of new log lines
  • apps/api/src/vcs-sync-runtime.ts:136: If this logs once per call, about 194 MB/month: the file's operations run 24k times a day.
What was checked
  • A batch cannot run twice: no-binding/no-token/400/401/404 fall back inline (vcs-sync.ts:35)
  • The new route is bearer-gated by isValidInternalBearer, the helper /internal/tick uses (vcs-sync-dispatch.ts:163)
  • One maple-vcs-sync SDK per isolate: the re-export keeps module identity (vcs-sync-runtime.ts:42)
Observability coverage: 3 of 3 changes observable
Change Kind Observable Evidence
POST /internal/vcs-sync/batch inbound route (self binding) server yes Batch runs under its own VcsSyncQueue.processBatch consumer span with maple.vcs.dispatch/maple.placement (vcs-sync-runtime.ts:175, :205); the maple-api server span is deliberately suppressed (api-observability.ts:36)
Outbound self-binding batch call client yes HttpClient.execute under a producer VcsSyncQueue.dispatch span (vcs-sync.ts:52, vcs-sync-dispatch.ts:107)
Dispatch and placed-batch failure paths error yes Outcome annotated on the span and the cause logged as Error (vcs-sync.ts:36, :42, :74)

d91f47b · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

Devin Review

Comment thread apps/api/src/worker/vcs-sync.ts Outdated
Comment thread apps/api/src/worker/vcs-sync-dispatch.ts
Comment thread apps/api/src/worker/vcs-sync-dispatch.ts Outdated

@maple-review-bot maple-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 inline note from Maple's review. The score and summary are in the review comment above.

Comment thread apps/api/src/worker/vcs-sync.ts Outdated
@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e394282d-a3ac-484d-a78a-46bec20a019d
📥 Commits

Reviewing files that changed from the base of the PR and between 0214234 and 15a8812.

📒 Files selected for processing: 11
  • apps/alerting/src/tick-dispatch.ts
  • apps/api/src/http/api-observability.ts
  • apps/api/src/vcs-sync-runtime.ts
  • apps/api/src/worker.ts
  • apps/api/src/worker/consumers.ts
  • apps/api/src/worker/http.ts
  • apps/api/src/worker/vcs-sync-dispatch.ts
  • apps/api/src/worker/vcs-sync.test.ts
  • apps/api/src/worker/vcs-sync.ts
  • packages/backend/src/platform/self-binding.ts
  • packages/domain/package.json

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

📝 Walkthrough

The API Worker adds a self-binding route for VCS sync batches. Queue consumers can dispatch batches to that route and apply returned outcomes, or process messages inline when dispatch is unavailable. The change also adds shared Worker binding utilities and updates alerting to use them.

Changes: VCS sync batch dispatch

Layer / File(s) Summary
Batch protocol and self-binding utilities
apps/api/src/worker/vcs-sync-dispatch.ts, packages/backend/src/platform/self-binding.ts, apps/alerting/src/tick-dispatch.ts, packages/domain/package.json
Defines batch request and outcome schemas, dispatch limits, and error types. Adds shared self-binding and secret helpers. Alerting uses the shared helpers, and the domain package exports ./http/vcs.
Batch dispatch and message settlement
apps/api/src/worker/vcs-sync-dispatch.ts, apps/api/src/vcs-sync-runtime.ts, apps/api/src/worker/vcs-sync.ts
Dispatches batches through the Worker self binding and validates request bodies and authorization. Runtime settlement returns ack or retry outcomes, records exhausted messages, and processes placed batches. The consumer handles placed, inline, and exhausted dispatch outcomes.
Worker wiring and dispatch validation
apps/api/src/worker.ts, apps/api/src/worker/consumers.ts, apps/api/src/worker/http.ts, apps/api/src/http/api-observability.ts, apps/api/src/worker/vcs-sync.test.ts
Passes the Worker environment to queue consumers, exposes the batch route through the HTTP handler, and excludes that route from automatic HTTP tracing. Tests cover dispatch outcomes, authorization, and request limits.

Priority: ⬇️ Low

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant QueueConsumer
  participant dispatchVcsSyncBatch
  participant WorkerSelfBinding
  participant vcsSyncBatchRoute
  participant processPlacedBatch
  participant applyOutcomes
  QueueConsumer->>dispatchVcsSyncBatch: send placed messages
  dispatchVcsSyncBatch->>WorkerSelfBinding: POST batch with worker bearer
  WorkerSelfBinding->>vcsSyncBatchRoute: forward batch request
  vcsSyncBatchRoute->>processPlacedBatch: settle placed messages
  processPlacedBatch-->>vcsSyncBatchRoute: return message outcomes
  vcsSyncBatchRoute-->>dispatchVcsSyncBatch: return outcomes
  dispatchVcsSyncBatch->>applyOutcomes: settle queue messages
Loading

Suggested reviewers: jeremyfunk

Merge Risk: ⚪ Minimal · up to 15a88

No actionable merge-blocking issue remains. The VCS sync batch handoff is ready to merge after normal checks.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: moving VCS sync batch execution to the placed fetch handler.
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 1…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 10 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

🍁 Maple PR preview

Note

Preview resources were removed when the preview label came off.

Final commit d91f47b · View workflow run

@Makisuo Makisuo removed the preview Deploy a full PR preview stack (Cloudflare + AWS ingest); removing it tears it down label Oct 19, 2026
@Makisuo
Makisuo deployed to pr-preview October 19, 2026 09:56 — with GitHub Actions Active
@maple-review-bot

maple-review-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 8/10 · likely safe to merge
The failed-hop fix reads correct at the head and is tested; placement hinges on cf-connecting-ip being absent on service bindings, and the fallbacks are safe if it is not.
quality 100/100 · no findings · tests covered · risk medium · 2/2 new units observable

Runs VCS sync batches through a placed fetch handler over a self binding, fixing the failed-hop terminal status and falling back to inline. CiTraceIngestError handling is untouched, so those open ingest issues are not addressed by this diff.

  • handOffVcsSyncBatch posts the batch to POST /internal/vcs-sync/batch over API_SELF
  • A failed hop retries only non-final messages; final ones record the exhausted failure and ack
  • A missing binding, token or 4xx answer falls back to the inline processBatch
  • The batch span gains maple.vcs.dispatch and maple.placement

Before merge

  • Manual · Confirm INTERNAL_SERVICE_TOKEN is set for the api Worker in prd; without it every batch silently falls back inline

Fixed since the last review

  • ✅ F1 · Retry-all after a failed placed hop drops the exhausted message's terminal status

Production impact

Open errors in the changed files
Issue Service Occurrences File
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 829 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 742 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 348 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 313 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 300 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 234 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 229 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 207 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 203 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 158 apps/api/src/vcs-sync-runtime.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
apps/api/src/vcs-sync-runtime.ts 24k VcsSyncQueue.processMessage 19k/day, 0.0% err
VcsSyncQueue.processBatch 5.2k/day, 0.0% err
VcsScheduledSync.tick 7/day, 0.0% err
apps/api/src/worker/http.ts 932 POST 777/day, 4.2% err
POST 155/day, 0.5% err
apps/api/src/http/api-observability.ts 6 GET /health 6/day, 0.0% err
apps/api/src/worker.ts 6 GET /health 6/day, 0.0% err
Telemetry this change adds and removes (4)
  • ➕ span name VcsSyncQueue.recordExhausted · apps/api/src/vcs-sync-runtime.ts:228
  • ➕ attribute maple.vcs.dispatch · apps/api/src/worker/vcs-sync.ts:42
  • ➕ attribute maple.vcs.dispatch.reason · apps/api/src/worker/vcs-sync.ts:48
  • ➕ span name VcsSyncQueue.dispatch · apps/api/src/worker/vcs-sync.ts:63
Ingest cost of new log lines
  • apps/api/src/vcs-sync-runtime.ts:137: If this logs once per call, about 195 MB/month: the file's operations run 24k times a day.
  • apps/api/src/vcs-sync-runtime.ts:220: If this logs once per call, about 195 MB/month: the file's operations run 24k times a day.
What was checked
  • Failed-hop path retries only non-final deliveries and records the rest, pinned by the final-delivery test (vcs-sync.test.ts:148); F1's defect is gone
  • Batch body is bounded by VCS_SYNC_BATCH_MAX_BYTES before buffering and the array capped at VCS_SYNC_BATCH_SIZE (vcs-sync-dispatch.ts)
  • The eager @maple/domain/http/vcs import costs a few ms against the ~80 ms of a 1 s startup budget (docs/infra-history.md)
Observability coverage: 2 of 2 changes observable
Change Kind Observable Evidence
POST /internal/vcs-sync/batch placed batch handler server yes route span suppressed, the batch runs under the VcsSyncQueue.processBatch consumer span carrying maple.vcs.dispatch/maple.placement (vcs-sync-runtime.ts)
self-binding hop to the placed handler client yes VcsSyncQueue.dispatch Producer span with messaging.batch.message_count (vcs-sync.ts:63)

c6d447b · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

Worker placement applies to fetch invocations only, so the VCS sync
queue consumer ran in whatever colo the queue picked, far from Postgres
and the ingest gateway. The consumer now posts its batch to the api
Worker's own fetch handler over a self service binding (API_SELF), the
way alerting hands off its cron ticks. The placed handler runs the batch
and answers one outcome per message (ack, or retry with an optional
delay), and the consumer applies them.

Without the binding or token, or when the handler refuses the request
(400/401/404), the batch runs inline exactly as before. A transport
failure or a 5xx retries every message, and a message missing from the
answer is retried, so an unknown outcome is never acked.

The self-binding helpers move from alerting into
@maple/backend/platform/self-binding so both Workers share them. The
route's server span is suppressed so the batch time stays on
maple-vcs-sync, not maple-api.
…final deliveries of a failed hop

The api Worker has public routes, so the batch route refuses any request that
carries cf-connecting-ip. Cloudflare stamps that header on every request from
the internet and a client cannot remove it; a service binding call carries only
the headers the consumer set. The refusal is a 404, which the consumer treats
as "did not run" and so falls back inline. A named entrypoint was not an
option: placement does not apply to RPC methods or named entrypoints.

The route decodes messages against the real VcsSyncJob union, caps a request
at one queue batch (10 messages) and at 10 x 128 KB plus an envelope, read
with a bounded stream so an oversize body is refused before it is buffered.

When the hop fails with an unknown outcome, messages on their final delivery
are no longer retried into nothing. Their terminal status is recorded through
VcsSyncService.recordExhaustedFailure and they are acked, as a failed final
attempt inline would be. Every other message is still retried, never acked.
…r holds

The batch route accepted the internal service token, which other Workers also
hold, so any of them could post jobs to it. The hop now carries a random
secret alchemy mints once and binds to the api Worker alone. The edge check on
cf-connecting-ip stays as a second guard. A consumer without the secret runs
its batch inline, as before.
@Makisuo
Makisuo force-pushed the perf/vcs-sync-placed-consumer branch from c6d447b to 15a8812 Compare October 19, 2026 11:45
@maple-review-bot

maple-review-bot Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 7/10 · likely safe to merge
Hop is well tested and its fallbacks are safe; unsettled only whether a placed batch keeps its CPU budget, since the api Worker sets no limits where alerting sets cpuMs.
quality 100/100 · no findings · tests covered · risk medium · 3/3 new units observable

VCS sync batches now hop to the api Worker's own placed fetch handler over a self binding, authenticated with a per-Worker secret, with inline fallback. The four changed-since files hold up. The 10 open CiTraceIngestError issues in apps/api/src/vcs-sync-runtime.ts stay open: the diff changes where a batch runs, not the CI-ingest call or its error handling.

  • handOffVcsSyncBatch posts a batch to the placed fetch handler over the self binding
  • POST /internal/vcs-sync/batch takes only VCS_SYNC_HOP_SECRET, 404s public callers
  • Inline fallback when the binding, token, or route is absent or refuses
  • selfFetcher/workerSecretFor/internalTokenFor shared from @maple/backend/platform/self-binding

Before merge

Production impact

Open errors in the changed files
Issue Service Occurrences File
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 854 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 767 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 363 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 318 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 305 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 244 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 237 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 207 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 204 apps/api/src/vcs-sync-runtime.ts
@maple/backend/integrations/CiTraceIngestError maple-vcs-sync 159 apps/api/src/vcs-sync-runtime.ts

After this merges, Maple checks whether they stop.

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
apps/api/src/vcs-sync-runtime.ts 24k VcsSyncQueue.processMessage 19k/day, 0.0% err
VcsSyncQueue.processBatch 5.3k/day, 0.0% err
VcsScheduledSync.tick 7/day, 0.0% err
apps/api/src/worker/http.ts 930 POST 775/day, 4.3% err
POST 155/day, 0.5% err
apps/api/src/http/api-observability.ts 6 GET /health 6/day, 0.0% err
apps/api/src/worker.ts 6 GET /health 6/day, 0.0% err
Telemetry this change adds and removes (4)
  • ➕ span name VcsSyncQueue.recordExhausted · apps/api/src/vcs-sync-runtime.ts:228
  • ➕ attribute maple.vcs.dispatch · apps/api/src/worker/vcs-sync.ts:42
  • ➕ attribute maple.vcs.dispatch.reason · apps/api/src/worker/vcs-sync.ts:48
  • ➕ span name VcsSyncQueue.dispatch · apps/api/src/worker/vcs-sync.ts:63
Ingest cost of new log lines
  • apps/api/src/vcs-sync-runtime.ts:137: If this logs once per call, about 197 MB/month: the file's operations run 24k times a day.
  • apps/api/src/vcs-sync-runtime.ts:220: If this logs once per call, about 197 MB/month: the file's operations run 24k times a day.
What was checked
  • Route auth: per-Worker secret plus the cf-connecting-ip 404 gate (vcs-sync-dispatch.ts:204), tested at vcs-sync.test.ts:225
  • Fallbacks: missing binding/token and 400/401/404 run inline; a failed hop retries every non-final message (vcs-sync.ts:44-61)
  • Final-delivery off-by-one: attempts > VCS_SYNC_MAX_RETRIES matches maxRetries: 3 and batchSize: 10 in consumers.ts:21-26
Observability coverage: 3 of 3 changes observable
Change Kind Observable Evidence
POST /internal/vcs-sync/batch (self binding only) server route yes answered in makeFetch; placed batch reports as maple-vcs-sync with maple.vcs.dispatch / maple.placement, api server span suppressed
Consumer to placed handler hop producer span yes VcsSyncQueue.dispatch producer span, vcs-sync.ts:63-66
VcsSyncQueue.processBatch placed/inline consumer span yes batchSpan carries maple.vcs.dispatch, vcs-sync-runtime.ts:176

15a8812 · Updated on every push. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo merged commit afde00e into main Oct 19, 2026
28 checks passed
@Makisuo
Makisuo deleted the perf/vcs-sync-placed-consumer branch October 19, 2026 21:06
@maple-review-bot

Copy link
Copy Markdown

Merged with 1 step from "Before merge" still open. If it is done, tick it on the review comment; if not, now is the time.

  • Secret · Add secret VCS_SYNC_HOP_SECRET to every environment

@maple-review-bot

Copy link
Copy Markdown

Maple: after this shipped

⚠️ Something changed after this shipped. Details below.

Deployed to scraper (production) at 2026-10-19 21:11 UTC as afde00e. Compared the 60 minutes before and after.

Errors in the changed files:

  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
  • ➖ @maple/backend/integrations/CiTraceIngestError: 0/h → 0/h
Operation Calls/h Errors p95
VcsSyncQueue.processMessage 534 → 1.0k 0.0% → 0.0% 3435 → 2784 ms
VcsSyncQueue.processBatch 243 → 377 0.0% → 0.0% 7278 → 7041 ms
⚠️ POST 53 → 96 0.0% → 0.0% 358 → 547 ms
POST 4 → 7 0.0% → 0.0% 861 → 521 ms
VcsScheduledSync.tick 0 → 0 0.0% → 0.0% 0 → 0 ms
GET /health 1 → 0 0.0% → 0.0% 2 → 0 ms

Maple looks once, an hour after the first deploy that carries the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant