Repository navigation
Conversation
Self-host checks for due scheduled runs and for queued provisioning jobs every second, and traced every check: a schedule.dispatch span with its storage and SQL spans, plus a root sql.execute span. An instance with no traffic exported about 800 spans a minute into the bundled Motel store, which reaches its 1 GiB bound within hours and then evicts on every export, writing to disk constantly. A check that finds nothing now records nothing. Each scheduled operation a check finds runs in its own schedule.dispatch trace, so the background work under it keeps its dispatcher root, and each provisioning job is traced as before. Local runs the same scheduler and stops tracing its idle checks too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Self-host and local look for due event deliveries every two seconds, and traced every pass: an events.dispatch span with its delivery, subscription and retention statements, about 15 spans each. On an instance with no traffic that is about 450 spans a minute in the bundled Motel store, the largest idle source since the event worker arrived. A pass that finds nothing to deliver now records nothing. Each delivery attempt and each subscription a pass expires is still traced, in its own trace. Cloud's coordinator calls the same delivery code with tracing on and is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2194.
An idle self-host instance traces three polling loops: the scheduler's check for due runs (every second), self-host's check for queued provisioning jobs (every second), and the event-delivery check (every two seconds, about 15 spans a pass). On 2.0.0-beta.13 an idle instance stores 10,724 spans in 10 minutes, about 96% of them from these polls, so the bundled Motel store reaches its 1 GiB cap within hours. From then on Motel evicts on every export, which is the CPU and disk churn in #2194 (on our Unraid host, running beta.8, about 530 GB of writes a day).
Fix
Two commits, one per worker, each with its own changeset:
SELECTrun withEffect.withTracerEnabled(false)(already used inpackages/telemetry/src/isolate.ts). Each operation the scheduler finds still runs traced, in its ownschedule.dispatchspan, sooauth-reconnect-schedulesstill sees dispatcher roots. Provisioning jobs are traced as before. Sampling can't be used instead, because the telemetry layer forcessampled: true.packages/sdk/src/implementation/event-worker.ts, added after [bug] v2 self-host: idle container burns ~13% CPU and writes ~190 GiB/day once bundled Motel hits its 1 GiB cap #2194 was filed). The pass runs untraced;deliverturns tracing back on for each delivery attempt and each subscription it expires. Cloud's coordinator already runs with tracing on, so Cloud is unchanged.Not in this PR
apps.repository.recoverstill trace at idle (about 100 spans a minute). Existing scenarios assert their dispatch roots, so I left them.fix/size-bound-hysteresis(373a47e) addresses that.Tested
bun run formatchanges nothing. The rest ofbun run checkpasses on the branch as rebased ontov241d3982 (format check, lint, typecheck, protocol and contract checks), with no diagnostics in the changed lines.patches:checkfails its own fixture self-test under git 2.43 on my build machine and passes under git 2.54; that is independent of this change.oauth-reconnect-schedulesanddeploy-setup(member setup wakes profile setup) pass on self-host; the first fails if the per-operation dispatch span is removed.mcp-eventson self-host andlocal-mcp-eventson local pass.🤖 Generated with Claude Code