Repository navigation
feat(spotlight): Share Local processing across server and UI - #1690
MathurAditya724 wants to merge 16 commits into
Conversation
Extract the portable store, Node decoding pipeline, queries, and trace assembly into @sentry/telemetry. Use shared models in the Local receiver and browser workspace, and prepare independently versioned package releases. Co-Authored-By: Codex <noreply@openai.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Publish the shared package as @sentry/spotlight and update workspace, consumer, and release paths. Preserve retained trace views, incomplete span summaries, AI metadata, and thread-aware profile processing with regression coverage. Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Merge main while preserving Local schema preparation and the CLI Sentry import. Keep decoded history current when delivery IDs are reused, ignore invalid token usage, and pair each surfaced response with its finish reason. Co-Authored-By: Codex <noreply@openai.com>
Guard staged-byte cleanup by generation so cancelled parsed work cannot remove a newly admitted delivery with the same ID after reset. Co-Authored-By: Codex <noreply@openai.com>
Return 415 for unsupported envelope media types, preserve usable AI prompts and tool calls, and clip profile intervals to the available gap. Build the shared package for direct CLI test entry points even with cached schemas. Co-Authored-By: Codex <noreply@openai.com>
Remove legacy subscriptions, disabled logging, unused private helpers and types, and unnecessary async test callbacks. Preserve the public API and executable examples. Co-Authored-By: Codex <noreply@openai.com>
Let the pipeline account for admitted writes and reserve facade slots only for writes deferred behind retention controls. Immediate validation failures no longer cause false queue-full responses. Co-Authored-By: Codex <noreply@openai.com>
Share message normalization between the GenAI and Vercel handlers. Keep raw fallbacks replaceable until the first valid conversation is found. Co-Authored-By: Codex <noreply@openai.com>
Read standard tool-call attributes when the Vercel alias is absent or malformed. Preserve valid proprietary values, including explicit empty arrays. Co-Authored-By: Codex <noreply@openai.com>
Extend synthetic parent time ranges as later siblings are attached. Avoid overlapping profile gaps while preserving real children and trailing samples. Co-Authored-By: Codex <noreply@openai.com>
Classify known JSON wire items separately from processed event types. Keep report fields available to subscribers and validate malformed payloads. Co-Authored-By: Codex <noreply@openai.com>
Re-sort visible siblings after hiding missing-parent wrappers. Preserve captured parent identifiers and keep the SDK span tree immutable. Co-Authored-By: Codex <noreply@openai.com>
Read nested system instructions and prefer structured conversations and tool-call arrays over malformed aliases while retaining raw inspection values. Treat null model and provider attributes as absent so later spans can supply metadata. Co-authored-by: Codex <noreply@openai.com>
Reparent merged children and lifted frames to their surviving parent spans. Co-authored-by: Codex <noreply@openai.com>
Use a neutral graft parent for fillers between independent trace roots. Co-authored-by: Codex <noreply@openai.com>
There was a problem hiding this comment.
Two things before this lands, plus one question about who owns the package.
-
Ingest cost grows with retained data, and Local's receiver now waits on it. In
packages/spotlight/src/telemetry/index.ts, everypublish()callsstateSnapshot(), and that deep-copiestraces,sdks, andprofilesthroughreadonlyCopy. Thestablecache only covers envelopes, events, and logs. After the retention cap is reached,enforceRetentionalso replays the whole journal throughrebuild. I ran a local benchmark:createTelemetryPipelinefrom builtdist/sdk/node.js, with the receiver's options, ingesting sequential 21-span transaction envelopes. Results:- with
maxEnvelopes: 500(the Local default): 14 ms per envelope at first, 125 ms at 500 retained, then about 310 ms once eviction starts (about 3 envelopes/s) - with
maxEnvelopes: 50000: still climbing linearly, at 216 ms per envelope by envelope 800 - CPU profile: about 60% of the time is in
publish → stateSnapshot → readonlyCopyand about 28% in evictionrebuild
ingestPayloadintelemetry-buffer.tsnow awaits this before returning 204, andmaxPending: 64turns any backlog into 503s. SDKs drop those envelopes, whilemainstores the raw envelope in O(1). The browser store has the same snapshot cost on the UI thread. Fix idea: build snapshots lazily on read, or share unchanged trace and profile graphs between snapshots, and make eviction incremental instead of replaying everything. Please also add a throughput regression test, for example 1,000 transaction envelopes at default retention within a time budget, so this stays fixed. - with
-
Envelopes with only malformed spans are now rejected instead of kept for inspection.
processEnvelopereturnsinvalidwhen every span item failsnormalizeSpanItemand the envelope has no other items. Local maps that to a 400 and doesn't retain it. The docs change indocs/cli/local-control-plane.mdmakes this official, but onmainthose raw envelopes stay inspectable. For a local debugger, a misbehaving SDK's output is exactly what people want to look at. I'd keep the envelope with aninvalid_spandiagnostic, the way unsupported containers are kept, rather than turning away the SDK. -
Ownership question: the code is taken from getsentry/spotlight#1374, which is still an unreviewed draft, and this PR sets up
@sentry/spotlightto publish from Toolkit. Have the Spotlight maintainers agreed on the name and on moving the processing code here? If so, linking that decision in the description would help. Otherwise the two copies will drift until Spotlight switches to this package.
Sentry Local now uses
@sentry/spotlightfor shared envelope processing, trace assembly, queries, AI interpretation, profiles, and Web Vitals. The CLI receiver consumes the Node pipeline; the browser workspace consumes the portable store and processing helpers. HTTP/SSE transport, capture sessions, raw inspection, and rendering remain application concerns.The Apache-2.0 package is extracted from Spotlight #1374 with bounded ingestion, immutable snapshots, delivery receipts, live retention settings, and JSON exports. Local preserves replay cursors, attachment downloads, session isolation, and existing inspection models.
Accepted delivery bytes remain stable across rejected redispatches and are replaced or cleared when an evicted ID is accepted again. Reset invalidates older cleanup work, and published views follow the retained capture window. Continuous profile chunks refresh merged samples and derived spans while preserving real topology; sample intervals are clipped at the next real span rather than dropped.
AI queries preserve root identity, prioritize structured conversations over prompt fallbacks, pair finish reasons with the surfaced response, ignore malformed token/tool-call attributes, and reconcile aggregate/step usage without double counting. Incomplete traces expose the strongest observed token subtotal. Local returns HTTP 415 for unsupported envelope media types, and direct CLI test commands build the SDK even when generated schema files are cached. Regression coverage exercises these paths through public readers and real receiver inputs, including reordered captures and concurrent session changes.
Toolkit becomes the home for shared processing. The release workflow builds and checks an independently versioned npm artifact. After the first package release, Spotlight can replace its extraction with this dependency and remove its local implementation; this PR does not publish a package.
Emitted declaration checks, executable examples, SDK tests, Local tests, workspace typechecking, and lint pass. The integration command also passes without prebuilt SDK output. The full local workspace run hits two unchanged macOS failures: the Linux-musl installer assertion and a DerivedData scan timeout. Hosted CI covers the Linux workspace and Local native integrations.