Skip to content

feat(spotlight): Share Local processing across server and UI - #1690

Draft
MathurAditya724 wants to merge 16 commits into
mainfrom
codex/shared-telemetry-package
Draft

MathurAditya724 wants to merge 16 commits into
mainfrom
codex/shared-telemetry-package

Conversation

@MathurAditya724

@MathurAditya724 MathurAditya724 commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Sentry Local now uses @sentry/spotlight for shared envelope processing, trace assembly, queries, AI interpretation, profiles, and Web Vitals. The CLI receiver consumes the Node pipeline; the browser workspace consumes the portable store and processing helpers. HTTP/SSE transport, capture sessions, raw inspection, and rendering remain application concerns.

import { createTelemetryStore } from "@sentry/spotlight";
import { createTelemetryPipeline } from "@sentry/spotlight/node";

The Apache-2.0 package is extracted from Spotlight #1374 with bounded ingestion, immutable snapshots, delivery receipts, live retention settings, and JSON exports. Local preserves replay cursors, attachment downloads, session isolation, and existing inspection models.

Accepted delivery bytes remain stable across rejected redispatches and are replaced or cleared when an evicted ID is accepted again. Reset invalidates older cleanup work, and published views follow the retained capture window. Continuous profile chunks refresh merged samples and derived spans while preserving real topology; sample intervals are clipped at the next real span rather than dropped.

AI queries preserve root identity, prioritize structured conversations over prompt fallbacks, pair finish reasons with the surfaced response, ignore malformed token/tool-call attributes, and reconcile aggregate/step usage without double counting. Incomplete traces expose the strongest observed token subtotal. Local returns HTTP 415 for unsupported envelope media types, and direct CLI test commands build the SDK even when generated schema files are cached. Regression coverage exercises these paths through public readers and real receiver inputs, including reordered captures and concurrent session changes.

Toolkit becomes the home for shared processing. The release workflow builds and checks an independently versioned npm artifact. After the first package release, Spotlight can replace its extraction with this dependency and remove its local implementation; this PR does not publish a package.

Emitted declaration checks, executable examples, SDK tests, Local tests, workspace typechecking, and lint pass. The integration command also passes without prebuilt SDK output. The full local workspace run hits two unchanged macOS failures: the Linux-musl installer assertion and a DerivedData scan timeout. Hosted CI covers the Linux workspace and Local native integrations.

Extract the portable store, Node decoding pipeline, queries, and trace assembly into @sentry/telemetry. Use shared models in the Local receiver and browser workspace, and prepare independently versioned package releases.

Co-Authored-By: Codex <noreply@openai.com>
@vercel

vercel Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
sentry-local Ready Ready Preview Oct 29, 2026 2:59pm UTC

Request Review

Comment thread apps/local/src/lib/performance.ts Resolved
Comment thread apps/local/src/lib/telemetry-store.ts Resolved
Comment thread apps/local/src/lib/trace-model.ts Resolved
Publish the shared package as @sentry/spotlight and update workspace, consumer, and release paths. Preserve retained trace views, incomplete span summaries, AI metadata, and thread-aware profile processing with regression coverage.

Co-Authored-By: Codex <noreply@openai.com>
@MathurAditya724 MathurAditya724 changed the title feat(telemetry): Share Local processing across server and UI feat(spotlight): Share Local processing across server and UI Oct 28, 2026
Comment thread packages/spotlight/src/sdk/node.ts Resolved
Co-Authored-By: Codex <noreply@openai.com>
Comment thread packages/spotlight/src/sdk/node.ts Resolved
Comment thread apps/local/package.json Resolved
Merge main while preserving Local schema preparation and the CLI Sentry import.
Keep decoded history current when delivery IDs are reused, ignore invalid token
usage, and pair each surfaced response with its finish reason.

Co-Authored-By: Codex <noreply@openai.com>
Guard staged-byte cleanup by generation so cancelled parsed work cannot
remove a newly admitted delivery with the same ID after reset.

Co-Authored-By: Codex <noreply@openai.com>
Comment thread packages/cli/package.json Resolved
Return 415 for unsupported envelope media types, preserve usable AI prompts
and tool calls, and clip profile intervals to the available gap. Build the
shared package for direct CLI test entry points even with cached schemas.

Co-Authored-By: Codex <noreply@openai.com>
Remove legacy subscriptions, disabled logging, unused private helpers and types, and unnecessary async test callbacks. Preserve the public API and executable examples.

Co-Authored-By: Codex <noreply@openai.com>
Let the pipeline account for admitted writes and reserve facade slots only for writes deferred behind retention controls. Immediate validation failures no longer cause false queue-full responses.

Co-Authored-By: Codex <noreply@openai.com>
Share message normalization between the GenAI and Vercel handlers.
Keep raw fallbacks replaceable until the first valid conversation is found.

Co-Authored-By: Codex <noreply@openai.com>
Read standard tool-call attributes when the Vercel alias is absent or malformed.
Preserve valid proprietary values, including explicit empty arrays.

Co-Authored-By: Codex <noreply@openai.com>
MathurAditya724 and others added 2 commits October 29, 2026 17:21
Extend synthetic parent time ranges as later siblings are attached.
Avoid overlapping profile gaps while preserving real children and trailing samples.

Co-Authored-By: Codex <noreply@openai.com>
Classify known JSON wire items separately from processed event types.
Keep report fields available to subscribers and validate malformed payloads.

Co-Authored-By: Codex <noreply@openai.com>
Comment thread apps/local/src/lib/trace-model.ts Resolved
Re-sort visible siblings after hiding missing-parent wrappers.
Preserve captured parent identifiers and keep the SDK span tree immutable.

Co-Authored-By: Codex <noreply@openai.com>
MathurAditya724 and others added 2 commits October 29, 2026 19:35
Read nested system instructions and prefer structured conversations and
tool-call arrays over malformed aliases while retaining raw inspection values.
Treat null model and provider attributes as absent so later spans can supply
metadata.

Co-authored-by: Codex <noreply@openai.com>
Reparent merged children and lifted frames to their surviving parent spans.

Co-authored-by: Codex <noreply@openai.com>
Use a neutral graft parent for fillers between independent trace roots.

Co-authored-by: Codex <noreply@openai.com>

@sentry-junior sentry-junior Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things before this lands, plus one question about who owns the package.

  • Ingest cost grows with retained data, and Local's receiver now waits on it. In packages/spotlight/src/telemetry/index.ts, every publish() calls stateSnapshot(), and that deep-copies traces, sdks, and profiles through readonlyCopy. The stable cache only covers envelopes, events, and logs. After the retention cap is reached, enforceRetention also replays the whole journal through rebuild. I ran a local benchmark: createTelemetryPipeline from built dist/sdk/node.js, with the receiver's options, ingesting sequential 21-span transaction envelopes. Results:

    • with maxEnvelopes: 500 (the Local default): 14 ms per envelope at first, 125 ms at 500 retained, then about 310 ms once eviction starts (about 3 envelopes/s)
    • with maxEnvelopes: 50000: still climbing linearly, at 216 ms per envelope by envelope 800
    • CPU profile: about 60% of the time is in publish → stateSnapshot → readonlyCopy and about 28% in eviction rebuild

    ingestPayload in telemetry-buffer.ts now awaits this before returning 204, and maxPending: 64 turns any backlog into 503s. SDKs drop those envelopes, while main stores the raw envelope in O(1). The browser store has the same snapshot cost on the UI thread. Fix idea: build snapshots lazily on read, or share unchanged trace and profile graphs between snapshots, and make eviction incremental instead of replaying everything. Please also add a throughput regression test, for example 1,000 transaction envelopes at default retention within a time budget, so this stays fixed.

  • Envelopes with only malformed spans are now rejected instead of kept for inspection. processEnvelope returns invalid when every span item fails normalizeSpanItem and the envelope has no other items. Local maps that to a 400 and doesn't retain it. The docs change in docs/cli/local-control-plane.md makes this official, but on main those raw envelopes stay inspectable. For a local debugger, a misbehaving SDK's output is exactly what people want to look at. I'd keep the envelope with an invalid_span diagnostic, the way unsupported containers are kept, rather than turning away the SDK.

  • Ownership question: the code is taken from getsentry/spotlight#1374, which is still an unreviewed draft, and this PR sets up @sentry/spotlight to publish from Toolkit. Have the Spotlight maintainers agreed on the name and on moving the processing code here? If so, linking that decision in the description would help. Otherwise the two copies will drift until Spotlight switches to this package.

This branch was successfully deployed

1 active deployment
Preview — c5b96e5f Deployed Oct 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant