Skip to content

fix(error-tracking): signals omit the event origin, so dev-host noise becomes duplicate reports and PRs #90178

Open

Description

@andrewm4894

Problem

An error-tracking signal tells the Signals pipeline what threw and where in the code, but not where the event came from. emit_issue_lifecycle_signal (products/error_tracking/backend/temporal/lifecycle/side_effects.py) builds the signal description as:

<preamble>:
<issue name>: <issue description>

```<rendered stack trace>```

It already has the full exception event properties in hand (fetch_event_properties), but only reads $exception_list out of them. $current_url, $host, $lib, and the environment never reach the description.

So every downstream consumer is blind to origin:

  • Grouping (products/signals/backend/temporal/grouping.py) compares signal text. Five different exceptions that all came from localhost:3000 on developers' machines look like five unrelated bugs, because the one thing they share is not in the text.
  • Actionability and priority judge the same text, so "one event, from a dev server, on an unmerged branch" reads like a production defect until research digs into it.
  • Research has to rediscover origin every run with its own $exception SQL query, and then explain it in prose in the report.

The visible result on one customer project: eight separate inbox reports over five days, all diagnosing the same root cause (an unguarded posthog.init letting local dev servers report into the production project), and three auto-started PRs that change the same file for the same reason. The signals were genuinely different exceptions (a ReferenceError on a dashboard route, Prisma schema drift, a native module resolution failure), so grouping could not have joined them from the text it had. The shared cause was only ever visible in $current_url.

Fixes

1. Add an origin block to the signal description. In emit_issue_lifecycle_signal, after the header and before the stack trace, render a short deterministic block from the event properties that are already fetched. Something like:

Origin: host localhost:3000 (local development host), lib posthog-js
  • Host: the hostname and port parsed from $current_url, falling back to $host. Hostname only, never the path or query, since those can carry identifiers.
  • Dev-host flag: mark localhost, 127.0.0.1, 0.0.0.0, ::1, *.local, *.localhost, and private-range IPs as a local development host, in words the grouping and actionability prompts will pick up.
  • $lib (and $lib_version if cheap). Skip any field that is missing rather than rendering an empty value.
  • Keep the line stable and low-entropy so it helps the embedding rather than adding noise.

The block counts against the same token budget as the header does today (SIGNAL_MAX_TOKENS in rendering.py), so the stack trace budget shrinks by the block's size. Rendering belongs next to render_stacktrace in rendering.py, not inline in side_effects.py.

2. Carry the same facts as structured data. Put host and is_dev_host into the extra dict passed to emit_signal, alongside fingerprint. That lets a later change filter or steer on them without parsing prose.

3. Cover all three lifecycle callers. issue_created, issue_reopened, and issue_spiking all go through emit_issue_lifecycle_signal, so one change covers them. Extend products/error_tracking/backend/temporal/lifecycle/test/test_side_effects.py with cases for: a production host, a localhost host with a port, $current_url absent with $host present, and both absent (no origin line at all).

Out of scope here, but the natural next step once this lands: an aggregate line ("N events from M hosts in the last 7 days, all local development hosts") from a fingerprint query, and a signals-side rule that treats dev-host-only issues as not actionable. Neither is needed for the description change, and the second is a product decision.

Why

The pipeline already spends a full research run per report to learn something the event carried from the start. Putting origin in the signal text lets grouping join dev-noise exceptions into one report, lets actionability rank them correctly, and stops the customer-visible outcome of several reports and competing PRs for one fix. It is a small change in one emitter with no new queries.

Origin and full timeline in this Slack thread; the investigation write-up is in the self-driving-debugging skill in the skills store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions