feat(error-tracking): overlay recent issue state by hpouillot · Pull Request #87915 · PostHog/posthog

hpouillot · 2026-08-24T14:59:52Z

Problem

Error tracking users can briefly see stale issue status, severity, and assignments after a mutation.

ClickHouse receives issue state asynchronously. The current workaround sends client-generated rows for each fingerprint and caps them at 50.nnFollow-up: #87950 removes those client rows after production validation.

Before

flowchart LR
    A[Issue mutation] --> B[Frontend fetches fingerprints]
    B --> C[Pending row per fingerprint]
    C --> D[ClickHouse UNION ALL and argMax]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class A phYellow;
    class B,C phBlue;
    class D phRed;

Changes

Issue queries now use recent Postgres status, severity, and assignments while ClickHouse catches up.
The query sends one temporary external row per issue and joins it by issue ID before filtering.
An explicit presence field preserves cleared descriptions, severities, and assignments.
The durable team watermark invalidates cached results without reverting after the 60-second overlay window.
The existing fingerprint pending rows remain available for merge and split mapping lag.
Nothing changes visually outside the corrected issue state.

After

flowchart LR
    A[Issue mutation] --> B[Postgres state timestamp]
    B --> C[Query loads recent issue state]
    C --> D[ClickHouse external overlay by issue ID]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class A phYellow;
    class B,C phBlue;
    class D phRed;

How did you test this code?

Added regressions for stale status, severity, detail projection, assignment, unassignment, expiry, team isolation, issues with 51 fingerprints, and unmapped legacy events.
Added a regression for durable cache watermark behavior after overlay expiry.
Ran targeted Ruff formatting and lint checks.
Ran the repository-wide mypy check.
Compiled optimized and legacy HogQL shapes with severity overrides and verified external-table registration.
Ran focused ClickHouse query regressions successfully.
The focused API test is blocked locally by an unrelated Pydantic v1 metaclass conflict during URL import.
Not run: hogli ci:preflight because it requires Git metadata unavailable in this Jujutsu workspace.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Automatic notifications

Publish to changelog?

Docs update

No docs update. This changes internal read-after-write consistency without changing an API or documented workflow.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Pi implemented the approved issue-state overlay design. The implementation uses the existing fingerprint overlay only for mapping changes.

Skills invoked: /error-tracking, /writing-tests, /writing-code-comments, /writing-dataclasses, /running-ci-preflight, /writing-pr-descriptions, and /pr.

The committed tests and identifiers use repository fixtures and invented values. No private customer material informed the change.

trunk-io · 2026-08-24T14:59:56Z

😎 Stack merged successfully - details.

github-actions · 2026-08-24T15:01:29Z

🤖 CI report

⚠️

Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Playwright — all passed

All tests passed.

View test results →

posthog · 2026-08-24T15:13:17Z

🦔 ReviewHog reviewed this pull request

Found 0 must fix, 3 should fix, 0 consider.

Published 3 findings (view the review).

Resolved comments: 2 fixed, 1 already settled

trunk-io · 2026-08-24T15:18:38Z

_{View Full Report ↗︎ ⋅ Docs}

posthog · 2026-08-24T15:28:41Z

ReviewHog Alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

posthog

ReviewHog Report

Found 3 should fix.

greptile-apps · 2026-08-25T08:36:35Z

Prompt To Fix All With AI

### Issue 1
products/error_tracking/backend/hogql_queries/error_tracking_query_builder.py:579-580
**Legacy overlay drops unmapped events**

When a mixed-filter query runs during the recent-state window and a matching exception fingerprint has not reached ClickHouse, this new fingerprint-state inner join discards the event before filtering and grouping, causing a valid issue to disappear from the results.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(error-tracking): keep issue-state wa..." | Re-trigger Greptile}

veria-ai · 2026-08-25T09:03:28Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

The bulk mutation endpoint does not cap how many issues one request changes, and every changed issue was loaded from Postgres and sent as an external-table row on each forced refresh for the whole overlay window. A large bulk resolve or assign could materialize an unbounded row list and ship it to ClickHouse repeatedly. Cap the loaded rows at MAX_RECENT_ISSUE_STATES. When a mutation touches more issues than the bound, skip the overlay and fall back to ClickHouse state instead of shipping the payload. The database read is limited to the bound plus one so the query itself stays cheap. Generated-By: PostHog Desktop Task-Id: a2355240-7fae-4452-b056-7d8eee6d040c

The issue-state overlay adds two primary-Postgres reads to every error tracking query. The watermark read runs on the cache-key path, before any cache lookup, so an OperationalError there fails a request whose result may already be cached. The recent-states read runs during calculation, so a failure there fails the whole query even though ClickHouse could serve a slightly stale result without the overlay. Wrap both reads in except OperationalError, mirroring AnalyticsQueryRunner._products_modifiers_for_cache: the watermark degrades to a stable "unavailable" cache marker, and the overlay falls back to an empty list so the query runs without it. Log each failure and count it on a new error_tracking_recent_issue_state_overlay_unavailable_total metric. Add a regression test for the cache-payload degradation. Generated-By: PostHog Desktop Task-Id: a1a0ffdb-a9ff-4893-b651-6d018645eb54

The cache watermark is derived as the maximum state_updated_at over live issue rows. A merge deletes the source rows without touching the target's timestamp, so a deleted source that held the team maximum drags the watermark backward. That can re-validate a pre-merge cache key and serve a snapshot from before the merge for the duration of the staleness window, showing a merged-away issue as still present. Stamp the merge target's state_updated_at inside merge(), so the watermark stays monotonic even when the deleted source held the maximum. This lives in the model's merge() so both the user mutation path and the Temporal auto-merge path are covered. Stamping is also semantically correct: a merge changes the target's composition. Add a regression test asserting the watermark does not move backward across a merge. Generated-By: PostHog Desktop Task-Id: a1a0ffdb-a9ff-4893-b651-6d018645eb54

greptile-apps · 2026-08-25T09:44:46Z

_{Reviews (2): Last reviewed commit: 6abdd27 | Re-trigger Greptile}

deployment-status-posthog · 2026-08-25T11:11:50Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-08-25 11:11 UTC	Run
prod-us	✅ Deployed	2026-08-25 11:34 UTC	Run
prod-eu	✅ Deployed	2026-08-25 11:36 UTC	Run

hpouillot self-assigned this Aug 24, 2026

hpouillot added the skip-inkeep-docs Use this label to skip an Inkeep docs PR in posthog.com label Aug 24, 2026

hpouillot added the reviewhog label Aug 24, 2026

hpouillot mentioned this pull request Aug 24, 2026

fix(error-tracking): retire client state overlays #87950

Merged

1 task

posthog Bot reviewed Aug 24, 2026

View reviewed changes

Comment thread products/error_tracking/backend/hogql_queries/issue_state_overlay.py Resolved

Comment thread products/error_tracking/backend/hogql_queries/error_tracking_query_runner.py Resolved

Comment thread products/error_tracking/backend/hogql_queries/issue_state_overlay.py Resolved

posthog Bot removed the reviewhog label Aug 24, 2026

hpouillot requested review from a team, ablaszkiewicz and cat-ph August 25, 2026 08:29

hpouillot marked this pull request as ready for review August 25, 2026 08:29

hpouillot requested a review from a team as a code owner August 25, 2026 08:29

pr-assigner-resolver-posthog Bot requested a review from a team August 25, 2026 08:30

greptile-apps Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread products/error_tracking/backend/hogql_queries/error_tracking_query_builder.py Resolved

hpouillot force-pushed the hp/error-tracking-issue-state-overlay branch 2 times, most recently from 0c711d5 to 6abdd27 Compare August 25, 2026 08:59

veria-ai Bot reviewed Aug 25, 2026

View reviewed changes

Comment thread products/error_tracking/backend/hogql_queries/issue_state_overlay.py Resolved

hpouillot marked this pull request as draft August 25, 2026 09:13

hpouillot removed the request for review from a team August 25, 2026 09:13

ablaszkiewicz approved these changes Aug 25, 2026

View reviewed changes

hpouillot marked this pull request as ready for review August 25, 2026 09:30

pr-assigner-resolver-posthog Bot requested review from a team August 25, 2026 09:31

ablaszkiewicz approved these changes Aug 25, 2026

View reviewed changes

hpouillot and others added 7 commits August 25, 2026 11:35

feat(error-tracking): overlay recent issue state

5dbab03

fix(error-tracking): overlay recent severity

9b8d927

fix(error-tracking): preserve unmapped legacy events

e24b1e4

fix(error-tracking): preserve ClickHouse state coverage

232abb3

hpouillot force-pushed the hp/error-tracking-issue-state-overlay branch from 6abdd27 to 232abb3 Compare August 25, 2026 09:35

sinan-ku approved these changes Aug 25, 2026

View reviewed changes

mariusandra approved these changes Aug 25, 2026

View reviewed changes

trunk-io Bot merged commit bef1125 into master Aug 25, 2026
250 checks passed

trunk-io Bot deleted the hp/error-tracking-issue-state-overlay branch August 25, 2026 10:42

hpouillot mentioned this pull request Aug 25, 2026

fix(error-tracking): overlay recent issue state #87882

Closed

1 task

hpouillot mentioned this pull request Sep 4, 2026

Sprint - September 7 to September 18 #92782

Closed

Conversation

hpouillot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Before

Changes

After

How did you test this code?

Automatic notifications

Docs update

🤖 Agent context

Uh oh!

trunk-io Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

Uh oh!

posthog Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦔 ReviewHog reviewed this pull request

Uh oh!

trunk-io Bot commented Aug 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

posthog Bot commented Aug 24, 2026

Uh oh!

posthog Bot left a comment

Choose a reason for hiding this comment

ReviewHog Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 25, 2026

Uh oh!

Uh oh!

Uh oh!

veria-ai Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

greptile-apps Bot commented Aug 25, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hpouillot commented Aug 24, 2026 •

edited

Loading

trunk-io Bot commented Aug 24, 2026 •

edited

Loading

github-actions Bot commented Aug 24, 2026 •

edited

Loading

posthog Bot commented Aug 24, 2026 •

edited

Loading

trunk-io Bot commented Aug 24, 2026 •

edited

Loading

veria-ai Bot commented Aug 25, 2026 •

edited

Loading

deployment-status-posthog Bot commented Aug 25, 2026 •

edited

Loading