Skip to content

fix(cohorts): stop rescanning every team cohort on each cohort write - #95271

Merged
trunk-io[bot] merged 3 commits into
masterfrom
matheus-vb/cohort-creation-limit
Sep 4, 2026
Merged

trunk-io[bot] merged 3 commits into
masterfrom
matheus-vb/cohort-creation-limit

Conversation

@matheus-vb

Copy link
Copy Markdown
Member

Problem

Creating, deleting, or editing one cohort costs time proportional to how many cohorts the project already holds, and that work runs inside the web request.

  • Each such write runs warm_team_cohort_dependency_cache inline, through transaction.on_commit with no ATOMIC_REQUESTS.
  • The warm loads every live cohort of the team from Postgres with full filters and does three to four Redis round trips per cohort.
  • A create always takes that path: the new cohort's cache key does not exist yet, so it reads as "dependencies changed".
  • The cost is independent of the request itself, so a project with many cohorts pays for all of them on every write.

Changes

  • A cohort write now costs one or two Redis round trips and no extra Postgres reads. Nothing user-visible changes.
  • _on_cohort_changed writes only the changed cohort's own keys and deletes the reverse lists of the cohorts it started or stopped referencing. DEL is idempotent, so concurrent writers cannot lose an update, and the next reader rebuilds the list.
  • Delete paths take the old dependency set from the instance's own filters, so an expired cache key no longer leaves a dangling reverse entry.
  • A read miss on dependents still warms the team, but the warm loads only filters and groups, writes with set_many, and never runs from a write path.
  • The two key families move to a writer-only cache alias, cohort_dependencies, mirroring organization_access. The create path reads its own keys back in the same request, and a replica-lag miss would rescan the team.
  • COHORT_DEPENDENCY_INCREMENTAL_MAINTENANCE (default on) falls back to the full warm. A wrong incremental update shows only as a shorter recalculation chain, which no dashboard surfaces, so a config flip has to beat a revert.
  • get_cohort_dependents no longer filters the team lookup on deleted, so a soft-deleted cohort with live dependents stops caching an empty list for a week.
  • New metrics: posthog_cohort_dependency_maintenance_total{operation,path} and the histogram posthog_cohort_dependency_warm_cohorts_scanned.
  • Supersedes fix(cohorts): batch the dependency cache warm's Redis calls #94251, which batched the warm's Redis calls but kept the warm on every save.

Note

Residual staleness: an update whose cached dependencies key expired leaves the cohort in a former dependency's reverse list until the next warm. Consumers filter extra ids out. Missing ids, the case consumers do not tolerate, are never produced by this path.

How did you test this code?

  • Existing test_dependencies.py cases pass with one helper reordered (the raw reverse-key assertion moves after the read that rebuilds it) and the TTL test rewritten for set_many.
  • New tests, each named for the regression it catches: a create on a team with existing cohorts issues no cohort table read and no warm; a create with no references writes only its own keys; a create with a reference invalidates the target's reverse list; a hard delete with the dependencies key absent still invalidates; soft delete then restore re-adds the reverse edge; a save without definition fields skips maintenance; the warm uses batched writes only; the reverse map spans batches; the kill switch restores the full warm.
  • Also run locally: test_cohort_update_recalculated_after_caching, the dependency-chain tests in test_calculate_cohort.py, the dependency tests in test_util.py, hogli lint:tach, repo-wide mypy, and hogli ci:preflight.
  • Not run: the full test_cohort.py and test_calculate_cohort.py suites.

Docs update

No.

@matheus-vb
matheus-vb requested a review from a team September 4, 2026 14:50
@trunk-io

trunk-io Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

@posthog

posthog Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

🦔 PostHog Review reviewed this pull request

Found 0 must fix, 2 should fix, 1 consider.

Published 3 findings (view the review).

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Playwright — all passed

All tests passed.

View test results →

⚠️ Backend coverage — 96.0% of changed backend lines covered — 8 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 96.0% (212 / 220)

File Patch Uncovered changed lines
products/cohorts/backend/models/dependencies.py 92.5% 301, 310–311, 333–334, 469, 716–717

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 33911573234 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)
Product Coverage Lines
platform_features ██░░░░░░░░░░░░░░░░░░ 12.1% 7 / 58
warehouse_sources_queue ██████░░░░░░░░░░░░░░ 29.1% 92 / 316
demo ███████████░░░░░░░░░ 57.0% 1,523 / 2,671
data_tools █████████████░░░░░░░ 66.2% 90 / 136
ai_gateway ███████████████░░░░░ 75.0% 9 / 12
batch_exports ████████████████░░░░ 79.8% 19,411 / 24,336
apm █████████████████░░░ 83.3% 1,234 / 1,481
signals █████████████████░░░ 85.2% 35,553 / 41,713
cdp █████████████████░░░ 85.3% 4,332 / 5,077
wizard █████████████████░░░ 87.3% 2,105 / 2,411
product_tours ██████████████████░░ 87.9% 1,303 / 1,482
notebooks ██████████████████░░ 88.4% 11,801 / 13,346
data_warehouse ██████████████████░░ 88.7% 12,732 / 14,359
business_knowledge ██████████████████░░ 89.1% 4,440 / 4,984
cohorts ██████████████████░░ 89.2% 7,526 / 8,433
engineering_analytics ██████████████████░░ 89.5% 8,163 / 9,124
dashboards ██████████████████░░ 89.6% 6,818 / 7,606
tasks ██████████████████░░ 90.1% 64,986 / 72,090
streamlit_apps ██████████████████░░ 90.2% 2,500 / 2,773
data_modeling ██████████████████░░ 90.4% 9,349 / 10,346
links ██████████████████░░ 90.6% 183 / 202
managed_warehouse ██████████████████░░ 90.7% 9,342 / 10,299
exports ██████████████████░░ 90.7% 9,120 / 10,051
alerts ██████████████████░░ 90.9% 5,224 / 5,745
visual_review ██████████████████░░ 91.0% 6,896 / 7,577
canvas ██████████████████░░ 91.1% 4,934 / 5,414
conversations ██████████████████░░ 91.2% 21,436 / 23,513
mcp_analytics ██████████████████░░ 91.2% 4,401 / 4,827
error_tracking ██████████████████░░ 91.8% 13,335 / 14,534
notifications ██████████████████░░ 92.4% 1,110 / 1,201
early_access_features ███████████████████░ 92.5% 1,323 / 1,430
managed_migrations ███████████████████░ 92.6% 1,556 / 1,681
slack_app ███████████████████░ 92.8% 12,596 / 13,578
surveys ███████████████████░ 93.1% 6,170 / 6,630
mcp_registry ███████████████████░ 93.2% 1,463 / 1,570
web_analytics ███████████████████░ 93.2% 18,857 / 20,230
context_layer ███████████████████░ 93.3% 2,593 / 2,779
stamphog ███████████████████░ 93.3% 6,533 / 6,999
reminders ███████████████████░ 93.4% 468 / 501
marketing_analytics ███████████████████░ 93.5% 17,337 / 18,541
posthog_ai ███████████████████░ 93.5% 1,385 / 1,481
mcp_store ███████████████████░ 93.5% 8,053 / 8,611
ai_observability ███████████████████░ 93.8% 18,195 / 19,403
approvals ███████████████████░ 93.9% 3,560 / 3,793
billing_alerts ███████████████████░ 94.0% 2,069 / 2,200
endpoints ███████████████████░ 94.2% 8,875 / 9,425
review_hog ███████████████████░ 94.3% 10,203 / 10,815
tracing ███████████████████░ 94.5% 2,709 / 2,868
legal_documents ███████████████████░ 94.7% 2,073 / 2,190
workflows ███████████████████░ 94.7% 11,779 / 12,433
access_control ███████████████████░ 94.9% 6,052 / 6,378
customer_analytics ███████████████████░ 95.1% 19,157 / 20,151
annotations ███████████████████░ 95.2% 795 / 835
autoresearch ███████████████████░ 95.3% 443 / 465
product_analytics ███████████████████░ 95.3% 19,348 / 20,304
actions ███████████████████░ 95.4% 740 / 776
replay_vision ███████████████████░ 95.5% 22,574 / 23,642
skills ███████████████████░ 95.6% 5,434 / 5,686
experiments ███████████████████░ 95.6% 30,050 / 31,434
messaging ███████████████████░ 95.8% 3,671 / 3,830
logs ███████████████████░ 95.9% 13,712 / 14,294
growth ███████████████████░ 95.9% 7,699 / 8,025
feature_flags ███████████████████░ 96.3% 20,303 / 21,089
revenue_analytics ███████████████████░ 96.3% 1,841 / 1,911
user_interviews ███████████████████░ 96.5% 2,639 / 2,735
warehouse_sources ███████████████████░ 97.2% 408,942 / 420,831
data_quality ████████████████████ 97.5% 4,217 / 4,323
analytics_platform ████████████████████ 98.1% 2,286 / 2,330
metrics ████████████████████ 98.2% 3,564 / 3,628
data_catalog ████████████████████ 98.4% 3,256 / 3,310
pulse ████████████████████ 98.4% 2,017 / 2,049
live_debugger ████████████████████ 99.2% 613 / 618
field_notes ████████████████████ 99.4% 158 / 159

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

@github-project-automation github-project-automation Bot moved this from In Review to Approved in Feature Flags Sep 4, 2026
@trunk-io

trunk-io Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

View Full Report ↗︎ ⋅ Docs

@posthog

posthog Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review

Found 2 should fix, 1 consider.

@posthog posthog Bot removed the reviewhog label Sep 4, 2026
@trunk-io
trunk-io Bot merged commit e3b3fca into master Sep 4, 2026
245 checks passed
@github-project-automation github-project-automation Bot moved this from Approved to Done in Feature Flags Sep 4, 2026
@trunk-io
trunk-io Bot deleted the matheus-vb/cohort-creation-limit branch September 4, 2026 21:20
@deployment-status-posthog

deployment-status-posthog Bot commented Sep 4, 2026 •

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-09-04 21:51 UTC Run
prod-us ✅ Deployed 2026-09-04 22:03 UTC Run
prod-eu ✅ Deployed 2026-09-04 22:03 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

2 participants