Skip to content

Behavioral cohorts: measure whether a streaming architecture unlocks greater scale #60582

New issue
New issue
Closed
Closed
Behavioral cohorts: measure whether a streaming architecture unlocks greater scale#60582
Assignees
matheus-vb
Labels
enhancementNew feature or requestfeature/cohortsFeature Tag: Cohortsfeature/feature-flagsFeature Tag: Feature flagsteam/feature-flags

Description

@matheus-vb
matheus-vb
opened on May 28, 2026

Investigate how far the current architecture can scale. Given the current performance issues, maybe a streaming alternative that keeps incremental per person state and emits only membership transitions unlocks meaningfully more. Also investigate whether that alternative's added complexity is worth it.

We should measure the streaming alternative's throughput, freshness, state size, and failure modes at the same target team counts, via a simple PoC.

Also estimate the full cost of the alternative (new services, topics, person merge handling, cohort reference cascades, durability) against the cost of fixing/evolving the current design in place.

Activity

  1. matheus-vb
    self-assigned this
    on May 28, 2026
  2. matheus-vb
    added this to Feature Flagson May 28, 2026
  3. matheus-vb
    added
    enhancementNew feature or request
    on May 28, 2026
  4. matheus-vb
    moved this to In Progress in Feature Flagson May 28, 2026
  5. haacked
    added
    feature/cohortsFeature Tag: Cohorts
    team/feature-flags
    on Jun 9, 2026

  6. matheus-vb commented on Jun 30, 2026

    @matheus-vb
    matheus-vb
    on Jun 30, 2026
    MemberAuthor

    Streaming cohorts - shadow mode status

    The streaming cohorts pipeline PoC was introduced by this PR. It is currently running in prod-us shadow mode against team 2 live traffic and keeping up in real time. Where we sit right now:

    • Throughput: ~75 events/s consumed, processed at the same rate - no backlog.
    • Consumer lag: bounded ~350–1,500 events (a few seconds' buffer), stable, not growing.
    • Per-event latency: p50 ~18 ms, p99 ~83 ms.
    • Membership output: ~66 changes/s written to the shadow topic (from ~170–320 leaf transitions/s).
    • Resources: CPU ~2/3.4 cores, memory ~4.8/12 GiB, 0 restarts, error/stall counters clean. The on-disk state store grows ~2 GB per hour under sustained load (the main to watch as we scale).

    Recent work: a few rounds of optimization: event-name gating of conditions, person-state memoization, and a memory bump - roughly halved per-event cost. Under the same load the processor previously fell behind; it now keeps pace with the numbers above.

    Not exercised yet:

    • Multi-pod: a single pod owns all 64 partitions today; cross-pod partition rebalancing is untested.
    • Person merging: the merge/transfer paths aren't validated under production conditions.
    • Parity against the existing pipeline, and scaling beyond one team.
  7. marandaneto
    added
    feature/feature-flagsFeature Tag: Feature flags
    on Aug 19, 2026

  8. matheus-vb commented on Sep 28, 2026

    @matheus-vb
    matheus-vb
    on Sep 28, 2026
    MemberAuthor

    Measured with the PoC in #61317 (shadow results above). The streaming architecture has since shipped to production.

  9. matheus-vb
    closed this as completedon Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestfeature/cohortsFeature Tag: Cohortsfeature/feature-flagsFeature Tag: Feature flagsteam/feature-flags

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions