[matheus-vb]

Problem

An operator cannot tell what a cohort backfill scan cost. A slow scan cannot be traced back to a query in ClickHouse, nothing records how much data a chunk moved, and the seeder's latency quantiles read zero because they are rolling window summaries that drop a rare long scan.

Separately, a chunk that fails for a durable reason is claimed again on the next poll tick, fails the same way, and spends its whole attempt budget in minutes. The run then fails, and its cohort cannot start a new run until an operator intervenes.

Changes

Warning

This PR does not make the current backfill run faster. Everything here is measurement, and the retry backoff deliberately adds wall clock. The two changes that would speed a run up, more concurrency and more CPU, are chart config and are not in this diff. Deploying also rolls the singleton pod and loses the in-flight chunks' byte accumulators.

• A failed chunk now waits before it can be claimed again. The wait doubles per attempt with full jitter and is capped, so a transient cause gets time to clear.
• The wait gates only the failed-to-claim transition. A pending chunk and an expired lease reclaim stay immediate, so a run never stalls on a chunk that did not fail.
• Every ClickHouse scan carries a JSON log_comment naming the team, run, chunk, day, band and claim epoch. query_log_archive extracts team_id, product, feature and cohort_id from it, so seeder scans now archive under the team that caused them instead of under team 0.
• Each chunk records the bytes its scan received and decoded, on the row and on per-kind counters. Halted scans are counted too, since those bytes did move.
• The person boundaries scan is metered as its own kind. It is one whole-team scan per run, so folding it into the per-chunk person series would hide it.
• Seeder histograms render as real Prometheus histograms, and each bucket ladder is defined once in a table the recorder and its test both read.
• The person planning timer gets a ladder with a millisecond floor. It spans a ClickHouse scan and a Postgres insert, so the scan ladder's one-second floor erased every fast planning pass.
• The image build gains frame pointers, so the profiler can walk the CPU-bound scan path.
• Mechanical: one Django migration adds three columns, the frozen Rust DDL fixture tracks it, and two cursor loops move into helper functions so byte volume is metered at one exit point.

Scan semantics, tile wire format, claim ordering and the processor are unchanged.

The wait is a sum over attempts, not the cap

The drain for one durably failing chunk is the sum of its per-attempt ceilings, and the expected drain is half that, because the jitter draws uniformly from [0, ceiling]. At base 30s and cap 1800s the doubling reaches the cap on the 7th attempt:

attempts	worst case	expected
5 (code default)	15.5 min	7.8 min
12	3.5 h	1.8 h
50	22.5 h	11.3 h

The deployed SEEDER_MAX_CHUNK_ATTEMPTS is far above the code default, so this needs a companion charts change lowering it before or with this deploy. A run holds its cohort's uniqueness slot for that whole window.

Deploy ordering

Warning

Migration 0012 must land before the image. claim_next reads next_attempt_at unconditionally, so an image that rolls first makes every claim throw column does not exist and the live run stops claiming entirely. The db_default on the byte columns protects the other direction only.

The posthog_cohort_backfill_chunks_failed alert also needs retargeting in charts. A failed chunk now stays in that status for its whole wait, so an ordinary trickle of retryable failures holds the gauge above zero across the alert's 30-minute window.

How did you test this code?

New tests and the regression each catches:

• Retry wiring, in the Postgres contract suite: a chunk failed through the executor's recovery path draws a wait sized from its own attempt count, and that wait reaches next_attempt_at. Confirmed to fail when the recovery path is changed to always ask for the first attempt's wait, which is the edit that would defeat the whole feature while every policy-level test stayed green.
• Backoff gate: a failed chunk is not claimable until its stamp passes, the claim clears the stamp, and pending, unclaim and produced reclaim ignore it.
• Bucket ladders: asserts every histogram renders bucketed and that per-metric overrides beat the global default. The exporter documents the opposite of what it implements, so a dependency bump could silently collapse the scan metrics onto the 60s ladder.
• log_comment shape: pinned byte for byte, plus a check that every phase carries the four keys query_log_archive extracts. A dropped key archives the scan under team 0.
• Byte volume round trip: mark_produced persists both counts, and planning leaves them at the database-side default. The default is load-bearing, since planning inserts an explicit column list naming neither column.
• Backoff policy: ceiling doubling, cap saturation at absurd attempt counts, jitter reaching short waits, and rejection of a cap Postgres cannot add to a timestamp.

Not run: nothing in this diff was exercised against a real ClickHouse cluster. The log_comment and byte counters are asserted against the types, not against system.query_log.

Docs update

No.

[trunk-io]

😎 This pull request was merged.

[github-actions]

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️ Django migration SQL — 1 new migration to review

We've detected new migrations on this PR. Review the SQL output for each migration:

products/cohorts/backend/migrations/0012_cohortbackfillchunk_next_attempt_at_and_more.py

BEGIN;
--
-- Add field next_attempt_at to cohortbackfillchunk
--
ALTER TABLE "cohort_backfill_chunks" ADD COLUMN "next_attempt_at" timestamp with time zone NULL;
--
-- Add field scan_decoded_bytes to cohortbackfillchunk
--
ALTER TABLE "cohort_backfill_chunks" ADD COLUMN "scan_decoded_bytes" bigint DEFAULT 0 NOT NULL;
--
-- Add field scan_received_bytes to cohortbackfillchunk
--
ALTER TABLE "cohort_backfill_chunks" ADD COLUMN "scan_received_bytes" bigint DEFAULT 0 NOT NULL;
COMMIT;

Last updated: 2026-08-31 14:17 UTC (433e6cf)

✅ Django migration risk — migration analysis complete

We've analyzed your migrations for potential risks.

Summary: 1 Safe | 0 Needs Review | 0 Blocked

✅ Safe

Brief or no lock, backwards compatible

cohorts.0012_cohortbackfillchunk_next_attempt_at_and_more
  └─ #1 ✅ AddField
     Adding nullable field requires brief lock
     model: cohortbackfillchunk, field: next_attempt_at
  └─ #2 ✅ AddField
     Adding NOT NULL field with constant default (safe in PG11+)
     model: cohortbackfillchunk, field: scan_decoded_bytes
  └─ #3 ✅ AddField
     Adding NOT NULL field with constant default (safe in PG11+)
     model: cohortbackfillchunk, field: scan_received_bytes

📚 How to Deploy These Changes Safely

AddField:

This operation acquires a brief lock but doesn't rewrite the table.

Deployment uses lock timeouts with automatic retries, so lock contention will cause retries rather than connection pile-up.

Last updated: 2026-08-31 14:17 UTC (433e6cf)

[trunk-io]

   

View Full Report ↗︎ ⋅ Docs

[posthog]

🦔 PostHog Review is reviewing this pull request

Step 5/6 · Validating findings · 7/12

Specialist review skills read the changed code in parallel each from their own perspective, a blind-spot sweep catches what they missed, and only validated findings are published back to this pull request.

_{This comment updates as the review progresses.}

[gustavohstrassburger]

Looks solid. A few non-blocking suggestions inline; the metrics one is worth pairing with the charts changes you already listed.

[gustavohstrassburger]

suggestion: configured_builder switches the exporter from rolling-window summaries to real bucketed histograms, and that changes the rendered shape of seven metrics master already exports. A panel querying seeder_chunk_scan_duration_seconds{quantile="0.99"} returns no data the moment this deploys, because the quantile series stops being emitted. histogram_quantile(0.99, rate(seeder_chunk_scan_duration_seconds_bucket[5m])) is what replaces it.

The full set that changes shape:

• seeder_chunk_scan_duration_seconds and seeder_person_chunk_scan_duration_seconds
• seeder_person_planning_duration_seconds
• seeder_aggregate_entries
• seeder_produce_ack_seconds, seeder_pacer_wait_seconds, seeder_reconcile_observation_pass_seconds

The last three take no HISTOGRAM_LADDERS entry, so they inherit DEFAULT_SECONDS_BUCKETS and flip too, even though nothing in the diff names them.

The description lists two charts changes this deploy needs, SEEDER_MAX_CHUNK_ATTEMPTS and the posthog_cohort_backfill_chunks_failed retarget, but not this one. A panel that goes blank reads as "the seeder stopped reporting" rather than "the metric changed shape". Worth a third line on the charts checklist naming these seven and the histogram_quantile rewrite, so whoever lands the charts PR fixes all three together.

[trunk-io]

This pull request was merged into master as part of stacked PR 91390.

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-08-31 17:31 UTC	Run
prod-us	✅ Deployed	2026-08-31 17:51 UTC	Run
prod-eu	✅ Deployed	2026-08-31 17:56 UTC	Run