Repository navigation
feat(signals): pre-compute suggested scouts per project - #85963
Merged
Merged
Conversation
|
😎 Merged successfully - details. |
Contributor
🤖 CI report
|
| File | Size | Δ vs base |
|---|---|---|
posthog-app/_parent/products/customer_analytics/frontend/CustomerAnalyticsScene.js |
9.5 KiB | 🟢 -256.7 KiB (-96.4%) |
render-query/src/render-query/render-query.js |
21.82 MiB | 🔺 +32.8 KiB (+0.1%) |
posthog-app/src/scenes/dashboard/dashboards/Dashboards.js |
47.2 KiB | 🔺 +22.0 KiB (+87.6%) |
posthog-app/_parent/products/tracing/frontend/TracingScene.js |
120.9 KiB | 🔺 +20.7 KiB (+20.7%) |
exporter/src/queries/schema.js |
1.20 MiB | 🔺 +17.5 KiB (+1.4%) |
posthog-app/src/queries/schema.js |
1.20 MiB | 🔺 +17.5 KiB (+1.4%) |
posthog-app/_parent/products/signals/frontend/inbox/InboxScene.js |
763.1 KiB | 🔺 +14.5 KiB (+1.9%) |
posthog-app/src/scenes/annotations/Annotations.js |
13.7 KiB | 🔺 +13.7 KiB (new) |
posthog-app/src/scenes/real-time-usage/RealTimeUsage.js |
12.8 KiB | 🔺 +12.8 KiB (new) |
posthog-app/_parent/products/customer_analytics/frontend/scenes/CustomerAnalyticsAccountScene/CustomerAnalyticsAccountScene.js |
12.4 KiB | 🔺 +12.4 KiB (new) |
posthog-app/_parent/products/mcp_analytics/frontend/MCPAnalyticsScene.js |
186.8 KiB | 🔺 +10.4 KiB (+5.9%) |
posthog-app/src/scenes/cohorts/Cohorts.js |
17.6 KiB | 🔺 +4.9 KiB (+39.0%) |
posthog-app/_parent/products/workflows/frontend/Workflows/WorkflowScene.js |
124.2 KiB | 🔺 +3.6 KiB (+3.0%) |
posthog-app/_parent/products/engineering_analytics/frontend/scenes/EngineeringAnalyticsScene.js |
61.9 KiB | 🔺 +3.4 KiB (+5.9%) |
toolbar/src/toolbar/debug/chunk-EventDebugMenu.js |
276.3 KiB | 🔺 +3.2 KiB (+1.2%) |
exporter/src/exporter/scenes/ExporterNotebookScene.js |
3.40 MiB | 🟢 -3.1 KiB (-0.1%) |
posthog-app/_parent/products/metrics/frontend/MetricsScene.js |
70.8 KiB | 🔺 +2.8 KiB (+4.0%) |
posthog-app/src/scenes/surveys/Surveys.js |
26.6 KiB | 🟢 -2.4 KiB (-8.4%) |
posthog-app/_parent/products/workflows/frontend/WorkflowsScene.js |
88.8 KiB | 🔺 +2.1 KiB (+2.4%) |
exporter/_parent/products/workflows/frontend/Workflows/Workflow.js |
97.7 KiB | 🔺 +2.0 KiB (+2.1%) |
exporter/_parent/products/signals/frontend/components/ScoutPicker/CyclotronJobInputScoutPicker.js |
1.9 KiB | 🔺 +1.9 KiB (new) |
posthog-app/_parent/products/signals/frontend/components/ScoutPicker/CyclotronJobInputScoutPicker.js |
1.8 KiB | 🔺 +1.8 KiB (new) |
posthog-app/_parent/products/tasks/frontend/SlackTaskContextScene.js |
12.1 KiB | 🔺 +1.1 KiB (+9.9%) |
Posted automatically by build-bundle-size-report · uncompressed bytes from dist-report
✅ Eager graph — within budget
How much code each root ships on the eager path — downloaded and parsed before the surface is interactive. Measured from the esbuild output chunks (post-tree-shake, static imports only); lazy import() / React.lazy chunks are not counted.
| Root | Eager (shipped) | Δ vs base | Budget |
|---|---|---|---|
entry (logged-out pages, app bootstrap)src/index.tsx |
1.43 MiB · 22 files | 🔺 +40.2 KiB (+2.8%) | ███░░░░░░░ 31.6% of 4.51 MiB |
authenticated shell (every logged-in page)src/scenes/AuthenticatedShell.tsx |
8.84 MiB · 3,249 files | 🔺 +15.6 KiB (+0.2%) | █████████░ 91.1% of 9.71 MiB |
🟢 node_modules/monaco-editor/ stays out of src/index.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 [object Object] stays out of src/index.tsx
🟢 node_modules/monaco-editor/ stays out of src/scenes/AuthenticatedShell.tsx
🟢 src/lib/components/ActivityLog/describers stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
🟢 [object Object] stays out of src/scenes/AuthenticatedShell.tsx
Largest files eagerly shipped from src/index.tsx
| Size | File |
|---|---|
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 24.6 KiB | ../node_modules/.pnpm/buffer@6.0.3/node_modules/buffer/index.js |
| 6.3 KiB | ../node_modules/.pnpm/react@18.3.1/node_modules/react/cjs/react.production.min.js |
| 4.5 KiB | ../node_modules/.pnpm/@jspm+core@2.1.0/node_modules/@jspm/core/nodelibs/browser/process.js |
| 3.9 KiB | ../node_modules/.pnpm/scheduler@0.23.2/node_modules/scheduler/cjs/scheduler.production.min.js |
| 1.4 KiB | ../node_modules/.pnpm/base64-js@1.5.1/node_modules/base64-js/index.js |
| 1.3 KiB | src/RootErrorBoundary.tsx |
| 912 B | ../node_modules/.pnpm/ieee754@1.2.1/node_modules/ieee754/index.js |
| 789 B | src/scenes/ChunkLoadErrorBoundary.tsx |
| 762 B | src/index.tsx |
Largest files eagerly shipped from src/scenes/AuthenticatedShell.tsx
| Size | File |
|---|---|
| 307.0 KiB | ../node_modules/.pnpm/posthog-js@1.422.5_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/rrweb.js |
| 267.7 KiB | ../node_modules/.pnpm/@posthog+icons@0.38.0_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
| 263.5 KiB | ../node_modules/.pnpm/posthog-js@1.422.5_@types+react@18.3.27_react@18.3.1/node_modules/posthog-js/dist/module.js |
| 256.1 KiB | src/taxonomy/core-filter-definitions-by-group.json |
| 154.2 KiB | ../node_modules/.pnpm/re2js@0.4.1/node_modules/re2js/build/index.esm.js |
| 126.8 KiB | ../node_modules/.pnpm/react-dom@18.3.1_react@18.3.1/node_modules/react-dom/cjs/react-dom.production.min.js |
| 104.5 KiB | src/lib/api.ts |
| 95.2 KiB | ../packages/quill/packages/quill/dist/index.js |
| 93.3 KiB | ../node_modules/.pnpm/prosemirror-view@1.40.1/node_modules/prosemirror-view/dist/index.js |
| 90.6 KiB | ../node_modules/.pnpm/@tiptap+core@3.20.6_@tiptap+pm@3.20.6/node_modules/@tiptap/core/dist/index.js |
Posted automatically by check-eager-graph · sizes are eager output bytes (shipped, post-tree-shake) from the esbuild metafile · part of #32479
✅ Toolbar bundle — eager 2.26 MiB within budget
What the toolbar ships to customer pages, measured from the esbuild output (minified, post-tree-shake). The eager set is the entry plus everything statically imported from it — fetched before any feature runs; deferred chunks load lazily. The eager guardrail is 5.72 MiB. Each output file must also stay below 10 MB, where CloudFront stops compressing it. The module boundary is enforced separately by check-toolbar-graph.
| Metric | Size | Δ vs base | Budget |
|---|---|---|---|
| Eager (shipped) entry + static imports |
2.26 MiB · 18 files | 🔺 +3.5 KiB (+0.2%) | ████░░░░░░ 39.5% of 5.72 MiB |
| Deferred (lazy) | 2.11 MiB · 45 files | 🔺 +3.2 KiB (+0.1%) | n/a — loads on demand |
Loader dist/toolbar.js |
1.1 KiB | no change | █░░░░░░░░░ 5.8% of 19.5 KiB |
Largest eagerly-shipped chunks
| Size | File |
|---|---|
| 751.7 KiB | dist/toolbar/toolbar-app-JNIXJPZH.css |
| 590.1 KiB | dist/toolbar/chunk-chunk-EFVXUHYC.js |
| 484.7 KiB | dist/toolbar/chunk-chunk-RNKUEDGA.js |
| 134.1 KiB | dist/toolbar/chunk-chunk-7F43S5WJ.js |
| 131.8 KiB | dist/toolbar/chunk-chunk-FDH2IBXT.js |
| 71.3 KiB | dist/toolbar/toolbar-app-3FOHUVAY.js |
| 69.0 KiB | dist/toolbar/chunk-chunk-TSAL54PB.js |
| 35.6 KiB | dist/toolbar/chunk-chunk-JDE4NYP5.js |
| 21.0 KiB | dist/toolbar/chunk-chunk-U37JV6N2.js |
| 6.8 KiB | dist/toolbar/chunk-chunk-DV7IWQNF.js |
Posted automatically by check-toolbar-size · sizes are toolbar output bytes (shipped, post-tree-shake) from the esbuild metafile
✅ Dist folder size — 🔺 +11.97 MiB (+0.8%)
Total size of the built frontend/dist folder (all assets), compared against the base branch.
Total: 1448.73 MiB · 🔺 +11.97 MiB (+0.8%)
ℹ️ MCP UI apps size — 33 app(s), 17662.9 KB JS
Built size of each MCP UI app (main.js + styles.css).
| App | JS | CSS |
|---|---|---|
| debug | 600.0 KB | 195.2 KB |
| action | 458.2 KB | 195.2 KB |
| action-list | 564.9 KB | 195.2 KB |
| cohort | 457.2 KB | 195.2 KB |
| cohort-list | 563.8 KB | 195.2 KB |
| email-template | 457.0 KB | 195.2 KB |
| error-details | 472.9 KB | 195.2 KB |
| error-issue | 457.9 KB | 195.2 KB |
| error-issue-list | 564.7 KB | 195.2 KB |
| experiment | 562.0 KB | 195.2 KB |
| experiment-list | 565.6 KB | 195.2 KB |
| experiment-results | 567.1 KB | 195.2 KB |
| feature-flag | 567.6 KB | 195.2 KB |
| feature-flag-list | 571.4 KB | 195.2 KB |
| feature-flag-testing | 461.4 KB | 195.2 KB |
| inline-scan | 457.7 KB | 195.2 KB |
| insight-actors | 563.0 KB | 195.2 KB |
| invite-email-preview | 456.4 KB | 195.2 KB |
| llm-costs | 560.0 KB | 195.2 KB |
| session-recording | 459.0 KB | 195.2 KB |
| survey | 458.8 KB | 195.2 KB |
| survey-global-stats | 562.7 KB | 195.2 KB |
| survey-list | 565.5 KB | 195.2 KB |
| survey-stats | 562.7 KB | 195.2 KB |
| trace-span | 457.6 KB | 195.2 KB |
| trace-span-list | 564.7 KB | 195.2 KB |
| vision-observation-list | 563.9 KB | 195.2 KB |
| workflow | 457.5 KB | 195.2 KB |
| workflow-list | 564.2 KB | 195.2 KB |
| loops-review | 461.9 KB | 195.2 KB |
| query-results | 755.2 KB | 195.2 KB |
| render-ui | 838.3 KB | 195.2 KB |
| visual-review-snapshots | 462.0 KB | 195.2 KB |
⚠️ Playwright — 2 flaky
🎭 Playwright report · View test results →
- Creating a SQL insight with a variable and overriding it on a dashboard (chromium)
- Hover chart to see tooltip with data point values (chromium)
These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!
⚠️ Backend coverage — 90.0% of changed backend lines covered — 114 uncovered
🧪 Backend test coverage
Patch coverage — changed backend lines (products + core): ██████████████████░░ 90.0% (1,109 / 1,223)
| File | Patch | Uncovered changed lines |
|---|---|---|
products/signals/backend/temporal/agentic/schedule.py |
0.0% | 26, 71, 78, 94–95, 97 |
products/signals/backend/management/commands/run_scout_suggestions.py |
0.0% | 8, 10, 12, 14, 16, 18–19, 27, 30–31, 33–36, 38–46, 48–49, 52, 54, 57–59, 66–71, 74, 78–80 |
products/tasks/backend/temporal/client.py |
50.0% | 419 |
products/signals/backend/temporal/agentic/scout_suggestions.py |
67.9% | 120–122, 128, 131–134, 144–145, 155, 157–160, 167, 171–173, 176, 178, 183, 186, 194, 211–213, 239, 243, 249–250, 254–261, 270–273, 275, 281 |
products/signals/backend/scout_chat.py |
77.8% | 79–80, 89–90 |
products/signals/backend/receivers.py |
81.8% | 632–633 |
products/signals/backend/scout_suggestions_api.py |
95.2% | 233, 240, 260–262 |
products/signals/backend/scout_harness/suggestions_runner.py |
96.9% | 139, 163–164, 217 |
products/signals/backend/scout_harness/suggestions.py |
97.6% | 322–323, 386, 442, 491–492, 698 |
🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 33550738901 -n patch-coverage), or the coverage-data block at the end of this comment.
Per-product line coverage (touched products)
| Product | Coverage | Lines |
|---|---|---|
platform_features |
██░░░░░░░░░░░░░░░░░░ 12.1% |
7 / 58 |
warehouse_sources_queue |
██████░░░░░░░░░░░░░░ 29.4% |
91 / 309 |
demo |
███████████░░░░░░░░░ 56.9% |
1,518 / 2,666 |
autoresearch |
████████████░░░░░░░░ 62.5% |
5 / 8 |
data_tools |
█████████████░░░░░░░ 66.2% |
90 / 136 |
ai_gateway |
███████████████░░░░░ 75.0% |
9 / 12 |
batch_exports |
████████████████░░░░ 79.9% |
19,716 / 24,669 |
apm |
█████████████████░░░ 83.3% |
1,234 / 1,481 |
signals |
█████████████████░░░ 84.5% |
32,960 / 39,026 |
cdp |
█████████████████░░░ 85.2% |
4,310 / 5,056 |
wizard |
█████████████████░░░ 86.7% |
1,090 / 1,257 |
data_modeling |
█████████████████░░░ 86.7% |
9,654 / 11,133 |
notebooks |
██████████████████░░ 87.7% |
11,202 / 12,770 |
product_tours |
██████████████████░░ 87.9% |
1,303 / 1,482 |
data_warehouse |
██████████████████░░ 88.6% |
12,782 / 14,425 |
engineering_analytics |
██████████████████░░ 89.0% |
7,707 / 8,656 |
cohorts |
██████████████████░░ 89.1% |
7,370 / 8,273 |
business_knowledge |
██████████████████░░ 89.1% |
4,440 / 4,984 |
dashboards |
██████████████████░░ 89.6% |
6,807 / 7,598 |
streamlit_apps |
██████████████████░░ 90.2% |
2,500 / 2,773 |
exports |
██████████████████░░ 90.5% |
8,685 / 9,594 |
links |
██████████████████░░ 90.6% |
183 / 202 |
managed_warehouse |
██████████████████░░ 90.7% |
8,563 / 9,436 |
alerts |
██████████████████░░ 90.9% |
5,182 / 5,700 |
visual_review |
██████████████████░░ 90.9% |
6,880 / 7,567 |
conversations |
██████████████████░░ 91.0% |
20,911 / 22,973 |
canvas |
██████████████████░░ 91.0% |
4,913 / 5,397 |
tasks |
██████████████████░░ 91.1% |
63,704 / 69,923 |
mcp_analytics |
██████████████████░░ 91.2% |
4,374 / 4,795 |
error_tracking |
██████████████████░░ 91.5% |
12,672 / 13,853 |
notifications |
██████████████████░░ 92.2% |
1,101 / 1,194 |
early_access_features |
███████████████████░ 92.5% |
1,323 / 1,430 |
managed_migrations |
███████████████████░ 92.6% |
1,556 / 1,681 |
slack_app |
███████████████████░ 92.8% |
12,354 / 13,315 |
stamphog |
███████████████████░ 92.9% |
5,775 / 6,216 |
context_layer |
███████████████████░ 93.3% |
2,573 / 2,758 |
surveys |
███████████████████░ 93.4% |
6,051 / 6,482 |
web_analytics |
███████████████████░ 93.4% |
18,575 / 19,892 |
reminders |
███████████████████░ 93.4% |
468 / 501 |
mcp_store |
███████████████████░ 93.4% |
7,609 / 8,144 |
posthog_ai |
███████████████████░ 93.5% |
1,372 / 1,468 |
marketing_analytics |
███████████████████░ 93.5% |
17,322 / 18,526 |
ai_observability |
███████████████████░ 93.7% |
17,988 / 19,194 |
approvals |
███████████████████░ 93.7% |
3,492 / 3,725 |
billing_alerts |
███████████████████░ 93.9% |
2,078 / 2,212 |
workflows |
███████████████████░ 94.1% |
11,042 / 11,739 |
legal_documents |
███████████████████░ 94.1% |
1,970 / 2,093 |
review_hog |
███████████████████░ 94.3% |
9,982 / 10,583 |
endpoints |
███████████████████░ 94.4% |
8,866 / 9,396 |
tracing |
███████████████████░ 94.4% |
2,678 / 2,837 |
access_control |
███████████████████░ 94.9% |
5,828 / 6,139 |
customer_analytics |
███████████████████░ 95.0% |
17,702 / 18,639 |
annotations |
███████████████████░ 95.2% |
795 / 835 |
product_analytics |
███████████████████░ 95.3% |
19,292 / 20,252 |
replay_vision |
███████████████████░ 95.4% |
26,086 / 27,355 |
experiments |
███████████████████░ 95.5% |
29,271 / 30,660 |
actions |
███████████████████░ 95.5% |
706 / 739 |
skills |
███████████████████░ 95.6% |
5,403 / 5,651 |
messaging |
███████████████████░ 95.8% |
3,636 / 3,795 |
logs |
███████████████████░ 95.9% |
13,495 / 14,077 |
growth |
███████████████████░ 95.9% |
7,644 / 7,969 |
feature_flags |
███████████████████░ 96.1% |
19,585 / 20,382 |
revenue_analytics |
███████████████████░ 96.3% |
1,841 / 1,911 |
user_interviews |
███████████████████░ 96.5% |
2,639 / 2,735 |
warehouse_sources |
███████████████████░ 97.2% |
405,790 / 417,574 |
data_quality |
███████████████████░ 97.5% |
4,103 / 4,209 |
analytics_platform |
████████████████████ 98.1% |
2,286 / 2,330 |
data_catalog |
████████████████████ 98.3% |
3,234 / 3,289 |
metrics |
████████████████████ 98.3% |
3,505 / 3,564 |
pulse |
████████████████████ 98.4% |
2,017 / 2,049 |
live_debugger |
████████████████████ 99.2% |
613 / 618 |
field_notes |
████████████████████ 99.4% |
158 / 159 |
Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.
⚠️ MCP snapshots — 1 updated (1 modified, 0 added, 0 deleted)
Snapshots: MCP unit test snapshots updated
Changes: 1 snapshots (1 modified, 0 added, 0 deleted)
What this means:
- Snapshots have been automatically updated to match current output
Next steps:
- Review the changes to ensure they're intentional
- If unexpected, investigate what caused the output to change
⚠️ Django migration SQL — 1 new migration to review
We've detected new migrations on this PR. Review the SQL output for each migration:
products/signals/backend/migrations/0109_signal_scout_suggestion_set.py
BEGIN;
--
-- Create model SignalScoutSuggestionSet
--
CREATE TABLE "signals_signalscoutsuggestionset" ("id" uuid NOT NULL PRIMARY KEY, "items" jsonb NOT NULL, "status" varchar(16) DEFAULT 'empty' NOT NULL, "generated_at" timestamp with time zone NULL, "task_run_id" uuid NULL, "model" varchar(200) DEFAULT '' NOT NULL, "fleet_snapshot" jsonb NOT NULL, "last_requested_at" timestamp with time zone NULL, "last_completed_at" timestamp with time zone NULL, "consecutive_failures" integer DEFAULT 0 NOT NULL CHECK ("consecutive_failures" >= 0), "created_at" timestamp with time zone NOT NULL, "updated_at" timestamp with time zone NOT NULL, "team_id" integer NOT NULL UNIQUE);
COMMIT;Last updated: 2026-09-01 19:44 UTC (acb6cf6)
✅ Django migration risk — migration analysis complete
We've analyzed your migrations for potential risks.
Summary: 1 Safe | 0 Needs Review | 0 Blocked
✅ Safe
Brief or no lock, backwards compatible
signals.0109_signal_scout_suggestion_set
└─ #1 ✅ CreateModel
Creating new table is safe
model: SignalScoutSuggestionSet
│
└──> ℹ️ INFO:
ℹ️ Skipped operations on newly created tables (empty tables
don't cause lock contention).
Last updated: 2026-09-01 19:45 UTC (acb6cf6)
ℹ️ ClickHouse migration SQL — 1 migration(s)
ClickHouse migration SQL per cloud environment
- unset
- all
CREATE TABLE IF NOT EXISTS posthog_test.logs34 ( `time_bucket` DateTime MATERIALIZED toStartOfDay(timestamp), `original_expiry_timestamp` DateTime64(6), `uuid` String, `team_id` Int32, `trace_id` String, `span_id` String, `trace_flags` Int32, `timestamp` DateTime64(6) CODEC(DoubleDelta), `observed_timestamp` DateTime64(6), `created_at` DateTime64(6) MATERIALIZED now(), `body` String, `severity_text` LowCardinality(String), `severity_number` Int32, `service_name` LowCardinality(String), `resource_attributes` Map(LowCardinality(String), String), `resource_fingerprint` UInt64 MATERIALIZED cityHash64(resource_attributes), `instrumentation_scope` String, `event_name` String, `attributes_map_str` Map(LowCardinality(String), String), `level` String ALIAS severity_text, `mat_body_ipv4_matches` Array(String) ALIAS extractAll(body, '(\d\.((25[0-5]|(2[0-4]|1(0, 1)[0-9])(0, 1)[0-9])\.)(2, 2)([0-9]))'), `time_minute` DateTime ALIAS toStartOfMinute(timestamp), `attributes` Map(LowCardinality(String), String) ALIAS mapApply((k, v) -> (left(k, -5), v), attributes_map_str), `attributes_map_float` Map(LowCardinality(String), Float64) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__float'), toFloat64OrNull(v)), attributes_map_str)), `attributes_map_datetime` Map(LowCardinality(String), DateTime64(6)) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__datetime'), parseDateTimeBestEffortOrNull(v, 6)), attributes_map_str)), `_partition` UInt32, `_topic` String, `_offset` UInt64, `_bytes_uncompressed` UInt64, `_bytes_compressed` UInt64, `_record_count` UInt64, `pattern` String, `pattern_version` UInt8, INDEX idx_severity_text_set severity_text TYPE set(10) GRANULARITY 1, INDEX idx_attributes_str_keys mapKeys(attributes_map_str) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_attributes_str_values mapValues(attributes_map_str) TYPE bloom_filter(0.001) GRANULARITY 1, INDEX idx_mat_body_ipv4_matches mat_body_ipv4_matches TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_body_ngram3 lower(body) TYPE ngrambf_v1(3, 25000, 2, 0) GRANULARITY 1, INDEX idx_uuid_bloom uuid TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_observed_minmax observed_timestamp TYPE minmax GRANULARITY 1, INDEX idx_timestamp_minmax timestamp TYPE minmax GRANULARITY 1, PROJECTION projection_aggregate_counts ( SELECT team_id, time_bucket, toStartOfMinute(timestamp), service_name, severity_text, resource_fingerprint, count() AS event_count GROUP BY team_id, time_bucket, toStartOfMinute(timestamp), service_name, severity_text, resource_fingerprint ) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.logs34', '{replica}-{shard}') PARTITION BY toDate(original_expiry_timestamp) PRIMARY KEY (team_id, time_bucket, service_name, resource_fingerprint, severity_text, timestamp) ORDER BY (team_id, time_bucket, service_name, resource_fingerprint, severity_text, timestamp) TTL original_expiry_timestamp SETTINGS allow_experimental_reverse_key = 1, index_granularity_bytes = 104857600, index_granularity = 8192, ttl_only_drop_parts = 1, add_minmax_index_for_numeric_columns = 1, map_serialization_version = 'with_buckets'
CREATE TABLE IF NOT EXISTS posthog_test.log_attributes2 ( `team_id` Int32, `time_bucket` DateTime64(0), `service_name` LowCardinality(String), `resource_fingerprint` UInt64 DEFAULT 0, `attribute_key` LowCardinality(String), `attribute_value` String CODEC(ZSTD(5)), `attribute_count` SimpleAggregateFunction(sum, UInt64), `attribute_type` LowCardinality(String) DEFAULT 'log', `original_expiry_time_bucket` DateTime DEFAULT now(), INDEX idx_attribute_key attribute_key TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_attribute_value attribute_value TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_attribute_key_n3 attribute_key TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 1, INDEX idx_attribute_value_n3 attribute_value TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 1 ) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.log_attributes2', '{replica}-{shard}') PARTITION BY toDate(original_expiry_time_bucket) ORDER BY (team_id, attribute_type, time_bucket, resource_fingerprint, attribute_key, attribute_value) TTL original_expiry_time_bucket SETTINGS deduplicate_merge_projection_mode = 'drop', index_granularity = 8192
CREATE TABLE IF NOT EXISTS posthog_test.logs_billing_metrics ( `team_id` Int32, `time_bucket` DateTime64(0), `service_name` LowCardinality(String), `bytes_uncompressed` SimpleAggregateFunction(sum, UInt64), `bytes_compressed` SimpleAggregateFunction(sum, UInt64), `record_count` SimpleAggregateFunction(sum, UInt64) ) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.logs_billing_metrics', '{replica}-{shard}') PARTITION BY toYYYYMM(time_bucket) ORDER BY (team_id, time_bucket, service_name) SETTINGS deduplicate_merge_projection_mode = 'rebuild', index_granularity = 8192
CREATE TABLE IF NOT EXISTS posthog_test.logs_kafka_metrics ( `_partition` UInt32, `_topic` String, `max_offset` SimpleAggregateFunction(max, UInt64), `max_observed_timestamp` SimpleAggregateFunction(max, DateTime64(9)), `max_timestamp` SimpleAggregateFunction(max, DateTime64(9)), `max_created_at` SimpleAggregateFunction(max, DateTime64(9)), `max_lag` SimpleAggregateFunction(max, UInt64) ) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.logs_kafka_metrics', '{replica}-{shard}') ORDER BY (_topic, _partition)
CREATE OR REPLACE TABLE posthog_test.logs_distributed AS posthog_test.logs34 ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'logs34')
CREATE OR REPLACE TABLE posthog_test.log_attributes_distributed AS posthog_test.log_attributes2 ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'log_attributes2')
CREATE TABLE IF NOT EXISTS posthog_test.logs_billing_metrics_distributed AS posthog_test.logs_billing_metrics ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'logs_billing_metrics')
CREATE TABLE IF NOT EXISTS posthog_test.logs_kafka_metrics_distributed AS posthog_test.logs_kafka_metrics ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'logs_kafka_metrics')
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.logs34_to_log_attributes TO posthog_test.log_attributes2 ( `team_id` Int32, `time_bucket` DateTime64(0), `original_expiry_time_bucket` DateTime64(0), `service_name` LowCardinality(String), `resource_fingerprint` UInt64, `attribute_key` LowCardinality(String), `attribute_value` String, `attribute_type` LowCardinality(String), `attribute_count` SimpleAggregateFunction(sum, UInt64) ) AS SELECT team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, attribute_key, attribute_value, attribute_type, attribute_count FROM ( SELECT team_id AS team_id, toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket, toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket, service_name AS service_name, resource_fingerprint, mapFilter((k, v) -> ((length(k) < 256) AND (length(v) < 256)), attributes) AS attributes, arrayJoin(attributes) AS attribute, 'log' AS attribute_type, attribute.1 AS attribute_key, attribute.2 AS attribute_value, sumSimpleState(1) AS attribute_count FROM posthog_test.logs34 GROUP BY team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, attributes )
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.logs34_to_resource_attributes TO posthog_test.log_attributes2 ( `team_id` Int32, `time_bucket` DateTime64(0), `original_expiry_time_bucket` DateTime64(0), `service_name` LowCardinality(String), `resource_fingerprint` UInt64, `attribute_key` LowCardinality(String), `attribute_value` String, `attribute_type` LowCardinality(String), `attribute_count` SimpleAggregateFunction(sum, UInt64) ) AS SELECT team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, attribute_key, attribute_value, attribute_type, attribute_count FROM ( SELECT team_id AS team_id, toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket, toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket, service_name AS service_name, resource_fingerprint, arrayJoin(resource_attributes) AS attribute, 'resource' AS attribute_type, attribute.1 AS attribute_key, attribute.2 AS attribute_value, sumSimpleState(1) AS attribute_count FROM posthog_test.logs34 GROUP BY team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, resource_attributes )
CREATE TABLE IF NOT EXISTS posthog_test.kafka_logs_avro ( `uuid` String, `trace_id` String, `span_id` String, `trace_flags` Int32, `timestamp` DateTime64(6), `observed_timestamp` DateTime64(6), `body` String, `severity_text` String, `severity_number` Int32, `service_name` String, `resource_attributes` Map(LowCardinality(String), String), `instrumentation_scope` String, `event_name` String, `attributes` Map(LowCardinality(String), String), `retention_days` Nullable(Int32), `pattern` Nullable(String), `pattern_version` Nullable(Int32) ) ENGINE = Kafka(warpstream_logs, kafka_topic_list = 'clickhouse_logs', kafka_group_name = 'clickhouse-logs-avro-new', kafka_format = 'Avro') SETTINGS kafka_skip_broken_messages = 100, kafka_num_consumers = 8, kafka_poll_timeout_ms = 3000, kafka_poll_max_batch_size = 1000, kafka_thread_per_consumer = 1, input_format_avro_allow_missing_fields = 1
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.kafka_logs34_avro_mv TO posthog_test.logs34 ( `uuid` String, `trace_id` String, `span_id` String, `trace_flags` Int32, `timestamp` DateTime64(6), `observed_timestamp` DateTime64(6), `body` String, `severity_text` String, `severity_number` Int32, `service_name` String, `instrumentation_scope` String, `event_name` String, `attributes_map_str` Map(String, String), `resource_attributes` Map(String, String), `team_id` Int32, `original_expiry_timestamp` DateTime64(6), `_partition` UInt64, `_topic` LowCardinality(String), `_offset` UInt64, `_record_count` Int64, `_bytes_uncompressed` Nullable(Int64), `_bytes_compressed` Nullable(Int64), `pattern` String, `pattern_version` UInt8 ) AS SELECT uuid, trace_id, span_id, trace_flags, timestamp, observed_timestamp, body, severity_text, severity_number, service_name, instrumentation_scope, event_name, mapSort(mapApply((k, v) -> (concat(k, '__str'), JSONExtractString(v)), attributes)) AS attributes_map_str, mapSort(mapApply((k, v) -> (k, JSONExtractString(v)), resource_attributes)) AS resource_attributes, toInt32OrZero(_headers.value[indexOf(_headers.name, 'team_id')]) AS team_id, observed_timestamp + toIntervalDay(if((retention_days IS NOT NULL) AND (retention_days > 0), retention_days, toInt32OrDefault(_headers.value[indexOf(_headers.name, 'retention-days')], toInt32(15)))) AS original_expiry_timestamp, _partition, _topic, _offset, toInt64OrDefault(_headers.value[indexOf(_headers.name, 'record_count')], toInt64(1)) AS _record_count, toInt64OrNull(_headers.value[indexOf(_headers.name, 'bytes_uncompressed')]) / _record_count AS _bytes_uncompressed, toInt64OrNull(_headers.value[indexOf(_headers.name, 'bytes_compressed')]) / _record_count AS _bytes_compressed, ifNull(pattern, '') AS pattern, toUInt8(ifNull(pattern_version, 0)) AS pattern_version FROM posthog_test.kafka_logs_avro
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.kafka_logs_avro_kafka_metrics_mv TO posthog_test.logs_kafka_metrics ( `_partition` UInt32, `_topic` String, `max_offset` SimpleAggregateFunction(max, UInt64), `max_observed_timestamp` SimpleAggregateFunction(max, DateTime64(6)), `max_timestamp` SimpleAggregateFunction(max, DateTime64(6)), `max_created_at` SimpleAggregateFunction(max, DateTime), `max_lag` SimpleAggregateFunction(max, Decimal(18, 6)) ) AS SELECT _partition, _topic, maxSimpleState(_offset) AS max_offset, maxSimpleState(observed_timestamp) AS max_observed_timestamp, maxSimpleState(timestamp) AS max_timestamp, maxSimpleState(now()) AS max_created_at, maxSimpleState(now() - observed_timestamp) AS max_lag FROM posthog_test.logs34 GROUP BY _partition, _topic
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.kafka_logs_avro_billing_metrics_mv TO posthog_test.logs_billing_metrics ( `team_id` Int32, `time_bucket` DateTime, `service_name` LowCardinality(String), `bytes_uncompressed` SimpleAggregateFunction(sum, Float64), `bytes_compressed` SimpleAggregateFunction(sum, Float64), `record_count` SimpleAggregateFunction(sum, UInt64) ) AS SELECT team_id, time_bucket, service_name, sumSimpleState(_bytes_uncompressed) AS bytes_uncompressed, sumSimpleState(_bytes_compressed) AS bytes_compressed, sumSimpleState(1) AS record_count FROM ( SELECT team_id, toStartOfInterval(timestamp, toIntervalMinute(1)) AS time_bucket, service_name AS service_name, _bytes_uncompressed, _bytes_compressed FROM posthog_test.logs34 ) GROUP BY team_id, time_bucket, service_name
- all
- US, EU, DEV
- logs
CREATE TABLE IF NOT EXISTS posthog_test.logs34 ( `time_bucket` DateTime MATERIALIZED toStartOfDay(timestamp), `original_expiry_timestamp` DateTime64(6), `uuid` String, `team_id` Int32, `trace_id` String, `span_id` String, `trace_flags` Int32, `timestamp` DateTime64(6) CODEC(DoubleDelta), `observed_timestamp` DateTime64(6), `created_at` DateTime64(6) MATERIALIZED now(), `body` String, `severity_text` LowCardinality(String), `severity_number` Int32, `service_name` LowCardinality(String), `resource_attributes` Map(LowCardinality(String), String), `resource_fingerprint` UInt64 MATERIALIZED cityHash64(resource_attributes), `instrumentation_scope` String, `event_name` String, `attributes_map_str` Map(LowCardinality(String), String), `level` String ALIAS severity_text, `mat_body_ipv4_matches` Array(String) ALIAS extractAll(body, '(\d\.((25[0-5]|(2[0-4]|1(0, 1)[0-9])(0, 1)[0-9])\.)(2, 2)([0-9]))'), `time_minute` DateTime ALIAS toStartOfMinute(timestamp), `attributes` Map(LowCardinality(String), String) ALIAS mapApply((k, v) -> (left(k, -5), v), attributes_map_str), `attributes_map_float` Map(LowCardinality(String), Float64) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__float'), toFloat64OrNull(v)), attributes_map_str)), `attributes_map_datetime` Map(LowCardinality(String), DateTime64(6)) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__datetime'), parseDateTimeBestEffortOrNull(v, 6)), attributes_map_str)), `_partition` UInt32, `_topic` String, `_offset` UInt64, `_bytes_uncompressed` UInt64, `_bytes_compressed` UInt64, `_record_count` UInt64, `pattern` String, `pattern_version` UInt8, INDEX idx_severity_text_set severity_text TYPE set(10) GRANULARITY 1, INDEX idx_attributes_str_keys mapKeys(attributes_map_str) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_attributes_str_values mapValues(attributes_map_str) TYPE bloom_filter(0.001) GRANULARITY 1, INDEX idx_mat_body_ipv4_matches mat_body_ipv4_matches TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_body_ngram3 lower(body) TYPE ngrambf_v1(3, 25000, 2, 0) GRANULARITY 1, INDEX idx_uuid_bloom uuid TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_observed_minmax observed_timestamp TYPE minmax GRANULARITY 1, INDEX idx_timestamp_minmax timestamp TYPE minmax GRANULARITY 1, PROJECTION projection_aggregate_counts ( SELECT team_id, time_bucket, toStartOfMinute(timestamp), service_name, severity_text, resource_fingerprint, count() AS event_count GROUP BY team_id, time_bucket, toStartOfMinute(timestamp), service_name, severity_text, resource_fingerprint ) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.logs34', '{replica}-{shard}') PARTITION BY toDate(original_expiry_timestamp) PRIMARY KEY (team_id, time_bucket, service_name, resource_fingerprint, severity_text, timestamp) ORDER BY (team_id, time_bucket, service_name, resource_fingerprint, severity_text, timestamp) TTL original_expiry_timestamp SETTINGS allow_experimental_reverse_key = 1, index_granularity_bytes = 104857600, index_granularity = 8192, ttl_only_drop_parts = 1, add_minmax_index_for_numeric_columns = 1, map_serialization_version = 'with_buckets'
CREATE TABLE IF NOT EXISTS posthog_test.log_attributes2 ( `team_id` Int32, `time_bucket` DateTime64(0), `service_name` LowCardinality(String), `resource_fingerprint` UInt64 DEFAULT 0, `attribute_key` LowCardinality(String), `attribute_value` String CODEC(ZSTD(5)), `attribute_count` SimpleAggregateFunction(sum, UInt64), `attribute_type` LowCardinality(String) DEFAULT 'log', `original_expiry_time_bucket` DateTime DEFAULT now(), INDEX idx_attribute_key attribute_key TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_attribute_value attribute_value TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_attribute_key_n3 attribute_key TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 1, INDEX idx_attribute_value_n3 attribute_value TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 1 ) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.log_attributes2', '{replica}-{shard}') PARTITION BY toDate(original_expiry_time_bucket) ORDER BY (team_id, attribute_type, time_bucket, resource_fingerprint, attribute_key, attribute_value) TTL original_expiry_time_bucket SETTINGS deduplicate_merge_projection_mode = 'drop', index_granularity = 8192
CREATE TABLE IF NOT EXISTS posthog_test.logs_billing_metrics ( `team_id` Int32, `time_bucket` DateTime64(0), `service_name` LowCardinality(String), `bytes_uncompressed` SimpleAggregateFunction(sum, UInt64), `bytes_compressed` SimpleAggregateFunction(sum, UInt64), `record_count` SimpleAggregateFunction(sum, UInt64) ) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.logs_billing_metrics', '{replica}-{shard}') PARTITION BY toYYYYMM(time_bucket) ORDER BY (team_id, time_bucket, service_name) SETTINGS deduplicate_merge_projection_mode = 'rebuild', index_granularity = 8192
CREATE TABLE IF NOT EXISTS posthog_test.logs_kafka_metrics ( `_partition` UInt32, `_topic` String, `max_offset` SimpleAggregateFunction(max, UInt64), `max_observed_timestamp` SimpleAggregateFunction(max, DateTime64(9)), `max_timestamp` SimpleAggregateFunction(max, DateTime64(9)), `max_created_at` SimpleAggregateFunction(max, DateTime64(9)), `max_lag` SimpleAggregateFunction(max, UInt64) ) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.logs_kafka_metrics', '{replica}-{shard}') ORDER BY (_topic, _partition)
CREATE OR REPLACE TABLE posthog_test.logs_distributed AS posthog_test.logs34 ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'logs34')
CREATE OR REPLACE TABLE posthog_test.log_attributes_distributed AS posthog_test.log_attributes2 ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'log_attributes2')
CREATE TABLE IF NOT EXISTS posthog_test.logs_billing_metrics_distributed AS posthog_test.logs_billing_metrics ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'logs_billing_metrics')
CREATE TABLE IF NOT EXISTS posthog_test.logs_kafka_metrics_distributed AS posthog_test.logs_kafka_metrics ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'logs_kafka_metrics')
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.logs34_to_log_attributes TO posthog_test.log_attributes2 ( `team_id` Int32, `time_bucket` DateTime64(0), `original_expiry_time_bucket` DateTime64(0), `service_name` LowCardinality(String), `resource_fingerprint` UInt64, `attribute_key` LowCardinality(String), `attribute_value` String, `attribute_type` LowCardinality(String), `attribute_count` SimpleAggregateFunction(sum, UInt64) ) AS SELECT team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, attribute_key, attribute_value, attribute_type, attribute_count FROM ( SELECT team_id AS team_id, toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket, toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket, service_name AS service_name, resource_fingerprint, mapFilter((k, v) -> ((length(k) < 256) AND (length(v) < 256)), attributes) AS attributes, arrayJoin(attributes) AS attribute, 'log' AS attribute_type, attribute.1 AS attribute_key, attribute.2 AS attribute_value, sumSimpleState(1) AS attribute_count FROM posthog_test.logs34 GROUP BY team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, attributes )
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.logs34_to_resource_attributes TO posthog_test.log_attributes2 ( `team_id` Int32, `time_bucket` DateTime64(0), `original_expiry_time_bucket` DateTime64(0), `service_name` LowCardinality(String), `resource_fingerprint` UInt64, `attribute_key` LowCardinality(String), `attribute_value` String, `attribute_type` LowCardinality(String), `attribute_count` SimpleAggregateFunction(sum, UInt64) ) AS SELECT team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, attribute_key, attribute_value, attribute_type, attribute_count FROM ( SELECT team_id AS team_id, toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket, toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket, service_name AS service_name, resource_fingerprint, arrayJoin(resource_attributes) AS attribute, 'resource' AS attribute_type, attribute.1 AS attribute_key, attribute.2 AS attribute_value, sumSimpleState(1) AS attribute_count FROM posthog_test.logs34 GROUP BY team_id, time_bucket, original_expiry_time_bucket, service_name, resource_fingerprint, resource_attributes )
CREATE TABLE IF NOT EXISTS posthog_test.kafka_logs_avro ( `uuid` String, `trace_id` String, `span_id` String, `trace_flags` Int32, `timestamp` DateTime64(6), `observed_timestamp` DateTime64(6), `body` String, `severity_text` String, `severity_number` Int32, `service_name` String, `resource_attributes` Map(LowCardinality(String), String), `instrumentation_scope` String, `event_name` String, `attributes` Map(LowCardinality(String), String), `retention_days` Nullable(Int32), `pattern` Nullable(String), `pattern_version` Nullable(Int32) ) ENGINE = Kafka(warpstream_logs, kafka_topic_list = 'clickhouse_logs', kafka_group_name = 'clickhouse-logs-avro-new', kafka_format = 'Avro') SETTINGS kafka_skip_broken_messages = 100, kafka_num_consumers = 8, kafka_poll_timeout_ms = 3000, kafka_poll_max_batch_size = 1000, kafka_thread_per_consumer = 1, input_format_avro_allow_missing_fields = 1
CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.kafka_logs34_avro_mv TO posthog_test.logs34 ( `uuid` String, `trace_id` String, `span_id` String, `trace_flags` Int32, `timestamp` DateTime64(6), `observed_timestamp` DateTime64(6), `body` String, `severity_text` String, `severity_number` Int32, `service_name` String, `instrumentation_scope` String, `event_name` String, `attributes_map_str` Map(String, String), `resource_attributes` Map(String, String), `team_id` Int32, `original_expiry_timestamp` DateTime64(6), `_partition` UInt64, `_topic` LowCardinality(String), `_offset` UInt64, `_record_count` Int64, `_bytes_uncompressed` Nullable(Int64), `_bytes_compressed` Nullable(Int64), `pattern` String, `pattern_version` UInt8 ) AS SELECT uuid, trace_id, span_id, trace_flags, timestamp, observed_timestamp, body, severity_text, severity_number, service_name, instrumentation_scope, event_name, mapSort(mapApply((k, v) -> (concat(k, '__str'), JSONExtractString(v)), attributes)) AS attributes_map_str, mapSort(mapApply((k, v) -> (k, JSONExtractString(v)), resource_attributes)) AS resource_attributes, toInt32OrZero(_headers.value[indexOf(_headers.name, 'team_id')]) AS team_id, observed_timestamp + toIntervalDay(if((retention_days IS NOT NULL) AND (retention_days > 0), retention_days, toInt32OrDefault(_headers.value[indexOf(_headers.name, 'retention-days')], toInt32(15)))) AS original_expiry_timestamp, _partition, _topic, _offset, toInt64OrDef
- logs
…truncated. See the full SQL in the workflow logs.
andrewm4894
marked this pull request as ready for review
August 19, 2026 19:20
Contributor
🦔 Hogbox preview · ❌ build failedThe preview didn't come up for commit Previews are optional and never block merging. A failure here is often a hogland or tailnet hiccup rather than anything in your PR, so the check stays green and this comment is the status. |
Contributor
Prompt To Fix All With AI### Issue 1
products/signals/backend/temporal/agentic/scout_suggestions.py:255-257
**Split workflow IDs break single-flight**
If a scheduled scan is running when a manual refresh starts, the separate workflow ID namespaces allow both scans to run for the same team, causing duplicate model spend and letting the last completion overwrite the other generated batch.
### Issue 2
products/signals/backend/scout_harness/suggestions.py:440-442
**Refresh can erase dismissals**
If a dismissal is saved after generation reads `row.items` but before it replaces the JSON field, the generation save discards `dismissed_at`, causing a suggestion documented as dismissed across refreshes to reappear.
### Issue 3
products/signals/backend/scout_harness/suggestions.py:443-448
**Outdated fleet marked fresh**
If a scout is enabled, disabled, or deleted after the runner captures its fleet snapshot, this unconditional `FRESH` write can overwrite the receiver's stale status, causing a batch generated and validated against the old fleet to be reported as fresh.
### Issue 4
products/signals/backend/temporal/agentic/scout_suggestions.py:240-242
**Skipped children are stamped dispatched**
When `start_child_workflow` raises `WorkflowAlreadyStartedError`, the team is still appended to `dispatched` and receives a new `last_requested_at`, causing the planner to defer a scan that was not started for up to the configured refresh interval.
### Issue 5
products/signals/backend/management/commands/run_scout_suggestions.py:57
**Per-team lookup bypasses scoped manager**
This management-command lookup targets one known team but uses the cross-team manager. Use the model's `for_team(team_id)` access path so fail-closed team scoping remains the default; the same per-team pattern also appears in the new suggestion persistence helpers.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "feat(signals): pre-compute suggested sco..." | Re-trigger Greptile |
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
PR overviewThis PR adds backend support to pre-compute suggested scouts for each project through the signals suggestions runner. Three issues have been addressed, but one significant security issue remains open. Attacker-controlled event or dashboard text can influence the scan, query sensitive project data, and exfiltrate it through an allowed public PostHog ingestion endpoint. The egress allowlist should be narrowed before release. Open issues (1)
Fixed/addressed: 3 · PR risk: 8/10 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Answers the bot review on the suggestions producer: - lock the team's row for every write to the JSON batch, keep dismissed suggestions as hidden tombstones across batches that omit them, and store a batch as stale when the fleet moved during the scan - count a failure anywhere after the gates toward the breaker, not only a failed run - stamp planner state on a manual dispatch so the coordinator does not double up, and honor the kill switch and blocklist on the refresh endpoint - canonicalize flag-supplied team ids, count child-environment source configs toward eligibility, and reject custom drafts that reuse a stored skill or config name - hide suggestions whose scout is already enabled, declare the list response as one object, and read per-team rows through for_team Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…suggestions # Conflicts: # posthog/settings/web.py # products/signals/backend/migrations/max_migration.txt # products/tasks/backend/migrations/max_migration.txt # products/tasks/frontend/generated/api.zod.ts # services/mcp/src/api/generated.ts # services/mcp/src/generated/tasks/api.ts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mount GitHub as a read-only token on suggestion scans, like scout runs - normalize a blank suggested cron to null so create accepts it - keep a manual dispatch successful when its planner stamp fails - exclude null parents when a root source config marks a project set up, so tiers 3-4 keep enrolling unconfigured projects, and query memberships by their related query name - run the breaker cooldown from the last attempt, not row activity - lock the row before the receiver flips a batch to stale - refund a refresh attempt when the dispatch conflicts or fails Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
- map the scout_suggestions origin to the signals gateway product in both resolvers, with the shared routing fixture covering it, so a scan mints a token the gateway accepts instead of falling to background_agents - canonicalize the management command's team before it stamps planner state - cover SignalScoutSuggestionSet in the IDOR team-scoping semgrep rule - sort planner candidates by an ordered frozen dataclass, not a tuple Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
React Doctor found 118 issues in 43 files · 7 errors & 111 warnings. Errors
111 warnings
61 more warnings not shown. Reviewed by React Doctor for commit |
eligibility_tier: 0 in the flag payload plans only the team_allowlist, giving the rollout a rung between manual-only and the whole tier-1 fleet. Claude-Session: https://claude.ai/code/session_01F51msiecN5G8mZ9ouwBDQ3
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
- A repo-less signals task no longer falls back to the creator's personal GitHub integration; no resolution is attempted at all. - A manual suggestions refresh runs the scan as its authenticated caller instead of the resolved (possibly more privileged) member. - The aggregate suggestions API requires resource-level access, so an object grant on one scout does not open the project batch. - An impossible cron calendar drops the one item instead of failing the whole batch (and no longer 500s the config API). - Custom drafts whose name a stored skill or config has since taken are hidden at read time; canonical items keep offering to enable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| try: | ||
| fleet = await database_sync_to_async(fleet_context, thread_sensitive=False)(team.id) | ||
| sandbox_env_id = await database_sync_to_async(get_or_create_signals_sandbox_env, thread_sensitive=False)( | ||
| team.id, SIGNALS_REPORT_RESEARCH_ENV_NAME, tasks_facade.SandboxNetworkAccessLevel.TRUSTED |
Contributor
There was a problem hiding this comment.
High: Prompt injection can exfiltrate project data
The scan reads attacker-controlled event and dashboard text, while TRUSTED permits *.posthog.com. An attacker can plant instructions that make the sandbox query sensitive project data and POST it to an attacker-owned PostHog project through a public ingestion endpoint. Use a dedicated CUSTOM allowlist containing only the exact API/MCP hosts required by this scan, excluding public ingestion hosts, or otherwise prevent arbitrary shell egress.
Member
Author
There was a problem hiding this comment.
Not changing in this PR. The scan runs in the same TRUSTED research env as every scout run and report research, which read the same member-writable project text under the same posture; narrowing only the suggestions egress would leave the larger surface as it is. Fleet-wide egress policy belongs with the headless-identity follow-up from the earlier thread on this file. What this run does hold to: a read_only MCP token as one member, no MCP Store connectors, no GitHub credential, and no writes.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d3117cf45
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
skoob13
approved these changes
Sep 1, 2026
- Anchor the suggestions viewset's resource-level RBAC to the canonical team - Close the TaskRun as completed only after the batch is stored, failed otherwise - Bound the per-team scan's queue wait with a schedule_to_close timeout - Count a scout a person created as engagement in the planner - Expire an aged "nothing to suggest" batch to stale like a fresh one - Share the 100-character cron cap between the producer and the config API Claude-Session: https://claude.ai/code/session_01T9XhsgASCXhB6UYsx3rz5a
Member
Author
|
/trunk merge |
Member
Author
|
/trunk cancel |
Member
Author
|
/trunk merge |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: acb6cf6f3c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+275
to
+277
| await workflow.execute_activity( | ||
| stamp_requested_scout_suggestions_activity, | ||
| StampRequestedInput(team_ids=dispatched), |
There was a problem hiding this comment.
When this coordinator times out, is terminated, or exhausts the final stamp activity's retries after launching any child, none of the already-started teams are stamped because all writes are deferred until the entire fan-out completes. The next 30-minute tick has a different tick ID and therefore launches another paid scan for each still-overdue team, allowing duplicate runs and last-writer-wins results; stamp each child or small batch immediately after it is successfully started, as the adjacent scout coordinator does.
AGENTS.md reference: products/signals/backend/scout_harness/AGENTS.md:L25-L26
Useful? React with 👍 / 👎.
| # caller's own access rather than a possibly more privileged resolved member. | ||
| user_id = acting_user_id | ||
| if user_id is None: | ||
| user_id = await database_sync_to_async(resolve_acting_user_id_for_team, thread_sensitive=False)(team.id) |
There was a problem hiding this comment.
When a scheduled run has no explicit actor, this helper may select a GitHub integration creator or the first active organization member without checking effective access to the project. On a private project that user can lack project membership, so the task is created under an identity whose PostHog MCP reads fail TeamMemberAccessPermission, even when another active project member could run the scan; select from team.all_users_with_access() or otherwise validate effective project access. .agents/security.mdL14-L15
Useful? React with 👍 / 👎.
Comment on lines
+133
to
+137
| "items": visible_items( | ||
| row, | ||
| enabled_skill_names=enabled_skill_names(team_id), | ||
| reserved_names=reserved_scout_names(team_id), | ||
| ), |
There was a problem hiding this comment.
When a canonical skill is added to the dynamic withheld_skills holdback after a batch was generated, this read path filters only enabled and reserved names, so the stored suggestion remains visible even though the config API hides and refuses that scout. Fresh evidence beyond the earlier generation-time fix is that holdbacks are now checked while producing the batch but are not rechecked after persistence, and a flag change emits no config receiver; filter canonical items against the current holdback on every read or invalidate affected batches.
AGENTS.md reference: products/signals/backend/scout_harness/AGENTS.md:L25-L26
Useful? React with 👍 / 👎.
Comment on lines
+389
to
+393
| state_by_team = { | ||
| row.team_id: row | ||
| for row in SignalScoutSuggestionSet.all_teams.only( | ||
| "team_id", "last_requested_at", "consecutive_failures", "last_completed_at" | ||
| ) |
There was a problem hiding this comment.
When a broad rollout has created suggestion rows for many projects and the flag is later narrowed to a small tier or allowlist, this still materializes every historical SignalScoutSuggestionSet every 30 minutes before considering the current tiers map. Those rows persist after eligibility changes, so a rollback from tier 4 to a small dogfood cohort can leave each coordinator tick transferring and instantiating the full fleet table and timing out before dispatch; query only current candidate team IDs, using chunking or an adaptive path when that set is large.
AGENTS.md reference: products/signals/backend/scout_harness/AGENTS.md:L25-L26
Useful? React with 👍 / 👎.
This was referenced Sep 3, 2026
dmarticus
added a commit
that referenced
this pull request
Sep 4, 2026
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: tests-posthog[bot] <250237707+tests-posthog[bot]@users.noreply.github.com>
dmarticus
added a commit
that referenced
this pull request
Sep 4, 2026
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: tests-posthog[bot] <250237707+tests-posthog[bot]@users.noreply.github.com>
dmarticus
added a commit
that referenced
this pull request
Sep 4, 2026
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: tests-posthog[bot] <250237707+tests-posthog[bot]@users.noreply.github.com>
1 task
This was referenced Sep 16, 2026
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Changes
GET signals/scout/suggestions/returns the stored batch for the project, minus dismissed and already-created items.POST .../<id>/dismiss/hides one; dismissal is remembered by skill name across refreshes.POST .../refresh/re-runs the scan now, capped at 3 per project per day.draft_body,description, andproposed_config. The producer drops any item the one-click create could not apply as-is (unknown canonical name, bad slug, invalid cron, already enabled), so nothing stored can fail on create.ScoutSuggestionsCoordinatorWorkflowticks every 30 minutes, plans the most-overdue eligible teams, and fans out oneRunScoutSuggestionsWorkflowchild per team. The child mints aread_onlyheadless task with the new reservedSIGNALS_SCOUT_SUGGESTIONSorigin, runs the same project scan the chat button uses, and persists the parsed batch. An unparseable close-out is a failed generation, not salvaged prose.signals-scout-suggestionsflag payload, read every tick:enabled(default off, so this ships dark),eligibility_tier1-4,refresh_days,max_children_per_tick,team_allowlist/team_blocklist, failure breaker. Tier 1 is engaged self-driving projects; 4 is every AI-approved team. The queue is recomputed from Postgres each tick, never stored.SignalScoutSuggestionSet, one row per team (items as JSON), which doubles as the planner's per-team state. A scout being enabled, disabled, or deleted marks a fresh batchstale, and a batch persisted after the fleet moved is storedstaletoo.signalsmigration (one table, FK withdb_constraint=False; the neworigin_productchoice needs notasksmigration since that field uses callable choices), theSCOUT_PROJECT_SCAN_GUIDANCEconstant shared between the chat prompt and the headless prompt, the cron rule lifted intoconfig_registry.cron_schedule_errorso the create API and the producer validate schedules identically, theConfidenceTierEnumoverride (growth's identity-matching tier shares the same low/medium/high set, so its generated type is renamed), regenerated OpenAPI/MCP types, management commandrun_scout_suggestions, and AGENTS.md updates.How did you test this code?
products/signals/backend/test/test_scout_suggestions.py: planner tiers / ordering / cap / allowlist / breaker (a regression here spends on the wrong projects or none); draft validation drops every shape create cannot apply; dismissal carries across a refresh and created items hide; a failed generation keeps prior items and counts toward the breaker; enabling a scout marks the batch stale; runner persists a validated batch withread_onlyscopes and no prose fallback, records a failed generation, and skips an unapproved org before any spend; API list / dismiss / refresh gating; dismissal tombstones across an omitting batch; stale-on-fleet-move; refresh 403 on the kill switch and blocklist; child-environment source configs and flag ids canonicalize to the project.enabled: trueis set; first dogfood target is project 2 viarun_scout_suggestions --team-id 2.Automatic notifications
Docs update
None yet. The user-facing surface lands with the follow-up PR;
products/signals/backend/scout_harness/AGENTS.mdandmanagement/AGENTS.mdare updated here.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
scouts-workskill store (idea 24). Skills invoked:/django-migrations,/improving-drf-endpoints,/writing-tests,/writing-code-comments,/writing-dataclasses,/writing-pr-descriptions,/simplify.suggestions_runner.pyso receivers and routes import the light module; the origin value shortened toscout_suggestionsto fit the 20-charorigin_productcolumn; the run stamped as the Scout built-in agent with an empty Store allowlist so it mounts no team connectors.