Skip to content

feat(workflows): report every committed workflow creation - #127193

Merged
trunk-io[bot] merged 10 commits into
masterfrom
workflows-tracking/creation-event
Nov 3, 2026
Merged

trunk-io[bot] merged 10 commits into
masterfrom
workflows-tracking/creation-event

Conversation

@Silthus

@Silthus Silthus commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • The Workflows team counts workflow creation with the user event hog_flow_created. Only the API create view and premade data-suggestion saves (#119724) emit it.
  • Workflows created any other way never count: generated data-suggestion drafts that a Celery task saves for PostHog AI, transactional email workflows (#122019), and plain ORM writes.
  • So a per-team or per-organization creation count misses every creation that skips those two code paths. A read-only production read on 2026-11-02 confirmed that recent workflows exist with no matching hog_flow_created; the counts stay in private proof storage.

Impact

Closes a gap in two Grow Workflows leading metrics: Started orgs (organizations whose first workflow creation falls in the month) and activation rate (the share of started orgs whose first billed action comes within 30 days of that creation). Both are defined on the first hog_flow_created, so an organization whose first workflow came from a transactional email workflow or an accepted data-suggestion draft is counted late or never. Once these metrics read workflows workflow created, they count every creation path. Expected move: no change in product usage; Started orgs stops undercounting, and activation rate is measured from the real first creation.

Merge Danger

Warning

🟡 Merge risk: medium · two-way door (merge-risk rubric)

  • Every committed workflow creation now enqueues one Celery task on the default queue after commit. The worst case is silent: creation events lost or wrong in internal analytics, which Started orgs and activation rate would read. Workflow creators reach: some teams.
  • The enqueue sits in a try/except behind on_commit, so it never fails or rolls back a creation, and the upload no longer runs in the request. A failed upload retries up to five times with backoff; the stable event uuid and creation timestamp keep a retry from counting twice. During the deploy, a new web pod can enqueue the task before an old worker knows it; those few events are lost, and nothing else breaks.
  • Revert: git revert the merge commit. Events already sent stay sent and are correct records.

Changes

Kind Files Lines
Production 2 +65 −1
Tests 2 +193 −1
  • Analysts can count each committed workflow creation with the team event workflows workflow created, whatever path created it.
    • Properties: workflow_id, origin_product, actions_count. The event carries the project and organization groups.
    • API and premade creations emit both events. Count teams and organizations with workflows workflow created; keep hog_flow_created for user and channel attribution.
  • The HogFlow post_save receiver enqueues the report_workflow_created Celery task with transaction.on_commit on the signal's own connection. The request never touches the analytics client.
    • Updates emit nothing. A rolled-back creation emits nothing.
    • The event carries the workflow's created_at as its timestamp and a uuid5 of the workflow id as its uuid, so a delayed task keeps the creation in its month and a replayed task does not count it twice.
  • The task sends through ph_scoped_capture, which flushes on exit. The global client can lose events in a Celery worker (AGENTS.md forbids it there), and the generated data-suggestion save runs in one.
    • The scoped client sends only in cloud run modes, so self-hosted and plain local dev emit nothing. Local E2E mode counts as cloud and sends events tagged region: E2E to the US project, so per-team counts should filter region to US and EU.
    • The event carries the instance region (US or EU) as its region property, like events from the global client.
  • A failed enqueue logs an exception with the team and workflow ids and never fails the creation. In the task, the scoped client runs with raise_on_error=True, so an upload error the SDK would otherwise drop with its own log line reaches Celery. The task retries it up to five times with jittered exponential backoff (60 s base, 600 s cap), like the neighbouring SES tasks, and then fails where Celery records it. A missing team fails at once, without retries.
  • Nothing user-visible changes.

Recreates #115349 from a branch in this repository so CI can run; before that Silthus/posthog#358 and #111727 (closed).

How did you test this code?

Test rationale: two seams.

  • The model seam every creation path passes (TestHogFlow in products/workflows/backend/test/test_hog_flow.py): a creation enqueues the task exactly once, after commit, with the values at creation time, and never builds an analytics client in the request; a failed enqueue is logged and the workflow still exists; a rolled-back creation enqueues nothing.
  • The task (products/workflows/backend/test/test_report_workflow_created.py): the event goes out through the scoped client with the instance region, creation timestamp, groups and properties; a replay sends the same event uuid; a failed upload is retried and the retry delivers the same event; an upload that keeps failing stops after five retries; a missing team fails after one attempt. The upload cases drive the real ph_scoped_capture and the task's real Celery retry config through apply(), with only the SDK client faked.

The new tests failed before each change: the task tests before the task existed, and the two retry tests before the retry config. The tests patch ph_scoped_capture or the task's delay, as the repo's other callers do.

Live run at c1b101c (this PR's head), through the real DRF API with a logged-in client on a non-atomic private test database, so every request commits and on_commit fires as in production. posthog.ph_client.is_cloud was forced to True and Posthog.capture was replaced by a recorder, so nothing reached PostHog Cloud. Test settings run Celery eagerly, so the enqueued task ran in the same process; the recorder checked that each capture ran inside report_workflow_created:

Step workflows workflow created events Task enqueues
PATCH a workflow written before the run 0 0
POST a new workflow (origin_product: broadcasts, one action) 1 1
PATCH the new workflow 0 0
ORM create outside the API 1 1

Both events came from the task, carried the team's UUID as distinct_id, the project and organization groups, the right properties, the workflow's created_at as timestamp and a uuid5 of the workflow id. They went to the scoped client for us.i.posthog.com with region: US; the global client saw no creation event.

Lane gate on c1b101c: ruff check, ruff format --check, ty check, lint-imports, test_hog_flow.py and test_report_workflow_created.py (14 passed), the repo guard tests in posthog/test/repo_invariants (289 passed), repo-wide mypy, and hogli ci:preflight --strict, all exit 0.

Proofs owed, per the agent proof ledger:

  • Live run: above, at c1b101c.
  • Production shape: at c1b101c, a read-only read of posthog_hogflow joined to hog_flow_created showed which creation paths lack the event. The counts stay private.
  • Post-deploy watch: owed after merge.

Gaps:

  • Delivery to PostHog analytics is unverified; the live run stops at the SDK client.
  • Cross-version: while old and new pods run side by side in a deploy, an old Celery worker discards the new task as unregistered, so creation events queued in that window are lost; creation itself is unaffected. Not staged into a separate PR; see Decisions.
  • A real Celery worker was not run; test settings execute the task eagerly in the request process, and eager retries run at once instead of after the backoff.
  • The Celery data-suggestion save was not live-run; it reaches the same receiver as the ORM create above.

Release status

  • No feature flag controls this change

Docs update

None. No existing doc covers Workflows analytics events.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: T3 Code with Claude Code, Claude Opus 5.5 (claude-opus-5-5). An earlier Codex worker (GPT-6.1 Sol) recreated the change test-first before its session dropped.

Question Answer Who Where
Keep this PR after closing the creation-intent PR? Keep it: product intents feed neither the Workflows funnel nor the goal tree, while Started orgs and activation rate need every creation counted; #115350 closed 👤 Human #115349
Which client sends the creation event? A report_workflow_created Celery task enqueued from on_commit, sending through ph_scoped_capture; replaces capturing inside the request's on_commit callback, which held the create response (PostHog Review P2) 🤖 Agent decisions
Second event on API creations, or extend hog_flow_created to every path? A separate team event below every path; hog_flow_created stays for user and channel attribution 🤖 Agent #115349
Event time and id for a queued send? The workflow's created_at as timestamp and a uuid5 of the workflow id as uuid, so delay keeps the month and replay dedupes 🤖 Agent decisions
Failure counter and alert for lost creation events? No: a failed enqueue logs with team and workflow ids, and a failed upload is retried, then fails the task; a pod alert on PostHog's own analytics ingestion would page on blips 🤖 Agent decisions
Retry a failed upload? Yes: five Celery retries with jittered exponential backoff, like the SES tasks; the stable uuid keeps retries idempotent, and a missing team fails at once (PostHog Review P2) 🤖 Agent decisions
Stage the new task's registration before its publisher? No: an old worker discards the task during a deploy, losing that window's creation events but never a creation; new tasks ship with their publisher in one PR (#125338) 🤖 Agent decisions

Involvement: merged master on 2026-11-02 three times with no conflicts; two QA panel rounds and two final reads on the fork PR; on this PR one review round (GPT-6.1 Sol: upload failures fixed, metric and alert declined) and four final reads (creation timestamp and replay-safe uuid fixed; two findings in master-only files rejected; the last two found nothing); two PostHog Review P2 threads fixed, answered and resolved (send moved into a Celery task; bounded retries); live run and production-shape read captured at the head.

  • Skills: modernize-pr, qa-swarm, tdd, babysit-pr, merge-risk, pr-description, writing-voice.
  • Public artifact: nothing from the session beyond this repository went into the diff or this description.

🤖 Generated with Claude Code

Silthus and others added 4 commits October 31, 2026 12:04
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A data suggestion saved by a Celery task creates its workflow inside the
worker, where the global analytics client can lose queued events when the
child exits. Capture through the scoped client, which flushes on exit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g/creation-event

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

😎 Merged successfully - details.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

✅ Duplication (Python) — clean

New Python code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Duplication (TypeScript) — clean

New TypeScript code duplication introduced by this branch. Fails at 70+ tokens in app code, or 150+ tokens when both copies live in test files. Advisory while the gate proves itself: extract a shared helper instead of copying.

✅ Hogbox preview — ready, open the preview

▶ Open the preview

🔑 Login test@posthog.com / 12345678 (demo data)
🧩 Running this PR's backend on the PostHog :master base (frontend unchanged by this PR)
🔗 Link stable across rebuilds: a re-push swaps the box underneath, the URL stays
🔒 Access tailnet only (PostHog VPN)
🛠️ Admin inspect & debug state in hogland
💤 Idle sleeps after ~30 min idle (snapshot to S3, zero node cost) and wakes on your next visit in ~30s, behind a brief "waking up" screen

commit c1b101c · box box-7b254b842218 · ready in 2159s (push → usable) · build log · rebuilds on every push, torn down on close

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed fd4aa91: 1 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

@hosthog

hosthog Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

HostHog preview — posthog-desktop-web

The previews for this PR have been torn down and no longer serve.

Silthus and others added 4 commits November 2, 2026 09:09
ph_scoped_capture swallowed SDK upload errors by default, so a lost creation event left no log with its workflow. Raise them at exit into the existing handler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The on_commit callback ran ph_scoped_capture in the request, and its
shutdown() waits through upload retries, so a stalled analytics ingest
held a committed workflow's create response. The callback now only
enqueues report_workflow_created; a failed enqueue is logged and never
fails the create.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stable uuid

A queued task would otherwise record its run time, which can move a creation
into the next month, and a replayed task would count the creation twice.
The task now sends the workflow's created_at as the event timestamp and a
uuid5 of the workflow id as the event uuid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decisions from the review rounds on this PR:

  • 🤖 Which client sends the creation event? A report_workflow_created Celery task, enqueued from on_commit, sends it through ph_scoped_capture. This replaces capturing inside the request's on_commit callback, which held the create response while the SDK drained its upload retries (PostHog Review P2 on 5759eb3).
  • 🤖 Event time and id for a queued send? The workflow's created_at is the event timestamp, and a uuid5 of the workflow id is the event uuid, so a backed-up queue keeps the creation in its month and a replayed task does not count it twice.
  • 🤖 Failure counter and alert for lost creation events? No. A failed enqueue logs with the team and workflow ids, and a failed upload is retried, then fails the Celery task where Celery records it. A pod alert on PostHog's own analytics ingestion would page on blips, so the reviewer's metric ask was declined.
  • 🤖 Retry a failed upload? Yes: up to five Celery retries with jittered exponential backoff (60 s base, 600 s cap), like the neighbouring SES tasks. The stable event uuid and creation timestamp keep each retry idempotent, and a missing team fails at once (PostHog Review P2 on eca322e).
  • 🤖 Stage the new task's registration before its publisher? No. While old and new pods run side by side in a deploy, an old worker discards the new task as unregistered, so creation events from that window are lost; creation itself is unaffected. PostHog ships a new task with its first publisher in one PR (feat(error-tracking): sync issue status from linear tickets #125338), and a second release cycle is not worth one deploy window of analytics events.

@Silthus Silthus added the hogbox-preview Spin up a per-PR PostHog preview on a hogland hogbox label Nov 2, 2026
@Silthus
Silthus marked this pull request as ready for review November 2, 2026 12:54
@parameterai

parameterai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Risk: No findings

The change since the last review adds bounded autoretry (5 attempts, 60–600s backoff, Team.DoesNotExist excluded) to the internal report_workflow_created analytics task, with tests. I traced the retry path end to end: the deterministic uuid5 event id makes retries replay-safe, the retry loop has a cap and terminal state, and no attacker-controlled input reaches the task. No new security risk.

Sentinel reviewed c1b101c · Review settings

@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team November 2, 2026 12:55

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed eca322e: 1 Should fix.

Silthus and others added 2 commits November 2, 2026 13:02
When analytics ingestion stayed down past the SDK's own retries, the task
failed once and the committed creation never reached the metrics. It now
retries up to five times with jittered exponential backoff; the stable
event uuid and creation timestamp keep each retry idempotent. A missing
team still fails at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@PostHog review

@posthog

posthog Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@Silthus PostHog Review is reviewing this pull request.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed c1b101c: no findings.

@Silthus Silthus added the stamphog Request AI approval (no full review) label Nov 3, 2026

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

The write-path changes have independent assurance from the owning-team author and a clean current-head review. No outstanding substantive concerns or policy refusal grounds were identified.

  • PostHog Review (Standard) reviewed the current head: no findings.
  • posthog[bot] reviewed the current head.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 66L, 2F substantive, 260L/4F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1c-medium (260L, 4F, single-area, feat)
stamphog 2.7.0 .stamphog/policy.yml @ c1b101c · reviewed head c1b101c

@Silthus

Silthus commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/trunk merge

@trunk-io
trunk-io Bot merged commit 1903bd3 into master Nov 3, 2026
316 checks passed
@trunk-io
trunk-io Bot deleted the workflows-tracking/creation-event branch November 3, 2026 11:23
@deployment-status-posthog

deployment-status-posthog Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-11-03 11:48 UTC Run
prod-us ✅ Deployed 2026-11-03 12:06 UTC Run
prod-eu ✅ Deployed 2026-11-03 12:09 UTC Run

This branch was successfully deployed

1 active deployment
hogbox-preview — c1b101c2 Deployed Nov 2, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hogbox-preview Spin up a per-PR PostHog preview on a hogland hogbox stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant