AI decision step: decide route and action type in Django, reviving #107832

Part of #240
Spec: #247 (Spec doc: AI decision step for workflows). Sections: "What is kept, changed, and dropped from PostHog#107832", "The action type", "The Django route", "Feature flag".

What to build

Revive PostHog/posthog#107832 ("Classify with Jev") on current upstream/master and reshape it into the Django half of the AI decision step: the ai_decision action type saves and validates, and the decide route answers a decision with every gate in place.

Carry the commits. Cherry-pick the six commits of feat(workflows): add a classify with jev workflow action PostHog/posthog#107832 (ea15ce462, e0e0bbba1, 4e452db49, 0149eaade, 9e0267ae8, 5af614295; branch workflows/jev-classify-step on upstream) onto a branch cut from upstream/master. Upstream moved views to products/workflows/backend/presentation/views/ and tests to products/workflows/backend/tests/api/, so relocate those files during the pick. Keep the original authorship on the carried commits. Every reshape commit carries Co-authored-by: Harley Alexander <harley.a@posthog.com>.
Reshape, per the spec's kept/changed/dropped table:
- Rename classify to AI decision: route workflow_ai_decisions, view WorkflowAIDecisionViewSet in presentation/views/workflow_ai_decisions.py, JWT purpose WORKFLOW_AI_DECISION, audience posthog:workflows:ai_decision in posthog/jwt.py and its mirror in nodejs/src/cdp/utils/jwt-utils.ts, secret WORKFLOW_AI_DECISION_JWT_SECRET(S) in posthog/settings/data_stores.py and nodejs/src/cdp/config.ts.
- Replace build_system_one_client with ml_inference decide_when_available (timeout_seconds=5, ai_product="workflows", privacy_mode=True, trace_id = invocation id, properties = hog_flow_id, action_id). noul for yes or no, choice for pick one, the question verbatim as instructions.
- Gates in the spec's order: flag workflows-ai-decision (local evaluation, organization group), AI data processing approval, is_team_over_ai_credit_budget, the 8,192-byte state cap, per-team then global token_bucket.consume admission (budgets as env-overridable settings with the spec's starting values).
- The status contract and terminal codes from the spec (200 envelopes, 429 and 503 with Retry-After).
- Add ai_decision to SUPPORTED_ACTION_TYPES, with save-time validation that reuses the route serializer. Compile config.inputs.context through HogFlowConfigFunctionInputsSerializer with a fixed server-side schema (context, dictionary, Hog templating). The question, options, and descriptions stay plain config fields.
- graph_validation.py _branch_slot_count for ai_decision (answers plus Unsure), and a strict save requires every answer edge.
- The serializer rejects a new ai_decision action when the flag is off and keeps stored ones.
- Add an optional mock_answer to the hog flow test invocation request and pass it through to the CDP. AI decision step: Node runtime that asks, branches and reschedules #249 uses it.
- Drop the function template, its bytecode fixture, postHogClassify, the callInternalApi timeoutMs change, the flag_gated_templates.py entry, and the frontend flag and toolbar node from feat(workflows): add a classify with jev workflow action PostHog/posthog#107832.
- Serializer help_text on every new field, then hogli build:openapi.

Acceptance criteria

A strict save of an ai_decision step with valid config and every answer edge returns 201. A save with 1 or 17 options, a duplicate option name, a 2,001-character question, a missing answer edge, or no_threshold at or above yes_threshold returns 400 with a field error.
A save that sends templating on the question cannot make the question render: the question is not an input.
With the flag off for the organization, a new ai_decision action is rejected and a stored one still saves.
The route returns a 200 decision with probabilities for both answer types, and the stubbed facade receives ai_product="workflows", privacy_mode=True, the trace id, and the user text only in state.
The route returns each terminal code in the spec for its cause, 429 with Retry-After when admission is denied or the gateway returns 429, and 503 for a gateway timeout, a gateway 5xx, or Redis unavailable.
A token for another audience gets 401.
No log line holds the state or a gateway body.
OpenAPI, generated clients, and MCP schema snapshots are regenerated and committed.

Test seam

Spec seams 1 and 2: the Django route contract with the DRF test client and the facade stubbed (extend PostHog#107832's test_workflow_classifications_api.py, renamed), and hog flow save validation in test_hog_flow.py, parameterized.

Base branch

New branch feat/workflows-ai-decision-route cut from upstream/master. This is the base of the stack; #249 (Node) and #250 (builder) both stack on it.

Cast

GPT-6.1 Sol through codex:codex-rescue (bounded backend work) when Codex answers, else Opus 5.5.

Mandatory skills

/tdd, /improving-drf-endpoints, /implementing-mcp-tools, /writing-tests, /writing-user-facing-copy (terminal error messages), /writing-code-comments, /writing-pr-descriptions, then qa-swarm.

Claimed by background implementation worker (Claude Code, Opus 5.5). Codex substituted: it returned 429 at dispatch. CodeRabbit slot reserved. Wall-time cap 240 minutes.

Progress (background worker, 2h30 into the 240-minute cap).

Draft PR: feat(workflows): add the ai_decision action type and its decide route PostHog/posthog#111612 (branch feat/workflows-ai-decision-route on the fork, head 6d4c21b).
The six feat(workflows): add a classify with jev workflow action PostHog/posthog#107832 commits are carried with their original authorship, then reshaped into the ai_decision action type and the workflow_ai_decisions route per the spec.
qa-swarm: three rounds done, every verified finding fixed (ledger kept locally). Round 4 is next on 6d4c21b.
Local gate on 6d4c21b: workflows API suite green, repo-wide mypy clean, hogli ci:preflight --strict no failures, hogli lint:tach and hogli product:lint workflows clean, diff-aware semgrep 0 findings. Fork CI is held for maintainer approval.

Next step: round 4 panel, then the resolution comment and close.

Stopping at the 240-minute wall-time cap. Not resolved yet, so the ticket stays open.

State

Draft PR: feat(workflows): add the ai_decision action type and its decide route PostHog/posthog#111612, branch feat/workflows-ai-decision-route on the fork, head f078cb3.
The six feat(workflows): add a classify with jev workflow action PostHog/posthog#107832 commits are carried with Harley Alexander's authorship. Every reshape commit carries his Co-authored-by trailer.
The acceptance criteria are covered by the route contract tests and the hog flow save tests (seams 1 and 2 of the spec), plus pure graph checks.
Local gate on f078cb3: the workflows API suite is green, repo-wide mypy is clean, hogli ci:preflight --strict has no failures, hogli lint:tach and hogli product:lint workflows are clean, diff-aware semgrep has 0 findings. OpenAPI, generated clients and MCP schemas are regenerated. Fork CI is held for maintainer approval.
qa-swarm: seven rounds, every verified finding fixed. Round 7 on 1ebe67b came back clean on four of seven lenses. Its valid findings are fixed in f078cb3 (pre-admission outages log at debug, an activation test). That commit has not had a fresh round. The GPT lane hit a 429 at dispatch, so every lens ran on Claude Opus. The CodeRabbit CLI was signed out; the CodeRabbit bot's threads are fixed and resolved.

Next step

Run one more qa-swarm round on f078cb3. If it is clean and CodeRabbit's incremental review of f078cb3 adds nothing, post the resolution and close this ticket.

Calls made

503 carries no Retry-After; the Node backoff owns the wait. CodeRabbit asked for the header twice. A draft reply for that review comment needs Michael's approval and is not posted.
A gateway 429 and an admission denial spread Retry-After with jitter.
Gateway 200 unreadable, 400, 413, 422 and an answer that does not fit the options fail as model_refused; other gateway 4xx fail as gateway_unavailable; only 5xx and unreachable answer 503.
A flag the SDK cannot evaluate answers 503 at run time and blocks a save.
A strict save that sends the graph requires every answer edge, the Unsure edge when on, and the continue edge unless on_error is abort.
answer_type is required.
The decision contracts live in the workflows facade with team_id; products.workflows declares a facade-only dependency on products.ml_inference in tach.toml.

Follow-ups for the conductor

Batch scale (recommend before the flag widens): the token bucket's Retry-After names the next token, not the queue position, so a large batch polls the route at about N/2.5 requests per second and many people hit the spec's 30-minute throttled bound. Options: queue-aware admission in Django (a GCRA-style next free slot) or a per-team concurrency cap in the Node handler (AI decision step: Node runtime that asks, branches and reschedules #249).
A per-organization admission bucket, so one organization with many projects cannot fill the global bucket.
The ml_inference facade's 5 s timeout applies per httpx phase, not as a total deadline; AI decision step: Node runtime that asks, branches and reschedules #249 should size its 10 s worker timeout against that.
Keep the flag off until the Node runtime PR deploys: ai_decision is in SUPPORTED_ACTION_TYPES before its CDP handler exists.

Successor worker (Claude Code, Opus 5.5) taking over from the capped predecessor to finish the final qa-swarm round. Wall-time cap 120 minutes.

Progress (successor worker, 47 minutes in).

Round 8 on f078cb3: correctness and security clean. Fixed in ed52b0e: tests for the refund's explicit client and for the log level per cause, a sharper failure-edge assertion, and one validation path per serializer.
Round 9 on ed52b0e: correctness, security, production and adversary clean. Fixed in 21b01e2: an order-dependent test (reproduced: 2 failed when run first, 2 passed after the fix), a docstring that overclaimed, a header comment, and Any to object.
Local gate on 21b01e2: 967 tests passed (workflows API suite plus token bucket), repo-wide mypy clean, hogli ci:preflight --strict 0 failures, diff-aware semgrep 0 findings.
Round 10 is running on 21b01e2. Codex returned 429 again, so every lens runs on Claude Opus.

Progress (successor worker, 88 minutes in).

Round 10 on 21b01e2: fixed in f4d1ff7. An unreadable gateway 200 now logs at warning as a gateway contract break. New tests cover a token for another team, the stored config defaults, and a blank context name.
Round 11 on f4d1ff7 found a CI blocker, fixed in 115d9d7. The workflows facade imported the decision service at module level, so django.setup() loaded the openai and anthropic SDKs, and posthog/test/repo_invariants/test_startup_import_budget.py failed (2 failed, reproduced). The facade now imports the service at call time. Also fixed: tests for the log level of every cause and for the Retry-After bounds.
Local gate on 115d9d7: 1,138 passed, which includes posthog/test/repo_invariants/ (the earlier gates left it out). Repo-wide mypy is clean, hogli lint:tach and hogli product:lint workflows are clean, diff-aware semgrep has 0 findings, and pre-push preflight has 0 failures.
Round 12 is running on 115d9d7. If it comes back clean, the resolution follows. If it does not, the next step goes in a comment before the cap.

Stopping at the successor's 120-minute wall-time cap. Not resolved yet, so the ticket stays open.

State

Draft PR: feat(workflows): add the ai_decision action type and its decide route PostHog/posthog#111612, branch feat/workflows-ai-decision-route on the fork, head 791565f. The PR body is updated to this state.
This worker ran rounds 8 to 12. Every verified finding is fixed except the two open items below. Each fix went red first (by mutation or by a failing case), then green.
Fixes by round:
- Round 8 (ed52b0e): tests only.
- Round 9 (21b01e2): an order-dependent test.
- Round 10 (f4d1ff7): an unreadable gateway 200 logs at warning.
- Round 11 (115d9d7): a CI blocker. The workflows facade loaded the openai and anthropic SDKs at django.setup(), which failed test_startup_import_budget.
- Round 12 (791565f): a stored decision carrying a flag-gated template_id no longer grandfathers that template.
Local gate on 791565f: the workflows API suite, the token bucket tests and the startup import guards pass (989). Repo-wide mypy is clean, diff-aware semgrep has 0 findings, and pre-push preflight has 0 failures. The full posthog/test/repo_invariants/ passed on 115d9d7 (1,138 with the suites).
Codex returned 429 at its one attempt, so every lens ran on Claude Opus. CodeRabbit paused its reviews after 46a8b77. Fork CI is held as action_required.

Next step

Fix the open call:
- Trigger: a strict raw-API PATCH with stage_draft sends only actions.
- Problem: the early AI decision edge check compares them against the live instance.edges, not the draft's edges. That gives a false 400 for a draft that is fully wired.
- Location: hog_flow.py, around line 3242.
- Recommendation: skip the early check for staged saves, because publish re-validates the whole draft strictly. The alternative is to fall back to the instance.draft edges and actions.
Fix the nit: the test helpers _post and _patch in TestAIDecisionActionValidation default flag_enabled to opposite values. Pass it explicitly.
Run round 13 on the new head. On a clean round, post the resolution and close this ticket. The ledger is at /tmp/jev-248-qa/ledger.md, the briefs are at /tmp/jev-248-qa/r12/, and a draft resolution is at /tmp/jev-248-qa/resolution.md.

Calls made (the predecessor's, still standing)

A 503 carries no Retry-After; the Node backoff owns the wait. CodeRabbit asked twice (feat(workflows): add the ai_decision action type and its decide route PostHog/posthog#111612 (review), feat(workflows): add the ai_decision action type and its decide route PostHog/posthog#111612 (review)). Draft reply for Michael, not posted:

Leaving the 503 without Retry-After on purpose. Django can't know when the gateway or Redis comes back, so any value would be a guess. The worker's bounded backoff (2, 4, 8, 16, 32 s, per the spec) applies when the header is missing, and the 503 response schema says so.
A gateway 429 and an admission denial get a spread Retry-After.
These fail as model_refused: a gateway 200 the facade cannot read, a gateway 400, 413 or 422, and an answer that does not fit the options.
Other gateway 4xx fail as gateway_unavailable. Only 5xx and an unreachable gateway answer 503.
answer_type is required.
A flag the SDK cannot evaluate answers 503 at run time and blocks a save.
This worker's call: the config docstring now promises only what the serializer enforces. The field limits stay as the spec sets them.

Follow-ups for the conductor

Batch admission scale and a per-organization bucket (AI decision step: let large batches through admission without throttling people #251).
The ml_inference 5 s timeout applies per httpx phase. ml_inference also opens a new httpx.Client per decision, so no connection is reused.
Keep the flag off until the Node runtime PR deploys.
A combined prompt budget for the question and options against the 8,192-token window. At the spec's field limits, the config alone can exceed the window.
An ai_decision edge that loops straight back to the same step. No action type has a cycle rule today.
The shared token_bucket logs a warning per Redis error.
The flag-gated template grandfathering set ignores the action type for other non-function steps. This existed before this PR. This PR closed it for ai_decision.

Second successor worker (Claude Code, Opus 5.5) taking over at head 791565f to fix the open round-12 items and finish qa-swarm. Wall-time cap 120 minutes.

Third successor worker (Claude Code, Opus 5.5) resuming after a server restart killed the previous worker. Round-15 fixes were staged but uncommitted at 18d14d7. Wall-time cap 120 minutes.

Resolved. qa-swarm handed off ready with a clean round 20 on e6ecec8. The PR stays a draft for Michael.

PR: PostHog#111612, branch feat/workflows-ai-decision-route on the fork, clean head e6ecec875cb72.

Proof

Red then green for every behavior. The first worker's red run on the tree before the reshape gave 44 failed, 1 passed. Every later fix had one failing test, or one mutation of the code under it, before going green.
This worker's rounds:
- Round 16: the outcome counter had no failure reason label. 3 failed, then 68 passed.
- Round 17: an ai_decision with filters and on_error: abort saved without its continue edge. abort_with_filters_that_skip_people failed, then 145 passed. Mutations proved the new tests: session auth added to the route gives 1 failed; throttled counter labels swapped gives 2 failed.
- Round 18: measuring the state without surrogatepass makes lone_surrogate_from_a_split_emoji fail.
Local gate on e6ecec875cb72:
- 742 passed: test_hog_flow, test_workflow_ai_decisions_api, test_graph_validation, test_token_bucket, posthog/test/repo_invariants/
- uv run mypy --cache-fine-grained .: Success, 22584 files
- diff-aware semgrep (devex + security rules): 0 findings
- pre-push hogli ci:preflight --strict: 0 failures
No user-facing surface: this PR has no UI, so there is no screenshot.

qa-swarm

Rounds 1 to 7: first worker. Rounds 8 to 12: successor. Rounds 13 to 15: second successor. Rounds 16 to 20: this worker.
This worker committed the staged round-15 fixes as 0042840 after checking they matched the ledger: pure moves plus the mirror grandfathering test.
Rounds 16 to 18 fixed 8 items:
- Round 16: failure reason on the outcome counter (call); model_refused copy (call); a redundant Redis flush in the tests.
- Round 17: the continue edge for filtered steps; a session-user 401 test; counter rows for every outcome.
- Round 18: measure the state before repairing it; help text for the continue edge.
Rejected or follow-up: 1, the timeout double-billing risk below.
Round 19: six lenses clean. Design raised one comment-only nit: a duplicated comment, folded into the docstring in e6ecec8.
Round 20: clean. The panel was skipped because the round-19 fix only touched a comment and a docstring (qa-swarm rule). The checks and bots on e6ecec8 were reviewed.
Lanes: Claude Opus on all seven lenses every round. Codex (GPT-6.1 Sol) answered correctness once, in round 16, and found nothing. Earlier Codex attempts returned 429.
CodeRabbit paused its reviews on this PR after 46a8b77 and stayed silent on later heads. Its two threads are fixed and resolved. No unresolved threads.
Fork CI is held for maintainer approval, so 26 checks show as skipping. That is a gap, not a failure. The 6 checks that ran passed.

Calls made

503 carries no Retry-After. Django cannot know when the gateway or Redis recovers, so the Node backoff owns the wait. Alternative: a fixed header value.
A gateway 429 and an admission denial answer 429 with a spread Retry-After, so a batch does not retry at once.
Gateway 200 unreadable, 400, 413, 422, and an answer that does not fit the options fail as model_refused. Other gateway 4xx fail as gateway_unavailable. Only 5xx and unreachable answer 503. Alternative: retry other 4xx as 503.
An unreadable gateway 200 logs at warning, as a gateway contract break.
answer_type is required. Alternative: infer it from the presence of options.
A flag the SDK cannot evaluate answers 503 at run time and blocks a save.
A strict save that sends the graph requires every answer edge, the Unsure edge when it is on, and the continue edge.
- Only a step without filters and with on_error: abort may leave continue out, because the runtime follows it for a person the filters skip (round 17).
- The check uses filters is not None, the same test as Node's shouldSkipAction. Empty filters match everyone, so this errs toward requiring the edge. Alternative: inspect the filter contents.
A staged-draft save on an active flow that sends only actions, or only edges, skips the early edge check. Publish re-validates the whole draft (conductor call, round 12). Alternative: fall back to the draft's other half.
The outcome counter labels are outcome, code, reason, so a failed decision carries its failure reason (round 16). Alternative: fold the reason into the second label.
The model_refused message now covers an unreadable gateway answer: "The AI model couldn't answer this request. Check the step's question, options, and context, and contact support if this keeps happening." Alternative: choose the copy by reason (round 16).
The decision contracts live in the workflows facade with team_id. tach.toml declares the facade-only dependency on products.ml_inference.

Draft for Michael's approval (not posted)

CodeRabbit asked twice for Retry-After on 503: PostHog#111612 (review) and PostHog#111612 (review). Draft reply:

Leaving the 503 without Retry-After on purpose. Django can't know when the gateway or Redis comes back, so any value would be a guess. The worker's bounded backoff (2, 4, 8, 16, 32 s, per the spec) applies when the header is missing, and the 503 response schema says so.

Follow-ups for the conductor

Batch admission scale (AI decision step: let large batches through admission without throttling people #251): the token bucket's Retry-After names the next token, not the queue position, so a large batch polls the route at about N/2.5 requests per second. Options: queue-aware admission, or a per-team concurrency cap in the Node handler (AI decision step: Node runtime that asks, branches and reschedules #249).
A per-organization admission bucket (AI decision step: let large batches through admission without throttling people #251).
A gateway that finishes after the 5 s timeout, or behind a 504, may still bill. The Node runtime retries the 503 up to five times, so one decision could bill several times. Not proven: whether the gateway bills a disconnected call. Owners: AI decision step: Node runtime that asks, branches and reschedules #249 for the timeout retry policy, the gateway for idempotency on the trace id (round 17).
The ml_inference facade's 5 s timeout applies per httpx phase, not as a total deadline. AI decision step: Node runtime that asks, branches and reschedules #249 should size its 10 s worker timeout against that.
ml_inference opens a new httpx client per decision, so connections are not reused (another product's code).
Keep the flag off until the Node runtime PR deploys: ai_decision is in SUPPORTED_ACTION_TYPES before its CDP handler exists.
A publish honors a stray "stage_draft": true in its own body, so a web-session publish validates the draft leniently. This predates the PR and the PR does not widen it. Fix idea: honor stage_draft only on update and partial update (round 13).
An option named "Unsure" with unsure_enabled makes mock_answer: "unsure" ambiguous and doubles a builder label. AI decision step: Node runtime that asks, branches and reschedules #249 owns the mapping, AI decision step: builder palette, config panel and test surfaces #250 the labels. Options: reject the reserved name (case-insensitive) when Unsure is on, or state the precedence in the help text (round 15).
Other lenient non-function action types and the template grandfathering set: a stored config can carry a flag-gated template_id. This predates the PR (round 12).
A combined budget for the question and options. At the spec's field limits, the config alone can exceed the model's 8,192-token window in dense scripts. That is a spec decision.
An ai_decision edge that loops straight back to the same step bills one decision per pass. No action type has a cycle rule today, and the runtime caps a run at 1,000 steps. Candidates: a builder guard (AI decision step: builder palette, config panel and test surfaces #250) or a general graph rule.
The shared posthog/token_bucket.py logs a warning on every Redis error, so a Redis outage logs once per decision attempt. It predates this PR, and four other callers share it.
Not proven here: a live gateway call and a request against a running dev server. The DRF test client with the facade stubbed is the proof.

Navigation Menu

Repository navigation

AI decision step: decide route and action type in Django, reviving #107832 #248

What to build

Acceptance criteria

Test seam

Base branch

Cast

Mandatory skills

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

AI decision step: decide route and action type in Django, reviving #107832 #248

Description

What to build

Acceptance criteria

Test seam

Base branch

Cast

Mandatory skills

Activity

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 4, 2026

Silthus commented on Oct 5, 2026

Silthus commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions