[Silthus]

Problem

• Workflow authors cannot let a model pick a path, for example "which team handles this ticket" or "is this spam".
• This revives feat(workflows): add a classify with jev workflow action #107832 ("Classify with Jev") by Harley Alexander (@mayteio) on current master, as he built it and with his name for the step.
• The fuller native-step design in feat(workflows): add the ai_decision action type and its decide route #111612 (closed) stays the starting point for a later version.

Note

🔵 Merge risk: low (merge-risk rubric)

• New internal route that spends AI credits per classification, behind a new scoped JWT purpose (posthog:workflows:classify).
• Limits: the template is hidden and flag-gated at save, the JWT secret is empty outside dev/test so the route refuses every token until provisioned, and the revert is clean.

Changes

Kind	Files	Lines
Production	16	+398 −1
Tests	4	+276 −0
Total	20	+674 −1

• With the workflow-jev-classify-action flag on, the builder palette offers "Classify with Jev".
• The author enters a question, a JSON context and 2 to 16 categories with descriptions.
• The step returns {category, confidence, probabilities}. The palette entry maps them to the jev_category and jev_confidence variables.
• The author branches on jev_category with the regular conditional branch.
• A test run in the step panel mocks the call and returns the first category, so it spends no credits.

Before (flag off)	After (flag on)

Step configuration with output variables

flowchart LR
  Step[Classify with Jev step]:::phYellow --> Fn[postHogClassify async function]:::phBlue
  Fn -- scoped service JWT --> Route[workflow_classifications route]:::phBlue
  Route -- AI approval, limits --> GW[AI gateway, hosted Jev]:::phRed
  GW --> Vars[jev_category, jev_confidence]:::phGray
  classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
  classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
  classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
  classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;

Loading

• The Django route sends the question only as instructions and the context only as state, so event data never becomes instructions.
• The route refuses an organization without AI data processing approval, a token for another audience or team or without a workflow claim, bad category counts, and category names over 100 characters.
• A gateway 429, 5xx or network error returns 503, which the worker retries. Any other gateway refusal returns 422 with the gateway status and fails the step. An answer the route cannot read also returns 422, so a broken gateway is never asked three times.
• The route uses build_system_one_client with ai_product="workflows" and the team, on the gateway only. No TypeSafe fallback.
• A strict save (API or MCP) applies the route's question and category limits, so a bad step fails at save rather than on every run.
• New scoped JWT purpose and secret: WORKFLOW_CLASSIFY_JWT_SECRET on web and the CDP worker.
• Mechanical: Harley's six commits keep their authorship. Fixes sit on top:
  • The LLM client import moves into the request path for the startup import budget.
  • A mypy ignore for the context field name.
  • The question length and blank category names are checked before a live run.
  • Refusal copy names the gateway status.
  • Category names are capped at 100 characters in the route and the worker.
  • The JWT purpose is imported through the workflows facade.

Not in v1: a native action type, an Unsure path, thresholds, token-bucket admission, a credit pre-check (a gateway 402 surfaces as an error), a byte cap beyond the character cap, and builder UI beyond the palette entry.

How did you test this code?

• Django route contract, DRF test client with the client builder stubbed: products/workflows/backend/tests/api/test_workflow_classifications_api.py.
• Save-time limits: TestJevClassifyActionValidation in test_hog_flow.py.
• Async function, live and mocked paths: nodejs/src/cdp/async-functions/classify.test.ts.
• Ran locally: the two Django files, posthog/test/repo_invariants/, the async-function jest suites, repo-wide mypy, frontend and Node typechecks, diff-aware semgrep, hogli ci:preflight --strict.
• Ran the real worker handler and tokens against a local Django server (gateway not configured). Output below.
• Screenshots and the GIF come from the real components in Storybook with mocked API responses.
• Not run: a live call against the AI gateway, and a manual walk through the running app.
• Not run: GitHub Actions. The fork's runs wait for maintainer approval, so the local gate stands in.

Test rationale: #107832's two seams. The route tests catch a context leaking into instructions, a missing approval check, and a retry status mapped to a failure. The jest tests catch the worker and the route disagreeing on limits or the JWT audience.

End-to-end proof against a local Django server

step, org without AI approval (team 4): {"status":403,"body":{"detail":"Your organization has not approved AI data processing. Approve it in organization settings."}}
step, approved org, no gateway configured (team 5): {"status":501,"body":{"detail":"Jev is not available on this PostHog deployment."}}
raw, token for another audience: 401 {"type":"authentication_error","code":"authentication_failed","detail":"Invalid or expired service token.","attr":null}
raw, token for another team: 401 {"type":"authentication_error","code":"authentication_failed","detail":"Service token team does not match the requested team.","attr":null}
raw, one category: 400 {"type":"validation_error","code":"invalid_input","detail":"Enter between 2 and 16 categories.","attr":"categories"}
raw, blank category name: 400 {"type":"validation_error","code":"invalid_input","detail":"Give every category a name.","attr":"categories"}

"step" lines run postHogClassify itself, so the worker mints the token. "raw" lines post with tokens minted by the worker's ScopedServiceJwt.

Release status

• No feature flag controls this change
• This change is behind a feature flag and is not available to users
• This change makes a previously flagged feature available to everyone

Keep the flag off until the CDP worker with the new async function deploys and both sides have WORKFLOW_CLASSIFY_JWT_SECRET.

Automatic notifications

• Publish to changelog?

Docs update

None. The step is hidden behind a flag.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Agent: Claude Code, claude-opus-5-5

• Revives feat(workflows): add a classify with jev workflow action #107832 with Harley Alexander's commits and step name; added commits carry his Co-authored-by.
• An earlier revision renamed the step to "AI decision (Jeeeeeeeeev)". Michael asked to keep Harley's name, so the branch was rebuilt on Harley's commits with the review fixes re-applied.
• Skills: /tdd, /improving-drf-endpoints, /writing-ui-components, /writing-pr-descriptions, /reviewing-with-coderabbit, /qa-frontend (reel).
• CodeRabbit CLI: skipped, signed out with no person available to sign in.
• QA swarm: two review rounds with GPT-6.1 Sol and Claude lanes on the earlier revision. Fixed the question length, blank names, and the refusal copy. Out-of-scope items went to the follow-up list on AI decision v1: revive Classify with Jev as one minimal PR Silthus/posthog#252.
• No duplicate: an open-PR search for Jev workflow steps found none.

🤖 Generated with Claude Code

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[coderabbitai]

Important

Review skipped

We couldn't safely recover the incremental review. No full review was started, and the last reviewed checkpoint was preserved. Retry later, or explicitly request a full review by commenting @coderabbitai full review.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a feature-flagged AI decision action to workflow editing. The Node.js template validates inputs and sends requests to a new authenticated backend endpoint. The endpoint checks AI data-processing approval and calls Jev. The action exposes decision and confidence outputs, and workflow validation enforces option limits.

Priority: ⬇️ Low

Merge Risk: 🔵 Low · up to fb2de

A persistent malformed AI response can cause repeated billed requests for one workflow step. This is a bounded risk to address or explicitly accept before enabling the feature.

Security Architecture Review

Security architecture risk: 🔵 Low · up to fb2de

Access is restricted to project-scoped service requests, and organizational approval is required before AI processing. No introduced security failure was established, but production configuration and the effects of repeated requests remain unconfirmed.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — A leaked decision token is limited to its claimed team, decision audience, and short lifetime. Compromise of the dedicated signing key could authorize decision requests across consenting teams, not merely one workflow. Workflow arguments do not select the destination, audience, gateway credential, or tool configuration.

Security Findings and Attack Paths

• observed — The supplied security assessment retains no findings. Its deferred candidate targets an audience-rejection test and lacks a valid verification receipt. Source independently requires a workflow claim; the recorded absence of a missing-claim regression test does not demonstrate a production bypass. The candidate remains deferred.

Trust Boundaries and Controls

• observed — The helper overwrites team identity from the trusted invocation. Authentication verifies the scoped token and URL-team match and requires a workflow claim. The endpoint validates input and checks the team's organization approval before constructing the gateway client. Workflow identity is trusted from the worker claim rather than separately looked up in workflow storage.

Resilience and Maintainability Implications

• observed — The gateway client excludes response error bodies from logged exceptions because they may echo workflow state. Service tokens are minted inline rather than persisted in job rows. These existing controls are preserved by the new caller.

Hardening Proposals

• proposed — Establish the gateway's charging and cancellation contract for repeated requests. If retries can duplicate billable effects, propagate a stable logical-request identifier and enforce downstream deduplication. This is conditional hardening, not an observed duplicate-charge finding.
• proposed — Document that flag dial-down prevents adoption rather than stopping active AI processing, and identify an incident-stop procedure. Add runtime gating only if immediate cessation is a required release-control guarantee.

🚥 Pre-merge checks | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description includes the required sections, but it describes a different classification feature. It names the wrong feature flag and JWT purpose, claims different inputs and outputs, and refers to…	Update the Problem and Changes sections to describe the AI decision step, its question/context/options inputs, decision and confidence outputs, actual feature flag, JWT purpose, route, and test names. Replace the classification screenshots …

Full details: Description check

Explanation

The description includes the required sections, but it describes a different classification feature. It names the wrong feature flag and JWT purpose, claims different inputs and outputs, and refers to different routes and tests than this AI decision change. It also omits the required before-flow diagram.

Resolution

Update the Problem and Changes sections to describe the AI decision step, its question/context/options inputs, decision and confidence outputs, actual feature flag, JWT purpose, route, and test names. Replace the classification screenshots and flow details with accurate material. Add separate before-and-after Mermaid flowcharts, and verify the test and release-status details against this change.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Note

Quiet mode is enabled, so only the most important comments were posted inline. Other review comments are grouped below.

🟡 Other comments (1)

products/workflows/backend/presentation/views/workflow_ai_decisions.py-145-148 (1)

145-148: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Keep malformed gateway responses out of the retryable 503 path.

When the gateway returns a missing or malformed answer, parse_system_one_response raises SystemOneRequestFailed without a status code. This handler maps that error to 503, and callInternalApi retries 503 up to three times. A gateway that keeps returning malformed answers can therefore trigger up to three billed model requests for one workflow action. Distinguish malformed-response failures from network failures and return a non-retriable result for malformed responses.

🧹 Nitpick comments (1)

products/workflows/backend/tests/api/test_workflow_ai_decisions_api.py (1)

122-125: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Test rejection of a valid token without hog_flow_id.

The endpoint rejects a missing workflow claim, but no API test protects this check. Add a correctly scoped token without the claim, then assert a 401 and that Jev construction does not run.

Suggested test

+    def test_rejects_a_token_without_a_workflow_claim(self) -> None:
+        token = encode_jwt(
+            {"team_id": self.team.id},
+            timedelta(minutes=5),
+            PosthogJwtAudience.WORKFLOW_AI_DECISION,
+            signing_key=SECRET,
+        )
+        with patch(_BUILD) as build:
+            response = self._post(token=token)
+
+        assert response.status_code == status.HTTP_401_UNAUTHORIZED
+        build.assert_not_called()

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: PostHog/posthog/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 175e76a7-c0c4-46db-b71f-245c6ce8f2d6

📥 Commits

Reviewing files that changed from the base of the PR and between d5e1441 and fb2de78.

📒 Files selected for processing (21)

• frontend/src/lib/constants.tsx
• nodejs/src/cdp/async-functions/ai-decision.test.ts
• nodejs/src/cdp/async-functions/ai-decision.ts
• nodejs/src/cdp/async-functions/index.ts
• nodejs/src/cdp/async-functions/internal-api-call.ts
• nodejs/src/cdp/config.ts
• nodejs/src/cdp/templates/_destinations/posthog_workflows/posthog-ai-decision.template.ts
• nodejs/src/cdp/templates/index.ts
• nodejs/src/cdp/utils/jwt-utils.ts
• nodejs/tests/fixtures/hog-bytecode/76959e3b7e5dee74f023dde37eedcfa8e23115b4284a9c903e3698c72adac389.json
• posthog/cdp/flag_gated_templates.py
• posthog/jwt.py
• posthog/settings/data_stores.py
• products/workflows/backend/facade/service_jwt.py
• products/workflows/backend/presentation/views/hog_flow.py
• products/workflows/backend/presentation/views/workflow_ai_decisions.py
• products/workflows/backend/routes.py
• products/workflows/backend/service_jwt.py
• products/workflows/backend/tests/api/test_hog_flow.py
• products/workflows/backend/tests/api/test_workflow_ai_decisions_api.py
• products/workflows/frontend/Workflows/hogflows/panel/HogFlowEditorPanelBuild.tsx

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 6 remain after this review.

[parameterai]

Risk: No findings

Since the last review, the PR's substantive change is the feature-flag gate on the classification route: workflow_classifications.py now refuses requests with 403 unless the workflow-jev-classify-action flag is enabled for the team, and the check fails closed when flag evaluation errors. The rest of the diff since the last review snapshot is rebase drift from master (public template route registration, unrelated feature-flag constants, CDP DLQ config), all of which already exists in the PR base. No new security issues found: the scoped-JWT auth chain, per-purpose secret, team-claim matching, bounded inputs, and save-time validation reuse all hold.

_{Sentinel reviewed b07fd97 · Review settings}

[pr-assigner-resolver-posthog]

👀 Auto-assigned reviewers

These soft owners were skipped because they only have minor changes here. Nothing blocks merge, so self-assign if you'd like a look:

• @PostHog/team-ingestion (owners.yaml)

Soft owners come from each directory's owners.yaml and each product's product.yaml (resolved nearest-file-wins). For a skipped owner, the locator is the file that decided it. Generated files and lockfiles are ignored when deciding ownership.

[Silthus]

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

@coderabbitai both findings from your review body are fixed in 3957d7e.

• Malformed gateway answers: _is_transient in workflow_classifications.py now treats only an unreached gateway (an httpx.HTTPError cause), a 429, or a 5xx as retriable. An unreadable answer returns 422, so the worker fails the step without asking Jev again. The new malformed_answer case covers it.
• Missing workflow claim: test_rejects_a_token_without_a_workflow_claim posts a correctly scoped token without hog_flow_id and asserts a 401 with no client built.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail
fd -t f 'workflow_(classifications|ai_decisions).*\.py$' .
fd -t f 'workflow_(classifications|ai_decisions).*\.py$' . --exec ast-grep outline {} --items all
rg -n -C 12 --type py 'def _is_transient\b|malformed_answer|def test_rejects_a_token_without_a_workflow_claim\b' products/workflows
rg -n -C 8 --type py 'hog_flow_id|_is_transient\(' products/workflows/backend/presentation/views/workflow* 2>/dev/null

Length of output: 46072

---

@Silthus Both findings are addressed in the inspected code. The regression cases cover malformed answers returning 422 and missing workflow claims returning 401 without building a client.

I checked the source only; I did not run the tests. No further changes are needed for these two findings.

You are interacting with an AI system.

[Silthus]

Note

🤖 GPT-6.1 Sol responding on behalf of Michael

Replaced by the draft fork PR #366 in its native fork stack. @meikelmosby will start migrating the stack from the fork into PostHog/posthog for upstream review and CI. Closing this master-targeted copy to keep the handoff clear.