Spec: Generate text — an LLM action node for workflows #22

Architecture blueprint (visual companion): https://claude.ai/code/artifact/443a4af9-b14a-4ec8-982f-3e32867f8ce7

Map: #1. This spec is the destination artifact of the "LLM action for workflows" map.
It records the surviving version of every decision. Where tickets amended each other, only the final version appears here.
The reasoning behind each decision lives in the linked ticket resolutions. This document carries outcomes plus a one-line why where the why prevents a wrong turn.

Problem Statement

A workflow author can send email and push messages, but the content is static text plus variable substitution.
The author cannot summarize, classify, or write text that adapts to the person the workflow runs for.
To personalize beyond {{ person.properties.name }}, the author must leave PostHog, call an LLM from their own backend, and write the result back. Most teams do not do this.

Solution

A Generate text node in a new AI category of the workflow builder.

The author writes a prompt with liquid templating, picks a model from a curated list of six, and declares output fields.
Each output field points at a declared workflow variable and carries an instruction, for example subject_line: "A short subject line for this email".

When the workflow runs, the node makes two LLM passes.
Pass 1 runs the author's prompt on the chosen model and produces free text.
Pass 2 reads that text and the field instructions and extracts one string per field, under a strict JSON schema.
The node writes the results into the declared workflow variables. Downstream nodes use them like any other variable.

The panel has a Run preview button. It runs the real prompt against the real model, spends real AI credits, and shows the generated text, the extracted fields, and the resulting variable values.

Each run bills the customer's PostHog AI credits. The org-level AI credit cap bounds spend.

User Stories

1. As a workflow author, I want a node that runs my prompt against an LLM, so that my workflow can produce text no template can express.
2. As a workflow author, I want to reference person properties, event properties, and workflow variables in my prompt, so that the output is personal to the person the run is for.
3. As a workflow author, I want {-triggered autocomplete in the prompt input that shows real fields from a recent event, so that I do not have to guess property names.
4. As a workflow author, I want to paste JSON examples and few-shot blocks into my prompt without escaping braces, so that prompt writing works like it does everywhere else.
5. As a workflow author, I want {% if %} blocks in my prompt, so that one prompt can handle several audience segments.
6. As a workflow author, I want to pick a model from a short list ordered by cost, so that I can trade cost against capability without reading model documentation.
7. As a workflow author, I want to declare named output fields with plain-language instructions, so that I get structured values instead of one blob of text.
8. As a workflow author, I want each output field to write into a workflow variable I declared, so that downstream steps consume the result the same way they consume any variable.
9. As a workflow author, I want a way to map the raw generated text to a variable, so that I can use the whole generation when I do not need fields.
10. As a workflow author, I want a preview that runs the real model, so that I see what a run will actually produce before I publish.
11. As a workflow author, I want the preview to show the resulting variable values, so that I can confirm the mapping end to end.
12. As a workflow author, I want the node to abort the run by default when generation fails, so that an email step never sends with empty variables.
13. As a workflow author, I want run-log errors that name the cause (quota_exceeded, deadline_exceeded, capacity_unavailable, …), so that I can act on a failure without filing a support ticket.
14. As a workflow author, I want a field the model cannot support to come back empty rather than invented, so that I never send fabricated claims to a customer.
15. As a workflow author testing a whole flow, I want the mocked test walk to return obviously fake placeholder values for this node, so that I can test flow logic without spending credits.
16. As a workflow author, I want the preview button and model input to tell me that runs spend AI credits, so that spend never surprises me.
17. As a billing admin, I want every generation billed through the standard AI credits path with $ai_billable, so that this feature appears on the existing PostHog AI line and respects the org spend cap.
18. As an org admin, I want the feature gated by one flag evaluated by organization, so that enabling it is one decision with one blast radius.
19. As a data subject whose properties feed prompts, I want templated content treated as data rather than instructions in both passes, so that a crafted property value cannot hijack the workflow.
20. As a PostHog operator, I want fleet-wide and per-team concurrency caps on generations, so that one team cannot exhaust shared Celery capacity.
21. As a PostHog operator, I want this feature to add no queue, no pod, and no chart change, so that rollout and rollback are a flag flip.
22. As a support engineer, I want each workflow run to be one LLM trace, so that I can inspect both passes of every node of a run in LLM analytics.
23. As an agent authoring workflows through MCP, I want the graph-schema doc to state the coupling between output_fields and output_variable, so that I never build a node that generates, bills, and discards.
24. As a workflow author with an in-flight run during a rollback, I want retrieve to keep working with the flag off, so that a started generation still completes.

Implementation Decisions

The node and its template

• A plain function action node. The action-type union is untouched.
• Template id: template-workflow-llm. Convention match with the workflow-only siblings (template-email, template-native-push, template-posthog-set-variable). (Settled at spec time; no ticket named an id.)
• Template source in TypeScript, in the nodejs CDP template registry, next to the async function and service.
• Template type: 'destination', status: 'hidden' like its siblings. The Beta toolbar tag carries the labeling. (Settled at spec time from tickets 15/17: 'beta' status would list the node in the CDP destinations catalog for everyone, bypassing the flag.)
• Category AI, node name Generate text, description "Run a prompt and store the result in workflow variables." Registered through registerActionNodeCategory with featureFlag: FEATURE_FLAGS.WORKFLOWS_LLM_ACTION.
• Node default on_error: 'abort', not the usual continue. A failure means the team was charged and downstream would act on empty variables.
• Node metering inherits 'fetch' as the billable invocation kind. Workflow billing and AI credits stay separate lines.
• inputs_schema has exactly three entries, in panel order:
  1. prompt — type: 'string', required, templating: 'liquid'. Description: "Type { to insert a person property, an event property, or a workflow variable." Liquid because a bare { is text there: JSON examples and few-shot blocks need no escaping, and {% if %} covers conditional prompts. All examples everywhere use the {{ … }} form.
  2. output_fields — type: 'dictionary', keyed by declared workflow variable key, value is the instruction string. Rendered by a bespoke branch; stored shape stays a plain dictionary.
  3. model — static choice, searchable, templating: false, required, default gpt-5-mini. Six choices grouped by provider, cheapest first: gpt-4.1-mini, gpt-5-mini, gpt-5.4, claude-haiku-4-5, claude-sonnet-4-6, claude-opus-4-8. Description: "Each run of this node uses your PostHog AI credits. More capable models cost more."
• No reasoning-effort input. A user who wants more reasoning picks a stronger model.
• The picker is a deliberate subset of the endpoint's validation set, so a picker refresh cannot break a saved workflow. A Python test pins the six ids to the model registry.
• The registered async function declares a mock. It returns the envelope shape with text = "This is example text. Turn on 'Make real HTTP requests' to run your prompt." and one "Example value for \`"` per declared field, plus a log line echoing the resolved prompt. Nothing in the mock reads like a real answer.

Frontend config panel

• Panel order: step name, prompt (the main input, gets the vertical space), output fields, model, then the existing accordion (Output variables / Skip conditions / Error handling).
• An output-field row is [pick a declared variable] + [instruction] on two lines. The variable picker is the existing select over declared variables, filtered to unused ones. Variable creation stays in the variables panel behind the existing link.
• Picking a variable writes a derived output_variable entry {key: <variable>, result_path: "fields.<variable>"}. Field-owned rows regenerate on any field change; rows no field owns survive untouched. A hand-added row with result_path: text is how the raw generation reaches a variable.
• Field-owned rows stay visible and editable in the Output variables accordion, tagged "Set by this step" with tooltip "This mapping comes from an output field above. To change it, edit the output field."
• Rendered by a branch on the literal template id in HogFlowFunctionConfiguration, following the two existing literal-id branches. No new shared input type.
• Run preview is the shipped "Pick from response" button relabeled, with tooltip "Runs your prompt against the selected model and shows the result. This uses your AI credits." It renders three sections read off the response: generated text, extracted fields, resulting variable values. The clickable result tree goes away. On this template the button stops stripping output_variable before dispatch.
• Preview spends real credits; the press is the consent. No mock toggle, no confirmation. The Test panel keeps its own mocked default behind "Make real HTTP requests"; the two surfaces disagreeing is correct.
• Preview deadline is 25 seconds, passed by the test path. Expiry is a preview-level message, not a node failure: "The preview stopped after 25 seconds. The step gets longer when the workflow runs. Try a shorter prompt or a faster model." The button keeps its loading state with elapsed seconds beside it.
• Discovery of referenceable data is the existing {-autocomplete plus the prompt's description string. No reference list, no new component.

Runtime transport (CDP)

• The node's Hog shell calls one registered async function that sets a new llmGenerate queue-parameter type. The async dispatch in executeWithAsyncFunctions gains a branch for it. This follows sendPushNotification (inline execution, self-reschedule), not email (dedicated queue).

• No new Cyclotron queue, no consumer, no capability, no pod, no chart change. Jobs stay on the existing hogflow queue.

• Each execution is one short HTTP call: a submit, or one retrieve. On 202 pending the service returns finished: false with the same queueParameters and queueScheduledAt = now + 2s. A reschedule does not resume the VM, so it consumes no async step.

• When options.isTest is set (preview), the service loops inline in process to completion or the 25-second preview deadline. Preview therefore never exercises the reschedule transport.

• The generation service owns submit, retrieve, timeouts, and failure handling. It does not use cdpTrackedFetch and inherits no fetch timeout. Values:

  Parameter	Value
  Poll interval	2 s
  Retrieve timeout	3 s, set explicitly
  Submit timeout	5 s, set explicitly (settled at spec time; submit only enqueues)
  Iteration bound	50, carried in queueMetadata; exceeding it is terminal (settled at spec time: 90 s / 2 s = 45, plus slack; the time give-up is the primary check, the bound is the backstop)
  Give-up	90 s, then terminal deadline_exceeded
  429 bounces	up to 10, reschedule at now + Retry-After, then terminal throttled

• Status handling: 202 → reschedule. 200 succeeded → push the envelope onto the VM stack. 200 failed → terminal, surface error.code and error.message into the run log, node inherits on_error. 404 → terminal, never poll, surface feature_unavailable. 429 → reschedule with backoff. Status codes are retry instructions; this rule now binds the service.

• Do not copy queuePriority + 1 from the fetch-retry path; it is a no-op on cyclotron v2. scheduledAt is the only lever.

• Metrics live in the service and the task, not on queue gauges: counters for started/finished/failed, a duration histogram, and a Celery queue-wait histogram from an enqueued_at stamp in the task payload. Saturation shows on the existing concurrency-limit-exceeded counter.

• Blocking precondition: the run-log flood fix. Widen routingOnlyReschedule's suppression to a same-action, same-queue continuation, without disturbing the fetch-retry and SES-throttle cases. Roughly ten lines. Must land before or with the feature; each reschedule currently writes two log lines a customer reads.

Django endpoint

• A workflows-owned pair, registered by the product's routes.py:
  • POST /api/projects/:team_id/workflow_llm_generations/
  • GET /api/projects/:team_id/workflow_llm_generations/<id>/
• Flat, not nested under hog_flows. Flow attribution rides in the body; preview may have no saved flow.
• Auth: TeamSecretTokenAuthentication + TeamSecretTokenPermission, with two entries added to the view-name allowlist (project_workflow_llm_generations-list, -detail). This is the credential the CDP already loads, through the DRF-native path, which buys schema annotations, generated types, and correctly keyed throttles. Not internal auth (unreachable via public SITE_URL), not PSAK (the CDP holds none), not a new JWT.
• Submit body: invocation_id, hog_flow_id (optional in preview), prompt (fully rendered), model, output_fields. No max_tokens, no byte budget on the wire.
• Response: {id, status: pending|succeeded|failed, result: {text, fields}, error: {code, message}}.
• Status codes are retry instructions, not descriptions: 202 pending; 200 for success and for terminal generation failure; 4xx/5xx only for request-level faults. Two carve-outs: 404 with code: "feature_unavailable" when the flag is off (terminal), and 429 from the submit throttle with an explicit Retry-After (a reschedule).
• Error codes: quota_exceeded, invalid_model, output_too_large, model_refused, malformed_output, gateway_unavailable, deadline_exceeded, capacity_unavailable, feature_unavailable; service-side throttled. These strings surface in run logs, so they are user-facing copy. gateway_unavailable is terminal, not retried. quota_exceeded is a 200 terminal failure, not 402.
• Over-quota submits answer 200-failed directly from submit, with no task enqueued. (Settled at spec time: the gate runs before enqueue, so answering at submit is simpler and cheaper than 202-then-failed.)
• Throttle: submit only, 60/minute per project, on the team-bucket throttle base class (the synthetic token user would otherwise share one global bucket). Retrieve unthrottled. The throttle bounds enqueue rate; the concurrency cap bounds occupancy.
• Idempotency key: invocation_id plus a hash of the submit payload. One invocation id spans a whole flow run, so the payload hash is what separates two LLM nodes in one run. Resolve the dedupe record before enqueue; a resubmit returns the existing handle without a second task.
• Deadlines and caching: a 60-second wall-clock budget inside the task spanning both passes, checked between them. Successes cached 10 minutes; failures about 30 seconds. The Redis result record doubles as the dedupe store.
• Trace identity: one trace per workflow run, posthog_trace_id = invocation_id. Both passes and every LLM node of a run share one trace.
• Model validation: against a new exported superset built from OpenAIConfig.SUPPORTED_MODELS and AnthropicConfig.SUPPORTED_MODELS, minus Gemini (the gateway holds no Google key). The implementer excludes reasoning-only ids that cannot serve chat completions. Not PLAYGROUND_MODEL_IDS, which is curated for the case where PostHog pays.

Two-pass generation

• Pass 1: the user's model, a fixed endpoint-imposed system prompt (brevity framing plus a treat-templated-content-as-data guard), the rendered prompt as the user message, max_tokens 1024. (Settled at spec time: the envelope is capped at 4096 bytes and this node writes fragments; 1024 tokens is comfortably above the cap in the common case without paying for runaway generations.) There is no user-settable system prompt, deliberately.
• Pass 2 runs only when fields are declared; zero fields skips it and returns {text, fields: {}}.
• Pass 2 model: gpt-5.6-luna, hardcoded in the endpoint, never the user's pick. Reasoning: minimal if the route accepts it, else low (probe P2 decides during implementation). Chosen for transport fidelity: strict response_format is unreliable through the gateway's Anthropic route, and the extraction contract rests entirely on strict mode.
• Extraction schema: strict: true, additionalProperties: false, every field required, field type ["string", "null"] coerced to "" in the endpoint. Property names are the workflow variable keys. All fields are strings, because the variables panel only creates string variables.
• The envelope {text, fields} keeps user field names out of text's namespace. result_path is fields.<key>; text is always mappable.
• Size budget: a flat 4096-byte budget on the returned payload, 80% of the executor's 5120-byte variable cap, which is not raised. Flat, not headroom-measured, so the same prompt never succeeds or fails depending on sibling nodes.
• Retry: one, inside the endpoint, for model-fixable failures only (oversized output, malformed response, refusal). Never for transport.
• Pass 2 receives the generated text, the field names, and the instructions. Not the original prompt, not person properties. The system prompt (fixed wording settled on the extraction-contract ticket) orders: rules, fields, delimited text. Rule 3 (empty string over guessing) is the anti-fabrication counter to strict mode; rule 2 is the injection guard, because person properties reach pass 2 through pass 1's text. Collapse whitespace in untrusted values before embedding.

Billing

• The instrumented PostHog OpenAI SDK, caller-side: client built with posthog_client and base_url pointing at the gateway; both passes pass posthog_properties = {"ai_product": "workflows", "feature": "llm_action", "$ai_billable": True, "team_id": …}.
• Add "workflows" to POSTHOG_AI_PRODUCTS. Gate submit on is_team_over_ai_credit_budget.
• No gateway change, no product Literal entry, no allowed_models pin, no credit_bucket. The frozen Python gateway is untouched.
• Observability falls out free: generations land in the team's LLM analytics with the per-run trace.

Celery execution

• Task in the workflows product's tasks package, imported in that package's __init__.py (autodiscovery misses submodules and nothing catches the miss).
• Routes to CeleryQueue.LONG_RUNNING. No dedicated queue, no dedicated pool: production reads worker queues from the charts repo, so a queue added only here is consumed by nothing in production, silently.
• Concurrency bound: two limit_concurrency decorators copied from process_query_task — 30 global, 5 per team, both from env settings, ttl=120 (must exceed the 90-second hard time limit so a dead worker cannot free a live slot).
• Task shape: soft_time_limit just above 60 s, hard time_limit 90 s, expires 90 s (a task nobody will collect is discarded, not billed), retry_backoff 2 up to 8, max_retries 4, on_failure writes the terminal record.
• Saturation is invisible by design: waiting tasks retry in the broker while retrieve returns 202; an exhausted retry budget writes capacity_unavailable as a 200 terminal failure via on_failure. Preview needs no special case.
• Graduation trigger: if Celery queue-wait p99 exceeds a few seconds during the pilot, split to a dedicated queue; the charts entry must land before the flag widens.

Feature flag

• One key: workflows-llm-action, declared in the frontend constants, read from two places: the builder category (ActionNodeCategory.featureFlag) and the submit endpoint (inline check, fail-closed, 404 with the feature_unavailable envelope: "Generate text is not available for this project. Contact support to request access.").
• Aggregate on organization — the only group whose key is byte-identical between browser and Django evaluation, and the same scope as the AI credit balance. Never roll out by person percentage or project group; identities do not match across the two ends.
• A saved workflow gets no flag-aware branch: the node renders, saves, publishes, and fails at submit with the terminal 404. The CDP cannot evaluate flags; hiding a saved node would make the canvas lie; every flag in this editor gates authoring only.
• Retrieve does not check the flag, so in-flight generations survive a rollback.
• At GA: delete the constant, the category's flag, and the endpoint check in one PR.

Docs and skills, same PR

Five in-repo files, none under docs/ (workflows has no page there; the posthog.com page is a named follow-up in the other repo):

1. products/workflows/skills/building-workflows/references/graph-schema.md — the array form of output_variable, plus an LLM step worked example carrying the three non-discoverable facts: output_fields keys must name existing variables; each field needs a matching derived output_variable entry with result_path: fields.<key> (a node without them runs, bills, and discards); result_path: text reaches the raw generation. Also: runs spend AI credits; default on_error is abort.
2. products/workflows/skills/building-workflows/SKILL.md — one clause: the step spends AI credits when mock_async_functions=false.
3. products/workflows/skills/building-workflows/references/lifecycle-and-debugging.md — correct the claim that the catalog never lists fixed-template steps.
4. products/workflows/mcp/tools.yaml — credit clause on the test-run tool description; regenerate the two MCP schema files.
5. products/workflows/CONTRIBUTING.md — a queue-routing subsection for the llmGenerate parameter type (schema union, dispatch branch, the options.isTest inline branch, reschedule-on-202); six lines on the per-template renderer as the cheap route; fix one stale path.

The skill ships in the same PR, departing from the product's own precedent, because a skill one PR behind is a published document telling agents a paid operation is free.

Testing Decisions

• Tests assert observable behavior at the highest seam that catches the bug. The seams, all existing:
  1. The Django endpoint contract (DRF test client): submit/retrieve shapes, 202/200/404/429 semantics, idempotency (same key returns the same handle, no second task), model validation, quota gate, flag-off 404, throttle keying by team. Celery dispatch is mocked here: CELERY_ALWAYS_EAGER would otherwise run the task inline and make the 202-then-poll protocol untestable.
  2. The two-pass logic (task-level, gateway calls stubbed): envelope assembly, null-to-"" coercion, zero-fields skip, the one model-fixable retry, the 4096-byte budget, deadline check between passes, failure caching.
  3. The concurrency limiter against fakeredis directly (note the memoized fake server leaks ZSET state between tests unless flushed).
  4. The CDP service (existing hogflow executor test harness): dispatch branch, reschedule-on-202 with interval and iteration bound, terminal handling of 404/200-failed, the isTest inline loop, no VM async-step consumption on reschedule.
  5. The frontend panel (existing logic-level Jest patterns): derived output_variable regeneration preserving unowned rows, preview rendering of the three sections, disabled/loading submit guard.
  6. A registry pin test: the six picker ids are members of the validation superset — the test that makes the subset relationship a fact rather than an intention.
• Prior art: endpoint tests beside the six async-function endpoints; executor tests beside the hogflow executor service; process_query_task tests for the limiter pattern.
• No end-to-end test across Django + Celery + CDP; the contract tests at each seam plus the shared shapes are the coverage. The rollout smoke test is a real activated workflow on an internal org, because preview routes inline and cannot smoke-test the transport.

Out of Scope

• Prompts database (LLMPrompt) integration; prompt versioning and rollback.
• An email template as LLM input.
• Deriving the output schema from a downstream action's required inputs.
• Raising the 5 KB workflow-variable cap, or any alternative transport for large output.
• Bring-your-own provider keys (which is what gates dynamic model listing).
• A per-feature AI spend cap; no such control exists anywhere in PostHog. Standing controls: trigger masking on run volume, the org AI cap on spend, the submit throttle on enqueue rate.
• A general park-and-wake primitive for the CDP runtime. The contract here is written so the node adopts it later without changing the endpoint or user-visible behavior.
• Server-side derivation of output_variable from output_fields; documented instead, revisit if that proves too thin.
• Per-team fair dequeue and waiter aging; graduates on the queue-wait signal.
• Fixing the unread MAX_ACTION_STEPS_HARD_LIMIT.

Further Notes

Companion PRs and ordering.

1. PR A, first and standalone: three autocomplete/liquid one-liners (feed the editor the merged real sample globals; add liquid to the cursor-offset condition; stop double-wrapping inserted suggestions in braces). These fix every templating field, not just this node.
2. The run-log flood fix rides PR A or the feature PR, never after.
3. The feature PR: template, async function, service, dispatch branch, endpoint, task, renderer, preview relabel, flag, five doc files.
4. PR B, independent: the cyclotron-v2 isHealthy fix (the check reads a poll timestamp that goes stale during any batch over 30 s while a Postgres heartbeat proves the pod alive; fix the check, not the window).
5. Follow-ups: posthog.com docs page; a dedicated Celery queue only if the queue-wait histogram demands it.

Rollout. Flag off at merge → enable for internal orgs, aggregated on organization → build and activate a real workflow (preview cannot smoke-test the transport) → widen → GA cleanup in one PR. Rollback is flag off; in-flight generations stay collectable.

Three probes before or during implementation (each one request, none a design decision):

• P1: strict response_format passes unaltered through the gateway's OpenAI route end to end. The extraction contract rests on this.
• P2: gpt-5.6-luna accepts minimal reasoning; fall back to low if not.
• P3: the gateway route this endpoint uses permits all six picker models (ownership of this claim was previously nobody's).

Values settled at spec time (no ticket named them; each is marked inline above): template id template-workflow-llm; template status: 'hidden'; pass-1 max_tokens 1024; submit timeout 5 s; iteration bound 50; over-quota answered at submit; pass-2 reasoning minimal-else-low.

Known pre-existing defects recorded, not fixed here: on_error: continue meters a failed action as succeeded; MAX_ACTION_STEPS_HARD_LIMIT is never read; the Variables panel prints a single-brace usage form while its copy button writes double braces.

Provenance. Charted and resolved on map #1 across tickets #2–#21. Amendment handling: this spec is the merged surviving record; for any conflict between this spec and a ticket, this spec wins, and the ticket trail holds the full reasoning.