Generation endpoint, task, and first pass #45

Parent

Spec: #22 — Generate text, an LLM action node for workflows.

What to build

The generation service behind the node: something submits a rendered prompt and a model, polls, and gets generated text back, billed to the team's AI credits.

A workflows-owned pair of endpoints, registered by the product's own routes, flat rather than nested under workflows — flow attribution rides in the request body because a preview may have no saved flow at all.

• Submit takes an invocation id, an optional flow id, the fully rendered prompt, the model, and the output fields. No token limit and no byte budget cross the wire.
• Retrieve returns a status of pending, succeeded or failed, with the result envelope or a structured error.

Status codes are retry instructions, not descriptions. This is the load-bearing rule of the contract. Pending is a 202. Both success and terminal generation failure are 200 — a model that refused, an over-quota team, or an unavailable gateway are all outcomes of a request that was handled correctly, and answering them as errors would make the caller retry something that will never succeed. Request-level faults keep their 4xx and 5xx. Two carve-outs: a 404 when the flag is off, carrying a feature-unavailable code and terminal, and a 429 from the submit throttle with an explicit retry hint, which is a reschedule.

The error codes surface directly into customer-read run logs, so they are user-facing copy, not internal enum names.

Auth is the team secret token, through the standard framework path rather than internal auth or a new token type. The workflow runtime already loads this credential, internal auth is unreachable over the public site URL, and going through the framework path is what buys schema annotations, generated frontend types, and throttles that key correctly. Both view names need adding to the authentication allowlist.

Guards on submit. The feature flag is checked inline and fails closed. The AI credit budget is checked before anything is enqueued, and an over-quota team is answered as a terminal failure straight from submit — the gate has already run, so answering immediately is simpler and cheaper than accepting and then failing. The model is validated against a superset assembled from the OpenAI and Anthropic supported-model sets, minus Gemini, since the gateway holds no Google key; exclude any reasoning-only ids that cannot serve chat completions. Do not build this from the playground model list, which is curated for the case where PostHog pays. A submit throttle of sixty a minute per project, on the team-bucket throttle base class — the synthetic user the token authenticates as would otherwise put every team in one global bucket. Retrieve is unthrottled.

Idempotency. The key is the invocation id plus a hash of the submit payload. One invocation id spans a whole workflow run, so the payload hash is the only thing separating two generation steps in the same run. Resolve the dedupe record before enqueueing; a resubmit returns the existing handle and must not start a second task.

The task. In the workflows product's task package, and imported in that package's init file — autodiscovery misses submodules and nothing catches the miss. Routed to the existing long-running queue. No dedicated queue and no dedicated pool: production reads its worker queues from the charts repository, so a queue added only here would be consumed by nothing, silently. A soft time limit just above the wall-clock budget and a hard limit above that; an expiry so a task nobody will collect is discarded rather than billed; bounded retries with backoff; and a failure hook that writes the terminal record.

Pass one. The author's chosen model, a fixed system prompt the endpoint imposes — brevity framing plus a guard that templated content is data and never instructions — and the rendered prompt as the user message. There is deliberately no user-settable system prompt. This ticket returns the envelope with generated text and an empty field map; extraction is the next ticket, and zero declared fields is a real product path that ends here.

Billing. The instrumented PostHog OpenAI SDK on the caller side, built with the PostHog client and pointed at the gateway, tagging every call as billable with the workflows product and team. Add workflows to the recognised AI products list. No gateway change at all — the Python gateway is under a code freeze and is not touched. Observability then falls out for free: generations land in the team's own LLM analytics.

Trace identity. One trace per workflow run, keyed on the invocation id, so both passes and every generation step of a run land under one trace a support engineer can open.

Storage and deadlines. A wall-clock budget inside the task covering both passes. Successes cached for minutes, failures for tens of seconds. The result record doubles as the dedupe store.

Acceptance criteria

• Submit with a valid team secret token returns a handle, and retrieve on that handle eventually returns generated text.
• Pending is answered 202; success and terminal generation failure are both answered 200 with the failure carrying a code and message.
• Flag off returns 404 with the feature-unavailable code, and it is terminal. Retrieve deliberately does not check the flag, so a generation already in flight survives a rollback.
• The flag is checked by the same key the builder reads, and the check fails closed.
• Workflows is added to the recognized AI products list, so the spend actually rolls into AI credits rather than being tagged and dropped.
• An over-quota team is answered as a terminal failure directly from submit, with no task enqueued.
• An unknown or unsupported model is rejected, and the validation set is built from the OpenAI and Anthropic supported-model sets minus Gemini, not from the playground list.
• Submit is throttled at sixty a minute keyed per project, not globally; a throttled submit carries an explicit retry hint; retrieve is unthrottled.
• Resubmitting the same invocation id and payload returns the existing handle and starts no second task.
• Two different generation steps in one workflow run get separate handles.
• Requests without a valid team secret token are rejected, and both view names are on the authentication allowlist.
• The task is registered in the product's task package init file and routes to the existing long-running queue.
• A generation appears on the team's AI credits line, tagged billable to the workflows product, and in the team's LLM analytics.
• Every generation of one workflow run shares a single trace keyed on the invocation id.
• Zero declared output fields returns generated text with an empty field map.
• The endpoint's system prompt treats templated content as data — a prompt carrying instruction-shaped person property values does not redirect the generation.
• The endpoint declares its request and response schemas, so generated frontend types are not empty.
• Endpoint contract tests use the test client with Celery dispatch mocked — running the task inline would make the accept-then-poll protocol untestable.

Blocked by

None — can start immediately.