Skip to content

AI decision step: Node runtime that asks, branches and reschedules #249

Closed as not planned

Description

@Silthus

Part of #240
Spec: #247 (Spec doc: AI decision step for workflows). Sections: "Answer selection", "The result and variables", "The Node runtime", "Test runs and the mocked walk", "Docs and MCP".

What to build

The CDP runs an ai_decision step end to end: it renders the Context, asks the decide route, picks the output from the probabilities and the author's thresholds, and reschedules the person when the route says "later". After this ticket, a decision step built through the API or MCP runs in a live workflow.

  • ai_decision in the Node action schema (nodejs/src/cdp/schema/hogflow.ts), and an AiDecisionHandler in the executor registry (hogflow-executor.service.ts).
  • A pure answer-selection function: yes or no with yes_threshold, the Unsure band with no_threshold, pick one with the top probability and min_pick_probability, ties to the earlier option.
  • A decision service that owns the call: render inputs.context with buildInputsWithGlobals, fail state_too_large over 8,192 bytes of compact JSON without a call, mint the WORKFLOW_AI_DECISION JWT (claims team_id, hog_flow_id), and POST to /api/projects/:team_id/workflow_ai_decisions/ with a 10 s timeout. No callInternalApi, no inline retry loop.
  • Map responses by the spec's status contract. On 429 and 503, return scheduledAt with the spec's reschedule bounds, carried in the action's state: 429 waits Retry-After capped at 60 s plus jitter, up to 30 minutes, then throttled; 503, a timeout, or a connection error waits Retry-After or 2 to 32 s, up to 5 times, then gateway_unavailable.
  • A reschedule writes no "Resuming..." / "Workflow will pause until..." pair into the run log.
  • Success returns nextAction on the answer's branch edge and the result {answer, probability, probabilities, model} for output_variable. The run log names the answer, all probabilities, and the model, never the state. A failure returns an error, so on_error sends the person down the continue edge ("If the decision fails") or ends the run.
  • Test runs: with mock_async_functions, no call; validate and render like a live run, answer with mock_answer or the first answer, say it was mocked in the run log, and return the rendered Context and its size. Without mocks, one real call; 429 or 503 ends the test with a busy message instead of rescheduling.
  • Docs and MCP, per the spec: the ai_decision section and worked example in products/workflows/skills/building-workflows/references/graph-schema.md, the node type and the credit clause plus mock_answer in products/workflows/mcp/tools.yaml (regenerate the MCP schemas), and a short section on the native handler in products/workflows/CONTRIBUTING.md.
  • Where feat(workflows): add a classify with jev workflow action PostHog/posthog#107832's classify.ts code fits (token minting, input parsing, the mock and live parity cases), carry it and add Co-authored-by: Harley Alexander <harley.a@posthog.com> to that commit.

Acceptance criteria

  • A live ai_decision step whose stubbed route answers P(yes) = 0.62 with yes_threshold 50 follows the Yes edge, and writes answer: "yes" to a mapped variable.
  • With Unsure on, P(yes) between no_threshold and yes_threshold follows the Unsure edge; a pick one whose top probability is under min_pick_probability follows the Unsure edge.
  • A terminal failure envelope follows the continue edge with on_error: continue, and ends the run with on_error: abort; the run log names the code.
  • A 429 reschedules at Retry-After and gives up with throttled after 30 minutes; a 503 reschedules at most 5 times, then fails with gateway_unavailable. Neither writes the resume log pair.
  • A Context over 8,192 bytes fails with state_too_large and makes no HTTP call.
  • The mocked and live paths reject the same configs the Django serializer rejects, from one shared cases table.
  • A mocked test run with mock_answer set to the second option follows that option's edge and spends nothing.
  • graph-schema.md and tools.yaml describe the step, its edges, and its credit use; MCP schemas are regenerated.

Test seam

Spec seams 3, 4, and 5: the answer-selection function (it.each at threshold boundaries), the handler in the existing hogflow executor test harness with the route stubbed, and mock and live parity over one cases table.

Base branch

Stacks on #248's branch feat/workflows-ai-decision-route. New branch feat/workflows-ai-decision-runtime. Independent of #250.

Cast

GPT-6.1 Sol through codex:codex-rescue (bounded Node work) when Codex answers, else Opus 5.5.

Mandatory skills

/tdd, /writing-tests, /implementing-mcp-tools, /writing-user-facing-copy (run-log messages), /writing-code-comments, /writing-pr-descriptions, then qa-swarm.

Activity

  1. Silthus commented on Oct 4, 2026

    @Silthus
    OwnerAuthor

    Scope note from the conductor, from the #248 build (see its progress comment):

    • The ml_inference facade's 5 s timeout applies to each httpx phase, not as a total deadline. Size the worker's 10 s timeout against that.
    • 503 from the decide route carries no Retry-After; the Node backoff owns that wait. A 429 carries a jittered Retry-After.
    • Batch admission (people failing as throttled in large batches) is its own ticket, AI decision step: let large batches through admission without throttling people #251, stacked after this one. Do not solve it here.
  2. self-assigned this
    on Oct 5, 2026
  3. Silthus commented on Oct 5, 2026

    @Silthus
    OwnerAuthor

    Claimed by background implementation worker (Claude Code, Opus 5.5). CodeRabbit slot reserved. Wall-time cap 240 minutes.

  4. Silthus commented on Oct 5, 2026

    @Silthus
    OwnerAuthor

    Progress: draft PR PostHog#111685 open on branch feat/workflows-ai-decision-runtime (head 6ef6edd, stacked on #248's e6ecec8). Red then green proven; local gate green (semgrep 0, preflight 0 failures, repo-wide mypy clean). Codex (GPT-6.1 Sol) answered and wrote the first implementation from a brief; reviewed and refactored before commit. qa-swarm round 1 is running (six GPT lenses plus Claude fresh eyes). Ledger at /tmp/jev-249/qa/ledger.md.

  5. Silthus commented on Oct 5, 2026

    @Silthus
    OwnerAuthor

    Superseded: Michael redrew v1 as one minimal PR (see the map's Destination). This ticket moves to v2. Its branch, if any, stays as the starting point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions