Skip to content

What a decision sees, and how it stays safe #243

Closed

Description

@Silthus

Question

What should a decision see when it runs for one person, and how is that kept safe?

Cover:

  • State. Which of the event, person properties, workflow variables, and earlier steps' outputs go into the System One state, and whether the author picks them or the step sends a default set. Size: the facade caps state at 65,536 characters, and token count drives cost.
  • Instructions. Author text only. Confirm that templated user data never reaches instructions, as in PostHog/posthog#107832 and the HogQL decide() function, and say how the builder makes that clear to the author.
  • Options. How the author writes 2 to 16 options with descriptions for a pick-one question, and how a yes/no question exposes its threshold.
  • Low confidence. Whether an "unsure" outcome (confidence under a threshold) is a first-class branch, a default branch, or nothing.
  • Privacy. When to set privacy_mode, what lands in LLM analytics traces, and what the AI data processing approval implies for the step's visibility.
  • What Jev is good at. From the Jev and System One docs and existing callers (signals, error tracking severity, replay vision), which question shapes work well and which do not (free-form reasoning, arithmetic over many events).

End with a recommended state contract and author-facing inputs. Use /research, and write the findings to products/workflows/docs/research/decision-step-context.md on branch research/decision-step-context.

Activity

  1. self-assigned this
    on Oct 4, 2026
  2. Silthus commented on Oct 4, 2026

    @Silthus
    OwnerAuthor

    Claimed by background research worker (Claude Code, Opus 5.5). Codex substituted: the Codex runtime returned 429 at dispatch. Wall-time cap 60 minutes.

  3. Silthus commented on Oct 4, 2026

    @Silthus
    OwnerAuthor

    Resolved. Recommendation: the author picks the state, the question stays untemplated, and the step always runs in privacy mode with an optional Unsure branch.

    • State: a "Context" dictionary (field name to Hog template) sent as one JSON object. No default bundle: an event trigger pre-fills event: {event.event} only. Earlier steps reach it only through workflow variables (5,120-byte total cap, hogflow-executor.service.ts:947-962).
    • Size: cap the state at 8 KiB, not 65,536 characters. The deployed model has an 8,192-token cap (signals/.../report_safety_judge.py:27-28). The 65,536 figure is only the REST serializer's guard (ml_inference/.../serializers.py:9); the facade itself is unbounded. Over the cap, fail loudly and never truncate.
    • Instructions: author text only, wrapped as {question, data} the way HogQL decide() wraps its literal (prompt_jev.py:160-166). Gap: on a function template, templating: false is only a UI convention. InputsItemSerializer keeps a client-sent templating: "liquid" and the worker renders it (cdp/validation.py:659-662,932-938, hog-inputs.service.ts:48-52). A native action type closes the gap, or Django reads the question from the saved action.
    • Options: 2 to 16 described options (JevK5's A to P letters, facade/contracts.py:38-39), 500 characters each. The option name is the branch label.
    • Yes/no threshold: "Answer yes at or above N%", default 50, range 1 to 99. Callers use 0.3 to 0.9, and JevK5 compresses some scales (jev_watch_feed.py:75-82).
    • Unsure: a first-class branch, off by default. Pick-one uses the top probability. Yes/no uses a "no at or below" band. JevK5's confidence formula can't be checked from this repo, so the step does not depend on it.
    • Privacy: always set privacy_mode=True (facade/contracts.py:72-74). PR feat(workflows): add a classify with jev workflow action PostHog/posthog#107832 did not. Of the callers, only Replay Vision sets it. Check approval on every run through decide_when_available. Hide the step where it's unavailable or the flag is off. Show it disabled with the reason when approval is missing. The approval switch's copy names only third-party providers.
    • Fit: narrow, typed judgments about text. Counting, arithmetic, generation and literal patterns belong to filters, HogQL or Generate text. Jev lost to regexes on CI routing (docs/internal/ci-things-already-tried.md:526-538).

    Full findings, recommended contract and author inputs: https://github.com/Silthus/posthog/blob/research/decision-step-context/products/workflows/docs/research/decision-step-context.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions