Skip to content

docs(flags): specify SDK-sent $experiment_exposure for config version 2 flags - #162

Merged
andehen merged 10 commits into
mainfrom
docs/experiment-exposure
Nov 5, 2026
Merged

docs(flags): specify SDK-sent $experiment_exposure for config version 2 flags#162
andehen merged 10 commits into
mainfrom
docs/experiment-exposure

Conversation

@andehen

@andehen andehen commented Oct 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

This adds the OpenSpec change add-experiment-exposure for config version 2 flags ("flag rules v2"), which are built from ordered rules. For these flags, an SDK must:

  • send $experiment_exposure itself when a read lands on an experiment split;
  • add rule and variant attribution to $feature_flag_called;
  • dedupe the two events with two separate keys.

Applied and archived in this PR.

The problem

Today ingestion copies some multivariate $feature_flag_called events into $experiment_exposure, and reads the variant from the string in $feature_flag_response. That cannot:

  • tell an experiment variant from an ordinary string value;
  • tell apart variants with the same value (A/A tests);
  • tell apart two experiment rules on one flag, or a variant from a holdout or rollout exclusion with the same value;
  • handle variants whose values are booleans, numbers or objects.

A /flags?v=4 record for a config version 2 flag names the reason code, rule ID, rule type and variant key, so the SDK knows when a read assigned a variant. Ingestion is ready: it keeps a directly sent exposure, and it does not copy a $feature_flag_called that has $feature_flag_config_version 2 (PostHog/posthog#125449).

The wire contract is contracts/feature_flag_rules_v2 in posthog-sdk-test-harness 1.27.0 (contract 8.0.0). One rule needs posthog-sdk-test-harness#101 (ready for review, contract 9.0.0): it lets a $feature_flag_called that drops a broken rule pair, or that has a missing or unknown reason code, pass validation.

When SDKs send $experiment_exposure

The rule

A read that reports $feature_flag_called (its feature flag event decision is true) also sends one $experiment_exposure, deduped on its own key, when the record that supplied the value has all of these:

  1. config version 2;
  2. reason code experiment_split;
  3. rule type experiment;
  4. a rule ID (a hyphenated UUID string) and a variant key;
  5. a value that is not failed, bootstrapped or overridden.

No other outcome sends it: not a targeted or percentage match, rollout exclusion, holdout, paused experiment, default, missing or failed flag, or config version 1 flag. A linked experiment is not required.

The SDK decides from the record, not the returned value. A variant whose value is false, or a read that returned its caller default for null or false, still counts.

Edge cases

  • Type mismatch. A typed read of the wrong type sends neither event, takes no dedupe slot, and logs an error (docs(flags): propose typed flag values and evaluation details #104).
  • Malformed split. If a readable record's rule type is not experiment, or the rule ID or variant key is missing, there is no exposure. $feature_flag_called keeps config version 2 and the reason code, and drops rule ID, rule type and variant. The SDK never guesses the missing parts. A rule ID that is not a UUID string counts as missing. A known field of the wrong JSON type is a per-record parse error, not a malformed split.
  • Unknown reason code. A config version 2 record with a missing or unknown reason code sends $feature_flag_called with config version 2 and the raw code (none if missing), no rule attribution, and no exposure. The SDK does not map the code to another one.
  • Event opt-out. The event decision (the per-read option, else the client-level default, else true) governs both events. When it is false, neither event is sent and neither dedupe store is touched. Payload reads send nothing by default. A payload read that opted into events and evaluates the flag follows the normal trigger. A caller-supplied match value never sends an exposure or reserves its key.
  • Local evaluation. Config version 2 local evaluation sends the same exposure with locally_evaluated true. Config version 1 flags keep the ingestion path.
  • Snapshots. A snapshot read reports the assignment the snapshot holds.
  • Best-effort building. One step builds both events from the record and read context. It never changes the value or throws, and building the events makes no flags or definitions request. The events then go through the normal capture pipeline. Each event is attempted independently, and its dedupe key is added only after capture handles it: capture accepts it, or the application's before_send hook (or equivalent) discards it. A failed capture (a throw, a full or closed queue, opt-out or a disabled SDK) uses up no key and is tried again on the next read. An OpenFeature hook can reuse it.

Properties

$experiment_exposure, required:

Property Value
$feature_flag flag key
$feature_flag_response same as the paired call (diagnostic only)
$feature_flag_variant metadata.variant_key
$feature_flag_rule_id metadata.rule_id
locally_evaluated true for local evaluation, else false

It omits the config version, rule type, reason code and $experiment_exposure_source, because they never change for a split. The rule ID identifies the experiment, and only SDK-sent exposures have one.

When available, it adds the flag ID and version, request ID, evaluation time, definitions-loaded time, $lib, $lib_version and $groups. It shares the paired call's identity and session but has its own UUID. It never carries an experiment ID, holdout ID, forced-variant marker or assignment seed. Minimal-event mode does not apply to it.

$feature_flag_called, new properties for config version 2 only:

Outcome config version, reason code rule ID, rule type variant
Experiment split yes yes yes
Malformed split yes no no
Targeted or percentage match yes yes no
Rollout exclusion, paused, holdout yes yes no
Any of those with a rule ID or rule type missing, or a rule type the reason does not allow yes no no
No rule match, missing group key, flag disabled, failed yes no no
Config version 1, missing flag, bootstrap, override unchanged event
  • Rule ID and rule type go together or not at all.
  • $feature_flag_config_version is not $feature_flag_version (the row version).
  • $feature_flag_reason_code replaces $feature_flag_reason on config version 2 events only.
  • $feature_flag_response keeps today's format on both events: true for numbers and objects, false for null.
  • Minimal-event mode keeps the new properties. Config version 1 keeps today's allowlist.

Dedupe

  • $feature_flag_called: for config version 2, the existing key adds config version, reason code, rule ID, rule type and variant key. Each new outcome sends a new event: another A/A variant, a new rule, a holdout after a variant, a holdout then a pause, shipping a winner, a move to config version 2.
  • $experiment_exposure: its own size-limited store, keyed on distinct ID, groups, flag key, config version, rule ID and variant key. It leaves out the value, request ID, evaluation time and source, so a remote result after the same local assignment is one exposure.
  • The two events dedupe independently: a repeated $feature_flag_called that the tracker suppresses does not suppress a new exposure, and the reverse.
  • A shared cache keeps the two namespaces distinct, for example with the event name in the key, so a flag-called entry cannot block its exposure.
  • A flags reload never clears the exposure store.
  • Bootstrapped and overridden values never take an exposure key.

Worth looking at first

  • The $feature_flag_called table and the rule-pair rule.
  • Malformed splits.
  • What each dedupe key leaves out.
  • The short exposure property list (design decision 14).

Relation to #104

#104 (typed flag values) leaves the exposure, attribution and dedupe to this PR. Either can land first. The two agree on caller defaults, type mismatches, $feature_flag_response, bootstrap and overrides.

Implementation

posthog-python goes first: draft PostHog/posthog-python#1154, based on main. It builds on PostHog/posthog-python#1031, released in posthog 7.83.0.

Out of scope

Files and process

The change is applied and archived:

  • Canonical specs: openspec/specs/experiment-exposure/spec.md (new), and new requirements in openspec/specs/feature-flag-called-tracker/spec.md and openspec/specs/evaluate-flags/spec.md.
  • Change history: openspec/changes/archive/2026-11-05-add-experiment-exposure/ (proposal, design, tasks, deltas).
  • Acceptance: acceptance/private/experiment-exposure.feature (new), and attribution, dedupe-key and minimal-event scenarios in acceptance/private/feature-flag-called-tracker.feature. They keep the legacy @both, @client and @server tags, so the integration suite does not select them yet.
  • README: "Experiment Exposure" in the capability index.

Three tasks stay open, because they are outside this repository: the harness step bindings for these scenarios, the release of contract 9.0.0 (posthog-sdk-test-harness#101), and the check that experiment analysis reads $feature_flag_variant.

openspec validate --specs --strict passes.

🤖 Generated with Claude Code

@andehen
docs(flags): propose SDK-sent $experiment_exposure for config version…
996b180 
… 2 flags

Add the OpenSpec change add-experiment-exposure. A read that emits
$feature_flag_called also emits $experiment_exposure when the record that
supplied its value is a complete config version 2 experiment split. It adds
rule attribution to $feature_flag_called, extends its dedupe key for config
version 2, and gives the exposure its own dedupe namespace.
@andehen
docs(flags): add reason code and rule type to the config version 2 fl…
9445362 
…ag-called key

The config version 2 $feature_flag_called dedupe key now carries the whole
attribution the event sends: config version, reason code, rule ID, rule type
and variant key. Two variant-less outcomes of one rule with the same value,
such as a holdout and then a paused experiment, no longer collapse. The
exposure key is unchanged. Name posthog-python#1154 as the first
implementation.
This was referenced Oct 30, 2026
Merged
Draft
@andehen
docs(flags): trim the exposure, drop $feature_flag_reason on v2 calls…
a2c4cc4 
…, skip wrong-type reads

- A typed read of the wrong type sends neither $feature_flag_called nor
  $experiment_exposure and touches neither dedupe namespace. A read that
  returned its default because the value is null or false still reports
  the assignment.
- $experiment_exposure carries $feature_flag, $feature_flag_response,
  $feature_flag_variant, $feature_flag_rule_id and locally_evaluated, plus
  the flag id, version, request id, evaluated time and groups. It drops
  the config version, reason code, rule type and source marker, which are
  constants for an experiment split; the rule id marks an SDK exposure.
- Config version 2 $feature_flag_called events drop $feature_flag_reason;
  $feature_flag_reason_code replaces it. Config version 1 events keep it.
- Replace the open question on the exposure's response, and mark task 2.4
  done now that add-typed-flag-values carries the dedupe rewording. Add a
  task for the harness exposure projection.
@andehen andehen mentioned this pull request Oct 30, 2026
Merged
andehen added a commit to PostHog/posthog-sdk-test-harness that referenced this pull request Oct 30, 2026
@andehen
feat(flags): trim the SDK experiment exposure contract (#98)
0e31ca6 
Follow PostHog/sdk-specs#162. An SDK sends $experiment_exposure only for
a config version 2 experiment split, so the config version, rule type,
reason code and source marker were constants on it. Require exactly
$feature_flag, $feature_flag_response, $feature_flag_variant,
$feature_flag_rule_id and locally_evaluated, and drop the other four
properties so the closed schema rejects them. The rule ID identifies
the experiment rule and marks the exposure as SDK-sent: ingestion no
longer copies a $feature_flag_called that carries config version 2
(PostHog/posthog#125449), and a copy never carries a rule ID.

A config version 2 $feature_flag_called carries $feature_flag_reason_code
in place of $feature_flag_reason. Declare $feature_flag_reason in the
called context and reject it in the version 2 branch; version 1 calls
keep it with any value.

The event schemas have no fixture sets, so meta-tests build the
projections: each exposure property missing, each forbidden exposure
property present, and $feature_flag_reason on every presence-matrix
row's version 2 call. Contract package 8.0.0.
@andehen
docs(flags): point the exposure change at harness 1.27.0
05a1320 
Contract 8.0.0 trims the exposure projection and rejects
$feature_flag_reason on config version 2 calls, so task 2.5 is done.
@andehen
docs(flags): tighten experiment exposure spec wording
b3f8d49 
- Send the rule ID and rule type on $feature_flag_called as a pair or not
  at all, and only with a rule type the reason code allows; cover a
  targeting match that names an experiment rule.
- Scope the wider minimal-event allowlist to config version 2 events, with
  a scenario for a minimized config version 1 event.
- Say the exposure dedupe key uses the distinct ID as the string the event
  carries.
- State which retained snapshot details the exposure needs, and point to
  the existing evaluation-metadata retention for the rest.
- Align design decisions 2 and 14 and the proposal with the spec, add the
  reason for forbidding $feature_flag_forced_variant, name the /flags API
  version the minimal-event gate comes from, and fix the harness version
  in task 1.1.
@andehen andehen mentioned this pull request Nov 4, 2026
Merged
andehen added a commit to PostHog/posthog-python that referenced this pull request Nov 4, 2026
@andehen
fix(flags): send the rule ID and rule type as a pair, or neither
34012f1 
Follow PostHog/sdk-specs#162. On a config version 2 $feature_flag_called,
$feature_flag_rule_id and $feature_flag_rule_type are sent together, and
only with a rule type the reason code allows: targeted_release or
percentage_rollout for targeting_match, percentage_rollout or experiment
for rollout_miss, and experiment for experiment_paused, holdout and
experiment_split. A record with no rule ID, no rule type or another rule
type, including one this SDK does not know, sends neither. The event
keeps the config version and reason code, and the value is unchanged.

A targeting match that names an experiment rule gets no attribution and
no exposure; the exposure already required a complete experiment split.

Contract 8.0.0 rejects these calls without their rule pair, so the new
cases are not validated against the vendored schema until a harness
release that accepts them is copied.
@andehen
andehen marked this pull request as ready for review November 4, 2026 08:54
@andehen
andehen requested a review from a team as a code owner November 4, 2026 08:54
@andehen
docs(flags): say variant instead of arm, point the rule-pair rule at …
84285c3 
…contract 9.0.0
@andehen
docs(flags): dedupe the two events independently, scope malformed spl…
e9c2b83 
…its to readable records

- A tracked read checks the exposure key even when the flag-called
  tracker suppresses a repeat, and the reverse. A shared cache keeps the
  two namespaces distinct; the atomic check-and-mark alternative is gone,
  since it contradicted the separate exposure namespace.
- A malformed split is a readable config version 2 record. A known field
  of the wrong JSON type is a per-record parse error, as the wire reader
  contract says.
- Building the events makes no flags or definitions request; delivery
  follows the normal capture pipeline, which may send immediately.
- The happy-path scenario omits `failed`, which producers send only as true.
@andehen
docs(flags): settle unknown reasons, capture acceptance, non-UUID rul…
a2b17cc 
…e IDs, payload reads and reloads

- A config version 2 record with a missing or unknown reason code keeps
  config version 2 and the raw code, if any, with no rule attribution and
  no exposure. Contract 9.0.0 (harness #101) accepts these events.
- Each event is attempted independently, and its dedupe key is added only
  after capture accepts it. A failed or dropped capture adds no key.
  Config version 1 flag-called events keep the tracker's existing rules.
- A rule ID that is not a hyphenated UUID string counts as missing: no
  exposure, no rule pair.
- Payload reads stay silent by default. An opted-in payload read that
  evaluates the flag follows the trigger; a caller-supplied match value
  never sends an exposure or reserves its key.
- A flags reload never clears the exposure namespace.
andehen added a commit to PostHog/posthog-python that referenced this pull request Nov 4, 2026
@andehen
fix(flags): keep exposure dedupe only for accepted captures
a5b6c34 
Follow the updated PostHog/sdk-specs#162.

- A config version 2 $feature_flag_called and every $experiment_exposure
  keep their dedupe key only when capture() returns an event UUID
  without raising. The key is reserved while capture runs, so concurrent
  reads do not send the event twice, and released when capture drops the
  event (before_send, opt-out, a full or closed queue) or raises. A
  raising capture is logged, the read returns its value, and the other
  event is still attempted. Config version 1 calls keep their existing
  dedupe.
- A rule ID counts as present only in the hyphenated 8-4-4-4-12 UUID
  form. Any other string drops the rule pair and sends no exposure.
- A variant key is present when it is a string, so an empty variant key
  completes a split and sends the exposure.
- Super properties are merged before the attribution of both config
  version 2 events is set again from the record. A super property can no
  longer change the rule ID or variant, or add the reason description, a
  forced variant, an experiment ID, or exposure-forbidden properties.
  Config version 1 events still take super properties as before.

Tests cover unknown and missing reason codes, non-UUID rule IDs, empty
variant keys, a dropped and a raising capture of each event followed by
a second read, unchanged config version 1 dedupe, a flags reload, and
super properties on both clients. The scenario and parity mocks of
_enqueue now return an event UUID, as the real queue does.
@andehen
docs(flags): a before-send discard uses up the dedupe key
7e77778 
Capture handles an event when it accepts it or the application's
before-send hook discards it, and only then is the key added. A throw, a
full or closed queue, opt-out or a disabled SDK still add no key. The
discard is the application's choice, a later read would usually be
discarded again, and async SDKs run the hook after capture returns.

The queue-rejection scenario replaces the before-send one for a dropped
flag-called event, and a new scenario shows a discarded exposure keeping
its key.
andehen added a commit to PostHog/posthog-python that referenced this pull request Nov 4, 2026
@andehen
fix(flags): keep the dedupe key of an event before_send drops
be680d3 
The owner settled this in PostHog/sdk-specs#162: an event that before_send
drops uses up its dedupe key on both clients. The application chose to
drop it, and every later read would be dropped the same way. A raising
capture, a full or closed queue, opt-out and a disabled client still
release the key, so a later read tries again.

The async client already behaved this way, because its before_send runs
after capture() has queued the event. On the sync client, _capture_once
now passes a private _before_send_drops list through capture() to
_enqueue, which records a before_send drop there. A before_send that
raises still counts as a failed capture.

Tests: a before_send drop of each event on the sync client, and a drop of
the exposure on the async client with its worker running, are not sent
again. A raising capture and an unqueued capture of each event are still
sent by the second read. The config version 1 test now uses an unqueued
capture.
dustinbyrne
dustinbyrne reviewed Nov 4, 2026
View reviewed changes

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you ask your agent to apply and archive this change and commit?

it should produce paths like openspec/specs/experiment-exposure/spec.md and move the changes into openspec/changes/archive/YYYY-MM-DD-add-experiment-exposure

@dustinbyrne dustinbyrne mentioned this pull request Nov 4, 2026
Merged
@andehen @claude
docs(flags): apply and archive add-experiment-exposure
8a13670 
Sync the deltas into openspec/specs/experiment-exposure (new),
feature-flag-called-tracker and evaluate-flags, and move the change to
openspec/changes/archive/2026-11-05-add-experiment-exposure.

- Add acceptance/private/experiment-exposure.feature and the config
  version 2 attribution, dedupe-key and minimal-event scenarios in
  feature-flag-called-tracker.feature.
- Write the Purpose and Applicability of the new spec, and point the
  tracker's Behavior and Interactions prose at the config version 2 key
  and experiment-exposure.
- Add Experiment Exposure to the README capability index.

Tasks 2.3, 2.6 and 3.3 stay open: they are outside this repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andehen
andehen merged commit 0a9b697 into main Nov 5, 2026
10 checks passed
@andehen
andehen deleted the docs/experiment-exposure branch November 5, 2026 09:12
andehen added a commit to PostHog/posthog-sdk-test-harness that referenced this pull request Nov 5, 2026
@andehen
feat(flags): accept a flag call that drops a broken rule pair (contra…
425e9cd 
…ct 9.0.0) (#101)

* feat(flags): accept a flag call that drops a broken rule pair

Follow PostHog/sdk-specs#162. On a config version 2 $feature_flag_called,
$feature_flag_rule_id and $feature_flag_rule_type are sent together or
not at all, and only with a rule type the reason code allows. A record
with a missing rule ID, a missing rule type or a disallowed rule type
loses the whole pair; the call keeps the config version and reason code.

The called-context schema only tolerated this for a malformed
experiment_split. The presence-schema generator now emits one
dropped-attribution branch per reason that names a rule in the matrix
(targeting_match, experiment_split, rollout_miss, experiment_paused,
holdout). The malformed-split branch is generated the same way, with the
same shape. A half pair and a disallowed rule type are still rejected.

The meta-tests check a call without attribution for each of those
reasons, the complete call for every matrix row, and each half pair and
disallowed rule type. The set of accepted calls grows, so this is
contract package 9.0.0.

* feat(flags): a failed record yields no legacy payload

A failed record reads as disabled, so the legacy payload accessor yields
no payload for it, even when the record still carries metadata.payload.
readers.v1_failed is such a record: a producer-valid failed config
version 1 record with payload {"columns":2}. The contract defined its
error code but said nothing about the payload.

Reader expectations gain an optional legacy_payload: null for no
payload, otherwise the JSON-encoded payload string. readers.v1_failed
pins null. A meta-test checks every declared legacy_payload against the
record, and requires one on any failed reader case that carries a
payload. Part of contract package 9.0.0.

* fix(flags): pin legacy_payload as the decoded value and test stray attribution

- `legacy_payload` is the decoded JSON value the payload accessor returns, not
  the encoded wire string, as the reader section already says accessors decode.
  The meta-test decodes the wire payload and treats invalid JSON as no payload.
  `readers.v1_at_v4` declares its decoded payload so the encoding is pinned.
- The called-context meta-tests add each attribution field to a call without
  attribution, for every reason that names a rule and for `flag_disabled`, and
  check that a `flag_disabled` call without attribution is valid. Before this, a
  generator that stopped forbidding a field on a dropped call still passed.

* feat(flags): accept a flag call with a missing or unregistered reason code

A config version 2 `$feature_flag_called` whose `$feature_flag_reason_code` is
missing, or is not a registered reason code, is now valid if it carries no rule,
variant or experiment attribution. It keeps config version 2 and the raw code,
if any.

- The generator adds one branch for these calls. It excludes every reason that
  has a branch of its own, and every other branch now requires its reason code,
  so the branches do not overlap. A registered code still goes through its own
  branches: a `targeting_match` call with a half pair is still rejected.
- The `then` clause no longer requires a reason code on every version 2 call.
- Meta-tests: an unknown and a missing code without attribution are valid; each
  attribution field, and each complete rule pair, on such a call is invalid; the
  catch-all excludes exactly the registry's reason codes.
posthog Bot added a commit that referenced this pull request Nov 9, 2026
@posthog
docs(compliance): weekly SDK compliance audit — php, ruby, go
8232a36 
Deep re-audit of posthog-php, posthog-ruby and posthog-go against the 69
cross-SDK contracts, at sdk-specs 5160f2f.

Selection: php and ruby were the two stalest pointers (6 weeks, outside the
~4-week window) and had the largest backfill; go is the remaining matrix SDK
named in the mcp-analytics applicability table, which five of this cycle's 19
spec commits touch. All three are server SDKs, where the Emails, Exception
Event Metadata, Tracing Headers and config-version-2 flag deltas land.

Contract surface grew to 69 with Experiment Exposure (#162).

Main findings:
- Emails went from one implementation to four. All three SDKs ship it and all
  three pass, including #163's flattened send body, Cloud app-host resolution,
  and a once-per-call idempotency key.
- #129's span-context bridge and #123's trace context on exception events
  landed in all three. posthog-go is alone in getting the detached-work
  scenario right, and alone in the trace-context path being reachable only
  through an opt-in helper in a separate module.
- None of the three reads /flags?v=4, so Experiment Exposure is ➖ on all three
  by the spec's own applicability sentence — raised as a scope question.
- posthog-ruby ships a 3,798-line MCP package the spec's applicability table
  records as "not yet"; scored 🟡, with the table flagged for update.
- The #132 jitter requirement is now unmet on five of seven SDKs checked;
  posthog-go passes jitter: 0 at every ingestion call site while its own
  Emails client jitters.

Scope: compliance/ only.

Generated-By: PostHog Desktop
Task-Id: 86b4f98b-a338-40c2-912e-9e3a9dc366fd
@posthog posthog Bot mentioned this pull request Nov 9, 2026
Draft
This was referenced Nov 9, 2026
Merged
Merged
@turnipdabeets turnipdabeets mentioned this pull request Nov 10, 2026
Merged
@jurajmajerik jurajmajerik mentioned this pull request Nov 12, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dustinbyrne dustinbyrne dustinbyrne approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@andehen @dustinbyrne