Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion acceptance/private/feature-flag-called-tracker.feature
Original file line number Diff line number Diff line change
Expand Up @@ -185,11 +185,29 @@ Feature: Feature Flag Called Tracker
Examples:
| outcome | flag | value | failed | reason_code | rule_type | rule_id | sent_reason_code |
| no rule match | beta-ui | false | absent | "no_rule_match" | absent | absent | no_rule_match |
| failed record | beta-ui | false | true | "dependency_error" | absent | absent | dependency_error |
| unknown reason code | beta-ui | true | absent | "future_reason" | "targeted_release" | "c41f9e07-2d6b-4a83-b5e1-7f0a3c8d2e96" | future_reason |
| rule id without rule type | beta-ui | true | absent | "targeting_match" | absent | "c41f9e07-2d6b-4a83-b5e1-7f0a3c8d2e96" | targeting_match |
| targeting match on experiment | checkout-flow | true | absent | "targeting_match" | "experiment" | "3f6b1c2e-7a4d-4b8e-9c21-5d0e8f7a6b13" | targeting_match |

@both
Scenario: A failed record carries no rule identity
Given the SDK is initialized with token "test-token"
And the flags response for distinct id "user-123" has a record for "beta-ui" with:
| field | value |
| value | false |
| failed | true |
| reason.code | "dependency_error" |
| metadata.config_version | 2 |
| metadata.rule_type | absent |
| metadata.rule_id | absent |
When get feature flag "beta-ui" is called for distinct id "user-123"
Then the enqueued event property "$feature_flag_config_version" should equal "2"
And the enqueued event property "$feature_flag_reason_code" should equal "dependency_error"
And the enqueued event properties should not include "$feature_flag_rule_id"
And the enqueued event properties should not include "$feature_flag_rule_type"
And the enqueued event properties should not include "$feature_flag_variant_key"
And no event named "$experiment_exposure" should be enqueued

@both
Scenario: A record without a reason code keeps its config version
Given the SDK is initialized with token "test-token"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-11-11
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
## Context

The backend's detailed `/flags` response distinguishes a failed evaluation from a successful boolean false using `flags[key].failed`. Existing getter specs describe no result on evaluation failure and partial bulk results, but do not connect those rules to this wire field. The experiment-exposure and config-version-2 attribution specs already mention failed records; they do not define direct getter values.

Reference evidence, inspected at immutable revisions:

- [Backend `FlagDetails`](https://github.com/PostHog/posthog/blob/b0558549cf700f9ae9cc640ccf05ead533951834/rust/feature-flags/src/api/types.rs#L459-L472) serializes the failure marker only when true. [`create_error`](https://github.com/PostHog/posthog/blob/b0558549cf700f9ae9cc640ccf05ead533951834/rust/feature-flags/src/api/types.rs#L749-L769) sets `enabled: false`, no variant/payload, and `failed: true`.
- [Go's remote-only getter](https://github.com/PostHog/posthog-go/blob/86e57623afc6382e2e8403855dc1c269a3f13ee8/posthog.go#L2096-L2105) already treats failed details as no value. Its [error classification](https://github.com/PostHog/posthog-go/blob/86e57623afc6382e2e8403855dc1c269a3f13ee8/feature_flag_errors.go#L150-L178) reports `evaluation_failed`. This is the single-flag reference, not the poller/bulk implementation on that revision.
- [Go PR #434](https://github.com/PostHog/posthog-go/pull/434), [commit 023c983](https://github.com/PostHog/posthog-go/commit/023c9832b4bb39c3c63204bcb0fb0edd42e6fadb), adds poller and bulk/capture filtering and tests preservation of a local true value. It is an implementation in progress, not evidence of a released cross-SDK contract.

Source-level audit of known divergences, not a runtime conformance result:

| SDK and inspected revision | Evidence | Required correction |
| --- | --- | --- |
| Node, `d2e010e5e40636cde886f7abf1e44914c8cf04cf` | [Direct getter](https://github.com/PostHog/posthog-js/blob/d2e010e5e40636cde886f7abf1e44914c8cf04cf/packages/node/src/client.ts#L1837-L1867) constructs a result from enabled/variant without checking failed; [core value conversion](https://github.com/PostHog/posthog-js/blob/d2e010e5e40636cde886f7abf1e44914c8cf04cf/packages/core/src/featureFlagUtils.ts#L76-L108) also copies failed placeholders. | Return no structured result for failure without a fallback, omit failed bulk values, and classify failed tracked reads. Separate SDK changes required. |
| Python, `9d154daecc9330b806d052a631547447df941941` | [Result conversion](https://github.com/PostHog/posthog-python/blob/9d154daecc9330b806d052a631547447df941941/posthog/types.py#L613-L648) copies enabled/variant; [bulk conversion](https://github.com/PostHog/posthog-python/blob/9d154daecc9330b806d052a631547447df941941/posthog/types.py#L702-L711) includes every parsed flag. The private failure marker does not make these legacy conversions omit the result. | Check per-record failure before legacy result/bulk conversion and preserve its error classification. Separate SDK changes required. |

## Goals / Non-Goals

**Goals:**

- Make the agreed no-decision interpretation explicit for direct getters and bulk/capture merges.
- Preserve healthy boolean false and variant values, including in partial-error responses.
- Keep successful local decisions for the same evaluation context when another flag triggers remote fallback.
- Distinguish a present-but-failed remote flag from an absent key in direct getter tracking.

**Non-Goals:**

- Change public signatures, legacy scalar/boolean missing-result sentinels, retries, persistent-cache fallback policy, or cohort evaluation.
- Redesign snapshot record inclusion, unresolved reasons, filtering, or lazy access tracking in `evaluate-flags`.
- Implement SDK fixes or change acceptance-harness bindings, integration tags, or compliance claims.

## Decisions

### Interpret the per-record marker, not the response-wide error bit

Only a record explicitly marked `failed: true` establishes this per-flag failure. A response-wide `errorsWhileComputingFlags` bit can accompany successful siblings and is not a reason to drop every flag. Conversely, a failed detail remains failed even if that response-wide bit is absent or false.

Treating `enabled: false` as authoritative despite the marker was rejected: it collapses evaluation failure into a real disabled decision. Treating every partial response as a total failure was rejected because it discards healthy siblings.

### Preserve a successful local value without treating failure as a tombstone

A failed remote detail supplies no replacement value. During a single bulk evaluation it must neither overwrite nor delete an already resolved local true, false, or variant for the same identity/context. This does not authorize reuse across users or introduce a stale-result retention policy. Successful remote values keep each API's existing precedence.

### Keep getter results separate from telemetry diagnostics

A direct getter with no successful value returns its idiomatic no-result outcome rather than manufacturing a disabled structured result. Existing scalar and boolean APIs may still map that outcome to their documented sentinel/default. If the read emits `$feature_flag_called`, its error identifies `evaluation_failed`; normal dedupe, event controls, response-level errors, and unavailable-value event sentinels remain in force.

### Add normative scenarios without claiming harness execution

The change adds scenarios to the three OpenSpec capabilities through delta specs and CLI archive/sync. It does not add executable `acceptance/` cases: shared-harness binding work and real-adapter execution are separate follow-ups, not implied by OpenSpec validation.

## Risks / Trade-offs

- Existing callers may observe a no-result outcome or a missing bulk key instead of false. → Document the distinction and retain legacy public sentinel/error conventions.
- Current SDKs differ. → Record inspected source revisions and leave implementation/conformance claims to SDK-specific follow-ups.
- Reusing a local value from another context would be unsafe. → Limit merge preservation to a successful local decision for the same evaluation context.
- Snapshot missing/unresolved tracking has its own precedence rules. → Keep this clarification scoped to direct getters and bulk/capture merging rather than silently changing that separate contract.

## Migration Plan

Sync and archive this spec change on the same branch. Implement the Node/Python corrections and finish the Go fix in their SDK repositories; add executable cross-SDK coverage separately. No dependency, release pin, or SDK code changes belong in this branch.

## Open Questions

None for the agreed direct-getter and bulk/capture contract. Snapshot-specific failed-record behavior remains outside this change.
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
## Why

The backend can return a per-flag `failed: true` record with an `enabled: false` placeholder. The specs describe evaluation failures generally but do not explicitly prevent direct getters and bulk conversions from presenting that placeholder as a successful disable.

## What Changes

This is a canonical specification clarification, not a claim that all SDKs already conform.

- Define a failed remote record as no evaluated value, independently of response-wide `errorsWhileComputingFlags`.
- Return no structured result when direct flag evaluation fails and no successful fallback supplies a value; preserve documented legacy scalar/boolean missing-result mappings.
- Omit failed remote values from bulk getters and automatic capture enrichment without overwriting a successful local decision for the same evaluation context.
- Report `evaluation_failed` when a tracked direct getter has no value because its remote record failed, rather than reporting a missing flag or a successful disable.
- Cover healthy disabled flags, absent/false failure markers, partial responses, local-value preservation, and tracking suppression with normative scenarios.

## Capabilities

### New Capabilities

None.

### Modified Capabilities

- `get-feature-flag-result`: Distinguish a failed remote detail from a valid disabled result.
- `get-feature-flags`: Filter failed remote values and preserve successful local values during bulk/capture merges.
- `feature-flag-called-tracker`: Identify failed direct getter evaluations with `evaluation_failed` while retaining existing tracking controls.

## Impact

No new public API or SDK implementation is added here. The backend error record and Go's existing remote-only single-flag path supply the reference behavior; PostHog/posthog-go#434 extends it to that SDK's poller and bulk paths. Inspected Node and Python getter/conversion paths still expose failed placeholders and need separate SDK fixes. Source evidence and scope limits are recorded in the design.
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
## ADDED Requirements

### Requirement: Failed direct getter evaluations report evaluation_failed

When a direct single-flag getter has no successfully evaluated value for its context because its detailed remote record is marked `failed: true`, a `$feature_flag_called` event emitted for that read SHALL include `evaluation_failed` in `$feature_flag_error`. A present-but-failed record MUST NOT be classified as `flag_missing` merely because it supplies no value. This per-flag classification SHALL NOT depend on the response-wide `errorsWhileComputingFlags` bit; existing response-level errors remain independently reportable.

The SDK SHALL preserve its existing unavailable-value response sentinel, dedupe behavior, and feature flag event controls. This requirement MUST NOT cause an event when the read's event decision is false. A successfully evaluated value from the source actually used for the read MUST NOT acquire `evaluation_failed` solely because an unused remote record or another key failed. Snapshot missing/unresolved access rules remain governed by `evaluate-flags`.

#### Scenario: A present failed flag is not reported as missing (@both)
- **GIVEN** a fresh SDK instance has no successful fallback for "checkout"
- **AND** remote evaluation returns "checkout" with enabled false and failed true
- **AND** the read's feature flag event decision is true
- **WHEN** get feature flag result "checkout" is called
- **AND** the public flush completes delivery to an isolated mock ingestion service
- **THEN** the received "$feature_flag_called" event for "checkout" reports "evaluation_failed" in "$feature_flag_error"
- **AND** it does not report "flag_missing"

#### Scenario: Failed flag diagnostics coexist with response-wide errors (@both)
- **GIVEN** a fresh SDK instance has no successful fallback for "checkout"
- **AND** remote evaluation returns "checkout" with failed true and reports errorsWhileComputingFlags true
- **AND** the read's feature flag event decision is true
- **WHEN** get feature flag result "checkout" is called
- **AND** the public flush completes delivery to an isolated mock ingestion service
- **THEN** the received "$feature_flag_called" event for "checkout" reports "evaluation_failed"
- **AND** the SDK's existing response-wide error diagnostics are preserved

#### Scenario: A healthy sibling does not inherit another flag's failure (@both)
- **GIVEN** remote evaluation returns "checkout" with failed true and "beta-ui" with enabled false and failed false
- **AND** the read's feature flag event decision is true
- **WHEN** get feature flag result "beta-ui" is called
- **AND** the public flush completes delivery to an isolated mock ingestion service
- **THEN** the received "$feature_flag_called" event for "beta-ui" reports response false
- **AND** it does not report "evaluation_failed"

#### Scenario: Event suppression also applies to failed direct getter reads (@both)
- **GIVEN** remote evaluation returns "checkout" with failed true and no successful fallback supplies a value
- **AND** the read's feature flag event decision is false
- **WHEN** get feature flag result "checkout" is called
- **AND** the public flush completes for an isolated mock ingestion service
- **THEN** no "$feature_flag_called" event for "checkout" is received
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
## ADDED Requirements

### Requirement: Failed remote flag details are not evaluated results

When a direct flag getter consumes a detailed `/flags` response, a per-flag `failed: true` marker SHALL mean that the server produced no evaluated value for that key. The SDK MUST NOT convert that record's `enabled: false` placeholder or its variant into a successful structured result. This rule SHALL apply whether the getter uses remote evaluation directly or falls back from local evaluation, and whether `errorsWhileComputingFlags` is true, false, or absent.

If no successful local or otherwise supported fallback supplies a value for the same evaluation context, `get-feature-flag-result` SHALL return the platform's no-result outcome. The SDK MAY return an idiomatic error alongside that outcome where its public API supports one. Scalar and boolean getters that share this evaluation path SHALL retain their documented no-result/default mappings, including a legacy false sentinel; such a sentinel MUST NOT be treated as a successfully evaluated disabled flag.

An absent `failed` marker or explicit `failed: false` SHALL NOT by itself invalidate a record. In particular, a healthy boolean false SHALL remain a valid disabled result even when another flag caused response-wide evaluation errors. Legacy response formats without per-flag failure markers SHALL retain their existing value interpretation.

#### Scenario: Failed direct remote evaluation returns no structured result (@both)
- **GIVEN** a fresh SDK instance has no successful local or cached fallback for "checkout"
- **AND** the detailed remote response contains "checkout" with enabled false and failed true
- **WHEN** get feature flag result "checkout" consumes that response
- **THEN** no feature flag result is returned
- **AND** no successful disabled result is manufactured from the placeholder

#### Scenario: Failed remote fallback does not turn an inconclusive local result into false (@server)
- **GIVEN** local evaluation for "checkout" is inconclusive
- **AND** remote fallback is enabled
- **AND** the remote response contains "checkout" with enabled false and failed true
- **WHEN** get feature flag result "checkout" is called for that evaluation context
- **THEN** no feature flag result is returned
- **AND** the documented no-result error convention remains permitted

#### Scenario: Per-flag failure does not depend on the response-wide error bit (@both)
- **GIVEN** a fresh SDK instance has no successful fallback for "checkout"
- **AND** a detailed remote response contains "checkout" with failed true
- **AND** errorsWhileComputingFlags is false or absent
- **WHEN** get feature flag result "checkout" consumes that response
- **THEN** no feature flag result is returned

#### Scenario: A healthy disabled sibling remains a valid result (@both)
- **GIVEN** a detailed remote response reports errorsWhileComputingFlags true
- **AND** "checkout" has enabled false and failed true
- **AND** "beta-ui" has enabled false with its failed marker absent or false
- **WHEN** get feature flag result "beta-ui" consumes that response
- **THEN** a feature flag result with key "beta-ui" and enabled false is returned
Loading
Loading