Skip to content

feat(v2): run the /flags?v=4 read cases in the Gherkin runner - #103

Merged
andehen merged 17 commits into
mainfrom
andehen/flags-v4-parse-gherkin
Nov 9, 2026
Merged

andehen merged 17 commits into
mainfrom
andehen/flags-v4-parse-gherkin

Conversation

@andehen

@andehen andehen commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Runs the /flags?v=4 read cases of the YAML feature_flags_v4 suite (#100) in the v2 Gherkin runner, so SDKs with a Gherkin adapter, such as posthog-node, run the feature flag rules v2 reader fixtures and value corpus in their existing compliance job. The scenarios are in PostHog/sdk-specs#174 (merged as dd11a6b):

Feature Scenarios Requires Same as YAML test
remote-flags-v4-legacy-getter.feature 37 reader + 38 value flags_response_v4 legacy_getter_reads_reader_fixture, legacy_getter_reads_typed_value
remote-flags-v4-legacy-payload.feature 37 reader + 38 value + feature_flag_payloads legacy_payload_reads_reader_fixture, legacy_payload_is_the_typed_value

150 scenarios, the same count as the YAML suite. Contract 9.0.0 (#101) gives readers.v1_failed a legacy_payload expectation of null. flags_v4.py reads that expectation (f8e16d1), so the payload feature and the YAML suite both run the case.

Design: generated features with inline data

The other migrated features keep their inputs and expected results inline, and the v2 runner has no fixture-backed steps. So instead of a Scenario Outline over case IDs that loads contract fixtures at run time, bin/update-flags-v4-features.py --specs <sdk-specs checkout> writes ordinary scenarios. Each one shows its mock /flags body and expected result inline. A reviewer of the specs PR sees what each case checks without reading Python. The specs repo needs nothing from the harness at run time, and the v2 image does not need to package the contract.

The generator does not fork the suite logic. It builds bodies and expectations with posthog_test_harness.flags_v4 (response(), expectation(), case_ids()), as the YAML actions do. Two small public helpers were added there: contract_version() and reader().

To keep the specs and the contract together:

  • --check fails when a feature differs from what the contract generates.
  • The release workflow runs --check on the pinned specs before it builds the v2 distribution (new step "Check generated flags v4 features"). A release cannot ship a specs pin that disagrees with its contract.
  • tests/test_v2_flags_v4_reads.py runs the same comparison when SDK_V2_SPECS is set.

New steps (v2/remote_flag_steps.py)

  • the mock serves this /flags response body: queues the JSON doc string for up to 10 /flags requests (FLAGS_V4_RESPONSE_REPEAT from feat(flags): run the v4 reader fixtures and value corpus through an SDK #100), with raw_body, so it is sent byte for byte and a v4 envelope gets no default featureFlags.
  • get feature flag payload is called with JSON arguments: invokes /get_feature_flag_payload with the /get_feature_flag arguments. A thrown getter fails the case.
  • the public flag getter should return exactly JSON <value> uses strict JSON types (true is not 1). The existing ... should return JSON <value> keeps its YAML-parity Python equality, so the existing migrated cases do not change.
  • the public flag payload getter should return JSON <value> and ... should return this JSON: compare parsed JSON values (member order and number spelling may differ, types may not).
  • the public flag getter should return no value and the public flag payload getter should return no payload accept native undefined or JSON null. The YAML adapters turned both into null. v2 adapters report the native outcome, and posthog-node's getters return undefined for a missing flag.

Every result check above first requires that the SDK read the served body: at least one /flags request received that scenario's body, and every such request used v=4. Otherwise the case fails with flag_fixture_read, and the diagnostics give fixture_reads and fixture_read_versions. This matches the YAML suite's check from #100: a "no value" or "no payload" result cannot pass for a read that failed before it reached the mock, or for a read on another version. A thrown getter fails earlier.

The steps are documented in docs/harness-v2.md and docs/gherkin-steps.md.

Tests

tests/test_v2_flags_v4_reads.py generates the features into a temporary specs tree, so it runs in CI without SDK_V2_SPECS:

  • the scenario names are exactly the YAML suite's case IDs (75 + 75), and each scenario serves flags_v4.response() and reads the expected key;
  • capability selection: both capabilities select both features, flags_response_v4 alone selects the getter feature, and neither or only feature_flag_payloads selects nothing;
  • a controlled v4 reader (tests/v2_flags_v4_host.py) answers each served body with its contract expectation: all 150 cases pass, and every body reaches it without a merged featureFlags;
  • reads that skip the fixture fail with flag_fixture_read, even when the expected result is no value: no /flags request, a request to a path that never receives the body, and a v=2 read. A throwing reader fails as a thrown getter. A getter that reads /flags twice gets the body both times and passes;
  • wrong answers fail at the expected assertion with diagnostics: 1 for true, null for false, false for a missing flag, a JSON-encoded payload string, and a payload for a string flag. Equivalent answers pass: null for no value, and 0.0 for 0;
  • --check reports a stale feature.

The migrated scope grows from 157 to 308 cases and the acceptance scope from 188 to 339, in the discovery and bundle tests.

Stacking and landing order

Validation

  • uv run --frozen pytest with SDK_V2_SPECS set to a checkout of sdk-specs at dd11a6b: 2117 passed, 1 skipped. Without SDK_V2_SPECS, as in CI: 1615 passed, 503 skipped. CI does not set SDK_V2_SPECS, so it does not run the companion-spec tests against the pin.
  • bin/update-flags-v4-features.py --specs <sdk-specs at dd11a6b> --check passes.
  • Release dry run on the pinned specs: scripts/build_v2_distribution.py and scripts/smoke_v2_distribution.py pass, and the installed bundle's discover --migration-suite --require-ready (308 cases) and discover --acceptance-suite --require-ready (339 cases) pass.
  • ruff check clean; black --check flags the same eight files as the base, none of them changed here.
  • Local v2 image built from this branch and the specs branch, run with the posthog-node v2 compliance adapter in both capture modes: in both capture modes, a posthog-node build from feat(node): read the /flags?v=4 response and send $experiment_exposure posthog-js#5538 at 05035e4 (which declares flags_response_v4 and, with its new adapter commit, feature_flag_payloads) passes all 149 new scenarios. A build from posthog-js main selects none of them. Compared with main, the only other difference is the query check from #173: the v4 query scenario passes and the v2 one is not selected. Both builds fail the same four cases, which also fail with released harness 1.19.0: the three snapshot payload cases (false, 0, null) and "Omitted GeoIP settings default to enabled in flag requests".

🤖 Generated with Claude Code

The feature_flags_v4 suite serves mock /flags?v=4 responses built from the
feature flag rules v2 contract package and reads them through the SDK's
legacy accessors, one test per fixture case.

It requires flags_response_v4. get_feature_flag must return the legacy
rendering of every reader fixture record and every successful value corpus
case. With feature_flag_payloads, the legacy payload accessor must return a
config version 2 number or object value as the payload, keep a config
version 1 record's metadata.payload, and return no payload otherwise.
Payloads compare as parsed JSON values.

Adds the optional adapter route /get_feature_flag_payload, `cases:`
expansion for contract tests, and raw mock response bodies. Adapters
without flags_response_v4 run no new tests.
…it excludes

An @excludes:<capability> tag names a capability that replaces a case.
The runner does not select the case when the profile declares any such
capability, and an explicitly selected excluded case reports
sdk_capability_excluded. This lets flags_response_v4 replace the ?v=2
/flags query check with a ?v=4 check, as excludes already does in the
YAML suite.
Port the YAML feature_flags_v4 suite to the v2 Gherkin runner as two
generated specs features, so SDKs with a Gherkin adapter run the
feature flag rules v2 reader fixtures and value corpus too.

- bin/update-flags-v4-features.py writes
  remote-flags-v4-legacy-getter.feature (75 scenarios) and
  remote-flags-v4-legacy-payload.feature (74) into an sdk-specs
  checkout. It builds each mock body and expected result with
  posthog_test_harness.flags_v4, as the YAML suite does, and writes
  them inline. --check fails on drift; the release workflow runs it on
  the pinned specs.
- New steps: serve a /flags body byte for byte, call the legacy
  payload getter (/get_feature_flag_payload), compare getter results
  with strict JSON types, compare payloads as parsed JSON values, and
  accept native undefined or JSON null for no value or payload.
- A controlled v4 reader answers every generated case from the
  contract expectations; representative wrong answers fail the
  expected assertion.
- The migrated scope grows from 158 to 307 cases and the acceptance
  scope from 189 to 338.
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

minimal-adapter Compliance Report

Date: 2026-11-06T09:01:55.807173+00:00
Duration: 2620ms

✅ All Tests Passed!

17/17 tests passed


Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 10ms
Request Payload.Flags Request Uses V2 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 6ms
Request Payload.Flags Request Omits Authorization Header ✅ 6ms
Request Payload.Token In Flags Body Matches Init ✅ 8ms
Request Payload.Groups Round Trip ✅ 6ms
Request Payload.Groups Default To Empty Object ✅ 6ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 6ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 5ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 6ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 7ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 10ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 6ms
Retry Behavior.Retries Flags On 502 ✅ 1009ms
Retry Behavior.Retries Flags On 504 ✅ 1009ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 9ms

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

parallel-adapter Compliance Report

Date: 2026-11-06T09:01:16.401615+00:00
Duration: 1822ms

✅ All Tests Passed!

17/17 tests passed


Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 29ms
Request Payload.Flags Request Uses V2 Query Param ✅ 27ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 26ms
Request Payload.Flags Request Omits Authorization Header ✅ 23ms
Request Payload.Token In Flags Body Matches Init ✅ 20ms
Request Payload.Groups Round Trip ✅ 28ms
Request Payload.Groups Default To Empty Object ✅ 25ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 25ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 21ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 20ms
Request Lifecycle.No Flags Request On Init Alone ✅ 8ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 31ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 33ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 21ms
Retry Behavior.Retries Flags On 502 ✅ 1024ms
Retry Behavior.Retries Flags On 504 ✅ 1020ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 22ms

@posthog posthog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed fc9a435: 1 Must fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread .github/workflows/release.yml Resolved
A null expectation passed when the adapter call failed or the SDK never
requested /flags, and a request with the wrong v also got the fixture.
Each case assertion now needs success: true and at least one /flags
request that read the case's response, all with v=4.

The case's response now serves up to 10 /flags requests, so a payload
accessor that makes a second request still reads the fixture.
- Canonical cases are no longer selected when the profile declares a
  capability they exclude. Before, they were selected and then failed as
  unsupported_binding.
- An explicitly selected excluded case reports sdk_capability_excluded
  before other missing prerequisites: SDK type, routes, fixtures and
  capabilities. The selection reason also names the exclusion first.
- A case that both requires and excludes a capability, or has an empty
  @excludes: tag, is invalid source. Before, it could never be selected,
  and nothing reported it.
The contract defines the legacy rendering of a successful value but not
what a legacy getter returns on a reader error. Write down the rule the
suite asserts: a missing flag returns no value, and any other reader
error returns false.
A case that the profile excludes is replaced, not missing support. When it
is selected by ID, its inventory applicability is now not_applicable with
rule sdk-capability-excluded-v1 and the exclusion as the reason. It no
longer reports unsupported_binding with sdk_capability_excluded.

The case never reaches preflight, so the exclusion is still reported
before any other missing prerequisite. The strict exit code treats it as
neutral, like other not_applicable cases, and a run with no passed case
still fails.
A "no value" or "no payload" result also passed when the SDK never
requested /flags, so a read that failed before reaching the mock could
match. Each read assertion now first requires that at least one /flags
request received the scenario's body, all of them with v=4
(flag_fixture_read otherwise), as the YAML suite now does. The body is
queued for up to FLAGS_V4_RESPONSE_REPEAT requests, for getters that
call /flags more than once.

The controlled reader gains defects that skip /flags, request a path
that never receives the body, read with v=2, throw, or read twice.
Base automatically changed from andehen/har-005-remote-v2-suite to main November 5, 2026 09:23
andehen and others added 5 commits November 5, 2026 10:24
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contract 9.0.0 gives a reader case an optional legacy_payload
expectation, the decoded JSON value the legacy payload accessor returns.
flags_v4 now uses it: a declared legacy_payload decides the case and is
its expected payload. readers.v1_failed, a failed record that still
carries metadata.payload, declares null, so it now runs in the YAML
suite's legacy_payload_reads_reader_fixture and in the generated
payload feature, which grows from 74 to 75 scenarios.

Without a declared legacy_payload the earlier rule still applies, and
the payload assertion still refuses a case the contract leaves open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move SDK_SPECS_COMMIT to dd11a6b8f8c7be3013cd26d97782452ce0f8a76d
(PostHog/sdk-specs#174), which contains the generated
remote-flags-v4-legacy-getter and remote-flags-v4-legacy-payload
features and the v4 query scenario from PostHog/sdk-specs#173. The new
release check now passes on the pinned specs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new sdk-specs pin adds scenarios that the companion tests count:

- on-feature-flags.feature gains a client-only failed-load listener
  scenario (PostHog/sdk-specs#94), so the server profile sees 4
  not-applicable cases.
- identify.feature gains two client-only opted-out identify scenarios
  (PostHog/sdk-specs#167), so the feature set has 11 cases.
- evaluate-flags.feature gains seven "Server snapshot" event opt-out
  scenarios tagged @server, not @SDK:server (PostHog/sdk-specs#111).
  The snapshot matrix now selects only @SDK:server scenarios, the same
  scope as the acceptance suite, so it keeps its 21 cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andehen
andehen marked this pull request as ready for review November 6, 2026 09:05
@andehen
andehen requested a review from a team as a code owner November 6, 2026 09:05

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

stamphog can't approve this pull request automatically, because two gates refused it. The deny-list gate matched infra_cicd, since the PR changes .github/workflows/release.yml (a new release-workflow step and a bumped specs pin). The tier gate classified the PR as T2-never: it is a cross-cutting feature touching 21 files and 845 lines, including the runner steps, the generator script, and the contract.

The size gate passed, so splitting is not required, but it would help. Moving the release workflow change into its own PR would make it easier for a human to review. Otherwise, ask a human reviewer to take the whole PR.

Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✗ matches: infra_cicd
size ✓ 378L, 5F substantive, 845L/21F incl. docs/generated/snapshots — within ceiling
tier ✗ classified as T2-never: T2-never (845L, 21F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ unknown · reviewed head 42be936

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for doing this - the Gherkin suite is still a work in progress with much more to be done, so the help is greatly appreciated!! 🙌

@andehen
andehen merged commit 168f46a into main Nov 9, 2026
14 checks passed
@andehen
andehen deleted the andehen/flags-v4-parse-gherkin branch November 9, 2026 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants