Repository navigation
feat(v2): run the /flags?v=4 read cases in the Gherkin runner - #103
Conversation
The feature_flags_v4 suite serves mock /flags?v=4 responses built from the feature flag rules v2 contract package and reads them through the SDK's legacy accessors, one test per fixture case. It requires flags_response_v4. get_feature_flag must return the legacy rendering of every reader fixture record and every successful value corpus case. With feature_flag_payloads, the legacy payload accessor must return a config version 2 number or object value as the payload, keep a config version 1 record's metadata.payload, and return no payload otherwise. Payloads compare as parsed JSON values. Adds the optional adapter route /get_feature_flag_payload, `cases:` expansion for contract tests, and raw mock response bodies. Adapters without flags_response_v4 run no new tests.
…it excludes An @excludes:<capability> tag names a capability that replaces a case. The runner does not select the case when the profile declares any such capability, and an explicitly selected excluded case reports sdk_capability_excluded. This lets flags_response_v4 replace the ?v=2 /flags query check with a ?v=4 check, as excludes already does in the YAML suite.
Port the YAML feature_flags_v4 suite to the v2 Gherkin runner as two generated specs features, so SDKs with a Gherkin adapter run the feature flag rules v2 reader fixtures and value corpus too. - bin/update-flags-v4-features.py writes remote-flags-v4-legacy-getter.feature (75 scenarios) and remote-flags-v4-legacy-payload.feature (74) into an sdk-specs checkout. It builds each mock body and expected result with posthog_test_harness.flags_v4, as the YAML suite does, and writes them inline. --check fails on drift; the release workflow runs it on the pinned specs. - New steps: serve a /flags body byte for byte, call the legacy payload getter (/get_feature_flag_payload), compare getter results with strict JSON types, compare payloads as parsed JSON values, and accept native undefined or JSON null for no value or payload. - A controlled v4 reader answers every generated case from the contract expectations; representative wrong answers fail the expected assertion. - The migrated scope grows from 158 to 307 cases and the acceptance scope from 189 to 338.
minimal-adapter Compliance ReportDate: 2026-11-06T09:01:55.807173+00:00 ✅ All Tests Passed!17/17 tests passed Feature_Flags Tests✅ 17/17 tests passed View Details
|
parallel-adapter Compliance ReportDate: 2026-11-06T09:01:16.401615+00:00 ✅ All Tests Passed!17/17 tests passed Feature_Flags Tests✅ 17/17 tests passed View Details
|
There was a problem hiding this comment.
PostHog Review · Standard · reviewed fc9a435: 1 Must fix.
This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
A null expectation passed when the adapter call failed or the SDK never requested /flags, and a request with the wrong v also got the fixture. Each case assertion now needs success: true and at least one /flags request that read the case's response, all with v=4. The case's response now serves up to 10 /flags requests, so a payload accessor that makes a second request still reads the fixture.
- Canonical cases are no longer selected when the profile declares a capability they exclude. Before, they were selected and then failed as unsupported_binding. - An explicitly selected excluded case reports sdk_capability_excluded before other missing prerequisites: SDK type, routes, fixtures and capabilities. The selection reason also names the exclusion first. - A case that both requires and excludes a capability, or has an empty @excludes: tag, is invalid source. Before, it could never be selected, and nothing reported it.
The contract defines the legacy rendering of a successful value but not what a legacy getter returns on a reader error. Write down the rule the suite asserts: a missing flag returns no value, and any other reader error returns false.
A case that the profile excludes is replaced, not missing support. When it is selected by ID, its inventory applicability is now not_applicable with rule sdk-capability-excluded-v1 and the exclusion as the reason. It no longer reports unsupported_binding with sdk_capability_excluded. The case never reaches preflight, so the exclusion is still reported before any other missing prerequisite. The strict exit code treats it as neutral, like other not_applicable cases, and a run with no passed case still fails.
A "no value" or "no payload" result also passed when the SDK never requested /flags, so a read that failed before reaching the mock could match. Each read assertion now first requires that at least one /flags request received the scenario's body, all of them with v=4 (flag_fixture_read otherwise), as the YAML suite now does. The body is queued for up to FLAGS_V4_RESPONSE_REPEAT requests, for getters that call /flags more than once. The controlled reader gains defects that skip /flags, request a path that never receives the body, read with v=2, throw, or read twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contract 9.0.0 gives a reader case an optional legacy_payload expectation, the decoded JSON value the legacy payload accessor returns. flags_v4 now uses it: a declared legacy_payload decides the case and is its expected payload. readers.v1_failed, a failed record that still carries metadata.payload, declares null, so it now runs in the YAML suite's legacy_payload_reads_reader_fixture and in the generated payload feature, which grows from 74 to 75 scenarios. Without a declared legacy_payload the earlier rule still applies, and the payload assertion still refuses a case the contract leaves open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Move SDK_SPECS_COMMIT to dd11a6b8f8c7be3013cd26d97782452ce0f8a76d (PostHog/sdk-specs#174), which contains the generated remote-flags-v4-legacy-getter and remote-flags-v4-legacy-payload features and the v4 query scenario from PostHog/sdk-specs#173. The new release check now passes on the pinned specs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new sdk-specs pin adds scenarios that the companion tests count: - on-feature-flags.feature gains a client-only failed-load listener scenario (PostHog/sdk-specs#94), so the server profile sees 4 not-applicable cases. - identify.feature gains two client-only opted-out identify scenarios (PostHog/sdk-specs#167), so the feature set has 11 cases. - evaluate-flags.feature gains seven "Server snapshot" event opt-out scenarios tagged @server, not @SDK:server (PostHog/sdk-specs#111). The snapshot matrix now selects only @SDK:server scenarios, the same scope as the acceptance suite, so it keeps its 21 cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… count Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
stamphog can't approve this pull request automatically, because two gates refused it. The deny-list gate matched infra_cicd, since the PR changes .github/workflows/release.yml (a new release-workflow step and a bumped specs pin). The tier gate classified the PR as T2-never: it is a cross-cutting feature touching 21 files and 845 lines, including the runner steps, the generator script, and the contract.
The size gate passed, so splitting is not required, but it would help. Moving the release workflow change into its own PR would make it easier for a human to review. Otherwise, ask a human reviewer to take the whole PR.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✗ | matches: infra_cicd |
| size | ✓ | 378L, 5F substantive, 845L/21F incl. docs/generated/snapshots — within ceiling |
| tier | ✗ | classified as T2-never: T2-never (845L, 21F, cross-cutting, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ unknown · reviewed head 42be936 |
dustinbyrne
left a comment
There was a problem hiding this comment.
thanks for doing this - the Gherkin suite is still a work in progress with much more to be done, so the help is greatly appreciated!! 🙌
Summary
Runs the
/flags?v=4read cases of the YAMLfeature_flags_v4suite (#100) in the v2 Gherkin runner, so SDKs with a Gherkin adapter, such as posthog-node, run the feature flag rules v2 reader fixtures and value corpus in their existing compliance job. The scenarios are in PostHog/sdk-specs#174 (merged asdd11a6b):remote-flags-v4-legacy-getter.featureflags_response_v4legacy_getter_reads_reader_fixture,legacy_getter_reads_typed_valueremote-flags-v4-legacy-payload.featurefeature_flag_payloadslegacy_payload_reads_reader_fixture,legacy_payload_is_the_typed_value150 scenarios, the same count as the YAML suite. Contract 9.0.0 (#101) gives
readers.v1_failedalegacy_payloadexpectation ofnull.flags_v4.pyreads that expectation (f8e16d1), so the payload feature and the YAML suite both run the case.Design: generated features with inline data
The other migrated features keep their inputs and expected results inline, and the v2 runner has no fixture-backed steps. So instead of a
Scenario Outlineover case IDs that loads contract fixtures at run time,bin/update-flags-v4-features.py --specs <sdk-specs checkout>writes ordinary scenarios. Each one shows its mock/flagsbody and expected result inline. A reviewer of the specs PR sees what each case checks without reading Python. The specs repo needs nothing from the harness at run time, and the v2 image does not need to package the contract.The generator does not fork the suite logic. It builds bodies and expectations with
posthog_test_harness.flags_v4(response(),expectation(),case_ids()), as the YAML actions do. Two small public helpers were added there:contract_version()andreader().To keep the specs and the contract together:
--checkfails when a feature differs from what the contract generates.--checkon the pinned specs before it builds the v2 distribution (new step "Check generated flags v4 features"). A release cannot ship a specs pin that disagrees with its contract.tests/test_v2_flags_v4_reads.pyruns the same comparison whenSDK_V2_SPECSis set.New steps (
v2/remote_flag_steps.py)the mock serves this /flags response body:queues the JSON doc string for up to 10/flagsrequests (FLAGS_V4_RESPONSE_REPEATfrom feat(flags): run the v4 reader fixtures and value corpus through an SDK #100), withraw_body, so it is sent byte for byte and a v4 envelope gets no defaultfeatureFlags.get feature flag payload is called with JSON arguments:invokes/get_feature_flag_payloadwith the/get_feature_flagarguments. A thrown getter fails the case.the public flag getter should return exactly JSON <value>uses strict JSON types (trueis not1). The existing... should return JSON <value>keeps its YAML-parity Python equality, so the existing migrated cases do not change.the public flag payload getter should return JSON <value>and... should return this JSON:compare parsed JSON values (member order and number spelling may differ, types may not).the public flag getter should return no valueandthe public flag payload getter should return no payloadaccept native undefined or JSONnull. The YAML adapters turned both intonull. v2 adapters report the native outcome, and posthog-node's getters returnundefinedfor a missing flag.Every result check above first requires that the SDK read the served body: at least one
/flagsrequest received that scenario's body, and every such request usedv=4. Otherwise the case fails withflag_fixture_read, and the diagnostics givefixture_readsandfixture_read_versions. This matches the YAML suite's check from #100: a "no value" or "no payload" result cannot pass for a read that failed before it reached the mock, or for a read on another version. A thrown getter fails earlier.The steps are documented in
docs/harness-v2.mdanddocs/gherkin-steps.md.Tests
tests/test_v2_flags_v4_reads.pygenerates the features into a temporary specs tree, so it runs in CI withoutSDK_V2_SPECS:flags_v4.response()and reads the expected key;flags_response_v4alone selects the getter feature, and neither or onlyfeature_flag_payloadsselects nothing;tests/v2_flags_v4_host.py) answers each served body with its contract expectation: all 150 cases pass, and every body reaches it without a mergedfeatureFlags;flag_fixture_read, even when the expected result is no value: no/flagsrequest, a request to a path that never receives the body, and av=2read. A throwing reader fails as a thrown getter. A getter that reads/flagstwice gets the body both times and passes;1fortrue,nullforfalse,falsefor a missing flag, a JSON-encoded payload string, and a payload for a string flag. Equivalent answers pass:nullfor no value, and0.0for0;--checkreports a stale feature.The migrated scope grows from 157 to 308 cases and the acceptance scope from 188 to 339, in the discovery and bundle tests.
Stacking and landing order
mainis merged into this branch at4a3da5a, so the diff againstmainholds only this PR's own commits:fc9a435,4e4ef57,f8e16d1,289da30and8c5ab00.7d6b950) and test(flags): read /flags?v=4 records through the legacy flag and payload getters sdk-specs#174 (merged asdd11a6b) are both merged.SDK_SPECS_COMMITinrelease.yml(and its copies inRELEASING.mdandtests/test_release_workflow.py) isdd11a6b8f8c7be3013cd26d97782452ce0f8a76d, the #174 merge commit, which contains the two features (289da30). The move also brings the other sdk-specs changes merged since the old pindef596d. Three companion tests counted those scenarios and are updated in8c5ab00:test_v2_boundary.py:on-feature-flags.featurehas 4 client-only cases (docs(flags): record the iOS onFeatureFlags listener and failed-load delivery sdk-specs#94).test_v2_identify_alias.py: the identify and alias features have 11 cases (fix(identify): keep identify running while opted out sdk-specs#167 adds two client-only ones).test_v2_snapshot_reads.py: docs(flags): honor the feature flag event opt-out on snapshot reads sdk-specs#111 adds seven "Server snapshot" event opt-out scenarios tagged@server, not@sdk:server. The healthy matrix now selects only@sdk:serverscenarios, the acceptance-suite scope, so it keeps its 21 cases.legacy_payloadswitch, which also changes the YAML suite.Validation
uv run --frozen pytestwithSDK_V2_SPECSset to a checkout of sdk-specs atdd11a6b: 2117 passed, 1 skipped. WithoutSDK_V2_SPECS, as in CI: 1615 passed, 503 skipped. CI does not setSDK_V2_SPECS, so it does not run the companion-spec tests against the pin.bin/update-flags-v4-features.py --specs <sdk-specs at dd11a6b> --checkpasses.scripts/build_v2_distribution.pyandscripts/smoke_v2_distribution.pypass, and the installed bundle'sdiscover --migration-suite --require-ready(308 cases) anddiscover --acceptance-suite --require-ready(339 cases) pass.ruff checkclean;black --checkflags the same eight files as the base, none of them changed here.$experiment_exposureposthog-js#5538 at05035e4(which declaresflags_response_v4and, with its new adapter commit,feature_flag_payloads) passes all 149 new scenarios. A build from posthog-jsmainselects none of them. Compared withmain, the only other difference is the query check from #173: the v4 query scenario passes and the v2 one is not selected. Both builds fail the same four cases, which also fail with released harness 1.19.0: the three snapshot payload cases (false,0,null) and "Omitted GeoIP settings default to enabled in flag requests".🤖 Generated with Claude Code