Repository navigation
feat(flags): run the v4 reader fixtures and value corpus through an SDK - #100
Conversation
The feature_flags_v4 suite serves mock /flags?v=4 responses built from the feature flag rules v2 contract package and reads them through the SDK's legacy accessors, one test per fixture case. It requires flags_response_v4. get_feature_flag must return the legacy rendering of every reader fixture record and every successful value corpus case. With feature_flag_payloads, the legacy payload accessor must return a config version 2 number or object value as the payload, keep a config version 1 record's metadata.payload, and return no payload otherwise. Payloads compare as parsed JSON values. Adds the optional adapter route /get_feature_flag_payload, `cases:` expansion for contract tests, and raw mock response bodies. Adapters without flags_response_v4 run no new tests.
parallel-adapter Compliance ReportDate: 2026-11-05T09:22:35.949959+00:00 ✅ All Tests Passed!17/17 tests passed Feature_Flags Tests✅ 17/17 tests passed View Details
|
minimal-adapter Compliance ReportDate: 2026-11-05T09:22:27.658033+00:00 ✅ All Tests Passed!17/17 tests passed Feature_Flags Tests✅ 17/17 tests passed View Details
|
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 6b250e5: 2 Should fix.
This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
| result = ctx.last_action_result or {} | ||
| expected = _case_expectation(params)["legacy_payload"] | ||
| _assert_json_equal(params["case"], "payload", result.get("value"), expected) |
There was a problem hiding this comment.
P2 · Require successful adapter results before comparing values
When an adapter returns HTTP 200 with {"success": false} and no value, result.get("value") becomes None, so this assertion passes cases expecting no payload despite the failed SDK call. The getter assertion also passes its missing-flag case this way. Require a result object with success is True before comparing either value.
| response_status=response_config.status_code, | ||
| response_headers=dict(response_config.headers), | ||
| response_body=response_config.body, | ||
| response_raw=response_config.raw_body, |
There was a problem hiding this comment.
P2 · Preserve raw_body when rebuilding capture-v1 responses
When a queued MockResponse(raw_body=True) targets a capture-v1 endpoint, record_request rebuilds the configuration without raw_body, so this assignment records False. The server then parses and reserializes the body; a deliberately malformed body produces HTTP 500 instead of the configured response. Preserve raw_body in the v1 reconstruction.
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
Both of @PostHog's substantive inline concerns remain unfixed, and its current-head review therefore does not provide concern-free assurance for the adapter contract addition. Address those comments, then request a current-head review from a teammate or agent; José Sequeira and Eli Reisman are suggested reviewers.
- Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
- PostHog Review (Standard) reviewed the current head: 2 findings, posted on the PR.
- posthog[bot] reviewed the current head.
- @PostHog on src/posthog_test_harness/actions.py: Both new assertions compare the returned value without requiring success to be true; failed adapter results with no value still pass cases expecting null. Require a successful result object in both assertions and add failure-result regression tests before re-requesting.
- @PostHog on src/posthog_test_harness/mock_server/state.py: The capture-v1 reconstruction at lines 272–277 omits raw_body, resetting it to false before response_raw is recorded; malformed raw bodies therefore reach JSON parsing rather than being served unchanged. Preserve raw_body during reconstruction and add a capture-v1 regression test before re-requesting.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 573L, 11F substantive, 909L/16F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1d-complex (909L, 16F, cross-cutting, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ 6b250e5 · reviewed head 6b250e5 |
A null expectation passed when the adapter call failed or the SDK never requested /flags, and a request with the wrong v also got the fixture. Each case assertion now needs success: true and at least one /flags request that read the case's response, all with v=4. The case's response now serves up to 10 /flags requests, so a payload accessor that makes a second request still reads the fixture.
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
The adapter contract addition has current-head review assurance, but @PostHog's substantive concern in mock_server/state.py remains unfixed. Preserve the raw-response setting through capture-v1 reconstruction and cover it with a regression test before re-requesting.
- Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
- PostHog Review (Standard) reviewed the current head: no findings.
- @PostHog on src/posthog_test_harness/mock_server/state.py: the capture-v1 response reconstruction at lines 272–277 still omits raw_body, resetting it to false before response_raw is recorded. Consequently, malformed raw response bodies reach json.loads in mock_server/server.py instead of being returned unchanged, causing HTTP 500. Preserve raw_body during reconstruction and add a capture-v1 regression test before re-requesting.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 598L, 11F substantive, 988L/16F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1d-complex (988L, 16F, cross-cutting, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ 670dcc8 · reviewed head 670dcc8 |
The contract defines the legacy rendering of a successful value but not what a legacy getter returns on a reader error. Write down the rule the suite asserts: a missing flag returns no value, and any other reader error returns false.
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
@PostHog's capture-v1 concern remains unfixed; the older concern about accepting unsuccessful adapter results is addressed. The adapter contract also needs current-head independent assurance: request review from José Sequeira, Eli Reisman, or an agent reviewer after fixing the open concern.
- Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
- @PostHog on src/posthog_test_harness/mock_server/state.py: the capture-v1 reconstruction at lines 272–277 still omits raw_body, resetting it to false before response_raw is recorded. Consequently, a configured malformed raw response is parsed by the server rather than sent unchanged. Preserve raw_body in that reconstruction and add a capture-v1 regression test before re-requesting.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 601L, 11F substantive, 998L/16F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1d-complex (998L, 16F, cross-cutting, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ 347a162 · reviewed head 347a162 |
There was a problem hiding this comment.
Not approved — this change needs a human reviewer.
@PostHog's raw-response concern remains unaddressed; preserve raw_body in the capture-v1 reconstruction and add a regression test before re-requesting. The adapter contract expansion also lacks current-head assurance; request review from José Sequeira, Eli Reisman, or an independent agent reviewer.
- Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
- @PostHog on src/posthog_test_harness/mock_server/state.py: the capture-v1 response reconstruction at lines 272–277 still omits raw_body, so the recorded response loses raw mode. src/posthog_test_harness/mock_server/server.py consequently parses deliberately malformed JSON instead of serving it unchanged, producing an error rather than the configured response.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 601L, 11F substantive, 998L/16F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1d-complex (998L, 16F, cross-cutting, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ b49938c · reviewed head b49938c |
dustinbyrne
left a comment
There was a problem hiding this comment.
Capability gating, fixture-derived projections, and successful-read guards look sound. One non-blocking raw-response edge case is noted inline.
AI-assisted review.
| response_status=response_config.status_code, | ||
| response_headers=dict(response_config.headers), | ||
| response_body=response_config.body, | ||
| response_raw=response_config.raw_body, |
There was a problem hiding this comment.
F1 — MINOR / OPTIONAL: Preserve raw mode through capture-v1 reconstruction
[P2] Preserve raw_body when rebuilding capture-v1 responses
When a harness caller queues MockResponse(raw_body=True) for a capture-v1 request, the mock still parses and reserializes the body. Deliberately malformed JSON therefore produces HTTP 500 instead of the configured HTTP 200 and exact body.
The reconstruction at mock_server/state.py:272–277 omits raw_body, so the new assignment at line 292 records False. mock_server/server.py:64–73 then reaches json.loads() instead of its verbatim-response branch. This contradicts the new MockResponse description in types.py:97–106.
Regression exercised on the reviewed head:
import pytest
from posthog_test_harness import MockResponse, MockServer, MockServerState
@pytest.mark.parametrize("path", ["/flags/?v=4", "/i/v1/analytics/events"])
@pytest.mark.parametrize("body", ["{broken", '{ "flags" : {} }'])
def test_raw_response_is_verbatim(path, body):
state = MockServerState()
state.set_response_queue([
MockResponse(status_code=200, body=body, raw_body=True)
])
response = MockServer(state).app.test_client().post(path, json={})
assert (response.status_code, response.get_data(as_text=True)) == (200, body)Observed: both /flags cases pass; capture-v1 returns 500 for {broken and reformats the valid spaced body. The full focused run reports 2 failed, 6 passed. Equivalent non-raw controls pass on base and head. Base lacks raw_body, so this is incomplete new functionality, not a regression in an existing raw API.
Consider preserving raw_body=response_config.raw_body in the capture-v1 reconstruction and adding this regression. Coverage for the shared AI capture-v1 route would also be useful; that route was source-inspected but not executed here.
Impact/disposition: Incorrect simulated response bytes/status for Python mock-server callers using the new option. The flags action scopes raw responses to /flags, and no existing raw capture-v1 consumer was found. This is therefore OPTIONAL, non-blocking feedback—not an SDK production crash or ingestion regression.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Not approved — escalated to a human reviewer.
The adapter contract addition needs independent assurance on the current head; both supplied reviews cover older commits, and the current code fixes all three inline concerns. Request a current-head review from José Sequeira, Eli Reisman, or an agent reviewer before re-requesting.
- Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
- Risky territory without independent assurance: Public API contracts — CONTRACT.yaml, sdk_adapter/interface.py and sdk_adapter/client.py introduce an optional payload-accessor endpoint and capability that SDK adapters must implement to participate.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✓ | 602L, 11F substantive, 1009L/17F incl. docs/generated/snapshots — within ceiling |
| tier | ✓ | T1-agent / T1d-complex (1009L, 17F, cross-cutting, feat) |
| stamphog 2.7.0 | .stamphog/policy.yml @ df103be · reviewed head df103be |
What this adds
A capability-gated
feature_flags_v4suite that runs the feature flag rules v2 reader fixtures and value corpus through a real SDK. Until now the v2 contract fixtures were only self-tested here. The only SDK-side v4 check was the?v=4request shape.Each test queues a mock
/flags?v=4response, built fromcontracts/feature_flag_rules_v2, and makes one remote read (force_remote: true). The response serves up to 10/flagsrequests, so an accessor may make more than one. A test withcases: <source>expands to one test per fixture case, named<test>[<case id>]. The mock builder issrc/posthog_test_harness/flags_v4.py. The contract package (contracts/feature_flag_rules_v2) does not change.legacy_getters.legacy_getter_reads_reader_fixtureflags_response_v4get_feature_flagreturns the legacy rendering of the record value, with strict JSON types (trueis not1). It returnsfalsefor a failed or unreadable record, no value for a missing flag, andvariant ?? enabledfor a record withoutvalue. Unknown fields, malformed siblings and avaluealongside staleenabled/variantdo not change the result.legacy_getters.legacy_getter_reads_typed_valueflags_response_v4legacy.valuefor string, number and object flags.legacy_payloads.legacy_payload_reads_reader_fixturefeature_flag_payloadsmetadata.payload. A boolean or string config version 2 value, a missing flag, and a failed or unreadable record without a payload all return no payload.legacy_payloads.legacy_payload_is_the_typed_valuefeature_flag_payloadslegacy.payload: a number or object value is the payload, even thoughmetadata.payloadis null for config version 2. Strings and null have no payload.Every test also requires
success: truefrom the adapter and at least one/flagsrequest that read the case's response, all withv=4. So a failed read or a read that never reached the mock cannot satisfy anullexpectation.Payloads compare as parsed JSON values, not as text. Key order and number spelling (
1.5e-7vs1.5e-07,2vs2.0) may differ, but the JSON type must match.Harness rule for legacy getters. The contract defines the legacy rendering of a successful value. It does not say what a legacy getter returns on a reader error. This suite uses one rule: a missing flag (
FLAG_NOT_FOUND) returns no value (null), and any other reader error returnsfalse. Other reader errors are a failed record, an unreadable record, or a known field with the wrong JSON type. ADAPTER_GUIDE.md and the suite description state the rule. posthog-python passes these cases. A draft posthog-node change passed the same cases in a local run of an unreleased Gherkin port of this suite; that is not yet checked in CI.Limit: in most reader cases the requested record's value is
false, the same result the legacy getter gives for a failed or unreadable record. So for those cases the getter test shows only that the SDK reads the response and returnsfalse. It cannot tell a reader that rejects a bad record from one that ignores the bad field, or one that drops a whole response for a malformed sibling. The typed details tests (not in this PR) can check that, because they see the error code.readers.v1_failedis left out of the payload test. It is a producer-valid failed config version 1 record that still carriesmetadata.payload, and the contract does not say whether the legacy payload accessor should return it. The legacy getter test still runs it.Capability gating
flags_response_v4. Without it, nothing is collected.feature_flag_payloads. That capability alone (withoutflags_response_v4) selects nothing.sdk_types: [server].feature_flag_payloads.tests/test_flags_v4_suites.pypins all of this.Adapter changes
POST /get_feature_flag_payload. It takes the same request as/get_feature_flagand returns{"success": true, "value": <payload as JSON, or null>}. See ADAPTER_GUIDE.md.raw_bodyso a v4 envelope is sent as is, without the defaultfeatureFlags.Not in this PR
get_*_details). No released SDK has them yet, and the accessor and metadata shape aren't specified. Their tests will follow once the SDK spec settles, gated on their own capability.$feature_flag_calledcontext and$experiment_exposure). They wait on the exposure spec.How to run
The adapter must declare
flags_response_v4, and alsofeature_flag_payloadswith the payload route for the payload tests.Local result with posthog-python
posthog-python
main(7.83.1), whose adapter was changed only to declarefeature_flag_payloadsand add the payload route. Full run, no docker:feature_flags_v4retry_behavior.retries_on_404andretry_behavior.retries_on_429. They are known capture v1 failures since harness 1.24.0 and are unrelated to this suite.Harness checks:
uv run --frozen pytest1444 passed,uv run ruff check .clean.🤖 Generated with Claude Code