Skip to content

feat(flags): run the v4 reader fixtures and value corpus through an SDK - #100

Merged
andehen merged 5 commits into
mainfrom
andehen/har-005-remote-v2-suite
Nov 5, 2026
Merged

andehen merged 5 commits into
mainfrom
andehen/har-005-remote-v2-suite

Conversation

@andehen

@andehen andehen commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

A capability-gated feature_flags_v4 suite that runs the feature flag rules v2 reader fixtures and value corpus through a real SDK. Until now the v2 contract fixtures were only self-tested here. The only SDK-side v4 check was the ?v=4 request shape.

Each test queues a mock /flags?v=4 response, built from contracts/feature_flag_rules_v2, and makes one remote read (force_remote: true). The response serves up to 10 /flags requests, so an accessor may make more than one. A test with cases: <source> expands to one test per fixture case, named <test>[<case id>]. The mock builder is src/posthog_test_harness/flags_v4.py. The contract package (contracts/feature_flag_rules_v2) does not change.

Test Cases Requires Checks
legacy_getters.legacy_getter_reads_reader_fixture 37 reader cases flags_response_v4 get_feature_flag returns the legacy rendering of the record value, with strict JSON types (true is not 1). It returns false for a failed or unreadable record, no value for a missing flag, and variant ?? enabled for a record without value. Unknown fields, malformed siblings and a value alongside stale enabled/variant do not change the result.
legacy_getters.legacy_getter_reads_typed_value 38 value corpus cases flags_response_v4 The corpus legacy.value for string, number and object flags.
legacy_payloads.legacy_payload_reads_reader_fixture 36 reader cases + feature_flag_payloads A config version 1 record keeps its decoded metadata.payload. A boolean or string config version 2 value, a missing flag, and a failed or unreadable record without a payload all return no payload.
legacy_payloads.legacy_payload_is_the_typed_value 38 value corpus cases + feature_flag_payloads The corpus legacy.payload: a number or object value is the payload, even though metadata.payload is null for config version 2. Strings and null have no payload.

Every test also requires success: true from the adapter and at least one /flags request that read the case's response, all with v=4. So a failed read or a read that never reached the mock cannot satisfy a null expectation.

Payloads compare as parsed JSON values, not as text. Key order and number spelling (1.5e-7 vs 1.5e-07, 2 vs 2.0) may differ, but the JSON type must match.

Harness rule for legacy getters. The contract defines the legacy rendering of a successful value. It does not say what a legacy getter returns on a reader error. This suite uses one rule: a missing flag (FLAG_NOT_FOUND) returns no value (null), and any other reader error returns false. Other reader errors are a failed record, an unreadable record, or a known field with the wrong JSON type. ADAPTER_GUIDE.md and the suite description state the rule. posthog-python passes these cases. A draft posthog-node change passed the same cases in a local run of an unreleased Gherkin port of this suite; that is not yet checked in CI.

Limit: in most reader cases the requested record's value is false, the same result the legacy getter gives for a failed or unreadable record. So for those cases the getter test shows only that the SDK reads the response and returns false. It cannot tell a reader that rejects a bad record from one that ignores the bad field, or one that drops a whole response for a malformed sibling. The typed details tests (not in this PR) can check that, because they see the error code.

readers.v1_failed is left out of the payload test. It is a producer-valid failed config version 1 record that still carries metadata.payload, and the contract does not say whether the legacy payload accessor should return it. The legacy getter test still runs it.

Capability gating

  • The suite requires flags_response_v4. Without it, nothing is collected.
  • The payload tests also require feature_flag_payloads. That capability alone (without flags_response_v4) selects nothing.
  • All tests are sdk_types: [server].
  • Tests the adapter is not eligible for are left out at collection, so they are skipped, never failed. Existing suites collect the same tests with or without feature_flag_payloads. tests/test_flags_v4_suites.py pins all of this.

Adapter changes

  • New optional route POST /get_feature_flag_payload. It takes the same request as /get_feature_flag and returns {"success": true, "value": <payload as JSON, or null>}. See ADAPTER_GUIDE.md.
  • Mock responses can set raw_body so a v4 envelope is sent as is, without the default featureFlags.

Not in this PR

  • Typed details accessors (get_*_details). No released SDK has them yet, and the accessor and metadata shape aren't specified. Their tests will follow once the SDK spec settles, gated on their own capability.
  • Flag events after v4 reads ($feature_flag_called context and $experiment_exposure). They wait on the exposure spec.

How to run

uv run --frozen posthog-test-harness run --adapter-url http://localhost:18080 \
  --mock-url http://localhost:18081 --mock-port 18081 --sdk-type server \
  --suite feature_flags_v4 --output text

The adapter must declare flags_response_v4, and also feature_flag_payloads with the payload route for the payload tests.

Local result with posthog-python

posthog-python main (7.83.1), whose adapter was changed only to declare feature_flag_payloads and add the payload route. Full run, no docker:

  • capture v0: 205/205 passed, 149 of them in feature_flags_v4
  • capture v1: 268/270 passed. The 2 failures are retry_behavior.retries_on_404 and retry_behavior.retries_on_429. They are known capture v1 failures since harness 1.24.0 and are unrelated to this suite.

Harness checks: uv run --frozen pytest 1444 passed, uv run ruff check . clean.

🤖 Generated with Claude Code

The feature_flags_v4 suite serves mock /flags?v=4 responses built from the
feature flag rules v2 contract package and reads them through the SDK's
legacy accessors, one test per fixture case.

It requires flags_response_v4. get_feature_flag must return the legacy
rendering of every reader fixture record and every successful value corpus
case. With feature_flag_payloads, the legacy payload accessor must return a
config version 2 number or object value as the payload, keep a config
version 1 record's metadata.payload, and return no payload otherwise.
Payloads compare as parsed JSON values.

Adds the optional adapter route /get_feature_flag_payload, `cases:`
expansion for contract tests, and raw mock response bodies. Adapters
without flags_response_v4 run no new tests.
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

parallel-adapter Compliance Report

Date: 2026-11-05T09:22:35.949959+00:00
Duration: 1851ms

✅ All Tests Passed!

17/17 tests passed


Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 29ms
Request Payload.Flags Request Uses V2 Query Param ✅ 27ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 21ms
Request Payload.Flags Request Omits Authorization Header ✅ 27ms
Request Payload.Token In Flags Body Matches Init ✅ 24ms
Request Payload.Groups Round Trip ✅ 21ms
Request Payload.Groups Default To Empty Object ✅ 24ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 23ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 23ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 19ms
Request Lifecycle.No Flags Request On Init Alone ✅ 7ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 29ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 40ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 17ms
Retry Behavior.Retries Flags On 502 ✅ 1030ms
Retry Behavior.Retries Flags On 504 ✅ 1022ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 23ms

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

minimal-adapter Compliance Report

Date: 2026-11-05T09:22:27.658033+00:00
Duration: 2841ms

✅ All Tests Passed!

17/17 tests passed


Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 10ms
Request Payload.Flags Request Uses V2 Query Param ✅ 8ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 7ms
Request Payload.Flags Request Omits Authorization Header ✅ 7ms
Request Payload.Token In Flags Body Matches Init ✅ 6ms
Request Payload.Groups Round Trip ✅ 7ms
Request Payload.Groups Default To Empty Object ✅ 7ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 7ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 7ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 9ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 10ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 7ms
Retry Behavior.Retries Flags On 502 ✅ 1009ms
Retry Behavior.Retries Flags On 504 ✅ 1010ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 10ms

@posthog posthog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 6b250e5: 2 Should fix.

This is the only automatic review while this pull request is a draft. Comment @posthog review to review new changes sooner. Otherwise, a review of the new changes runs when you mark it ready for review.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Comment thread src/posthog_test_harness/actions.py Outdated
Comment on lines +2456 to +2458
result = ctx.last_action_result or {}
expected = _case_expectation(params)["legacy_payload"]
_assert_json_equal(params["case"], "payload", result.get("value"), expected)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Require successful adapter results before comparing values

When an adapter returns HTTP 200 with {"success": false} and no value, result.get("value") becomes None, so this assertion passes cases expecting no payload despite the failed SDK call. The getter assertion also passes its missing-flag case this way. Require a result object with success is True before comparing either value.

response_status=response_config.status_code,
response_headers=dict(response_config.headers),
response_body=response_config.body,
response_raw=response_config.raw_body,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 · Preserve raw_body when rebuilding capture-v1 responses

When a queued MockResponse(raw_body=True) targets a capture-v1 endpoint, record_request rebuilds the configuration without raw_body, so this assignment records False. The server then parses and reserializes the body; a deliberately malformed body produces HTTP 500 instead of the configured response. Preserve raw_body in the v1 reconstruction.

@andehen
andehen marked this pull request as ready for review November 4, 2026 09:47
@andehen
andehen requested a review from a team as a code owner November 4, 2026 09:47

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

Both of @PostHog's substantive inline concerns remain unfixed, and its current-head review therefore does not provide concern-free assurance for the adapter contract addition. Address those comments, then request a current-head review from a teammate or agent; José Sequeira and Eli Reisman are suggested reviewers.

  • Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
  • PostHog Review (Standard) reviewed the current head: 2 findings, posted on the PR.
  • posthog[bot] reviewed the current head.
  • @PostHog on src/posthog_test_harness/actions.py: Both new assertions compare the returned value without requiring success to be true; failed adapter results with no value still pass cases expecting null. Require a successful result object in both assertions and add failure-result regression tests before re-requesting.
  • @PostHog on src/posthog_test_harness/mock_server/state.py: The capture-v1 reconstruction at lines 272–277 omits raw_body, resetting it to false before response_raw is recorded; malformed raw bodies therefore reach JSON parsing rather than being served unchanged. Preserve raw_body during reconstruction and add a capture-v1 regression test before re-requesting.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 573L, 11F substantive, 909L/16F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1d-complex (909L, 16F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ 6b250e5 · reviewed head 6b250e5

A null expectation passed when the adapter call failed or the SDK never
requested /flags, and a request with the wrong v also got the fixture.
Each case assertion now needs success: true and at least one /flags
request that read the case's response, all with v=4.

The case's response now serves up to 10 /flags requests, so a payload
accessor that makes a second request still reads the fixture.

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

The adapter contract addition has current-head review assurance, but @PostHog's substantive concern in mock_server/state.py remains unfixed. Preserve the raw-response setting through capture-v1 reconstruction and cover it with a regression test before re-requesting.

  • Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
  • PostHog Review (Standard) reviewed the current head: no findings.
  • @PostHog on src/posthog_test_harness/mock_server/state.py: the capture-v1 response reconstruction at lines 272–277 still omits raw_body, resetting it to false before response_raw is recorded. Consequently, malformed raw response bodies reach json.loads in mock_server/server.py instead of being returned unchanged, causing HTTP 500. Preserve raw_body during reconstruction and add a capture-v1 regression test before re-requesting.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 598L, 11F substantive, 988L/16F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1d-complex (988L, 16F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ 670dcc8 · reviewed head 670dcc8

The contract defines the legacy rendering of a successful value but not
what a legacy getter returns on a reader error. Write down the rule the
suite asserts: a missing flag returns no value, and any other reader
error returns false.

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

@PostHog's capture-v1 concern remains unfixed; the older concern about accepting unsuccessful adapter results is addressed. The adapter contract also needs current-head independent assurance: request review from José Sequeira, Eli Reisman, or an agent reviewer after fixing the open concern.

  • Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
  • @PostHog on src/posthog_test_harness/mock_server/state.py: the capture-v1 reconstruction at lines 272–277 still omits raw_body, resetting it to false before response_raw is recorded. Consequently, a configured malformed raw response is parsed by the server rather than sent unchanged. Preserve raw_body in that reconstruction and add a capture-v1 regression test before re-requesting.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 601L, 11F substantive, 998L/16F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1d-complex (998L, 16F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ 347a162 · reviewed head 347a162

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — this change needs a human reviewer.

@PostHog's raw-response concern remains unaddressed; preserve raw_body in the capture-v1 reconstruction and add a regression test before re-requesting. The adapter contract expansion also lacks current-head assurance; request review from José Sequeira, Eli Reisman, or an independent agent reviewer.

  • Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
  • @PostHog on src/posthog_test_harness/mock_server/state.py: the capture-v1 response reconstruction at lines 272–277 still omits raw_body, so the recorded response loses raw mode. src/posthog_test_harness/mock_server/server.py consequently parses deliberately malformed JSON instead of serving it unchanged, producing an error rather than the configured response.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 601L, 11F substantive, 998L/16F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1d-complex (998L, 16F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ b49938c · reviewed head b49938c

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capability gating, fixture-derived projections, and successful-read guards look sound. One non-blocking raw-response edge case is noted inline.

AI-assisted review.

response_status=response_config.status_code,
response_headers=dict(response_config.headers),
response_body=response_config.body,
response_raw=response_config.raw_body,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

F1 — MINOR / OPTIONAL: Preserve raw mode through capture-v1 reconstruction

[P2] Preserve raw_body when rebuilding capture-v1 responses

When a harness caller queues MockResponse(raw_body=True) for a capture-v1 request, the mock still parses and reserializes the body. Deliberately malformed JSON therefore produces HTTP 500 instead of the configured HTTP 200 and exact body.

The reconstruction at mock_server/state.py:272–277 omits raw_body, so the new assignment at line 292 records False. mock_server/server.py:64–73 then reaches json.loads() instead of its verbatim-response branch. This contradicts the new MockResponse description in types.py:97–106.

Regression exercised on the reviewed head:

import pytest
from posthog_test_harness import MockResponse, MockServer, MockServerState

@pytest.mark.parametrize("path", ["/flags/?v=4", "/i/v1/analytics/events"])
@pytest.mark.parametrize("body", ["{broken", '{ "flags" : {} }'])
def test_raw_response_is_verbatim(path, body):
    state = MockServerState()
    state.set_response_queue([
        MockResponse(status_code=200, body=body, raw_body=True)
    ])
    response = MockServer(state).app.test_client().post(path, json={})
    assert (response.status_code, response.get_data(as_text=True)) == (200, body)

Observed: both /flags cases pass; capture-v1 returns 500 for {broken and reformats the valid spaced body. The full focused run reports 2 failed, 6 passed. Equivalent non-raw controls pass on base and head. Base lacks raw_body, so this is incomplete new functionality, not a regression in an existing raw API.

Consider preserving raw_body=response_config.raw_body in the capture-v1 reconstruction and adding this regression. Coverage for the shared AI capture-v1 route would also be useful; that route was source-inspected but not executed here.

Impact/disposition: Incorrect simulated response bytes/status for Python mock-server callers using the new option. The flags action scopes raw responses to /flags, and no existing raw capture-v1 consumer was found. This is therefore OPTIONAL, non-blocking feedback—not an SDK production crash or ingestion regression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved — escalated to a human reviewer.

The adapter contract addition needs independent assurance on the current head; both supplied reviews cover older commits, and the current code fixes all three inline concerns. Request a current-head review from José Sequeira, Eli Reisman, or an agent reviewer before re-requesting.

  • Author wrote 33% of the modified lines and has 14 merged PRs in these paths (familiarity MODERATE).
  • Risky territory without independent assurance: Public API contracts — CONTRACT.yaml, sdk_adapter/interface.py and sdk_adapter/client.py introduce an optional payload-accessor endpoint and capability that SDK adapters must implement to participate.
Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✓ 602L, 11F substantive, 1009L/17F incl. docs/generated/snapshots — within ceiling
tier ✓ T1-agent / T1d-complex (1009L, 17F, cross-cutting, feat)
stamphog 2.7.0 .stamphog/policy.yml @ df103be · reviewed head df103be

@andehen
andehen merged commit a1a5658 into main Nov 5, 2026
14 checks passed
@andehen
andehen deleted the andehen/har-005-remote-v2-suite branch November 5, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants