Skip to content

test(flags): generate rules v2 presence negatives from the matrix - #86

Merged
andehen merged 1 commit into
mainfrom
chore/generate-wire-presence-negatives
Oct 20, 2026
Merged

andehen merged 1 commit into
mainfrom
chore/generate-wire-presence-negatives

Conversation

@andehen

@andehen andehen commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem

fixtures/wire/responses.json committed 77 per-field negatives that are single mechanical mutations of a presence-matrix row's template: for every row of rules/response_presence.json, one case per missing required field (including has_experiment), one per forbidden field with a constant value, and one wrong condition index. A meta-test required each of them by id.

  • They restate the matrix. The matrix and the presence-schema branches generated from it are already committed and reviewed, and the case ids, mutations and expected failures follow from the matrix row by row.
  • They are only checked against the harness's own schemas. The PostHog Rust test (rust/feature-flags/tests/test_rules_v2_evaluation.rs) runs every responses case through its port of the validator, checks the declared layer and pins the case counts. The draft feat(flags): read the /flags?v=4 response posthog-python#1031 uses only the valid cases and templates. No consumer runs them against producer or SDK code.
  • Every matrix change has to edit them by hand. feat(flags): add the experiment rule evaluation corpus without experiment or holdout links #74 adds about 350 lines of required negatives to responses.json for four new matrix rows, and needs helpers to work out which mutations land on a sibling row.

Changes

  • tests/test_feature_flag_rules_v2_wire.py: test_presence_matrix_negatives generates the per-field cases from the matrix and runs each through the same check as the committed producer fixtures (check_producer_case, extracted from test_wire_producer_fixtures). For every row it builds, from the row's template:

    • <row>/missing/<field> for has_experiment and each required field: required at the record's metadata, with the field name as the message fragment;
    • <row>/forbidden/<field> for each forbidden field, set to a fixed sample value (the constants the committed cases used): not at metadata;
    • <row>/wrong/condition_index: null for rule rows, 0 for no-rule rows, type at reason/condition_index.

    Each case must fail at its declared layer and at no earlier layer, as committed cases do. The layer is schema where the frozen producer schema restates the check (the split row's context and the no-rule rows' forbidden context and condition index, already pinned by test_frozen_producer_rules_agree_with_the_matrix) and presence otherwise, so the exact schema keeps its deliberate gaps. A mutation that a sibling row with the same reason and rule type accepts is not a negative (another_row_accepts, computed from the matrix). No such sibling rows exist yet; feat(flags): add the experiment rule evaluation corpus without experiment or holdout links #74 adds them.

  • fixtures/wire/responses.json: remove 76 of the 77 committed responses.<row>_missing_<field>, responses.<row>_forbidden_<field> and responses.<row>_wrong_condition_index cases. Before deleting them I checked that each generated case equals its committed case in template, mutation and expected failure (layer, keyword, instance path, message fragment). responses.holdout_missing_holdout_id stays committed, byte-identical, because no hand-made case fails a presence-layer required check, and a consumer that ports the validator should see one; the generated holdout/missing/holdout_id duplicates it harmlessly. The templates and all 25 valid cases stay, and 49 invalid cases remain: 25 at the schema layer, 20 at the presence layer, 3 at the seed layer and 1 at the semantic layer. A consumer that ports the validator still sees failures at every layer.

  • Pattern negatives that are not uniform per row stay committed:

    • <row>_forbidden_failed: rule rows fail on the exact schema's failed-record rule, no-rule rows on the matrix.
    • <row>_null_value: the rows come from the schema generator, not from the matrix.
    • <row>_wrong_rule_type: each row has its own value, and targeting_match_percentage_rollout has none because the obvious value lands on the sibling row.
    • experiment_split_contradicted_has_experiment: a single row.
    • The missing_failed and nonnull_value cases.
  • test_every_terminal_reason_has_required_and_forbidden_metadata_fixtures becomes test_every_terminal_reason_has_fixtures. It still requires a valid case per row, the missing_id/missing_version/missing_config_version cases, the split wrong-type cases and the partial-split reader cases. test_frozen_producer_rules_agree_with_the_matrix, the regenerate-is-a-no-op test and the reader tests are unchanged. No snapshot of the generated list is committed.

  • README: per-field response negatives are generated from the matrix by the meta-tests; the committed invalid response cases are the hand-made ones, plus responses.holdout_missing_holdout_id.

  • Bump the contract version to 5.0.0. refactor(flags): drop rules v2 wire fixture sets that nothing runs #85 shipped contract 4.0.0 in harness 1.17.0 with the committed negatives still in it, and removing cases is a major change. Add a changeset.

Consumers that pin the responses.json case counts have to update them when they re-vendor: the PostHog Rust test pins 25 valid and 125 invalid, which becomes 25 valid and 49 invalid.

How did you test this code?

  • uv run --no-sync pytest: 1120 passed, 502 skipped. The base branch has 1119 passed: the 77 generated cases replace 76 deleted parametrized fixture cases.
  • uv run --no-sync ruff check ., black --check on the changed test file and git diff --check pass. uv.lock, pyproject.toml and the schemas are unchanged.
  • Mutation checks, each reverted afterwards:
    • Drop experiment_id from the holdout row's required fields and regenerate the schemas: the generated set shrinks by one (holdout/missing/experiment_id) and everything passes. The row is not frozen, so the matrix is the only source.
    • Drop variant_key from the experiment_split row and regenerate: the generated set shrinks by one and test_frozen_producer_rules_agree_with_the_matrix fails.
    • Make the schema generator skip the forbidden holdout_id and regenerate: the five generated <row>/forbidden/holdout_id cases for non-frozen rows fail. The split row's case still passes because the exact schema catches it. check_schema also fails, on the split row's now empty anyOf.
    • Add a sibling rollout_miss experiment row without an experiment (as in feat(flags): add the experiment rule evaluation corpus without experiment or holdout links #74) with a template and regenerate: rollout_miss_experiment/missing/experiment_id and the sibling's forbidden/experiment_id are not generated, and every generated case passes. The only failure is the committed hand-made responses.rollout_miss_percentage_rollout_wrong_rule_type, which the new row accepts.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

parallel-adapter Compliance Report

Date: 2026-10-20T08:09:26.913986+00:00
Duration: 1506ms

✅ All Tests Passed!

17/17 tests passed


Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 22ms
Request Payload.Flags Request Uses V2 Query Param ✅ 20ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 16ms
Request Payload.Flags Request Omits Authorization Header ✅ 18ms
Request Payload.Token In Flags Body Matches Init ✅ 16ms
Request Payload.Groups Round Trip ✅ 14ms
Request Payload.Groups Default To Empty Object ✅ 14ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 14ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 13ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 18ms
Request Lifecycle.No Flags Request On Init Alone ✅ 6ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 21ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 25ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 18ms
Retry Behavior.Retries Flags On 502 ✅ 1020ms
Retry Behavior.Retries Flags On 504 ✅ 1015ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 11ms

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

minimal-adapter Compliance Report

Date: 2026-10-20T08:09:31.362396+00:00
Duration: 2459ms

✅ All Tests Passed!

17/17 tests passed


Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 10ms
Request Payload.Flags Request Uses V2 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 5ms
Request Payload.Flags Request Omits Authorization Header ✅ 6ms
Request Payload.Token In Flags Body Matches Init ✅ 6ms
Request Payload.Groups Round Trip ✅ 5ms
Request Payload.Groups Default To Empty Object ✅ 6ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 5ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 6ms
Request Lifecycle.No Flags Request On Init Alone ✅ 2ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 6ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 9ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 5ms
Retry Behavior.Retries Flags On 502 ✅ 1009ms
Retry Behavior.Retries Flags On 504 ✅ 1009ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 8ms

@andehen
andehen force-pushed the chore/generate-wire-presence-negatives branch from 2364959 to 134f5dc Compare October 15, 2026 11:04
@andehen
andehen force-pushed the chore/drop-unconsumed-wire-fixtures branch from 58f8ffa to b384f0b Compare October 19, 2026 16:42
@andehen
andehen force-pushed the chore/generate-wire-presence-negatives branch from 134f5dc to 145e227 Compare October 19, 2026 16:49
Base automatically changed from chore/drop-unconsumed-wire-fixtures to main October 19, 2026 17:47
The per-field response negatives in fixtures/wire/responses.json were
single mechanical mutations of each presence-matrix row's template: one
case per missing required field, per forbidden field and for a wrong
condition index. The meta-test now builds them from
rules/response_presence.json and checks each against the same layer,
keyword and instance path the committed cases declared. A mutation that a
sibling row with the same reason and rule type accepts is not a negative.

Remove 76 of the 77 committed cases the generator reproduces exactly.
Keep responses.holdout_missing_holdout_id so that the committed set still
has a presence-layer required failure for consumers that port the
validator. The hand-made invalid cases, all valid cases and all templates
stay; responses.json now has 25 valid and 49 invalid cases.
@andehen
andehen force-pushed the chore/generate-wire-presence-negatives branch from 145e227 to 2752487 Compare October 20, 2026 08:08
@andehen
andehen marked this pull request as ready for review October 20, 2026 10:18
@andehen
andehen requested a review from a team as a code owner October 20, 2026 10:18

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved yet — waiting on the conditions below.

stamphog declined to auto-review this pull request because it failed the size gate. It has 1206 substantive lines of change, and the ceiling is 800 (the total including docs, generated files and snapshots is 1298 lines across 5 files). The other gates (prerequisites, deny-list and tier) passed, so size is the only reason for the refusal.

To move forward, ask a human reviewer to look at it. Alternatively, split it into smaller pull requests, for example one for the generator in tests/test_feature_flag_rules_v2_wire.py and one for the responses.json fixture removals and version bump, so each stays under the ceiling.

Gate mechanics and policy version
Gate Result
prerequisites ✓ all clear
deny-list ✓ no deny categories matched
size ✗ too large for auto-review (1206L substantive in global — ceiling is 800L; 1206L, 2F total, 1298L/5F incl. docs/generated/snapshots)
tier ✓ T1-agent / T1d-complex (1298L, 5F, cross-cutting, test)
stamphog 2.6.0 .stamphog/policy.yml @ unknown · reviewed head 2752487

@andehen
andehen merged commit a6e61f6 into main Oct 20, 2026
14 checks passed
@andehen
andehen deleted the chore/generate-wire-presence-negatives branch October 20, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants