Repository navigation
test(flags): generate rules v2 presence negatives from the matrix - #86
Conversation
parallel-adapter Compliance ReportDate: 2026-10-20T08:09:26.913986+00:00 ✅ All Tests Passed!17/17 tests passed Feature_Flags Tests✅ 17/17 tests passed View Details
|
minimal-adapter Compliance ReportDate: 2026-10-20T08:09:31.362396+00:00 ✅ All Tests Passed!17/17 tests passed Feature_Flags Tests✅ 17/17 tests passed View Details
|
2364959 to
134f5dc
Compare
58f8ffa to
b384f0b
Compare
134f5dc to
145e227
Compare
The per-field response negatives in fixtures/wire/responses.json were single mechanical mutations of each presence-matrix row's template: one case per missing required field, per forbidden field and for a wrong condition index. The meta-test now builds them from rules/response_presence.json and checks each against the same layer, keyword and instance path the committed cases declared. A mutation that a sibling row with the same reason and rule type accepts is not a negative. Remove 76 of the 77 committed cases the generator reproduces exactly. Keep responses.holdout_missing_holdout_id so that the committed set still has a presence-layer required failure for consumers that port the validator. The hand-made invalid cases, all valid cases and all templates stay; responses.json now has 25 valid and 49 invalid cases.
145e227 to
2752487
Compare
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
There was a problem hiding this comment.
Not approved yet — waiting on the conditions below.
stamphog declined to auto-review this pull request because it failed the size gate. It has 1206 substantive lines of change, and the ceiling is 800 (the total including docs, generated files and snapshots is 1298 lines across 5 files). The other gates (prerequisites, deny-list and tier) passed, so size is the only reason for the refusal.
To move forward, ask a human reviewer to look at it. Alternatively, split it into smaller pull requests, for example one for the generator in tests/test_feature_flag_rules_v2_wire.py and one for the responses.json fixture removals and version bump, so each stays under the ceiling.
Gate mechanics and policy version
| Gate | Result | |
|---|---|---|
| prerequisites | ✓ | all clear |
| deny-list | ✓ | no deny categories matched |
| size | ✗ | too large for auto-review (1206L substantive in global — ceiling is 800L; 1206L, 2F total, 1298L/5F incl. docs/generated/snapshots) |
| tier | ✓ | T1-agent / T1d-complex (1298L, 5F, cross-cutting, test) |
| stamphog 2.6.0 | .stamphog/policy.yml @ unknown · reviewed head 2752487 |
Problem
fixtures/wire/responses.jsoncommitted 77 per-field negatives that are single mechanical mutations of a presence-matrix row's template: for every row ofrules/response_presence.json, one case per missing required field (includinghas_experiment), one per forbidden field with a constant value, and one wrong condition index. A meta-test required each of them by id.rust/feature-flags/tests/test_rules_v2_evaluation.rs) runs every responses case through its port of the validator, checks the declared layer and pins the case counts. The draft feat(flags): read the /flags?v=4 response posthog-python#1031 uses only the valid cases and templates. No consumer runs them against producer or SDK code.responses.jsonfor four new matrix rows, and needs helpers to work out which mutations land on a sibling row.Changes
tests/test_feature_flag_rules_v2_wire.py:test_presence_matrix_negativesgenerates the per-field cases from the matrix and runs each through the same check as the committed producer fixtures (check_producer_case, extracted fromtest_wire_producer_fixtures). For every row it builds, from the row's template:<row>/missing/<field>forhas_experimentand each required field:requiredat the record'smetadata, with the field name as the message fragment;<row>/forbidden/<field>for each forbidden field, set to a fixed sample value (the constants the committed cases used):notatmetadata;<row>/wrong/condition_index:nullfor rule rows,0for no-rule rows,typeatreason/condition_index.Each case must fail at its declared layer and at no earlier layer, as committed cases do. The layer is
schemawhere the frozen producer schema restates the check (the split row's context and the no-rule rows' forbidden context and condition index, already pinned bytest_frozen_producer_rules_agree_with_the_matrix) andpresenceotherwise, so the exact schema keeps its deliberate gaps. A mutation that a sibling row with the same reason and rule type accepts is not a negative (another_row_accepts, computed from the matrix). No such sibling rows exist yet; feat(flags): add the experiment rule evaluation corpus without experiment or holdout links #74 adds them.fixtures/wire/responses.json: remove 76 of the 77 committedresponses.<row>_missing_<field>,responses.<row>_forbidden_<field>andresponses.<row>_wrong_condition_indexcases. Before deleting them I checked that each generated case equals its committed case in template, mutation and expected failure (layer, keyword, instance path, message fragment).responses.holdout_missing_holdout_idstays committed, byte-identical, because no hand-made case fails a presence-layerrequiredcheck, and a consumer that ports the validator should see one; the generatedholdout/missing/holdout_idduplicates it harmlessly. The templates and all 25 valid cases stay, and 49 invalid cases remain: 25 at the schema layer, 20 at the presence layer, 3 at the seed layer and 1 at the semantic layer. A consumer that ports the validator still sees failures at every layer.Pattern negatives that are not uniform per row stay committed:
<row>_forbidden_failed: rule rows fail on the exact schema's failed-record rule, no-rule rows on the matrix.<row>_null_value: the rows come from the schema generator, not from the matrix.<row>_wrong_rule_type: each row has its own value, andtargeting_match_percentage_rollouthas none because the obvious value lands on the sibling row.experiment_split_contradicted_has_experiment: a single row.missing_failedandnonnull_valuecases.test_every_terminal_reason_has_required_and_forbidden_metadata_fixturesbecomestest_every_terminal_reason_has_fixtures. It still requires a valid case per row, themissing_id/missing_version/missing_config_versioncases, the split wrong-type cases and the partial-split reader cases.test_frozen_producer_rules_agree_with_the_matrix, the regenerate-is-a-no-op test and the reader tests are unchanged. No snapshot of the generated list is committed.README: per-field response negatives are generated from the matrix by the meta-tests; the committed invalid response cases are the hand-made ones, plus
responses.holdout_missing_holdout_id.Bump the contract version to
5.0.0. refactor(flags): drop rules v2 wire fixture sets that nothing runs #85 shipped contract 4.0.0 in harness 1.17.0 with the committed negatives still in it, and removing cases is a major change. Add a changeset.Consumers that pin the
responses.jsoncase counts have to update them when they re-vendor: the PostHog Rust test pins 25 valid and 125 invalid, which becomes 25 valid and 49 invalid.How did you test this code?
uv run --no-sync pytest: 1120 passed, 502 skipped. The base branch has 1119 passed: the 77 generated cases replace 76 deleted parametrized fixture cases.uv run --no-sync ruff check .,black --checkon the changed test file andgit diff --checkpass.uv.lock,pyproject.tomland the schemas are unchanged.experiment_idfrom theholdoutrow's required fields and regenerate the schemas: the generated set shrinks by one (holdout/missing/experiment_id) and everything passes. The row is not frozen, so the matrix is the only source.variant_keyfrom theexperiment_splitrow and regenerate: the generated set shrinks by one andtest_frozen_producer_rules_agree_with_the_matrixfails.holdout_idand regenerate: the five generated<row>/forbidden/holdout_idcases for non-frozen rows fail. The split row's case still passes because the exact schema catches it.check_schemaalso fails, on the split row's now emptyanyOf.rollout_missexperiment row without an experiment (as in feat(flags): add the experiment rule evaluation corpus without experiment or holdout links #74) with a template and regenerate:rollout_miss_experiment/missing/experiment_idand the sibling'sforbidden/experiment_idare not generated, and every generated case passes. The only failure is the committed hand-maderesponses.rollout_miss_percentage_rollout_wrong_rule_type, which the new row accepts.🤖 Generated with Claude Code