test: add schema and pure-logic test suites#6
Merged
Merged
Conversation
Adds the first two test layers for OpenHack: JSON Schema golden tests for all five durable artifact shapes, and pure-logic unit tests for the modules whose correctness gates the scenario→finding pipeline (paths, coverage scoring, routing-unit clustering, backlog validation). - 150 tests, runs in <0.2s, no LLM or network dependencies - pytest added to dev extras and wired into CI alongside ruff/mypy - conftest pins OPENHACK_ROOT so on-disk schema/expert lookups resolve deterministically regardless of where pytest is invoked from Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Gaps: - Add coverage for write_coverage and write_routing_units (the disk-side entry points called from the CLI). They were the largest untested seam. - Add record_backlog test for the expert_scope restriction path via run-config.yaml, plus an audit-event assertion on events.jsonl. - Add full coverage of boundary_requirements in coverage_errors: flagged-when-missing, satisfied-by-scenario-boundary-id, satisfied-by-covered_boundary_ids list, satisfied-by-recon_item_id fallback, satisfied-by-boundary-id decision, and the negative case where a decision without a boundary_id does not satisfy. - Rewrite scenario/decision predicate tests to go through coverage_errors rather than calling the private _scenario_covers_* / _validate_decisions helpers. Same coverage, resilient to internal refactors. Cleanup: - Fix test_kind_for_terms_first_match_wins to actually exercise priority (it now uses the 'template' overlap between html_template_dom_sink and parser_deserialization_integrity, plus the 'token' overlap). - Drop test_baselines_are_independent — the helpers return fresh dicts by construction so the assertion was tautological. - Replace the brittle bullet-count assertion in test_validator_reports_multiple_errors with field-name substring checks. - Drop change-detector constants tests (DECISIONS, PRODUCTIVE_CLASSES, MAX_REQUIREMENTS_PER_PATH) — they only fired when someone updated the constant. Nice-to-haves: - CI now caches pip via setup-python's cache: pip. - Split CI into a lint job (3.11) and a test job that matrixes pytest across 3.9, 3.11, 3.12 — pyproject declares requires-python>=3.9. - Remove empty tests/__init__.py (pytest discovers without it). - Hoist the ALL_RUN_DIRS import in conftest to module-level. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the first two test layers for OpenHack, grounded in the project's existing architecture (schema-first artifacts, pure-logic routing modules):
tests/test_schemas.py): a minimum-valid baseline for each of the five durable artifact schemas (scenario,scenario-result,finding,finding-candidate,finding-triage), plus single-field mutations asserting the validator raises and points at the correct JSON path. Catches silent loosening of schema rules.tests/test_paths.py—OPENHACK_ROOTresolution, walk-up fallback,ensure_run_dirsidempotency.tests/test_coverage.py—_path_class(27 cases),_tokens,_score_pairbranch-by-branch (boundary-mandatory, supply-chain manifest, non-productive path-class, no-strong-terms, suggestion vs high), end-to-endrouting_requirements/coverage_opportunities/coverage_suggestions.tests/test_routing_units.py—_kind_for_termspriority ordering,_compact_rowtruncation,_dedupe_rowscap,build_routing_unitsID assignment + required/suggested separation + boundary-field preservation + mandatory-path fallback.tests/test_backlog.py—_scenario_paths,_scenario_covers_*predicates,_validate_decisionserror cases,coverage_errorswith on-diskcoverage-gaps.json+routing-units.jsonl, andrecord_backloghappy-path + six error gates (unknown expert, duplicate id, duplicate obligation, missing required field, schema failure, coverage gap).150 tests, ~0.16s, no network or LLM dependencies.
Wiring
pytest>=7.0added to[project.optional-dependencies].devand a[tool.pytest.ini_options]block set inpyproject.toml.pyteststep added to the existing CI workflow alongsideruffandmypy.tests/conftest.pyautouse-pinsOPENHACK_ROOTto the repo root so the schema/expert lookups inschemas._schemaandbacklog.coverage_errorsresolve deterministically regardless of the pytest invocation directory.Test plan
pytest— 150 passedruff check .— cleanmypy— clean