Skip to content

feat(policy): turn measured findings into deterministic decision policies - #211

Draft
kvnloo wants to merge 12 commits into
feat/shadow-learning-portfoliofrom
feat/decision-policy
Draft

kvnloo wants to merge 12 commits into
feat/shadow-learning-portfoliofrom
feat/decision-policy

Conversation

@kvnloo

@kvnloo kvnloo commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Scope

Adds z0.finding.v1 canonical records, locked hash-chained append/read, source-hash-idempotent structured ingesters, pure z0.decision_policy.v1 compiler, atomic policy/history persistence, stage changelog, and policy CLI commands. Serving entries preserve model/quant/engine/backend/device plus kind. Canary/live rules without an exact measured endpoint carry serving_missing=true, also exposed by the changelog.

Evidence governance

Promotions require measured/live-shadow/replay/outcome metadata. Inference never promotes. Corrupt or noncanonical ledgers stop compilation without replacing the current policy. Narrow structural scopes override broad rules. Only measured findings with explicit endpoint URLs and complete physical scope produce serving entries.

Contract interpretations

Finding prev hashes the full prior canonical line excluding newline; finding id excludes id/prev. Policy version excludes version only. Shadow retention uses bar_shadow, canary/live use bar. Latest CI regressions demote immediately. Expired stage prerequisites retain only the highest freshly supported lower stage. Explicit arm dimensions constrain scope; prose never relaxes them. Noninferiority and outcome verification are explicit metric fields. Unknown/prose-only sources are recorded and skipped, not guessed. README describes source adapter layouts and all interpretation details.

Verification

Parent later explicitly authorized remote MBP verification, superseding the original no-test instruction. No builds, tests, linters or formatters ran on groot.

MBP Kueue job fleet-mbp-de69132c757b4f51: 57 passed in 0.87s for tests/test_policy_findings.py and tests/test_policy_compile.py, using a throwaway venv with pinned pytest 8.4.2 and PyYAML 6.0.3, 1 CPU / 2G.

Initial MBP job fleet-mbp-5e178f15fa244bd4 ran zero tests because system Python lacked pytest. The isolated-venv job corrected the environment prerequisite.

Tests cover hash/chain/whitespace tampering, ingest adapters and idempotency, byte-identical determinism, all promotion edges, immediate demotions, prerequisite expiry, scope precedence, serving scope/provenance/missing annotations, root CLI behavior, and corrupt-ledger fail-closed behavior. Integration with consumer/guard branches remains the parent integration check.

Risk-derived option capacity (2026-10-25)

  • Replaces the fixed capacity accuracy bar with per-risk expected-cost comparison: call + (1-p_lower)*error + abstain_rate*frontier < frontier. Accuracy promotion thresholds remain independent.
  • Nested exact-scope entries for read, reversible mutation and destructive risk; no evidence means capacity 0. Destructive decisions never route to an SLM.
  • Cost inputs carry measured/unmeasured status and finding provenance. Measured values must match fresh risk-bound findings; recovery proxy evidence does not promote accuracy.
  • MBP regression job fleet-mbp-4a414c30dd2342d9: 90 passed (compiler/findings).
  • Real observed frontier recovery costs are reported separately; they are not counterfactual executed-shadow recovery measurements. Private session traces are not published.

Canonical identity and final compiler gate

  • Canonical schema: schemas/decision-policy.scope.v1.schema.json; model is sha256:<GGUF digest>, engine is live /props.build_info unchanged. Friendly aliases cannot grant capacity.
  • Automatic evidence adoption requires complete fresh exact-scope cost findings with consistent units; no manual route flip.
  • Pushed head 7308400dc6c9a12111882324300824d1845fd6d6: MBP job fleet-mbp-889570419ca14f5c, 92 passed.
  • Observed recovery means: read 7,542.944 ms (n=196), mutation-tool proxy 12,678.077 ms (n=13). Median/P95: read 6,034/18,554 ms; mutations 7,764/64,339 ms. These are observed-frontier recovery proxies, not causal SLM intervention or proof of reversibility.
  • Observed gpt-6.1-sol whole-request median 7,999.089 ms (n=53,426); TTFT median 2,283.853 ms is already inside duration, not added twice. Model identity is a task-role proxy, not verified role attribution.

Final risk-capacity contract (2026-10-24)

Head 0332032. Deliberate parent-authorized contract change: unmeasured economics now yield allowed_k=0; assumed defaults/bar are diagnostic only. Missing/stale evidence routes to frontier/Jev. Exact model scope is the GGUF SHA-256 plus quant, observed build_info engine, backend and device. MBP fleet-mbp-b7e4673af94d4b2c: 93 passed.

@github-actions github-actions Bot added area:docs Documentation area:verify Tests, mutation, evidence labels Oct 24, 2026
@kvnloo

kvnloo commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

Follow-up commits aaf6d4b and 41dbbfd add synthetic provenance and explicit measured serving identity for the parent-owned isolated smoke. Optional finding synthetic=true participates in its content hash and chain. Affected rules/endpoints and the whole policy/changelog retain synthetic provenance. The Markdown changelog warns that this is not a real experiment promotion. The compiler and ingester preserve explicit expected_model_id, engine_build, model_path, model_resolved_path and model_sha256 without guessing or replacing actual receipt file provenance. Promotion gates remain unchanged. Regression tests were added but were NOT RUN for these follow-ups, per the smoke coordination instruction. The earlier 57-pass MBP result applies to 5a05b5d, not these later commits.

@kvnloo

kvnloo commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

Parent-added O13 projection is now pushed at 8a821e9. The content-addressed policy includes option_capacity keyed by canonical exact model/quant/engine/backend/device scope. Values contain allowed_k, finding_ids, wilson_lower, task (required task list), bar and expires_at. Default required tasks are T1-T4. The compiler takes the largest passing k per task and the minimum across all required tasks, using the smaller of stated and recomputed Wilson lower bounds. Missing/pending/failed/expired evidence defaults to 2. Routing stage bars remain unchanged; guards read the compiled projection, not live findings.

Final remote verification at HEAD 8a821e9: MBP Kueue job fleet-mbp-8cafeecea11e43e9 passed all 73 tests in tests/test_policy_findings.py and tests/test_policy_compile.py in 0.81s. This covers the earlier synthetic/identity follow-ups and the new capacity tests. The job used 1 CPU / 2G and a throwaway venv with pinned pytest/PyYAML. No groot tests or builds ran.

@kvnloo

kvnloo commented Oct 24, 2026

Copy link
Copy Markdown
Owner Author

Completed agent-outcomes proxy/product-evidence adapter in commits 89aab0b and 0f1ceca. Push/PR/tests-only outcomes remain non-promoting proxies. Product evidence requires a SHA-256-verified, run-bound receipt with structured observed product target; each receipt yields one task_success observation using receipt success, not ledger claims or asserted aggregates. JSON/JSONL and frozen SQLite ingestion, duplicate exclusion, WAL rejection, z0eval laundering exclusion and README contract are included. MBP job fleet-mbp-216013157c2f4ae6: 87 passed in 1.48s across tests/test_policy_findings.py and tests/test_policy_compile.py, exit 0. No tests/builds ran on groot.

…sion

Deliberately supersede default-cost routing: missing/stale economics route to frontier/Jev (law E measurement provenance; #224). Preserve assumed costs and derived bar only as diagnostics.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:docs Documentation area:verify Tests, mutation, evidence

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant