Repository navigation
RFC: Experience → outcome → counterfactual credit contract #54
Description
Parent: kvnloo/z0#15
Research follow-up — repair lineage and prioritization (2026-10-07)
Use the existing #137 recurrence join, EventLog, DecisionReceipt extensions and #62 identities. Do not add another failure database, judge or priority engine. #137 owns the reuse workstream; #22 owns continuity; #47 and kvnloo/company-os#2 / kvnloo/company-os#3 project the result read-only.
The research exposed a concrete canonicalization fixture: #140's active packet implementation referenced closed duplicate #138 instead of #137. Its PR metadata is now corrected; its historical branch/commits remain intact. Canonical tracker identity and historical aliases must be separate, so the same work does not become two repair queues.
Bounded acceptance
- Derive one traceable view: original occurrence -> canonical tracker -> exact implementation -> regression evidence -> actual consumer -> independently checked outcome -> later recurrence. Preserve symptom versus root-cause hypothesis.
- Canonicalize the feat: add revision-bound architecture reuse packet #140 -> memory/reuse: continuously fresh code intelligence + reuse-before-write gate #137 relation while retaining memory: continuously fresh architecture memory + reuse gate (child of #22) #138 only as duplicate/history. Do not rename branches, rewrite commit history or fabricate merge/deployment evidence to make lineage look clean.
- Replay and multiple summaries of one source-native occurrence count once; a genuinely new occurrence remains distinct. Missing history/telemetry is unknown coverage, not zero failures.
- Keep
implemented-on-branch,merged,configured,loaded,retrieved,model-visible,used,behavior-verifiedanddeployed-acceptanceas independent evidence dimensions. Freshness and scope coverage are orthogonal. feat: add revision-bound architecture reuse packet #140's successful unit workflow is not consumer acceptance. - A later recurrence reopens the outcome assessment, preserving the original record. Any GitHub status mutation remains in the existing authorized workflow; the visibility layer does not become a controller.
- Rank bounded repairs from observed impact, recurrence/exposure, affected consumers, blocked dependencies and estimated effort. Report uncertainty and reasons; do not treat the research's illustrative multiplicative formula or dates as measured scores or commitments.
- Evaluate on held-out episodes: fewer duplicate implementations, false novelty decisions, stale-evidence escapes and user corrections; include false blocks, latency and all measured costs. More receipts/issues or fewer tokens alone cannot win.
- Preserve original human authorship/contribution lineage; influence is not endorsement. Public projections exclude raw private transcripts and secrets.
The next action for each row must be explicit: source repair, implementation, integration, behavioral validation, or no change. These are acceptance requirements, not newly observed runtime outcomes.
Problem
Observed trajectories provide supervision, but behavioral action ≠ optimal label. A successful trajectory also does not prove which retriever/router/model/tool/verifier caused success.
z0intelligence already has receipts, outcomes, episode work, disagreement auditing, credit helpers, and counterfactual machinery. The missing contract is how experience becomes trustworthy learning signal.
Contract
Define a versioned episode/credit record joining:
- DecisionOpportunity identity;
- trace/work-item/attempt lineage;
- state/evidence revision;
- candidate action set;
- action actually taken;
- policy/backend/model revision and full distribution when available;
- tool/execution result;
- independent verifier/outcome;
- retries/corrections/escalations;
- measurement completeness;
- label source + confidence;
- counterfactual availability;
- credit level: observational / ablation / paired counterfactual / causal;
- privacy/provenance class.
Required invariants
- execution completion is not verified success;
- teacher/reference agreement is not gold;
- JEV is an observer/reference backend, not truth;
- user/agent historical action is not automatically the best action;
- missing telemetry cannot be interpreted as zero cost;
- optimizer-generated labels cannot silently become its own sealed judge.
Existing work to reuse
- J2: observer-first eval receipts, stability replay, and outcome-grounded calibration #14 observer-first eval receipts
- Mine confident observer/label disagreements into an episode-audit queue #27 disagreement audit queue
- Jev on real code-review diffs, and o8 tables for the episode compiler #11 real-diff transfer evidence
- Evolution Lab local cognition: add tool-calling and orchestration SLM portfolio #20 grouped observer evals
- Evolution Lab P0c: integrate RLCDAlignBench as an external Jev-style detector benchmark #24 verified-trajectory learning
- Tokenomics outcome tiers/measurement completeness
- z0evals frozen-study conventions
Acceptance criteria
- stable IDs join state→decision→execution→outcome across restart/replay;
- provenance can distinguish deterministic gold, negative gold, execution-only, soft/teacher, and unknown;
- counterfactual claims carry an explicit evidence level;
- a training pipeline can exclude teacher/self-label leakage;
- negative/inconclusive episodes remain first-class;
- the same record supports simple baselines before complex learners.
Falsification
Reject added schema complexity that does not change downstream attribution, leakage prevention, or learning quality relative to existing receipt/outcome records.
Non-goals
- inventing labels for unlabeled history;
- using JEV as gold;
- requiring causal identification when only observational evidence exists.
Activity
kvnloo commentedon Oct 5, 2026
Memory consolidation pointer: #22 now owns continuity priorities and the audited implementation state; #66 owns memory identity/scope/snapshot enforcement. This issue keeps broader outcome-credit/learning ownership. Memory experiments must reuse its existing receipt and frozen-evaluation boundaries, not create a separate reward or judge. Distinguish retrieval, actual model-visible injection, use and independent verification; captured text or task completion alone is not positive memory credit.
kvnloo commentedon Oct 7, 2026
#137 follow-up: make recurring failures actionable through the existing episode/credit join
The revised #137 maps 12 failure classes to current owners and separates tracked, implemented, merged, model-visible, used and independently verified states. This is a proposed extension of this issue's existing join—not another failure database or receipt schema.
Smallest next slice: derive a repair view from admitted EventLog observations + existing DecisionReceipt/outcome fields + canonical issue/PR references. For each admitted occurrence, retain the original event identity, symptom versus root-cause hypothesis, relevant intent/source revision, owning tracker, implementation ref, affected consumer, independent outcome, remaining blocker and next bounded action.
Required negative controls:
Acceptance: one original failure/correction can be traced through existing tracker → exact implementation → production-importing regression or appropriate live proof → actual consumer → independently checked outcome. The report must say whether the next action is implementation, wiring, validation, source repair or no change.
Reuse #62 for delivery identity and #47/CompanyOS#2 for a bounded read-only projection. No learned priority policy or broad history migration is needed to prove this join.
kvnloo commentedon Oct 10, 2026
CLAIM — bounded backlog engineering on existing receipt.py and test_receipt.py only. Start at own live-shadow-tested integration6770954. Ambient closes with whitespace/non-string verification_source can currently escape missing-verifier demotion through Python truthiness. Requirement: #54 execution completion is not verified quality; receipt Outcome documents a verifier source exception, not malformed truthy data.
Reproduce through production Outcome/join/readback. Make the smallest guard, preserve valid CI references, explicit negatives, historical rows and all prior unresolved/DISCARD/OBSERVE outcomes. No new outcome store/schema/judge, source-policy relaxation, default merge, paid inference, protected evaluator change or D2 repeat. This backlog slice is separate from Hermes host/producer work and Claude extractor/ctx work. Lease ends after the slice or2026-10-10T07:11:05Z.
kvnloo commentedon Oct 10, 2026
READY for independent review — draft PR149 targets the existing feature6770954, so reviewers see only two files. No default merge/deployment.
Production receipt guard now requires a nonblank string verifier reference for the ambient-close exception. RED2 failures/20pass, GREEN41 focused receipt/outcome tests. Actual identical CLI emit/join input: frozen6770954, installed c76f621, canonical State Packet source dcbcc88 each produced false gold; candidate3350a17 produces execution/verified_success null. Both current host receipt files hash8916fea265e7e70ddecb53aa19950ff8414899518c5f179d6ef4ce45d3b305e8. Private CLI comparison artifactSHA4eda9a9866c0b31d789872b81827d1e9e871a73bdbb0d0752bec84f026ea8a83. Explicit negative joins remainnegative, stored source evidence preserved, append-only rows retained; real skill DISCARD/OBSERVE outcomes untouched.
Exact-head external CoreCI38026695152 all4jobs PASS. Local tests-only offline1269/3skip/5deselect; CI uses full root and its own optional dependency environment, so counts are not directly interchangeable. Raw hash check and69 published claims pass. No protected evaluator mutation, new receipt store/schema/judge, model inference, authorization shortcut or tournament verdict. Nonblank reference alone is not authentication or independent success; existing higher qualification gates remain.
Lease released. Prior source integration199a849/Claude history/a90 latency correction/677coldFFF fix retained through ancestry. Next queue check stays backlog; no active Codex supervisor/frozen tournament assignment found, no claim of unattended multi-hour host execution.
kvnloo commentedon Oct 10, 2026
CLAIM — next independent engineering slice, existing receipt.py summary and its owning test only. #54 requires replay/multiple summaries of one source-native occurrence to count once, while a later recurrence can reopen the outcome assessment without losing history. Current summarize_tokenomics aggregates every append-only receipt revision, including repeated joins and canonical/bridge mirrors; verified-task and savings totals can inflate and historical positive revisions survive a later negative in aggregate.
Freeze3350a17, reproduce through real append/join/close functions and exact trace IDs. Keep canonical receipt precedence and legacy fallback consistent with find_receipt. Retain all raw/history rows, distinct traces, explicit negatives and unknown identities. No new store/schema/evaluator, protected split edits, paid inference, deployment, default merge or tournament judgment. Lease ends after slice or2026-10-10T07:11:05Z.
kvnloo commentedon Oct 10, 2026
READY for independent review — stacked draftPR150, exact candidate5216e2a from3350a17; no default merge/deployment. Nine added production lines, no new function/class/store/schema. Existing summary now counts the latest receipt once per exact trace, with canonical precedence; raw history, explicit negatives, distinct traces, legacy-only fallback and unidentified rows retained.
Corrected RED3 failures/22pass against frozen3350a17; same three failures independently reproduced against installed c76f621 and canonical-wrapper dcbcc88. An initial fixture-directory failure was corrected and retained separately, not counted as behavioral proof. GREEN44 focused receipt/outcome tests. Full localoffline1276/3skip/5deselect; exact-head CoreCI38027215938 all4jobs PASS: bothPython versions1270/9skip/5deselect, owning256Python,82TypeScript/1skip/510assertions. Differences reflect optional runtime availability, not more coverage. Raw hash checks and69 published claims pass. New owning-test sourceSHA61003421141718b46659639fe76b66ffbc6c5d5082afa3535f46be2eecf13f56.
Independent review stillopen; no contestant-awarded tournament KEEP, model improvement, measured provider savings, protected evaluator change or successful native reuse claim. Trace grouping does not invent cross-trace native work-item lineage. Existing per-file summary window remains a limitation.
Lease released; source3350a17/6770954/199a849 and Claude history retained. PR149 remains separate reviewed-source candidate; prior skill DISCARD, actual OBSERVE, failed proposals, D2 unresolved strict verdicts and two-upstream latency correction unchanged. Next queue refresh checks host supervision/readiness; no active Codex supervisor/frozen tournament assignment observed.