Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/gittensory-miner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,14 @@ The package also includes an append-only prediction ledger: `initPredictionLedge
codes, plus the producing `ENGINE_VERSION`) in local SQLite, so a later self-improve pass can score predictions
against realized outcomes. Insert-only. (#4263)

The package also includes the Phase 7 calibration runner: `runHistoricalReplayCalibrationCycle`
(`lib/calibration-run.js`) scores a completed historical-replay run with the deterministic objective-anchor scorer,
folds the composite into the engine's `computePhase7CalibrationLoop` combine alongside the existing `pr_outcome`
signal, and persists the combined snapshot as a `calibration_snapshot` event — queryable with
`gittensory-miner ledger list --type calibration_snapshot` or `readCalibrationSnapshots` /
`latestCalibrationSnapshot`. It measures and records only; acting on the metric (autonomy bumps, threshold tuning)
stays maintainer-only. See [`docs/miner-selfimprove-calibration.md`](docs/miner-selfimprove-calibration.md). (#4248)

`gittensory-miner manage status` now also folds each tracked repo's current discover/plan/prepare run state
(`run-state.js`) alongside its managed PR rows into a "run portfolio" view — `collectRunPortfolio` /
`renderRunPortfolioTable` — so a repo actively being discovered or planned shows up even with zero PRs yet.
Expand Down
21 changes: 21 additions & 0 deletions packages/gittensory-miner/docs/miner-selfimprove-calibration.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,27 @@ project, how many predictions were `wouldMerge` vs `wouldClose`, and how each re
error)**, and symmetrically for close. `mergePrecision` / `closePrecision` are the headline accuracy numbers a
dashboard renders.

## The runner: wiring the replay scorer into the combine (#4248)

`computePhase7CalibrationLoop` is a **pure combine contract** — by design it cannot schedule replay runs or read
ledgers (the miner depends on the engine, not the reverse), so it waits for an external caller to feed it a real
`historical_replay` composite. #3014 (PR #3225) shipped only that engine side; a 2026-07-08 audit found the closed
issue claimed the two halves were "wired" when in fact **no miner-side runner ever called the replay scorer
(`computeObjectiveAnchor`, #3012) and passed its result into the combine** — two finished pieces, never connected.

[`lib/calibration-run.js`](../lib/calibration-run.js) (#4248) is that missing runner.
`runHistoricalReplayCalibrationCycle` scores a completed replay run with the deterministic objective-anchor scorer,
reduces the per-task scores to one composite `[0, 1]`, folds it into the `HistoricalReplayCalibrationInput` shape the
engine expects, calls `computePhase7CalibrationLoop` with that **plus the existing `pr_outcome` signal**, and
persists the combined snapshot as a `calibration_snapshot` event on the local append-only event ledger (the same
typed-event-over-`event-ledger.js` pattern as `pr-outcome.js`). The persisted metric is queryable with
`gittensory-miner ledger list --type calibration_snapshot` and via `readCalibrationSnapshots` /
`latestCalibrationSnapshot`.

Consistent with the boundary below, the runner is **read/measure-only**: it produces and persists the tracked
metric but never acts on it (no autonomy bump, no threshold tune). Acting on the combined accuracy — the
calibration-gated circuit-breaker — remains maintainer-only (#2352).

## Value-weighting: durable correctness, not volume

Raw merge/close precision is not the real objective, and a contributor reading a dashboard number should understand
Expand Down
161 changes: 161 additions & 0 deletions packages/gittensory-miner/lib/calibration-run.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
import type {
HistoricalReplayCalibrationInput,
Phase7CalibrationConfig,
Phase7CalibrationLoopResult,
Phase7CalibrationManifest,
PrOutcomeCalibrationInput,
ReplayHarnessStatus,
} from "@jsonbored/gittensory-engine";

import type { AppendEventInput, LedgerEntry } from "./event-ledger.js";
import type {
ObjectiveAnchorResult,
ReplayPlanInput,
RevealedHistoryEntry,
} from "./replay-objective-anchor.js";

export const MINER_CALIBRATION_SNAPSHOT_EVENT: "calibration_snapshot";

/** One completed replay-run task result: what the replay targeted, and the revealed post-T history to score it. */
export interface ReplayTaskResult {
replayPlan?: ReplayPlanInput | null;
revealedHistory?: RevealedHistoryEntry[] | RevealedHistoryEntry | null;
}

export interface ScoreCompositeOptions {
computeObjectiveAnchor?: (
input: { replayPlan?: ReplayPlanInput | null; revealedHistory?: RevealedHistoryEntry[] | RevealedHistoryEntry | null },
) => ObjectiveAnchorResult;
}

export interface HistoricalReplayCompositeScore {
compositeScore: number | null;
sampleSize: number;
scores: number[];
}

export function scoreHistoricalReplayComposite(
replayResults: readonly ReplayTaskResult[] | null | undefined,
options?: ScoreCompositeOptions,
): HistoricalReplayCompositeScore;

/** A completed replay run's descriptor: its per-task results plus the run's identity/freshness/harness health. */
export interface ReplayRunDescriptor {
replayResults?: readonly ReplayTaskResult[] | null;
replayRunId?: string;
observedAt?: string;
harnessStatus?: ReplayHarnessStatus;
}

export interface BuiltHistoricalReplayInput {
historicalReplay: HistoricalReplayCalibrationInput | null;
compositeScore: number | null;
sampleSize: number;
scores: number[];
}

export function buildHistoricalReplayCalibrationInput(
replayRun: ReplayRunDescriptor | null | undefined,
options?: ScoreCompositeOptions,
): BuiltHistoricalReplayInput;

/** The persisted, public-safe projection of a Phase7CalibrationLoopResult. */
export interface CalibrationSnapshotPayload {
enabled: boolean;
combinedAccuracy: number | null;
baselineAccuracy: number;
deltaFromBaseline: number | null;
autonomyIncreasePermitted: boolean;
replayHarnessHold: boolean;
replayHarnessStatus: string;
replayRunDue: boolean;
holdReasons: string[];
contributingSources: string[];
replayRunId: string | null;
observedAt: string | null;
replaySampleSize: number;
}

export interface SnapshotMeta {
replayRunId?: string | null;
observedAt?: string | null;
sampleSize?: number;
}

export function snapshotPayloadFromResult(
result: Phase7CalibrationLoopResult,
meta?: SnapshotMeta,
): CalibrationSnapshotPayload;

export function normalizeCalibrationSnapshotPayload(payload: unknown): CalibrationSnapshotPayload | null;

export interface RecordCalibrationSnapshotOptions {
/** Optional at the type level so a caller can pass an unusable ledger to exercise the fail-closed guard; the
* writer throws `invalid_event_ledger` at runtime when this is absent or lacks `appendEvent`. */
eventLedger?: { appendEvent(event: AppendEventInput): LedgerEntry };
repoFullName?: string;
}

export function recordCalibrationSnapshot(
input: unknown,
options?: RecordCalibrationSnapshotOptions,
): LedgerEntry | null;

export interface CalibrationSnapshotReader {
readEvents(filter?: { since?: number | null; repoFullName?: string | null }): unknown[];
}

export interface CalibrationSnapshotFilter {
since?: number | null;
repoFullName?: string | null;
}

export interface PersistedCalibrationSnapshot extends CalibrationSnapshotPayload {
repoFullName: string | null;
seq: number | null;
createdAt: string | null;
}

export function readCalibrationSnapshots(
eventLedger: CalibrationSnapshotReader,
filter?: CalibrationSnapshotFilter,
): PersistedCalibrationSnapshot[];

export function latestCalibrationSnapshot(
eventLedger: CalibrationSnapshotReader,
filter?: CalibrationSnapshotFilter,
): PersistedCalibrationSnapshot | null;

export interface RunCalibrationCycleInput {
config?: Phase7CalibrationConfig | Phase7CalibrationManifest | Record<string, unknown> | null;
prOutcome?: PrOutcomeCalibrationInput | null;
replayRun?: ReplayRunDescriptor | null;
now?: string | Date | null;
observedAt?: string | null;
repoFullName?: string;
}

export interface RunCalibrationCycleDeps extends ScoreCompositeOptions {
computeLoop?: (input: {
config?: Phase7CalibrationConfig | Phase7CalibrationManifest | Record<string, unknown> | null;
prOutcome?: PrOutcomeCalibrationInput | null;
historicalReplay?: HistoricalReplayCalibrationInput | null;
now?: string | Date | null;
}) => Phase7CalibrationLoopResult;
eventLedger?: { appendEvent(event: AppendEventInput): LedgerEntry };
}

export interface RunCalibrationCycleResult {
result: Phase7CalibrationLoopResult;
snapshot: CalibrationSnapshotPayload;
recorded: LedgerEntry | null;
historicalReplay: HistoricalReplayCalibrationInput | null;
compositeScore: number | null;
sampleSize: number;
scores: number[];
}

export function runHistoricalReplayCalibrationCycle(
input?: RunCalibrationCycleInput,
deps?: RunCalibrationCycleDeps,
): RunCalibrationCycleResult;
Loading