Problem
DecisionRecord's three cryptographic commitments — configDigest, promptDigest, modelId — do not
commit to the things that actually decided the PR. All three are computed at the single call site,
src/queue/processors.ts:3426-3442.
1. modelId is hardcoded null on every path
The field is documented as "Model + prompt commitments when an AI review contributed"
(decision-record.ts:72), and renderDecisionRecordSection therefore always prints **model**: n/a.
The site's own comment concedes it ("modelId stays null at this site") and points at
reviewDiagnostics as holding the real per-run model identities — so the data exists and is simply not
threaded through.
2. promptDigest commits to the base template, not the prompt that was sent
3439: promptDigest: aiJudgment !== undefined ? await contentDigest({ version: REVIEW_PROMPT_VERSION, template: REVIEW_SYSTEM_PROMPT }) : null,
The prompt actually sent is buildSystemPrompt() (src/services/ai-review.ts ~1194), which is
REVIEW_SYSTEM_PROMPT plus up to ten suffixes: grounding, enrichment, profile/tone, security-focus,
path instructions, review.instructions (arbitrary maintainer prose that steers the judge), screenshot
evidence, inline, category, and improvement-signal. None enter the digest.
ai-review.ts:76 claims "a silent template edit changes every subsequent record digest" — true only of
the base constant. Two repos, one with review.instructions: "close anything touching src/billing" and
one with none, publish byte-identical promptDigest values for materially different judges. An
operator can edit review.instructions between two decisions with zero digest movement — which is the
single easiest way to bias the reviewer, via a .loopover.yml key.
3. configDigest hashes raw settings, not the resolved policy
3435: configDigest: await contentDigest(settings),
documented as "Digest of the RESOLVED effective settings … commits the operator to the exact config that
judged this PR" (decision-record.ts:65-67). But the policy that decided is
gateCheckPolicy(settings, …, aiReviewCloseConfidenceOverride), and that last argument is resolved
outside settings from live DB state (src/queue/gate-checks.ts:93-97). For any repo without an
explicit gate.aiReview.closeConfidence — the default — the close floor comes from
readCalibratedThreshold reading system_flags, which the daily runRiskControlRecalibration tick
rewrites or deletes (src/review/risk-control-wire.ts:123-136).
So a PR closed Monday at λ̂=0.93 and one held Tuesday at λ̂=0.97 publish the same configDigest.
Two further live inputs are outside the commitment too: readUntrustworthyRuleCodes(env)
(processors.ts:3335, cron-refreshed) and the CI aggregate — the latter notable because ciState is
hardcoded null despite being documented as "CI aggregate consumed by the decision" and despite red CI
being a primary close reason.
Impact
A configDigest + promptDigest + modelId triple that captures neither the model, nor the effective
prompt, nor the calibrated threshold cannot support "replay under the same config" — the property #8838
and the public verification story both rest on. The operator is not, in fact, committed to the policy
that judged the PR.
Note the replay path is unaffected: decision_replay_inputs.replay_json stores the fully-resolved
GateCheckPolicy. This breaks the public commitment only — which is precisely the half a third party
can see.
Requirements
- Digest the resolved
gateCheckPolicy(...) object (the thing that actually decided) rather than raw
settings; keep the raw-settings digest as a separate field if it is independently useful.
- Digest
buildSystemPrompt(input)'s actual output, or the resolved suffix set, rather than the base
constant.
- Thread the concrete provider/model ids from
reviewDiagnostics into modelId; record the full set when
more than one model ran, not a representative one.
- Populate
ciState from the live aggregate already in scope at the call site.
- Bump
DECISION_RECORD_SCHEMA_VERSION (currently "3") and state in the schema doc exactly what each
digest covers, so the next reader does not have to derive it from the call site.
Test Coverage Requirements
99%+ patch coverage, branch-counted. Regression tests proving: a changed review.instructions moves
promptDigest; a changed calibrated close floor moves configDigest; modelId is non-null whenever an
AI judgment shaped the decision.
Links & Resources
Boundaries
Record contents only. No change to the chain, the endpoint, or which decisions are recorded.
maintainer-only — public trust-surface authority.
Problem
DecisionRecord's three cryptographic commitments —configDigest,promptDigest,modelId— do notcommit to the things that actually decided the PR. All three are computed at the single call site,
src/queue/processors.ts:3426-3442.1.
modelIdis hardcodednullon every pathThe field is documented as "Model + prompt commitments when an AI review contributed"
(
decision-record.ts:72), andrenderDecisionRecordSectiontherefore always prints**model**: n/a.The site's own comment concedes it ("modelId stays null at this site") and points at
reviewDiagnosticsas holding the real per-run model identities — so the data exists and is simply notthreaded through.
2.
promptDigestcommits to the base template, not the prompt that was sentThe prompt actually sent is
buildSystemPrompt()(src/services/ai-review.ts~1194), which isREVIEW_SYSTEM_PROMPTplus up to ten suffixes: grounding, enrichment, profile/tone, security-focus,path instructions,
review.instructions(arbitrary maintainer prose that steers the judge), screenshotevidence, inline, category, and improvement-signal. None enter the digest.
ai-review.ts:76claims "a silent template edit changes every subsequent record digest" — true only ofthe base constant. Two repos, one with
review.instructions: "close anything touching src/billing"andone with none, publish byte-identical
promptDigestvalues for materially different judges. Anoperator can edit
review.instructionsbetween two decisions with zero digest movement — which is thesingle easiest way to bias the reviewer, via a
.loopover.ymlkey.3.
configDigesthashes rawsettings, not the resolved policydocumented as "Digest of the RESOLVED effective settings … commits the operator to the exact config that
judged this PR" (
decision-record.ts:65-67). But the policy that decided isgateCheckPolicy(settings, …, aiReviewCloseConfidenceOverride), and that last argument is resolvedoutside
settingsfrom live DB state (src/queue/gate-checks.ts:93-97). For any repo without anexplicit
gate.aiReview.closeConfidence— the default — the close floor comes fromreadCalibratedThresholdreadingsystem_flags, which the dailyrunRiskControlRecalibrationtickrewrites or deletes (
src/review/risk-control-wire.ts:123-136).So a PR closed Monday at λ̂=0.93 and one held Tuesday at λ̂=0.97 publish the same
configDigest.Two further live inputs are outside the commitment too:
readUntrustworthyRuleCodes(env)(
processors.ts:3335, cron-refreshed) and the CI aggregate — the latter notable becauseciStateishardcoded
nulldespite being documented as "CI aggregate consumed by the decision" and despite red CIbeing a primary close reason.
Impact
A
configDigest+promptDigest+modelIdtriple that captures neither the model, nor the effectiveprompt, nor the calibrated threshold cannot support "replay under the same config" — the property #8838
and the public verification story both rest on. The operator is not, in fact, committed to the policy
that judged the PR.
Note the replay path is unaffected:
decision_replay_inputs.replay_jsonstores the fully-resolvedGateCheckPolicy. This breaks the public commitment only — which is precisely the half a third partycan see.
Requirements
gateCheckPolicy(...)object (the thing that actually decided) rather than rawsettings; keep the raw-settings digest as a separate field if it is independently useful.buildSystemPrompt(input)'s actual output, or the resolved suffix set, rather than the baseconstant.
reviewDiagnosticsintomodelId; record the full set whenmore than one model ran, not a representative one.
ciStatefrom the live aggregate already in scope at the call site.DECISION_RECORD_SCHEMA_VERSION(currently"3") and state in the schema doc exactly what eachdigest covers, so the next reader does not have to derive it from the call site.
Test Coverage Requirements
99%+ patch coverage, branch-counted. Regression tests proving: a changed
review.instructionsmovespromptDigest; a changed calibrated close floor movesconfigDigest;modelIdis non-null whenever anAI judgment shaped the decision.
Links & Resources
src/queue/processors.ts~3426-3442;src/review/decision-record.ts~55-80;src/services/ai-review.ts~76, ~1166-1194;src/queue/gate-checks.ts~93-97;src/review/risk-control-wire.ts~123-136Boundaries
Record contents only. No change to the chain, the endpoint, or which decisions are recorded.
maintainer-only — public trust-surface authority.