Skip to content

orb(trust): the decision record's commitments don't commit — modelId is always null, promptDigest ignores ten prompt suffixes, configDigest misses the calibrated close floor #9124

Description

@JSONbored

Problem

DecisionRecord's three cryptographic commitments — configDigest, promptDigest, modelId — do not
commit to the things that actually decided the PR. All three are computed at the single call site,
src/queue/processors.ts:3426-3442.

1. modelId is hardcoded null on every path

3438:  modelId: null,

The field is documented as "Model + prompt commitments when an AI review contributed"
(decision-record.ts:72), and renderDecisionRecordSection therefore always prints **model**: n/a.
The site's own comment concedes it ("modelId stays null at this site") and points at
reviewDiagnostics as holding the real per-run model identities — so the data exists and is simply not
threaded through.

2. promptDigest commits to the base template, not the prompt that was sent

3439:  promptDigest: aiJudgment !== undefined ? await contentDigest({ version: REVIEW_PROMPT_VERSION, template: REVIEW_SYSTEM_PROMPT }) : null,

The prompt actually sent is buildSystemPrompt() (src/services/ai-review.ts ~1194), which is
REVIEW_SYSTEM_PROMPT plus up to ten suffixes: grounding, enrichment, profile/tone, security-focus,
path instructions, review.instructions (arbitrary maintainer prose that steers the judge), screenshot
evidence, inline, category, and improvement-signal. None enter the digest.

ai-review.ts:76 claims "a silent template edit changes every subsequent record digest" — true only of
the base constant. Two repos, one with review.instructions: "close anything touching src/billing" and
one with none, publish byte-identical promptDigest values for materially different judges. An
operator can edit review.instructions between two decisions with zero digest movement — which is the
single easiest way to bias the reviewer, via a .loopover.yml key.

3. configDigest hashes raw settings, not the resolved policy

3435:  configDigest: await contentDigest(settings),

documented as "Digest of the RESOLVED effective settings … commits the operator to the exact config that
judged this PR"
(decision-record.ts:65-67). But the policy that decided is
gateCheckPolicy(settings, …, aiReviewCloseConfidenceOverride), and that last argument is resolved
outside settings from live DB state (src/queue/gate-checks.ts:93-97). For any repo without an
explicit gate.aiReview.closeConfidence — the default — the close floor comes from
readCalibratedThreshold reading system_flags, which the daily runRiskControlRecalibration tick
rewrites or deletes (src/review/risk-control-wire.ts:123-136).

So a PR closed Monday at λ̂=0.93 and one held Tuesday at λ̂=0.97 publish the same configDigest.
Two further live inputs are outside the commitment too: readUntrustworthyRuleCodes(env)
(processors.ts:3335, cron-refreshed) and the CI aggregate — the latter notable because ciState is
hardcoded null despite being documented as "CI aggregate consumed by the decision" and despite red CI
being a primary close reason.

Impact

A configDigest + promptDigest + modelId triple that captures neither the model, nor the effective
prompt, nor the calibrated threshold cannot support "replay under the same config" — the property #8838
and the public verification story both rest on. The operator is not, in fact, committed to the policy
that judged the PR.

Note the replay path is unaffected: decision_replay_inputs.replay_json stores the fully-resolved
GateCheckPolicy. This breaks the public commitment only — which is precisely the half a third party
can see.

Requirements

  • Digest the resolved gateCheckPolicy(...) object (the thing that actually decided) rather than raw
    settings; keep the raw-settings digest as a separate field if it is independently useful.
  • Digest buildSystemPrompt(input)'s actual output, or the resolved suffix set, rather than the base
    constant.
  • Thread the concrete provider/model ids from reviewDiagnostics into modelId; record the full set when
    more than one model ran, not a representative one.
  • Populate ciState from the live aggregate already in scope at the call site.
  • Bump DECISION_RECORD_SCHEMA_VERSION (currently "3") and state in the schema doc exactly what each
    digest covers, so the next reader does not have to derive it from the call site.

Test Coverage Requirements

99%+ patch coverage, branch-counted. Regression tests proving: a changed review.instructions moves
promptDigest; a changed calibrated close floor moves configDigest; modelId is non-null whenever an
AI judgment shaped the decision.

Links & Resources

Boundaries

Record contents only. No change to the chain, the endpoint, or which decisions are recorded.

maintainer-only — public trust-surface authority.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions