Skip to content

api(eval): loopover.ai publishes a 460-case corpus and zero records committing to it #9805

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue.

Context

Verified live on 2026-07-29 against api.loopover.ai:

GET /v1/public/stats        -> rulePrecision.rules[0] = { ruleId: "ai_consensus_defect",
                                                          decided: 460, confirmed: 287, precision: 0.624 }
                               rulePrecision.latestBacktestRun = null
GET /v1/public/eval-corpus?ruleId=ai_consensus_defect
                            -> caseCount 460, truncated false,
                               checksum 470e234e8923461e8642807c46265f01393f6dc9e4e40e52363f41767b5c3c10
GET /v1/public/eval-scores  -> { "records": [] }

The corpus is real, complete, and its checksum is independently re-derivable: running
checksumPublicEvalCorpus over the cases array exactly as downloaded reproduces 470e234e…. The 460
published cases also match the 460 decided in rulePrecision exactly — they are the same dataset.

Yet the records array is empty, because buildEvalScoreRecordsFromRulePrecision
(src/review/eval-score-records.ts) sources its commitment from precision.latestBacktestRun, which
loadPublicRulePrecision reads from a persisted calibration.*_backtest_run audit event. #9639 fixes the
in-Worker writer, but that writer runs inside a review pass — and hosted review execution is retired
(src/index.ts acks-and-drops isReviewExecutionJob messages when !isSelfHostedReviewRuntime(env)). So on
loopover.ai that event never exists, and #9639 alone leaves this surface empty.

The user-visible consequence is that apps/loopover-ui/content/docs/verify-this-review.mdx §4 tells an
anonymous reader to curl .../v1/public/eval-scores | jq '.records' and re-derive recordDigest. They get
[]. The doc explains the empty array as "the latest backtest run's corpus is empty" — which is now
inaccurate: the corpus is not empty, it is 460 cases and downloadable.

The fix

Commit each record to its own rule's public-corpus checksum — the exact bytes /v1/public/eval-corpus
serves for that ruleId — when no persisted backtest run supplies one. That is not a synthetic or placeholder
commitment (which #9639 explicitly forbids): it is a checksum over an artifact the reader can download and
re-hash, which is the property reproducible trust tier asserts.

This also fixes a latent per-rule bug in the current code: latestBacktestRun.corpusChecksum is a single
value stamped onto every rule's record. With more than one rule in rulePrecision.rules, every record but
one would commit to a different rule's corpus. Only one rule is published today, which is why it has not
surfaced.

Requirements

  • The commitment for a rule's record must be the checksum of that rule's own corpus, never another rule's.
  • Window alignment is load-bearing: the corpus endpoint and PUBLIC_PRECISION_WINDOW_DAYS must agree, or the
    record commits to a different slice than the one its decided/confirmed were computed over. Assert it.
  • A truncated corpus (truncated: true, at PUBLIC_EVAL_CORPUS_MAX_CASES) must NOT be published as a
    commitment: the checksum would cover a subset while the score covers the whole window.
  • EMPTY_CORPUS_CHECKSUM must still be refused, for the reason eval-score-records.ts already documents.
  • A persisted backtest run, when one exists, must still win — self-host must not silently change behaviour.
  • verify-this-review.mdx §4's explanation of an empty records array must be corrected: it currently
    attributes it to an empty corpus, which is no longer the only cause.

Deliverables

  • Records published for every rule whose corpus is non-empty and untruncated, each committing to that
    rule's own checksum, on a deployment with no persisted backtest run.
  • A test asserting two rules get two DIFFERENT checksums (the per-rule bug, pinned).
  • A test asserting a truncated corpus yields no record for that rule.
  • A test asserting a persisted backtest run still takes precedence.
  • A test asserting the published checksum re-derives from the /v1/public/eval-corpus payload for the
    same rule — the reader's actual workflow, end to end.
  • verify-this-review.mdx §4 updated to describe the real conditions for an empty array.

Test Coverage Requirements

99%+ Codecov patch coverage, branch-counted, on src/**. Both arms of every changed conditional.

Expected Outcome

An anonymous reader can complete verify-this-review.mdx end to end against loopover.ai: fetch the corpus,
re-derive its checksum, fetch the records, confirm each record's recordDigest and confirm its
commitments.corpusChecksum equals the corpus they just hashed.

Links & Resources

src/review/eval-score-records.ts, src/review/public-eval-corpus.ts, src/review/public-rule-precision.ts,
src/api/routes.ts (the /v1/public/eval-scores handler), apps/loopover-ui/content/docs/verify-this-review.mdx.
Related: #9639, #9636, #9266, #9215.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions