Part of #82 (calibrated honesty).
Problem
The LLM self-rates almost every entity at confidence 0.9+ (a real 3-doc run had 0 entities below 0.6). A confidence signal that's always high is useless — it can't tell a human curator or a consuming agent which entities to doubt.
Goal
Confidence values that actually discriminate trustworthy from shaky entities/relationships, so they're usable as a trust signal.
Approaches to evaluate
- Corroboration-based (leverages #provenance): entity asserted from 1 doc → lower; corroborated across N docs → higher. This is likely the strongest, cheapest signal.
- Evidence-grounding: was the claim explicitly stated vs inferred? (the prompt already asks for
hedged_statements — use them.)
- A calibration pass / rubric that down-weights single-source, inference-heavy, or hedged claims.
Acceptance Criteria
Out of scope
- Provenance data model itself (separate issue) — this consumes it.
Part of #82 (calibrated honesty).
Problem
The LLM self-rates almost every entity at confidence 0.9+ (a real 3-doc run had 0 entities below 0.6). A confidence signal that's always high is useless — it can't tell a human curator or a consuming agent which entities to doubt.
Goal
Confidence values that actually discriminate trustworthy from shaky entities/relationships, so they're usable as a trust signal.
Approaches to evaluate
hedged_statements— use them.)Acceptance Criteria
Out of scope