Skip to content

feat: calibrated entity/relationship confidence (fix uniform 0.9 self-ratings) #83

Description

@rajnavakoti

Part of #82 (calibrated honesty).

Problem

The LLM self-rates almost every entity at confidence 0.9+ (a real 3-doc run had 0 entities below 0.6). A confidence signal that's always high is useless — it can't tell a human curator or a consuming agent which entities to doubt.

Goal

Confidence values that actually discriminate trustworthy from shaky entities/relationships, so they're usable as a trust signal.

Approaches to evaluate

  • Corroboration-based (leverages #provenance): entity asserted from 1 doc → lower; corroborated across N docs → higher. This is likely the strongest, cheapest signal.
  • Evidence-grounding: was the claim explicitly stated vs inferred? (the prompt already asks for hedged_statements — use them.)
  • A calibration pass / rubric that down-weights single-source, inference-heavy, or hedged claims.

Acceptance Criteria

  • Confidence is derived from signals (corroboration / evidence / hedging), not just the LLM's self-rating
  • Distribution actually spreads (not all clustered at 0.9) on a representative corpus
  • Single-source, inference-only entities score measurably lower than multi-source, explicitly-stated ones
  • Relationships get their own confidence (not just entities)
  • Tests: a single-doc inferred claim scores lower than a corroborated explicit one

Out of scope

  • Provenance data model itself (separate issue) — this consumes it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    frameworkScientific frameworks

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions