Skip to content

bug: Claim model has no min-evidence validator — uncited claims land via bundle import, put_claim, update_claim #81

Description

@galuis116

What happened

The Claim model has no min-evidence validator
(src/vouch/models.py:184-187), so a claim with evidence: [] is
schema-valid. As a result, the README and CONTRIBUTING contract —

Claims must cite sources. A claim without at least one Source /
Evidence id is a validation error, not a warning.

is enforced in exactly one place: proposals.propose_claim
(src/vouch/proposals.py:89-90). Every other write path to
claims/<id>.yaml accepts uncited claims silently:

  • Bundle importbundle._validate_content defers to
    Claim.model_validate (src/vouch/bundle.py:43,222-228), which
    accepts evidence: []. A hand-crafted bundle with one such claim
    passes import_check (ok=True, no issues) and import_apply
    writes it to disk under the legitimate bundle_id. The audit
    log records a clean bundle.import event for a KB that now
    contains an uncited claim.

  • store.put_claim directsrc/vouch/storage.py:283-291
    iterates claim.evidence to verify each cited id exists, but an
    empty list iterates zero times. Lifecycle ops
    (supersede, contradict, archive, confirm) and any caller
    that constructs a Claim directly can land an uncited one.

  • store.update_claimsrc/vouch/storage.py:308-… writes
    the YAML without any citation check. A previously-cited claim
    can be mutated to evidence=[] and the change persists.

This is exactly the failure mode CONTRIBUTING calls out under
"Things we won't merge":

Validation relaxations that let claims land without citations.
A claim without evidence is a working note at best — register
a source.

— except the enforcement was never on the model in the first
place, so every write path except the one explicit gate already
qualifies as the relaxed shape the team won't accept.

What you expected

Constructing a Claim with evidence=[] raises pydantic.ValidationError.
That closes all three bypass paths in one stroke:

  • bundle.import_apply raises (via _validate_content) before
    writing.
  • store.put_claim(Claim(evidence=[])) raises at the Claim(...)
    call.
  • store.update_claim(claim_with_emptied_evidence) raises the
    moment a caller mutates claim.evidence = [] and tries to pass
    it back through Claim(...) or pydantic re-validation.

The duplicate check inside propose_claim
(if not evidence: raise ProposalError(...)) can stay as the
nicer user-facing error message, but the load-bearing invariant
moves to the model where it belongs.

Reproduction

$ python uncited_repro.py
work dir: /tmp/vouch-uncited-XXXXXXXX
Bypass 1 (put_claim): SUCCEEDED — uncited claim on disk
  YAML on disk has 'evidence: []' → True
Bypass 2 (update_claim): SUCCEEDED — citations stripped to []

bundle.import_check(uncited): ok=True issues=[]
bundle.import_apply: wrote 1 files
landed claim text: 'shipped via bundle, no citations'
landed claim evidence: []

BYPASS 3 CONFIRMED: bundle import landed an uncited claim — the
README contract was broken.

uncited_repro.py exercises all three paths against a fresh KB:

  1. store.put_claim(Claim(id="direct-uncited", text="…", evidence=[]))
    — writes claims/direct-uncited.yaml with evidence: [].
  2. Create a cited claim, then c.evidence = []; store.update_claim(c)
    — the empty list persists.
  3. Build a bundle whose manifest is internally consistent and whose
    claims/bundle-uncited.yaml member has evidence: []. Call
    import_check (returns ok=True) and import_apply (lands the
    uncited claim). Read it back via store.get_claim — the
    stored claim has evidence == [].

Environment

  • vouch version: post-fix(bundle): verify per-member sha256 against manifest on import #75 (any commit since evidence: list[str] = Field(default_factory=list, ...) was the model shape — the
    validator was never there)
  • Python version: Python 3.12.13
  • OS: any (model-level; platform-independent)
  • Host: any (CLI / MCP / JSONL — every transport eventually calls
    one of the three uncited code paths)

.vouch/ state

$ vouch doctor

Not relevant — the bug reproduces on a freshly-initialised KB.

Anything else

Suggested fix

src/vouch/models.py, on the Claim model:

class Claim(BaseModel):
    ...
    evidence: list[str] = Field(
        default_factory=list,
        description="Source ids OR Evidence ids — both are valid citations",
    )

    @field_validator("evidence")
    @classmethod
    def _at_least_one_citation(cls, v: list[str]) -> list[str]:
        if not v:
            raise ValueError(
                "claim must cite at least one Source or Evidence id "
                "(README §'Object model'; CONTRIBUTING §'Things we won't merge')"
            )
        return v

That's the entire fix on the source side. The existing guard in
proposals.propose_claim becomes redundant but can stay as a
nicer error string for the CLI/JSONL front door.

Tests to add in tests/test_storage.py (round-trip per CONTRIBUTING):

  • test_claim_model_rejects_empty_evidenceClaim(evidence=[])
    raises pydantic.ValidationError.
  • test_put_claim_rejects_empty_evidence — direct
    store.put_claim raises.
  • test_update_claim_rejects_empty_evidence — mutating an
    existing claim to evidence=[] raises before disk write.
  • In tests/test_bundle.py: test_import_rejects_uncited_claim
    — bundle with a manifest-consistent claim that has
    evidence: [] is rejected by import_check with a clean
    schema validation failed: issue, and import_apply raises.

Why it's worth fixing

Checked for duplicates

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions