Repository navigation
A ledger with no entries and invented probe text satisfies both invariant evidence gates #292
Description
Surfaced independently by both reviews of #290. Pre-existing on main, not introduced by that PR — I am filing it so it stops being rediscovered.
This ledger passes the project-discovery gate:
{
"schema_version": "ultrafuzz.invariant-evidence-ledger.v1",
"entries": [],
"inventory_rows": [],
"scan_probes": [{ "id": "p1", "source_path": ".", "query": "...", "result": "invented result text" }]
}Verified by one reviewer against origin/main as well as the #290 branch, so the workspace-root allowance already granted exactly this. A probe naming a nonexistent path is likewise accepted, by design (// Scan probes may intentionally target optional files. When a probe path is absent, its result text is the durable evidence of that absence.) — and its result text is equally unchecked.
Two facts make this weaker than it looks, which is why it is worth reasoning about rather than panicking about:
- Nothing anywhere reads
probe.result. Every consumer is a presence check:artifact-gates.ts(missing scan probe results,Property fan-in requires scan probe results from project discovery) andminItemsininvariant-ledger.ts. No code compares probe text to file bytes. - The schema forces
entries: []andinventory_rows: []for the no-evidence shape, so nothing can be laundered into the invariant campaign. Ledger entries remain strictly byte-checked against the pinned commit.
So the exposure is that "the agent searched and found nothing" is unfalsifiable, and the fan-in gate's scan_probes requirement is satisfiable without work. Whether that matters is a policy question, same shape as #285:
- accept it — a probe is a record of intent, and the entries are where provenance is enforced;
- require at least one probe whose path exists and is snapshotted, so a run cannot claim a scan it did not do;
- keep accepting absent/directory probes but mark the derived proof as partial, so downstream consumers can tell.
Needs an owner with a view on the threat model. Not blocking any current run.
Activity
fix(runtime): accept directory invariant scan probes (#290)
docs(prompts): correct a false claim I put in the probe guidance
aviggiano commentedon Aug 7, 2026
The hole is larger than filed.
The fan-in gate returns at
packages/artifacts/src/artifact-gates.ts:417-437for this shape and never callsverifyInvariantProbePath— so fan-in does not check probe containment at all, independently of the empty-ledger question. Rejecting empty ledgers will not fix that; it needs its own change.Recommended verdict: reject silence, accept explicit emptiness. Not
partial— that adds a third state every consumer must handle, andprobe.resultcurrently has zero consumers. Require ano_invariants_justificationwhen the ledger is empty, so a genuinely invariant-free target stays possible but is auditable.The comment at
:260-261(and its duplicate at:311-312) currently asserts "accept" as a fact rather than as a decision; whichever way this lands, that comment needs to record it as a decision.aviggiano commentedon Aug 7, 2026
Path correction to my earlier comment.
artifact-gates.tslives atpackages/runtime/src/artifact-gates.ts, notpackages/artifacts/src/. All the line numbers I cited (:260-261,:311-312,:417-437) resolve correctly against the runtime file.invariant-ledger.tsis inpackages/artifacts/src, which is where the confusion came from.Implemented on local branch
fix/invariant-evidence-gates(not pushed — a branch push triggers a paid Modal run):no_invariants_justificationfield. New diagnosticINVARIANT_LEDGER_NO_INVARIANTS_UNJUSTIFIED. No thirdpartialstate.verifyInvariantProbePath, so it checked probe containment for no shape at all. It now re-checks containment against discovery's workspace..ultrafuzz/prompts/setup/project-discovery.mdwas updated to ask for the field — without that, a genuinely invariant-free target would fail with no way for the agent to know what was wanted.Behaviour change to be aware of: three existing empty-ledger fixtures were updated. Any other fixture or golden artifact pinning an empty ledger will now fail the gate. That is intended, but it is a change.
aviggiano commentedon Aug 7, 2026
Cross-branch interaction found by integration testing. Merging all twelve triage branches together surfaced a failure that neither branch produces alone.
fix/scan-probe-shared-validator(#301) adds a test — "project discovery gate accepts a pinned and unchanged scan probe source" — whoseinvariantProbeLedger()helper builds a ledger withentries: []. That is deliberate: the fixture exercises probe containment and has no invariants to record.fix/invariant-evidence-gates(#292) now rejects an empty ledger withoutno_invariants_justification. So the #301 fixture fails under the #292 rule:Both branches are green in isolation. This is exactly the class of breakage the #292 work flagged in its own caveats ("any other fixture or golden artifact pinning an empty ledger will now fail the gate") — this is the first confirmed instance.
Resolution is one line: the fixture declares its justification. With that applied, the twelve-branch integration is fully green —
pnpm -r testexit 0 (prompts 46, evmbench 20, config 28, topology 34, evals 165, modal 340, runtime 396 pass / 15 skipped), plusformat:check,lintandvalidate:releaseall clean.Whichever of #292 / #301 lands second must carry this fixture change.
fix(runtime): reject an unjustified empty invariant ledger and name a…