Lens reference-expectation authority: the prompt and the gate disagree about what "supplied" means

Summary

The property-lens prompt and the provenance gate use two different definitions of a "supplied" reference expectation, so in the configuration these benchmark runs actually use, every citation a lens emits is unauthorized. Raised by adversarial review of PR #284; splitting it out because it is a policy decision, not a defect in that diff.

The disagreement

The prompt tells the model to copy identifiers out of the pinned-reference documents:

Populate reference_expectations only from exact identifiers present in the supplied pinned-reference artifacts. Preserve one identifier per named expectation and carry the supplied identifier unchanged.

— .ultrafuzz/prompts/properties/0kn0t-lens.md:31-34

The gate only authorizes identifiers that appear in a structured ultrafuzz/reference-expectations@1 catalogue carrying operator-supplied provenance (packages/runtime/src/artifact-gates.ts, readLensSuppliedExpectationIds). That catalogue comes solely from the optional --reference-expectations CLI flag.

For most lenses the pinned reference is a Markdown corpus, not such a catalogue. .ultrafuzz/references.yml pins properties.0kn0t to 0kn0t/properties_framework with README.md, 01_PROPERTY_CLASSIFICATION_TAXONOMY.md and friends — documents, containing identifiers like LEND_ACC_01.

And when no dependency declares a catalogue, readLensSuppliedExpectationIds returns an empty id set with no diagnostic:

if (expectationPaths.length === 0) continue;   // artifact-gates.ts

So "absent from every supplied catalogue" is the universal case, not the exceptional one.

Observed consequence

Aave v4 run R44 (aave-v4-v0012-main-issue272-r44-invariant-only) supplied no catalogue — its config keys are app_name, benchmark_execution, braintrust, eval_reporting, ground_truth, image_name, models, node_timeout_seconds, run_id, schema_version, target. Node property-specification-0kn0t cited twelve identifiers and failed on all of them:

LEND_ACC_01 LEND_ACC_02 LEND_ACC_03 LEND_ACL_01 LEND_BND_01 LEND_BND_02
LEND_COL_01 LEND_DIR_01 LEND_DIR_02 LEND_DIR_03 LEND_ORC_01 LEND_STM_01

Review confirmed all twelve are physically present in that node's own pinned reference. The model did exactly what the prompt asked; the gate rejected it. Because property-specification-fanin depends on all eight lenses, this blocked the run.

The decision

Given a lens citing identifiers present in its supplied pinned-reference documents but absent from any structured catalogue, the gate should:

Strip them with a warning. Today's behaviour, deliberately pinned by property lens gate strips unsupported external labels without weakening benchmark authority and five other tests. reference_expectations then only ever carries verifiable provenance, at the cost of discarding a mapping the prompt asked for.
Keep them, marked unverifiable. Preserves the mapping; reference_expectations then contains content the gate could not authorize.
Supply a real catalogue for these runs, so the question does not arise.

Option 3 looks like the actual fix to me: the prompt and the gate should agree on what "supplied" means, and today they do not. Options 1 and 2 both paper over that.

Note the choice is currently made implicitly by identifier shape: isUnsupportedExternalReferenceLabel strips label-shaped identifiers and lets everything else fail the node. PR #284 makes that shape test consistent across - and _ separators — worth landing on its own merits, but it does not settle this.

Lens reference-expectation sanitizer only recognizes hyphenated labels, so underscore catalog IDs fail the node #283 / PR fix(runtime): sanitize underscore-separated reference labels #284 — the shape inconsistency that surfaced this.
fix(runtime): sanitize unsupported lens reference expectations #269 — introduced the sanitizer.
Rewritten published artifact strands runs: verified dependency digest mismatch fails fan-in forever with zero failed nodes #275 / PR fix(runtime): keep the verification marker consistent when sanitizing a property lens #279 — same function, marker consistency.

fix(runtime): stop lens expectation authority from turning on label s…

Correction: my earlier recommendation rested on a false premise.

I said option 3 was cheap because benchmarks/private-ground-truth/aave-v4-scfuzzbench/findings.yml is shape-compatible with the v1 reference-expectation schema. It is not. That file has top-level source / canaries / bugs keys — it is a scorer ground-truth bug list, not a reference-expectation catalogue. Verified directly against origin/main. So part (b) of my staged proposal is not a wiring exercise; it needs a catalogue that does not exist yet.

Also a path correction: artifact-gates.ts is at packages/runtime/src/artifact-gates.ts, not packages/artifacts/src/. The line numbers I cited resolve correctly against the runtime file.

Part (a) is implemented on local branch fix/invariant-evidence-gates (not pushed): the no-catalogue path now emits PROPERTY_REFERENCE_EXPECTATION_CATALOG_ABSENT as a warning instead of returning an empty id set in silence, and the doc comment at the deferral site records the decision — including why #240's verifyImplementationSelectionCoverage was inert on that path. Escalation is switchable via a new [invariants] reference_expectation_enforcement config key, defaulted to warn.

Part (b) remains open and now needs a scoping decision: either author a real reference-expectation catalogue for the benchmark targets, or accept that the gate stays warn-only on that path indefinitely. Note the new warning will appear on every property-lens node of every current benchmark run, since no run supplies a catalogue today.

fix(runtime): reject an unjustified empty invariant ledger and name a…

R52 reproduced this, and I measured the whole chain rather than inferring it. The fan-in node failed:

property-specification-fanin  failed  retry_count=0  category=artifact-contract
Canonical property "property-1" carries reference expectations not present in its source lens
artifacts: ["testPreviewDepositSameOrLessThanDeposit"]; ... property-2 ... property-3 ... property-4
... property-9 ["testConvertToSharesZeroAmountReturnsZero"] ... property-10 ... property-11 ...
property-12 ... property-125 ["testDepositSupportsEIP20ApproveTransferFromAssets"] ... property-126

Ten canonical properties. What the artifacts actually contain:

Every lens emitted zero reference expectations — 605 properties across all 8 lenses:

property-specification-0kn0t                :: props=69  carrying=0  distinct_exp=0
property-specification-a16z                 :: props=70  carrying=0  distinct_exp=0
property-specification-aviggiano            :: props=82  carrying=0  distinct_exp=0
property-specification-certora              :: props=101 carrying=0  distinct_exp=0
property-specification-crytic               :: props=62  carrying=0  distinct_exp=0
property-specification-josselin-feist       :: props=64  carrying=0  distinct_exp=0
property-specification-recon                :: props=82  carrying=0  distinct_exp=0
property-specification-runtime-verification :: props=75  carrying=0  distinct_exp=0
TOTAL distinct expectation ids across all lenses: 0

No reference artifact contains any of the disputed identifiers, and no reference-properties-*
artifact mentions reference_expectations at all. I probed all of them for the three sample ids above:
zero hits.

So the lenses behaved correctly. Their prompt says "Populate reference_expectations only from exact
identifiers present in the supplied pinned-reference artifacts"; no identifiers were supplied, so
emitting none was right.

The fan-in then emitted ten identifiers that appear nowhere in the run. They are real-world
a16z/ERC-4626 test names, which is exactly why they look plausible — the agent supplied them from model
knowledge, not from its inputs. Topology confirms it could not have read them from its inputs:

property-specification-<lens>  depends_on: [base-test-setup, reference-properties-<lens>]
property-specification-fanin   depends_on: [the 8 lens nodes]        # no reference artifact

The gate caught a genuine fabrication. That reframes this issue: the disagreement between prompt and
gate is real, but the resolution is not to loosen the gate — loosening it would admit fabricated
reference expectations into the canonical catalog, which is worse than failing. The fan-in prompt's
"or pinned-reference artifacts" clause permits sourcing the agent cannot actually perform, and that
clause is the thing that is wrong.

One further datum for whoever takes this: retry_count=0. The verifier is a separate task with
retries={0} depending on the agent task (workflow.tsx:4587-4600), so when a strict verifier rejects an
agentic node's content, the agent is never re-prompted with the diagnostic. A probabilistic
fabrication is therefore unconditionally fatal to the run. Whether that should stay true is a separate
policy question from this issue, but it is why one hallucinated identifier ends a multi-hour run.

Note this was not what blocked R52's report.md — that was #357. This node's failure only blocks
aggregate-test-files.

Part (b) scoping decision.

Part (a) shipped in #343: the no-catalogue path emits PROPERTY_REFERENCE_EXPECTATION_CATALOG_ABSENT instead of silently returning an empty identifier set, switchable through [invariants] reference_expectation_enforcement, defaulted to warn.

Part (b) is not the wiring exercise I originally described, and I want that on the record because I got it wrong twice. benchmarks/private-ground-truth/aave-v4-scfuzzbench/findings.yml is a scorer ground-truth bug list — top-level source / canaries / bugs — not a reference-expectation catalogue. There is no catalogue for any benchmark target, so there is nothing to thread through packages/evals and packages/modal.

Recommendation: leave enforcement at warn and do not author a catalogue for v0.1.0.

Reasons: the warning now makes the gap visible on every property-lens node, which was the actual defect; #240's verifyImplementationSelectionCoverage is inert on the benchmark path without a catalogue, so authoring one changes what benchmark runs assert and would invalidate comparison against the published 99 observations; and a catalogue authored to make a gate pass, rather than from real reference material, is worse than an honest warning.

If you disagree, the work is: define the v1 catalogue shape, author it per target, thread referenceExpectationsPath from evals/modal (today only cli/src/commands/run.ts:49 supplies it), observe one smoke lane at warn, then flip to enforce. That is a v0.2 shaped task, not a v0.1.0 blocker.

Tracked in #327.

Correcting my own comment above. I wrote:

So the lenses behaved correctly. Their prompt says "Populate reference_expectations only from
exact identifiers present in the supplied pinned-reference artifacts"; no identifiers were supplied,
so emitting none was right.

The second half is false. Identifiers were supplied. My probe searched for test[A-Z]…, and a16z
names its properties prop_previewDeposit / test_previewDeposit — lowercase after the separator — so
my regex could not match them. I concluded absence from a search that could not have found them.

reference-properties-0kn0t                  17   echidna_propertyName, echidna_supply_equals_balances, invariant_propertyName
reference-properties-a16z-erc4626           25   prop_asset, prop_totalAssets, prop_convertToShares
reference-properties-aviggiano               0
reference-properties-certora-sanity          0
reference-properties-certora-thinking        0
reference-properties-crytic                 88   testMode, testLimit, testAllContracts
reference-properties-montyly-rounding        0
reference-properties-recon                   8   invariant_number_never_zero, echidna_number_never_zero, checkError
reference-properties-runtime-verification   86   testFail_, testFailName, testDealIntendedAssetsToDummyUsers

Caveat I want to state rather than let the numbers oversell: several of these are not property names.
testMode / testLimit / testAllContracts are Echidna config keys, and echidna_propertyName /
invariant_propertyName are documentation placeholders. The unambiguous ones are a16z's
(prop_previewDeposit, prop_RT_deposit_withdraw, …, from the pinned ERC4626.prop.sol) and a good
part of runtime-verification's.

So the corrected picture is a three-stage failure, not the two-stage one I described:

The reference artifacts DO carry named identifiers — at minimum a16z's 25, pinned at
a16z/erc4626-tests@ac48546.
Every lens emitted zero reference_expectations anyway (0 of 605 properties). For the lenses
with real identifiers available — a16z above all — that is a lens failure, not correct behaviour as
I claimed.
The fan-in then invented testPreviewDepositSameOrLessThanDeposit and friends. Note these match
neither the run's inputs nor a16z's actual naming convention (prop_previewDeposit), which is
further evidence of fabrication rather than mis-copying.

What does not change: the gate was right to reject stage 3, and loosening it would admit fabricated
identifiers. What does change is that stage 2 is a real defect too — the lens prompt asks for exactly
this and the data is present — so "fix the fan-in prompt" alone would not make this feature work.

This is the second time I have had to correct a claim on this issue, both times because I asserted more
than my probe could support. The measurements above are reproducible from R52's volume.

Refining once more, and narrowing rather than widening this time — I nearly overcorrected in the
opposite direction.

I called stage 2 "a lens failure". I am withdrawing that as a firm claim. Here is the configuration
fact I had not checked, which reframes it:

The reference-expectations catalog is an optional user-supplied input that this benchmark does not
supply. Measured:

planRun takes input.referenceExpectationsPath and only calls
provisionReferenceExpectationOutput / provisionReferenceExpectationArtifacts /
materializeReferenceNodesForPlan when it is set (plan-run.ts:132-202).
The Aave v4 benchmark config sets no such key — zero matches for referenceExpectation in it.
Zero nodes in the shipped topology declare the ultrafuzz/reference-expectations@1 contract.
.ultrafuzz/references.yml declares zero expectations.
packages/artifacts/schema/reference-expectations.schema.json nevertheless exists, and
artifact-gates.ts enforces the canonical ⊆ lens rule regardless.

So the lens prompt's sanctioned path — "When a pinned-reference node declares a catalog with the
ultrafuzz/reference-expectations@1 contract, read its declared artifact and validate it…" — is
never satisfied in this configuration. The channel is empty by design, not by failure.

That leaves the fallback: "Populate reference_expectations only from exact identifiers present in the
supplied pinned-reference artifacts … when the supplied inputs contain no named expectation, leave the
field absent." The a16z artifact does contain prop_previewDeposit and friends — but those appear
inside a README and a pinned .sol listing, and whether a helper in a code listing is "a named
expectation" is genuinely ambiguous. Leaving the field absent is a defensible reading. I should not have
called it a failure on the evidence I had.

What remains a clear defect, unchanged through all three of my passes: the fan-in emitted ten
identifiers that exist nowhere in the run and do not even follow a16z's actual convention. The gate
caught it correctly.

What I would now say this issue is really about: an enforcement gate, a schema, and a prompt pathway
all ship for a feature whose only supply channel is an optional input the benchmark never sets. In that
configuration the sanctioned way to populate the field does not exist, and the only remaining way is for
an agent to scrape identifiers out of prose — which is exactly the situation where fabrication is most
likely. Whoever picks this up should decide whether the benchmark ought to supply a catalog, or whether
the gate should not be armed when no catalog is configured. I am deliberately not choosing that here.

Corrections logged: I claimed the lenses behaved correctly (wrong — my regex could not match prop_*),
then that they had failed (unsupported — the sanctioned channel is unconfigured). The measurements in
each comment stand; the conclusions I drew from them did not.

Evidence update from #531 and v0.0.16

The 2026-08-13 paired Aave run reproduces this authority disagreement at fan-in.

The two arms did not have identical fan-in diagnostics:

Treatment: 49 PROPERTY_REFERENCE_EXPECTATION_DROPPED-class diagnostics, in addition to the Markdown diagnostics tracked by Fan-in markdown parity reports a missing description for properties that render it #583.
Control: 30 reference-expectation diagnostics and no missing-description diagnostics.

A concrete treatment example is canonical property-1, which carries reference_expectations: ["LEND_ACC_03"]. Every contributing source-lens JSON row has reference_expectations absent. LEND_ACC_03 does appear in the pinned 0kn0t Markdown corpus, but the run supplied no structured ultrafuzz/reference-expectations@1 catalog and the fan-in node did not depend directly on one.

Therefore the gate was correct under its current authority rule: fan-in added an expectation ID not carried by any authenticated source-lens row. Loosening the gate would admit fabricated or prose-scraped provenance.

v0.0.16 does not resolve this part. It upgrades the schemas to property-lens@2 / reference-expectations@2, but the shipped topology's reference nodes still publish Markdown plus a reference manifest, not a structured expectation catalog. The prompt continues to discuss identifiers in "pinned-reference artifacts," while the semantic gate authorizes the structured channel.

Because v0.0.16 now fails descendants closed, leaving this unresolved will correctly stop campaigns rather than let them continue, but it also means a legitimate campaign can be lost to one fan-in fabrication.

Suggested resolution:

Choose one authority rule: structured catalogs only, or a typed derivation of named expectations from pinned references.
Make every lens and fan-in prompt describe that same rule without a prose-scraping fallback.
In the no-catalog configuration, require every lens and fan-in record to omit reference_expectations.
Preserve the current exact canonical-subset-of-lens gate.
Give fan-in producer-side feedback or a bounded correction attempt when this semantic gate fails.
Add the real Aave v4: release matched NoFuzz control and restart paired evaluation #531 fan-in artifacts as regression fixtures, including LEND_ACC_03 and the control's ERC-4626-style identifiers.

This is the root cause to fix before treating fail-closed fan-in as safe for expensive campaigns.

v0.1.0 bounded rule: a structured expectation catalog is the sole authority for reference_expectations. If no structured catalog is supplied, omit reference_expectations; do not infer or synthesize it from prompt prose. Keep the fix to that authority boundary rather than adding a new catalog feature.

Navigation Menu

Repository navigation

Lens reference-expectation authority: the prompt and the gate disagree about what "supplied" means #285

Summary

The disagreement

Observed consequence

The decision

Related

Evidence update from #531 and v0.0.16

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Lens reference-expectation authority: the prompt and the gate disagree about what "supplied" means #285

Description

Summary

The disagreement

Observed consequence

The decision

Related

Activity

aviggiano commented on Aug 7, 2026

aviggiano commented on Aug 7, 2026

aviggiano commented on Aug 7, 2026

aviggiano commented on Aug 7, 2026

aviggiano commented on Aug 7, 2026

aviggiano commented on Aug 14, 2026

Evidence update from #531 and v0.0.16

aviggiano commented on Aug 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions