Skip to content
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/0kn0t-lens.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/aviggiano-lens.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/certora-thinking-lens.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/josselin-feist-lens.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/property-specification-a16z.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/property-specification-crytic.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
18 changes: 8 additions & 10 deletions .ultrafuzz/prompts/properties/property-specification-fanin.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,14 @@ source row, preserve:
table, such as `property-specification-certora`;
- `source_property_id`: the prefixed property ID copied exactly from that
source table.
- `reference_expectations`: copy the exact identifiers present on contributing
source rows when those identifiers are supplied by the reference artifacts or
pinned-reference artifacts. Merge equivalent rows by taking the union
of supplied expectation identifiers, preserving each identifier unchanged.
When source rows carry no supplied expectation identifier, leave the field
absent.

When a pinned-reference node declares a catalog with
`ultrafuzz/reference-expectations@2`, read its declared artifact and validate it
with `{{schema_path}}/reference-expectations.schema.json`.
- `reference_expectations`: preserve only exact identifiers already carried by
contributing source rows; the lens gate has authorized those identifiers
against a structured `ultrafuzz/reference-expectations@2` catalog. Merge
equivalent rows by taking their union, preserving each identifier unchanged.
Never add identifiers from Markdown, prose, code listings, pinned-reference
artifacts, or model knowledge. When no contributing source row carries an
authorized identifier, omit the field entirely; an empty array is not
omission.

When several source rows describe one equivalent property, emit one canonical
property with every distinct contributing source in `sources`. Never keep only
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
13 changes: 6 additions & 7 deletions .ultrafuzz/prompts/properties/recon-lens.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,12 @@ You must produce a typed JSON catalog and a matching Markdown companion. Use
the catalog, and set every property priority to `high`, `medium`, or `low`.
The JSON catalog is the machine-readable source of truth.

Populate `reference_expectations` only from exact identifiers present in the
supplied pinned-reference artifacts. Preserve one identifier per named
expectation and carry the supplied identifier unchanged. When the supplied
inputs contain no named expectation, do not invent expectation IDs; use the
pinned property-lens schema's no-expectation representation.

When a pinned-reference node declares a catalog with the `ultrafuzz/reference-expectations@2` contract, read its declared artifact and validate it with `{{schema_path}}/reference-expectations.schema.json` before copying identifiers.
A structured catalog declared with the `ultrafuzz/reference-expectations@2`
contract is the sole authority for `reference_expectations`. Validate that
catalog with `{{schema_path}}/reference-expectations.schema.json`, and copy only
its exact identifiers. Never derive identifiers from Markdown, prose, code
listings, or model knowledge. When no structured catalog is supplied, omit
`reference_expectations` entirely; an empty array is not omission.

## Target-derived invariant extraction

Expand Down
20 changes: 14 additions & 6 deletions packages/prompts/test/semantic-anchors.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,22 @@ describe("prompt semantic anchors", () => {
);
expect(propertyPrompts).toHaveLength(8);
for (const asset of propertyPrompts) {
expect(asset.markdown, asset.relativePath).toContain("exact identifiers present in the");
expect(asset.markdown, asset.relativePath).toContain("{{schema_path}}/reference-expectations.schema.json");
expect(asset.markdown, asset.relativePath).toContain("do not invent expectation IDs");
expect(asset.markdown, asset.relativePath).toContain("schema's no-expectation representation");
expect(asset.markdown, asset.relativePath).not.toContain("scfuzzbench:aave-v4:iSpoke_supply");
const markdown = asset.markdown.replace(/\s+/gu, " ");
expect(markdown, asset.relativePath).toContain("is the sole authority for `reference_expectations`");
expect(markdown, asset.relativePath).toContain("{{schema_path}}/reference-expectations.schema.json");
expect(markdown, asset.relativePath).toContain(
"Never derive identifiers from Markdown, prose, code listings, or model knowledge"
);
expect(markdown, asset.relativePath).toContain("omit `reference_expectations` entirely");
expect(markdown, asset.relativePath).toContain("an empty array is not omission");
expect(markdown, asset.relativePath).not.toContain("scfuzzbench:aave-v4:iSpoke_supply");
}
const fanin = prompt("properties/property-specification-fanin.md");
expect(fanin).toContain("{{schema_path}}/reference-expectations.schema.json");
expect(fanin).toContain("preserve only exact identifiers already carried by");
expect(fanin).toContain("the lens gate has authorized those identifiers");
expect(fanin).toContain("omit the field entirely; an empty array is not");
expect(fanin).not.toContain("identifiers are supplied by the reference artifacts");
expect(fanin).not.toContain("read its declared artifact");
expect(fanin).not.toContain("scfuzzbench:aave-v4:iSpoke_supply");
expect(prompt("strategies/invariants/implement-properties.md")).not.toContain("scfuzzbench:aave-v4:iSpoke_supply");
});
Expand Down
51 changes: 40 additions & 11 deletions packages/runtime/src/artifact-gates.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1301,7 +1301,6 @@ function verifyLensReferenceExpectationPreservation(

for (const [propertyIndex, property] of catalog.properties.entries()) {
const expectedReferenceIds = property.reference_expectations ?? [];
if (expectedReferenceIds.length === 0) continue;
const sourcePairs = new Set(
property.sources.map((source) => `${source.source_node_id}\u0000${source.source_property_id}`)
);
Expand All @@ -1310,6 +1309,20 @@ function verifyLensReferenceExpectationPreservation(
.filter((row) => sourcePairs.has(`${row.sourceNodeId}\u0000${row.propertyId}`))
.flatMap((row) => row.expectationIds)
);
if (
property.reference_expectations !== undefined &&
expectedReferenceIds.length === 0 &&
lensReferenceIds.size === 0
) {
diagnostics.push({
code: "PROPERTY_REFERENCE_EXPECTATION_OMISSION_REQUIRED",
message: `Canonical property ${JSON.stringify(property.id)} must omit reference_expectations when no authorized identifier is carried from its source lens artifacts`,
severity: "error",
source: "property-fanin",
Comment thread
aviggiano marked this conversation as resolved.
path: `${catalogPath}#$.properties[${propertyIndex}].reference_expectations`
});
}
if (expectedReferenceIds.length === 0) continue;
const missingReferenceIds = expectedReferenceIds.filter((expectationId) => !lensReferenceIds.has(expectationId));
if (missingReferenceIds.length > 0) {
diagnostics.push({
Expand Down Expand Up @@ -5809,9 +5822,9 @@ function verifyImplementationPropertyReferences(

/**
* A reference expectation is provenance, not free-form model metadata. A lens
* may copy an ID only when the exact token is present in a declared pinned
* reference input artifact. This keeps illustrative prompt text from becoming
* an apparently authorized benchmark mapping.
* may copy an ID only when the exact token is present in a declared structured
* reference expectation catalog. Without such a catalog the field must be
* absent, keeping prose and code listings from becoming apparent authority.
*/
function verifyLensReferenceExpectationAuthority(
layout: RunLayout,
Expand Down Expand Up @@ -5883,6 +5896,25 @@ function verifyLensReferenceExpectationAuthority(
const suppliedExpectationIds = supplied.ids;
const diagnostics: RuntimeDiagnostic[] = [...supplied.diagnostics];
for (const [propertyIndex, property] of lens.value.properties.entries()) {
if (property.reference_expectations !== undefined && property.reference_expectations.length === 0) {
diagnostics.push({
code: "PROPERTY_REFERENCE_EXPECTATION_OMISSION_REQUIRED",
message: `Property lens ${JSON.stringify(property.id)} must omit reference_expectations when it carries no authorized identifiers`,
severity: "error",
source: "property-provenance",
path: `${lensPath}#$.properties[${propertyIndex}].reference_expectations`
});
}
if (!supplied.catalogSupplied && property.reference_expectations !== undefined) {
Comment thread
greptile-apps[bot] marked this conversation as resolved.
diagnostics.push({
code: "PROPERTY_REFERENCE_EXPECTATION_UNAUTHORIZED",
message: `Property lens ${JSON.stringify(property.id)} must omit reference_expectations because no structured catalog was supplied`,
severity: "error",
source: "property-provenance",
path: `${lensPath}#$.properties[${propertyIndex}].reference_expectations`
});
continue;
}
for (const [expectationIndex, expectationId] of (property.reference_expectations ?? []).entries()) {
if (suppliedExpectationIds.has(expectationId)) continue;
diagnostics.push({
Expand Down Expand Up @@ -5947,16 +5979,13 @@ function readLensSuppliedExpectationIds(
}
}
if (!catalogSupplied) {
// Issue #285, part (a): say so. This path used to return an empty id set in silence, which made
// EVERY citation a lens emitted unauthorized and every reference-expectation check downstream
// inert — including `verifyImplementationSelectionCoverage`, whose expectation-forced selection
// can never fire when no property is allowed to keep an expectation. On a benchmark run with no
// `--reference-expectations` catalogue that is the normal case, not the exceptional one, so the
// gate was quietly doing nothing exactly where it was supposed to be doing the most.
// Issue #285: structured catalogs are the sole authority. Most benchmark
// runs intentionally supply none, so record that expectation-backed
// coverage is unavailable while requiring every lens row to omit the field.
diagnostics.push({
code: "PROPERTY_REFERENCE_EXPECTATION_CATALOG_ABSENT",
message:
"No pinned-reference dependency supplies a reference expectation catalog, so every lens reference expectation is unauthorized and the provenance gate is inert",
"No pinned-reference dependency supplies a structured reference expectation catalog, so lens artifacts must omit reference_expectations",
severity: "warning",
source: "property-provenance",
path: `state.nodes.${node.id}.depends_on`
Expand Down
Loading