Goal planner inlines entire JSON records into the goal prompt, making each goal statement a multi-thousand-token wall #680

Summary

The goal planner is instructed to put entire JSON records into each dynamic goal's replacements map, so a 161-character goal prompt expands into a multi-thousand-token JSON wall — which the hunter prompt then introduces as "the first sentence above is the authoritative focused goal". This is the root cause of the goal-node timeouts previously attributed to model speed in #675.

Evidence

Measured across all 88 goals of one real default-profile run:

The template itself is exactly the crisp instruction you would want:

Your /goal is to find a vulnerability of type {{class:<class-id>}} using threat model {{<threat-id>}}.

Both placeholders resolve from the item's replacements map. For one representative class goal, that map held two entries of 10,051 and 4,744 characters — each a full raw JSON record: the vulnerability-class record with its guidance sub-object, and the threat-model entry with nested attack_surfaces, assets, actors, and every evidence array.

It is mandated by the planner prompt

.ultrafuzz/prompts/setup/goal-plan.md, under "## MDX replacement contract":

The replacement value for each threat key contains the full selected threat, assets, actors, surfaces, preconditions, invariants, assumptions, unknowns, and evidence. The class replacement contains the database record's focused hunter instructions and relevant examples. Never flatten a placeholder to a bare literal ID.

So the bloat is the specified behaviour, and the last sentence explicitly forbids the fix.

Nothing constrains the size

replacementsSchema in packages/artifacts/src/goal-plan.ts accepts any non-empty string:

const replacementsSchema = z
  .record(replacementKey, z.union([nonEmptyString, z.number().finite(), z.boolean()]))
  .refine((value) => Object.keys(value).length > 0, { message: "At least one item-scoped replacement is required" });

A 42,483-character "replacement" is contract-valid.

Impact

At maximum reasoning effort, every goal node must ingest and reason over several thousand tokens of dense nested JSON before it begins analysing any repository code, and it is told that blob is its goal statement.

In the run measured: 9 nodes were killed at exactly the 7200000ms group timeout, and only 3 of 77 class-goal nodes ever produced output. The goal phase is the most valuable phase of the audit and it completed 4% of its work.

This also explains why #675 looked like a timeout problem. Raising the timeout is the wrong fix: with a 14k-character goal statement, more time buys a longer failure and doubles the worst-case cost of a stuck node.

The fix is already the pattern used three lines away

goal-hunter.mdx references its other large inputs by path, not by value:

Canonical threat model:
{{artifact_path:threat-model}}/threat-model.json

Goal plan and node mapping:
{{artifact_path:goal-plan}}/goal-plan.json

Goal-plan artifact directory containing the selected records under
`vulnerability-db/selected`:
{{artifact_path:goal-plan}}

The agent already has every full record available on disk and is already told where. Inlining the same JSON into the goal sentence is redundant with the file's own established convention.

Proposed:

• Replacement values become a short human-readable label — the record's title, optionally with its id. The hunter reads the full record from the paths it is already given.
• Drop the "never flatten to a bare literal ID" instruction; keep the requirement that placeholders stay brace-wrapped in goal_prompt until the dynamic renderer resolves them, since validatePromptReplacements depends on it.
• Make replacementsSchema enforce a maximum length and reject values that are structurally JSON, so a planner that ignores the prompt cannot silently reintroduce the regression.

Acceptance criteria

• A goal node's rendered goal statement is on the order of hundreds of characters, not tens of thousands.
• The hunter can still locate the complete class record and threat-model entry for its assigned goal without guessing.
• An oversized or JSON-shaped replacement value fails contract validation with an actionable message.
• A regression test fails if a contract-valid goal plan can carry a replacement large enough to be a JSON record.