Repository navigation
Goal-hunter record-lookup instructions name fields that do not exist for threat goals and a manifest path that never resolves #686
Description
Summary
The goal-hunter prompt's record-lookup instructions name fields that do not exist for one of the two goal kinds, point at a manifest field that resolves to a non-existent path, and omit a join the old inlined payload had pre-done. Measured against a real 88-goal plan, the instructions are wrong for all 88 goals in at least one respect.
These surfaced when full records stopped being inlined into the goal sentence and the hunter began having to locate them itself.
Defect 1 — 11 of 88 hunters are told to read fields that do not exist
The instructions name class_id, threat_ids, attack_surface_ids, selected_record, and selection_rationale as "the definition of your scope", and tell the agent to read selected_record.path.
Threat-goal entries have no class_id and no selected_record at all. What a threat goal carries is class_ids — plural, between 1 and 29 entries — which the instructions never mention. Those 11 hunters have to guess.
Every threat-goal class_ids value does resolve into the plan's selected_class_records, with zero misses, so the information is available; only the instruction is wrong. The fix is to distinguish the two goal kinds and name the plural key, stating that a threat goal selects among the selected class records by those IDs.
Defect 2 — a 77-of-77 foot-gun that yields ENOENT every time
The instructions say the record is "indexed by vulnerability-db-manifest.json". Each selected_records[] entry in that manifest carries both:
path— database-relativeartifact_path— goal-plan-artifact-relative
They differ for all 77 entries. Only artifact_path resolves under the artifact directory, and it equals the plan's own selected_record.path for all 77. The manifest additionally has a 141-entry records array whose files were never copied into the artifact — only vulnerability-db/selected/ exists on disk.
An agent that follows "the index" and takes the obvious field name path gets ENOENT every time. Either name selected_records[].artifact_path explicitly, or drop the manifest clause entirely, since selected_record.path is already correct and sufficient.
Defect 3 — a missing dereference the old payload had pre-done
The instructions tell the agent to read the threat entry's "assets, actors, attack surfaces, preconditions, invariants, assumptions, unknowns". A threat entry carries only *_ids for six of those; the objects live in the threat model's top-level assets / actors / attack_surfaces / invariants / assumptions / unknowns arrays.
The previously inlined replacement value carried these joins resolved — it held three denormalized keys beyond the 14 present in the file — so removing the inlining removed a join the agent is now expected to perform but is not told to.
Information loss — 18 coverage-gap goals
Coverage-gap class goals previously carried a roughly 360-character planner-synthesized description that exists in no file, so shortening the replacement genuinely loses it.
Their coverage_gaps ID list does resolve into the threat model's coverage_gaps section, but the instructions say a coverage-gap goal's scope is "the class record plus selection_rationale" and never point at that section. selection_rationale measures 83 / 154 / 294 characters (min / median / max) and does not substitute. Coverage-gap hunters should be pointed at the threat model's coverage_gaps section, joined by the entry's coverage_gaps IDs.
Verified as correct, for contrast
{{item.node_id}}does render:isDynamicItemTemplateVariableadmitsitem.*anddynamicItemVariablesflattensnode_idas a top-level scalar. It is schema-required and unique across all 88 entries.- All 77 class-goal
selected_record.pathvalues resolve to files that exist under the goal-plan artifact directory. - Every threat ID referenced by any goal exists in the threat model.
So step 1 and the class-goal happy path are guess-free once defects 2 and 3 are fixed.
Acceptance criteria
- The instructions are correct for both goal kinds, naming only fields that exist on each.
- Following the instructions literally locates the selected class record on the first attempt for all class goals.
- The threat-entry ID-to-object dereference is stated explicitly.
- Coverage-gap goals are pointed at a real, readable source for their scope.
- No information the hunter needs remains unrecoverable from a file it is told to read.
Activity
aviggiano commentedon Aug 20, 2026
Addressed by PR #690 (
fix/goal-prompt-and-coverage-integrity), which lists this as itsCloses. Full fix — all four of the reported wrong facts are corrected in.ultrafuzz/prompts/strategies/goal-hunter.mdx.This issue only exists because #680 stopped inlining records into the goal prompt: once the records are referenced by path, the hunter has to locate them, and the first version of those instructions was wrong for all 88 goals in at least one respect.
class_idand oneselected_record, while a threat goal carriesclass_idsplural, resolved against the plan'sselected_class_records, and explicitly has noclass_id, noselected_record, and nocoverage_gap. That was 11 hunters reading fields that do not exist.pathnever resolved. L34-36 now points atselected_record.pathor the matching entry under{{artifact_path:goal-plan}}, where onlyvulnerability-db/selectedresolves. The manifest's ownpathis database-relative, so following it was an ENOENT for all 77 class goals.*_idsdereference is now stated. The inlined payload used to pre-do this expansion into the document's top-level arrays; with the payload gone, the step had to be written down.coverage_gapssection and theirclass_id's element. 18 goals had lost their only description.Anchor phrases for these facts are pinned in
packages/prompts/test/semantic-anchors.test.ts, so a later prose trim cannot quietly drop them — which matters, because a trim that deletes one of these reintroduces this bug.fix(prompts): trim the prose these fixes added, keeping only what cha…
aviggiano commentedon Aug 20, 2026
Shipped in #690, merged to
release/v0.1.0(f1b8c223).All four defects fixed and verified against the real 88-goal plan, not just read in the diff:
class_idorselected_record, which they do not have; it namesclass_ids(plural) and states they carry noclass_id,selected_record, orcoverage_gap.selected_record.path. Verified on disk: all 141 manifestpathvalues are ENOENT under the artifact directory, and all 77 instructed paths exist.*_idsdereference — now explicitly instructed against the threat model's top-level arrays. Verified: 98 of 98 references resolve, 0 unresolved.coverage_gapssection, recovering the structured content the removed synthesizeddescriptioncarried.Independent check found no information still lost: the on-disk records are a superset of what had been inlined.
Closed manually: default branch is
main.