Skip to content

Goal-hunter record-lookup instructions name fields that do not exist for threat goals and a manifest path that never resolves #686

Closed

Description

@aviggiano

Summary

The goal-hunter prompt's record-lookup instructions name fields that do not exist for one of the two goal kinds, point at a manifest field that resolves to a non-existent path, and omit a join the old inlined payload had pre-done. Measured against a real 88-goal plan, the instructions are wrong for all 88 goals in at least one respect.

These surfaced when full records stopped being inlined into the goal sentence and the hunter began having to locate them itself.

Defect 1 — 11 of 88 hunters are told to read fields that do not exist

The instructions name class_id, threat_ids, attack_surface_ids, selected_record, and selection_rationale as "the definition of your scope", and tell the agent to read selected_record.path.

Threat-goal entries have no class_id and no selected_record at all. What a threat goal carries is class_ids — plural, between 1 and 29 entries — which the instructions never mention. Those 11 hunters have to guess.

Every threat-goal class_ids value does resolve into the plan's selected_class_records, with zero misses, so the information is available; only the instruction is wrong. The fix is to distinguish the two goal kinds and name the plural key, stating that a threat goal selects among the selected class records by those IDs.

Defect 2 — a 77-of-77 foot-gun that yields ENOENT every time

The instructions say the record is "indexed by vulnerability-db-manifest.json". Each selected_records[] entry in that manifest carries both:

  • path — database-relative
  • artifact_path — goal-plan-artifact-relative

They differ for all 77 entries. Only artifact_path resolves under the artifact directory, and it equals the plan's own selected_record.path for all 77. The manifest additionally has a 141-entry records array whose files were never copied into the artifact — only vulnerability-db/selected/ exists on disk.

An agent that follows "the index" and takes the obvious field name path gets ENOENT every time. Either name selected_records[].artifact_path explicitly, or drop the manifest clause entirely, since selected_record.path is already correct and sufficient.

Defect 3 — a missing dereference the old payload had pre-done

The instructions tell the agent to read the threat entry's "assets, actors, attack surfaces, preconditions, invariants, assumptions, unknowns". A threat entry carries only *_ids for six of those; the objects live in the threat model's top-level assets / actors / attack_surfaces / invariants / assumptions / unknowns arrays.

The previously inlined replacement value carried these joins resolved — it held three denormalized keys beyond the 14 present in the file — so removing the inlining removed a join the agent is now expected to perform but is not told to.

Information loss — 18 coverage-gap goals

Coverage-gap class goals previously carried a roughly 360-character planner-synthesized description that exists in no file, so shortening the replacement genuinely loses it.

Their coverage_gaps ID list does resolve into the threat model's coverage_gaps section, but the instructions say a coverage-gap goal's scope is "the class record plus selection_rationale" and never point at that section. selection_rationale measures 83 / 154 / 294 characters (min / median / max) and does not substitute. Coverage-gap hunters should be pointed at the threat model's coverage_gaps section, joined by the entry's coverage_gaps IDs.

Verified as correct, for contrast

  • {{item.node_id}} does render: isDynamicItemTemplateVariable admits item.* and dynamicItemVariables flattens node_id as a top-level scalar. It is schema-required and unique across all 88 entries.
  • All 77 class-goal selected_record.path values resolve to files that exist under the goal-plan artifact directory.
  • Every threat ID referenced by any goal exists in the threat model.

So step 1 and the class-goal happy path are guess-free once defects 2 and 3 are fixed.

Acceptance criteria

  • The instructions are correct for both goal kinds, naming only fields that exist on each.
  • Following the instructions literally locates the selected class record on the first attempt for all class goals.
  • The threat-entry ID-to-object dereference is stated explicitly.
  • Coverage-gap goals are pointed at a real, readable source for their scope.
  • No information the hunter needs remains unrecoverable from a file it is told to read.

Activity

  1. aviggiano commented on Aug 20, 2026

    @aviggiano
    CollaboratorAuthor

    Addressed by PR #690 (fix/goal-prompt-and-coverage-integrity), which lists this as its Closes. Full fix — all four of the reported wrong facts are corrected in .ultrafuzz/prompts/strategies/goal-hunter.mdx.

    This issue only exists because #680 stopped inlining records into the goal prompt: once the records are referenced by path, the hunter has to locate them, and the first version of those instructions was wrong for all 88 goals in at least one respect.

    • Threat goals were told to read absent fields. L29-33 now states that a class goal carries one class_id and one selected_record, while a threat goal carries class_ids plural, resolved against the plan's selected_class_records, and explicitly has no class_id, no selected_record, and no coverage_gap. That was 11 hunters reading fields that do not exist.
    • The manifest path never resolved. L34-36 now points at selected_record.path or the matching entry under {{artifact_path:goal-plan}}, where only vulnerability-db/selected resolves. The manifest's own path is database-relative, so following it was an ENOENT for all 77 class goals.
    • The threat-entry *_ids dereference is now stated. The inlined payload used to pre-do this expansion into the document's top-level arrays; with the payload gone, the step had to be written down.
    • Coverage-gap goals got their description back. L47 points them at the threat model's coverage_gaps section and their class_id's element. 18 goals had lost their only description.

    Anchor phrases for these facts are pinned in packages/prompts/test/semantic-anchors.test.ts, so a later prose trim cannot quietly drop them — which matters, because a trim that deletes one of these reintroduces this bug.

  2. aviggiano commented on Aug 20, 2026

    @aviggiano
    CollaboratorAuthor

    Shipped in #690, merged to release/v0.1.0 (f1b8c223).

    All four defects fixed and verified against the real 88-goal plan, not just read in the diff:

    1. Threat goals (11 of 88) — the prompt no longer tells them to read class_id or selected_record, which they do not have; it names class_ids (plural) and states they carry no class_id, selected_record, or coverage_gap.
    2. The manifest ENOENT foot-gun (77 of 77) — resolved to the plan's own selected_record.path. Verified on disk: all 141 manifest path values are ENOENT under the artifact directory, and all 77 instructed paths exist.
    3. The *_ids dereference — now explicitly instructed against the threat model's top-level arrays. Verified: 98 of 98 references resolve, 0 unresolved.
    4. Coverage-gap goals (18) — pointed at the threat model's coverage_gaps section, recovering the structured content the removed synthesized description carried.

    Independent check found no information still lost: the on-disk records are a superset of what had been inlined.

    Closed manually: default branch is main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions