Add first-class dynamic runtime nodes to topology v2 #179

Summary

Add first-class data-driven node expansion to topology v2. An upstream node should be able to return a validated array and a static topology template should expand that array into independently scheduled runtime nodes.

This is foundational v0.1.0 work. Threat-model discovery may produce 100 or more threats, so fixed loops or a small fixed hunter pool is not an acceptable substitute.

Motivation

Topology v2 currently supports fixed loop counts and model-profile fanout, but it cannot create runtime nodes from earlier node output. Workflows that discover their work at runtime must either over-provision fixed attempts, spawn opaque sub-agents inside one durable node, or cap coverage prematurely.

Dynamic children must remain ordinary Ultrafuzz nodes: visible, durable, independently retryable, subject to existing concurrency limits, and attributable in final findings.

Proposed topology contract

The exact YAML spelling should follow existing topology conventions, but the contract should express the equivalent of:

- id: threat-goals
  needs:
    - goal-plan
  dynamic:
    from:
      node: goal-plan
      path: $.threat_goals
    key: id
    node_id: "dynamic.threat.{{ item.id }}"
  prompt: prompts/goal-hunter.mdx

Semantics:

• from.node must be a completed upstream dependency.
• from.path resolves to a schema-validated array, not executable topology supplied by an agent.
• key is required, stable, and unique within the expansion.
• The static node supplies the child model, prompt, permissions, retries, timeout, artifact contract, and other execution policy.
• Item fields are available to existing MDX/template replacement machinery.
• The group declaration is a downstream join: dependents become ready after all generated children reach the required terminal state.
• An empty expansion is valid.

Dynamic item context must work with MDX replacement syntax. A generated prompt must be able to retain and resolve placeholders such as:

Your /goal is to find any vulnerability affecting overdue liquidation using threat model threat {{liquidation:overdue}}.

Do not reduce this to a literal liquidation:overdue ID without the template replacement step.

Runtime identity and provenance

Generated node IDs must be deterministic, globally unique within the run, and useful to humans, for example:

dynamic.threat.liquidation.overdue
dynamic.class.liquidation.fixed-term-before-overdue

Reject invalid or duplicate IDs explicitly; never silently rename them.

Every child must carry runtime lineage:

• template/group node ID;
• source node ID and source output digest;
• expansion key and canonical item digest;
• generated node ID;
• ordinary attempt/retry identity from the existing attempt ledger.

Artifacts and candidate findings must automatically record their producer_node_id. This is the minimal primitive required for downstream finding provenance.

Durability and resume

Persist an expansion manifest at the first successful expansion. It must contain the exact source digest, canonical items, generated IDs, and ordering.

On resume or controller takeover:

• reuse the persisted expansion;
• do not silently re-evaluate changed planner output;
• do not duplicate completed children;
• preserve retries and terminal outcomes per child;
• require an explicit incompatible-workflow/change path if the expansion contract changed.

This should build on the existing artifact contracts, attempt ledger, checkpointing, wait-state, and effective-concurrency work rather than introducing parallel lifecycle state.

Scheduling and limits

• Dynamic children use the existing global concurrency scheduler.
• Concurrency limits only bound active work, not total expanded nodes.
• Add a configurable max_dynamic_nodes (or analogous) safety limit.
• Exceeding the safety limit is an explicit planning/runtime error; never silently truncate the array.
• The limit must comfortably allow hundreds of nodes and will later be configured by the audit_profile workstream.
• Status should distinguish generated, ready, active, terminal, failed, and skipped dynamic children.

v0.1.0 scope

In scope:

• one child node from each array item;
• one expansion level;
• runtime graph/status/CLI/dashboard visibility;
• ordinary retries, artifacts, accounting, and downstream joins;
• deterministic expansion and resume;
• MDX/template replacement using item context.

Out of scope for v0.1.0:

• agent-generated executable topology fragments;
• recursive/nested dynamic expansion;
• a multi-node dynamic subgraph per item;
• audit_profile or token-budget policy;
• vulnerability-specific behavior in the generic runtime.

Failure semantics

Use existing node failure policy wherever possible. The dynamic group must expose enough aggregate state for a topology to choose strict or best-effort behavior. A failed child must remain visible and must not be represented as an empty successful result.

Tests

Do not add fake-agent tests. Runtime behavior can be tested with deterministic structured producer nodes and ordinary task executors.

Cover at least:

• empty, one-item, 100-item, and safety-limit expansions;
• stable IDs and duplicate/invalid key rejection;
• MDX replacement from item context;
• global concurrency enforcement;
• child success, failure, retry, skip, and timeout;
• downstream join readiness;
• controller interruption and resume without duplicate work;
• changed source/template rejection after persisted expansion;
• first-class status, graph, events, usage, and artifact provenance;
• backward compatibility for topologies without dynamic nodes.

Acceptance criteria

• Topology schema and validation support one-level data-driven expansion.
• Dynamic children are first-class runtime nodes with stable IDs.
• Item data participates in the existing MDX replacement pipeline.
• Expansion manifests make resume deterministic and idempotent.
• Existing concurrency limits govern dynamic children.
• max_dynamic_nodes fails explicitly rather than truncating work.
• Downstream dependencies can join on a dynamic group.
• Artifacts/findings record the generated producer node ID.
• CLI/status/graph/dashboard surfaces show dynamic children and lineage.
• Deterministic tests cover large expansion, failure, and resume.
• Existing static and fixed-loop topologies remain compatible.

Implementation handoff

Suggested branch/worktree: feat/dynamic-topology-nodes, created from origin/main.

Likely touch points include topology types/validation, plan rendering, runtime scheduling and durable state, artifact manifests, status/event projections, and dashboard graph rendering. Final file ownership should be confirmed after tracing the current topology-v2 and checkpoint paths.