performance: Expand hydrates full destination rows and copies them per edge — OOM / Arrow 2 GB offset overflow on wide destinations #703

Summary

An Expand hydrates full destination rows (every non-blob column) and then replicates them once per matched edge. When the destination has a wide column — e.g. a multi-KB text field — a traversal from a large source multiplies that column by the edge count. Memory explodes and the aligned string column overflows Arrow's i32 offsets. On a served node this OOM-kills the server.

This is the general, non-ranked form of #563 (which fixed the bm25/rrf + join path).

Environment

• 0.10.0 release (revision c02153de); still present at main (e16955b).
• Graph: 213,109 TranscriptSegment nodes; ~380 Artifact transcripts whose content: String is several KB each; edge TranscriptSegmentOf: TranscriptSegment -> Artifact.

Where

crates/omnigraph/src/exec/query.rs, expand_hydrate_and_align:

let dst_batch = hydrate_nodes(snapshot, catalog, dst_type, &unique_dst_list, dst_filters, params).await?;
// ...
let dst_prefixed = prefix_batch(&dst_batch, dst_var)?;
let aligned_dst = take_batch(&dst_prefixed, &dst_take)?;   // one copy of the full row per surviving pair
*wide = hconcat_batches(&expanded_wide, &aligned_dst)?;

hydrate_nodes takes no projection ("Load full node rows for a set of IDs"). The #564 comment in execute_node_scan documents the gap: "Only NodeScan bindings prune: hydrate_nodes (Expand dst hydration) and edge-property attach keep the full non-blob width."

Repro

query verify_segment_counts() {
  match {
    $s: TranscriptSegment
    $s transcriptSegmentOf $t
    $t: Artifact { source: "granola", kind: "transcript" }
  }
  return { $t.slug, count($s) as segments }
  order { segments desc }
  limit 10
}

The query only needs $t.slug and a count, but every transcript's full content is copied once per segment.

For comparison, scoping to one transcript by slug and counting count($s.slug) peaks at 0.26 GiB.

Ask

1. Project hydrate_nodes to the destination's needed columns (RETURN / order / filters) plus id and key columns — the same NeededColumns derivation execute_node_scan already uses.
2. When a destination is referenced only through an aggregate or not at all, hydrate id only.
3. Guard the aligned output so a single string buffer cannot exceed i32 offsets (LargeUtf8, or chunked output batches), returning a typed error rather than overflowing.

Related: #704 (bare-variable aggregates fail open to full width), #702 (unbounded data-session caches), #705 (full-width merge scans). Together these let a single agent-written read take down a served node; see also #565.