spike(storage): evaluate spill, buffering, and memory-pool reuse under construction pressure (#1504)

[DecisionNerd]

Purpose

Evaluate external-sort/spill and memory-pool facilities under forced memory pressure for construction, including coexistence with GraphForge receipts and recovery.

Prerequisite

Blocked by the #1504 inventory/protocol child. Use that protocol.

Scope

• Prototype applicable external-sort/spill and memory-pool facilities under forced memory pressure, not merely in-memory inputs.
• Account for input/output batches, sort state, queues, completed results awaiting consumption, compression and writer buffers, adapters, spill files, and unregistered allocations. Distinguish reservation bounds, observed RSS, and page cache.
• Evaluate transient-file ownership, cleanup, disk limits, corruption detection, descriptor authentication, crash/restart behavior, and coexistence with receipts/recovery. A library spill file is not automatically a durable checkpoint.
• Measure bytes copied/read/written, spill amplification, file counts, durability barriers, simultaneous scratch, and refusal behavior.

Acceptance criteria

• Runnable spill/memory-pool replacement or hybrid experiment exists under forced pressure, or a specific independently reviewed incompatibility proof for each inapplicable candidate.
• Evidence distinguishes library spill from GraphForge durable checkpoints/recovery authority.
• Failure cases include malformed/truncated/corrupt spills, disk exhaustion, and admission refusal where applicable.
• Fair repeated measurements and maintenance/correctness tradeoff assessment are recorded for this mechanism.
• Results stay bounded; no claim that encode-only evidence settles spill.

Non-goals

Production migration; weakening authentication for benchmark convenience; closing #1504 alone.

Parent

Native sub-issue and blocker of #1504. Blocked by inventory/protocol child.

[DecisionNerd]

Evidence relocated from the docs tree (#1625). The 3 file(s) below were committed under docs/development/ as measurement evidence for this issue. They are removed from the tree by #1628 (the #1625 pull request); the repository keeps method and content digests, results live here. Every file remains available at its permalink on commit 29a7b34ebe44, and its SHA-256 is recorded next to it.

Directory permalinks:

• https://github.com/CurateLabs/graphforge/tree/29a7b34ebe441a85ffb9274164d58aaeeb68dc8a/docs/development/evidence/spill-memory-pool-1507

Files (3 total, 3,204 lines)

• docs/development/evidence/spill-memory-pool-1507.json sha256:805faa18046fd830edf72fae1e0040bc44a51a1212ccf0acc9ea304472203690
• docs/development/evidence/spill-memory-pool-1507.md sha256:d7873c0930c6b846454cc6f85b6bcd928bb9496210a61c543a2b0f903bf9f489
• docs/development/evidence/spill-memory-pool-1507/measure.py sha256:1e6bc3114e202fc5e020525820b6df648e39b9a7825af376eb044d655d8cf7e7