feat(io): chunked WDC Index/Arc Arrow bulk ingest tool

[DecisionNerd]

Problem

WDC Index/Arc graphs are too large for scalar add_node/add_edge loops. GraphForge’s supported path is offline convert → Arrow → chunked publish_bulk_nodes / publish_bulk_edges, but there is no checked-in tool that maps WDC integer IDs to UUIDv7 and streams batches without holding the full edge list in RAM.

Objective

Ship a focused offline ingest helper that reads Index/Arc (gzip TSV), assigns stable UUIDs, and publishes chunked bulk batches into a Parquet-backed project in exploratory mode. The tool must support capped samples (T1/T2) and be usable for larger Index/Arc tiers (T4, T5b, and Page shards at T6) when the #401 first-fail ladder reaches them.

Debt / regime

• Debt type: development and data
• Quality regime: A compute

Requirements

1. Accept paths to Index and Arc files (plain or .gz).
2. Stream nodes then edges in configurable Arrow batch sizes.
3. Map WDC integer node IDs to UUIDv7 with a deterministic or persisted mapping sufficient for reopen/replay of the same ingest.
4. Use public bulk APIs only (publish_bulk_nodes / publish_bulk_edges or thin Python helpers); Rust remains the engine.
5. Support a max-nodes / max-arcs limit for tier samples without loading the full 2012 PLD into memory.
6. Optionally warm CSR via index_adjacency after ingest and print disk/RSS summary hooks for spike issues.

Acceptance Criteria

• Tool ingests the WDC example (106/141) end-to-end into a project directory and reopens successfully.
• Tool ingests a bounded Index/Arc sample (documented tier) with peak RSS independent of full-file size (streaming).
• Deterministic operation UUIDs / idempotent retry behavior documented.
• Failure modes fail closed (bad TSV, missing endpoints, UUID conflicts) without partial silent success.

BDD Completion Scenarios

Scenario: Example ingest
Given fetched WDC example Index/Arc files
When the ingest tool runs against a fresh project directory
Then node and edge counts match 106 / 141
And GraphForge(path) reopen yields the same counts

Scenario: Bounded sample without full-file RAM
Given 2012 PLD Index/Arc on disk
When the tool runs with a documented node/arc cap for an early tier
Then it completes without reading the entire arc file into a single in-memory list
And Cypher MATCH ()-[r]->() RETURN count(r) equals the capped arc count

Implementation Notes

• Blocked by docs/retrieval docs(datasets): WDC Hyperlink Graph tiered scale guide and retrieval #399 for cache layout and format contracts.
• Do not implement BVGraph/Pajek parsers here; accept Index/Arc only (2014 PLD/Host edges require a prior conversion step owned elsewhere or documented externally for test(scale): WDC T0→T6 first-fail escalation spike #401).

Observability

• Log batch sizes, rows published, elapsed, and peak RSS if available. No UUID property dumps.

Security And Privacy

• Local filesystem only; no network in the ingest tool.

Testing

• Unit tests for TSV parsing / ID mapping on the example files.
• Integration test gated or marked heavy for sample ingest if CI-appropriate; otherwise scripted manual evidence for larger tiers.

Documentation

• Update docs/guide/datasets/wdc-hyperlink-graph.md ingest section with exact commands once the tool lands.

Non-Goals

• BVGraph/Pajek native parsers (conversion stays out of tree for this issue)
• graphforge.datasets catalog API
• Analyst-verb correctness beyond reopen + LIMIT Cypher smoke
• Owning the test(scale): WDC T0→T6 first-fail escalation spike #401 ladder run itself (this tool enables tiers; spike owns first-fail execution)

Related Issues

• Blocks / follows docs(datasets): WDC Hyperlink Graph tiered scale guide and retrieval #399
• Feeds escalation spike test(scale): WDC T0→T6 first-fail escalation spike #401 and CSR measurement test(performance): measure CSR build and reopen cost on WDC tiers #402
• Enables Host T5b / Page T6 Index/Arc paths when test(scale): WDC T0→T6 first-fail escalation spike #401 reaches them (scale(wdc): Host aggregation ladder step (T5) #403 / scale(wdc): Page aggregation ladder step (T6) #404)
• Uses construction contracts in docs/guide/graph-construction.md

Open Questions

• None

[DecisionNerd]

Blocked by #399 for cache layout/format contracts. Feeds #401 and #402.

[coderabbitai]

🔗 Related PRs

#217 - feat(api): add embedded write modes [merged]
#351 - fix(bindings): repair Node bulk API contract [merged]
#380 - test(exec): prove write execution and dispatch boundaries [merged]
#382 - test(storage-api): close remaining coverage floors for #369 [merged]
#405 - docs(datasets): WDC Hyperlink Graph tiered scale guide [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[DecisionNerd]

Not planned — WDC scale track cancelled

GraphForge scale testing pivots off WDC Hyperlink Graphs.

New track (docs in this repo; execution external):

• docs(scale): Graph500 × GSI size-ladder specification #408 — Graph500 × GSI size-ladder specification
• docs(scale): LDBC full-suite benchmark specification #409 — LDBC full-suite benchmark specification
• docs(scale): external Graph500 + LDBC harness contract #410 — external Graph500 + LDBC harness contract

WDC T0–T6 is no longer the execution ladder. Optional M4 external scale validation retargets to Graph500 + LDBC (still non-blocking for #335).

[DecisionNerd]

Closing as not planned: WDC scale track cancelled in favor of #408 / #409 / #410.