test(scale): WDC T0→T6 first-fail escalation spike

[DecisionNerd]

Boundary (harness location)

Ladder execution (orchestration, runners, first-fail evidence packaging) belongs in an external scale harness repository, not GraphForge core CI or Makefile. GraphForge docs define the spec/contract (GSI, WDC guide); thin reference fetch helpers may be called from the harness.

---

Problem

M4 has LiveJournal LIMIT evidence (~4M / ~35M) but no accepted external hyperlink-graph escalation ladder. Without a bounded WDC spike that runs the full ordered ladder until first fail, scale claims either stop artificially at PLD or jump to unsupported Host/Page ambitions without recorded stop evidence.

Objective

Run the WDC guide ladder T0 → T6 (including Host/Page year sub-steps) under the documented first-fail policy: each tier’s acceptance criteria must pass before the next starts; on the first failed tier, stop and record which tier failed and why. Do not skip tiers. Do not proceed past a red tier.

Debt / regime

• Debt type: test/proof
• Quality regime: A compute

Requirements

1. Follow the exact tier order and exit/acceptance criteria in docs/guide/datasets/wdc-hyperlink-graph.md:
   • T0 example 106/141
   • T1 ~100K 2012 PLD sample
   • T2 ~1M / ~5–15M 2012 PLD shard
   • T3 full 2014 PLD (~13M / ~56M)
   • T4 full 2012 PLD (~43M / ~623M)
   • T5a 2014 Host (~22M / ~123M) then T5b 2012 Host (~101M / ~2043M)
   • T6a 2014 Page (~1727M / ~64422M) then T6b 2012 Page (~3563M / ~128736M)
2. Use retrieved artifacts (docs(datasets): WDC Hyperlink Graph tiered scale guide and retrieval #399) and the chunked ingest tool (feat(io): chunked WDC Index/Arc Arrow bulk ingest tool #400) when available; document WebGraph→Arc conversion when required (2014 PLD/Host).
3. Record for each attempted tier: node/edge counts, project bytes, peak RSS, CSR build time if warmed, one-hop and two-hop LIMIT 1000 shape metrics aligned with test(performance): establish the M4 embedded baseline and entry gate #334 evidence fields where applicable, and green/red disposition.
4. First-fail: stop at the first tier that violates acceptance or documented resource stop conditions; do not attempt any larger tier (including Host/Page after a PLD fail, or T5b after T5a fail, or T6b after T6a fail).
5. Success of a tier is required before attempting the next. A first-fail stop at or before Host/Page is a valid complete outcome for this issue.

Acceptance Criteria

• Ladder executed in order from T0 without skipping; evidence attached for every attempted tier through the first red (or through T6b if all green).
• Tier T0 and at least one Index/Arc sample tier (T1 or T2) produce green LIMIT + reopen evidence attached to this issue (unless even T0 fails — then record T0 red and stop).
• Written first-fail report: last green tier, first red tier (if any), failure class, and why larger tiers were not run.
• Host (T5) and Page (T6) are attempted only if all prior tiers are green; otherwise explicitly recorded as not-run due to first-fail (not as “deferred forever”).
• Evidence does not claim analyst-verb or full-graph PageRank success unless separately measured and in posture.

BDD Completion Scenarios

Scenario: Early-tier green
Given a project ingested from the WDC example or sample tier
When fixed-hop Cypher with LIMIT 1000 runs after CSR warm
Then results return without full edge scans on an adjacency hit (or document adjacency miss honestly)
And reopen preserves counts

Scenario: First-fail stop honored
Given a tier whose acceptance criteria fail (disk/RSS projection, ingest, CSR/reopen, or LIMIT miss)
When the spike evaluates go/no-go for the next tier
Then it records a stop at that tier without attempting any larger tier (including Host/Page)

Scenario: Host/Page only after prior green
Given T0–T4 are green
When the spike continues
Then T5a is attempted before T5b, and T6 only after T5b green
And failure at any sub-step stops further escalation

Implementation Notes

• Blocked by docs(datasets): WDC Hyperlink Graph tiered scale guide and retrieval #399 (guide + retrieval). Prefer blocked by ingest tool feat(io): chunked WDC Index/Arc Arrow bulk ingest tool #400 once available.
• Does not expand M4 close criteria; optional evidence for test(performance): reconcile final M4 exit evidence and scale claims #345 if results are accepted.
• Owns the ladder-run evidence; scale(wdc): Host aggregation ladder step (T5) #403/scale(wdc): Page aggregation ladder step (T6) #404 track Host/Page step ownership/prep but do not authorize skipping the first-fail order.

Observability

• Use test(performance): establish the M4 embedded baseline and entry gate #334 structural counters / evidence format fields where available.

Security And Privacy

• No publishing of corpus contents; evidence is counts, timings, RSS, paths under a local cache root.

Testing

• Manual/scheduled heavy evidence; not a CI default.

Documentation

• Link evidence from this issue into the WDC guide “Validation” section or a short note under benchmarks/ if a durable artifact is warranted.

Non-Goals

• Closing M4
• Pretending Host/Page will pass
• Deterministic CPU parallelization work owned by perf(exec): parallelize exact cosine KNN by canonical source #342–perf(exec): parallelize Node2Vec walk generation deterministically #344
• Implementing Host/Page ingest engines (use feat(io): chunked WDC Index/Arc Arrow bulk ingest tool #400 + conversion notes)

Related Issues

• docs(datasets): WDC Hyperlink Graph tiered scale guide and retrieval #399 docs/retrieval
• feat(io): chunked WDC Index/Arc Arrow bulk ingest tool #400 ingest tool
• test(performance): measure CSR build and reopen cost on WDC tiers #402 CSR/reopen costs on attempted tiers
• scale(wdc): Host aggregation ladder step (T5) #403 Host T5 step tracker
• scale(wdc): Page aggregation ladder step (T6) #404 Page T6 step tracker
• test(performance): establish the M4 embedded baseline and entry gate #334 / test(performance): reconcile final M4 exit evidence and scale claims #345 M4 evidence
• fix(storage): stream adjacency index builds past the 134,217,727-edge Arrow limit #336 / perf(exec): consume persisted CSR without edge-scale hash-map expansion #340 adjacency/CSR scale (consumers of this evidence)

Open Questions

• None

[coderabbitai]

🔗 Related PRs

#278 - fix(ci): retain certification evidence as artifacts [merged]
#358 - test(api): make public BDD fail closed [merged]
#378 - test(core): enforce coverage ledger and foundational contracts [merged]
#382 - test(storage-api): close remaining coverage floors for #369 [merged]
#405 - docs(datasets): WDC Hyperlink Graph first-fail scale ladder [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[DecisionNerd]

Blocked by #399; prefer also blocked by #400 for ingest. Optional evidence toward #334/#345. Stop conditions cite #403/#404.

[DecisionNerd]

Strategy update (authoritative)

WDC escalation is full ladder from the beginning, end at first fail:

• Exact order: T0 → T1 → T2 → T3 → T4 → T5a (2014 Host) → T5b (2012 Host) → T6a (2014 Page) → T6b (2012 Page)
• Prior tier must be green before the next starts
• On first red: stop; do not escalate; record failed tier + why
• Host/Page (scale(wdc): Host aggregation ladder step (T5) #403/scale(wdc): Page aggregation ladder step (T6) #404) are later ladder steps, not deferred-forever non-goals
• Guide: docs/guide/datasets/wdc-hyperlink-graph.md (PR docs(scale): GSI taxonomy + scale-evaluation harness contract (Graph500/LDBC) #405)

[DecisionNerd]

Controlled WDC mirror for non-CI scale runners is tracked in #406 (R2 provision + T0–T3 sync) with client/docs scaffolding on PR #405. Prefer GF_WDC_SOURCE=mirror-only on first-fail ladder machines once the public base URL exists. Runbook: #407. Full Page mirror is not required to start T0–T4.

[DecisionNerd]

Boundary clarification (scale harness location)

Ladder execution (T0→T6 orchestration, first-fail stop, evidence packaging, dedicated runners) belongs in an external scale harness repository, not GraphForge core CI/Makefile.

This repo owns the spec / contract:

• docs/reference/graph-scale-index.md — GSI axis, Graph500 notches, WDC placement, harness interface expectations
• docs/guide/datasets/wdc-hyperlink-graph.md — tier order, acceptance, reference fetch (GF_WDC_*)

Thin reference clients (scripts/datasets/fetch_wdc_hyperlink.py) may be called by the harness; they are not the harness.

When closing this issue, attach harness-produced first-fail evidence against the published spec — do not treat GraphForge core CI green as ladder proof.

[DecisionNerd]

Not planned — WDC scale track cancelled

GraphForge scale testing pivots off WDC Hyperlink Graphs.

New track (docs in this repo; execution external):

• docs(scale): Graph500 × GSI size-ladder specification #408 — Graph500 × GSI size-ladder specification
• docs(scale): LDBC full-suite benchmark specification #409 — LDBC full-suite benchmark specification
• docs(scale): external Graph500 + LDBC harness contract #410 — external Graph500 + LDBC harness contract

WDC T0–T6 is no longer the execution ladder. Optional M4 external scale validation retargets to Graph500 + LDBC (still non-blocking for #335).

[DecisionNerd]

Closing as not planned: WDC scale track cancelled in favor of #408 / #409 / #410.