test(performance): reconcile final M4 exit evidence and scale claims

[DecisionNerd]

Reopened after closure audit

The previous exit reconciliation is stale because it freezes 2b1bfda8145c7f9fe125a75faf695792fc6bbfcb, which predates the later #498/#499–#588 M4 expansion. Re-run the contract against one exact accepted commit after that expansion, include the complete algorithm disposition ledger, and repair the duplicated/truncated CPU-kernel paragraph in docs/development/m4-exit-evidence.md.

This issue remains gated by #336 and #338 unless those outcome requirements are explicitly waived and documented. M5 (#735) is separate forward work and is not substitute M4 exit evidence.

Tracker Purpose

Produce the final, exact-tree M4 exit evidence and documentation reconciliation after every implementation child has reached an accepted disposition. This is the last child before #335 can close; it performs no new performance optimization.

Scope

• Rerun the test(performance): establish the M4 embedded baseline and entry gate #334 contract on the final accepted M4 tree.
• Reproduce the public persistence, adjacency scale, streaming-memory, CSR representation, Arrow shaping, and deterministic CPU-kernel outcomes.
• Reconcile public Rust and thin Python/Node behavior, exact-head CI history, live issue relationships, and evidence-backed documentation.
• GPU is intentionally outside this finite plan and is not required for M4 closure.

Closure Standard

Problem

Individual performance PRs can pass focused tests while the combined tree regresses small workloads, changes deterministic contracts, fails to expose large graphs through the public facade, or leaves documentation claiming unsupported scale. M4 needs one final evidence reconciliation rather than relying on a collection of issue-closing anecdotes.

Objective

Freeze the final accepted M4 tree, rerun the versioned entry contract, publish one complete before/after evidence ledger, correct all affected documentation, and leave #335 with no unresolved child, relationship, test, or claim.

Requirements

1. Verify every M4 milestone issue is a native child of epic(performance): close M4 embedded performance and scale #335, blocks epic(performance): close M4 embedded performance and scale #335, and is closed through merged work or an explicit evidence-backed disposition. This issue must be blocked by every earlier M4 child.
2. Rerun the test(performance): establish the M4 embedded baseline and entry gate #334 short deterministic matrix and the applicable manual/scheduled large matrix on the final accepted tree and exact commit.
3. Exercise the public Rust facade for fixed-hop LIMIT, full scan/count, aggregate/top-N, iterative PageRank, exact cosine KNN, Node2Vec embedding, project close/reopen, and applicable failure/cancellation/resource cases.
4. Prove file-backed public reopen at the accepted 8M-node/128M-edge class and bounded adjacency construction beyond the former Arrow boundary under the documented resource stops, or record a specific accepted blocker without overstating product scale.
5. Record structural work, partitions, rows scanned/decoded/materialized, index representation, output batches, threads/path, iterations, spill, peak RSS, result fingerprints, and hardware-specific timings for changed workload classes.
6. Prove CPU-only embedded operation remains complete, Rust-owned, network-free, and deterministic at every supported 1/2/4/8/automatic configuration.
7. Reconcile required exact-head CI/CI Gate and review/thread evidence for every merged child; do not rerun unchanged historical trees solely to create duplicate evidence.
8. Update scale limits, project/execution/algorithm architecture, public API/resource configuration, benchmark methodology, contributor guidance, and binding documentation to match shipped behavior exactly.
9. Post a final evidence reconciliation to epic(performance): close M4 embedded performance and scale #335 and close this issue only when no required evidence, docs correction, or relationship repair remains.

Acceptance Criteria

• Every prior M4 child is closed or explicitly dispositioned and the live parent/blocked-by graph is consistent.
• One final-tree evidence artifact compares test(performance): establish the M4 embedded baseline and entry gate #334 entry and exit for every changed workload class with exact commit, fixture, hardware, structural metrics, RSS/spill, timing, and fingerprints.
• Public GraphForge::new reopens and queries the accepted large file-backed fixture; lower-level-only evidence is not mislabeled as a product claim.
• The prior 134,217,727-row adjacency boundary no longer governs the accepted public build path, with bounded-memory evidence beyond it.
• Streaming, CSR-native, and Arrow-shaping structural gates prove the removed eager/duplicate representations rather than inferring them from elapsed time.
• KNN, PageRank, and Node2Vec walk generation retain exact thread parity and documented automatic crossovers without small-workload regression beyond accepted thresholds.
• Required final correctness, concurrency, cancellation, recovery, Rust-facade, and binding surfaces are green.
• Documentation contains no universal scale claim, stale 1/2 GiB public guidance, unsupported thread/GPU claim, or unqualified hardware timing.
• epic(performance): close M4 embedded performance and scale #335 receives the final evidence ledger and has no open child other than itself after this issue closes.

BDD Completion Scenarios

Scenario: Final M4 evidence is reproducible

Given the accepted M4 implementation tree and declared fixtures
When the versioned entry/exit commands run
Then they emit reproducible structural, resource, correctness, and hardware-specific timing evidence
And every public result fingerprint matches the accepted contract.

Scenario: Large embedded work is publicly representable and bounded

Given the accepted large graph fixtures and resource policy
When GraphForge publishes, reopens, indexes, scans, and runs bounded traversal through the public facade
Then it completes or returns the documented structured resource outcome without legacy snapshot or whole-file materialization ceilings
And the evidence names the exact workload-specific envelope.

Scenario: M4 closes without a GPU requirement

Given every CPU/storage/execution child is complete and no GPU child exists
When the milestone is reconciled
Then CPU-only embedded operation satisfies the full closure contract
And no accelerator capability, dependency, documentation, or performance claim is required or implied.

Scenario: Tracker closure follows the live ledger

Given all evidence and documentation are accepted
When this issue closes
Then #335 has no unresolved native child or blocked-by dependency
And its final comment links the exact evidence and dispositions needed to close M4.

Observability

Evidence may include aggregate counters, build/fixture taxonomy identity, hardware identity, resource policy, and result fingerprints. It must exclude query parameters, UUIDs, properties, graph contents, local paths, and sensitive system data beyond reproduction needs.

Security And Privacy

No network or authorization surface is added. Verify spill/path containment and cleanup evidence, ensure benchmark artifacts contain no graph content or sensitive paths, and keep large external datasets explicit opt-in local inputs.

Testing

• Run the test(performance): establish the M4 embedded baseline and entry gate #334 short deterministic matrix and applicable large manual/scheduled matrix at the final tree.
• Verify relevant targeted gates and the repository-required exact-head CI history for each changed child surface.
• Map Scenarios 1 and 2 to benchmark/evidence artifacts plus public-facade integration, Scenario 3 to CPU-only capability acceptance, and Scenario 4 to live GitHub relationship/issue verification.
• Inspect intentional snapshots and documentation claims mechanically where possible.

Documentation

Update and cross-check scale limits, architecture, public resource configuration, project format/open behavior, algorithm determinism, benchmark reproduction, binding guidance, contributor workflow, and any release-facing performance statement.

Non-Goals

• Implementing new optimizations or repairing unrelated failures.
• Creating or shipping a GPU backend.
• Distributed or server execution.
• Claiming a universal graph size, SLO, or cross-machine timing guarantee.

Related Issues

• test(performance): establish the M4 embedded baseline and entry gate #334 — authoritative entry contract.
• epic(performance): close M4 embedded performance and scale #335 — canonical M4 close tracker; this issue is its final child and blocks closure.
• Every other M4 implementation child blocks this issue.

Optional Graph500 / LDBC Evidence (non-blocking)

If the external harness produces accepted Graph500 × GSI progressive-scale evidence and/or LDBC suite runs before exit reconciliation, include a summary (last green Graph500 SCALE / GSI notch, first red if any; LDBC benchmark + SF coverage) in the final ledger. Absence of that evidence must not block #345 or #335. Full LDBC audit completion is not an M4 close requirement. WDC is retired from this track (#399–#407 not planned).