Skip to content

feat(storage): add persistent UUID membership indexes for bounded ingest validation #737

Description

@DecisionNerd

Problem

Bulk edge validation currently rebuilds in-memory sets of all persisted node UUIDs, and publication checks can load all edge UUIDs. Repeated O(V/E) scans and unbounded sets cannot support billion-edge staged ingest.

Objective

Provide authoritative persistent UUID membership indexes that allow bounded-memory node and edge existence checks without changing UUID semantics.

Requirements

  • Define the on-disk index format, versioning, checksums, atomic publication, and reopen behavior.
  • Support batched membership probes and duplicate detection for node and edge UUIDs.
  • Build or update indexes transactionally with graph publication.
  • Detect missing, stale, or corrupt indexes and fail safely or rebuild through an explicit bounded path.
  • Preserve Rust ownership; bindings consume the public facade only.
  • Expose useful probe/build metrics without leaking graph data.

Acceptance criteria

  • Edge validation no longer requires materializing every persisted node UUID.
  • Publication duplicate checks no longer require materializing every persisted edge UUID.
  • Peak index-build/probe memory is bounded by configured batch/work limits.
  • Reopen, crash-before-publish, corruption, and concurrent-reader tests are deterministic.
  • Index contents reconcile with canonical graph counts and UUID lookup behavior.
  • Format/version documentation and migration/rebuild behavior are documented.

Scenarios

  • Given a batch whose endpoints are partly missing, when membership is probed, then missing UUIDs are reported deterministically without loading the full vertex set.
  • Given a crash before the new index manifest is published, when the project reopens, then readers observe the prior valid index and graph.
  • Given an index checksum mismatch, when the graph opens, then GraphForge returns a typed integrity error and never silently accepts stale membership.

Non-goals

A general secondary-index framework or foreign key engine.

Related issues

Canonical tracker: #735. Direct prerequisite: #736.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    coreCore source code changesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions