Skip to content

feat(api): add resumable staged Arrow and Parquet graph-import sessions #738

Description

@DecisionNerd

Problem

Large graph construction needs a bounded-memory, resumable public API. Existing batch calls repeatedly validate against global state and do not define durable staging, checkpoint, cancellation, or atomic multi-file publication semantics.

Objective

Add Rust-facade import sessions for Arrow and Parquet node/edge sources that can resume safely and publish atomically.

Requirements

  • Define begin, append/register source, validate, checkpoint, resume, commit, abort, and status operations.
  • Stream record batches/files with explicit memory and concurrency budgets.
  • Use persistent UUID membership indexes for endpoint and duplicate validation.
  • Persist a versioned journal/manifest sufficient to resume after process interruption.
  • Make commit atomic: no partial graph or index publication is visible.
  • Reject incompatible schemas, corrupt inputs, path traversal, symlink escapes, and resource-limit violations with typed errors.
  • Make abort and stale-session cleanup deterministic and observable.

Acceptance criteria

  • Multi-file node and edge imports do not materialize the complete dataset.
  • An interrupted session resumes without duplicating committed work.
  • Commit publishes graph data, metadata, and required indexes as one recoverable state transition.
  • Abort leaves the prior project readable and removes or quarantines staged artifacts.
  • Progress includes rows/bytes/files accepted, rejected, pending, elapsed, and bounded resource usage.
  • Public API docs and small end-to-end tests cover Arrow and Parquet.

Scenarios

  • Given a process stops after several checkpoints, when the session resumes, then it continues from durable progress without duplicate live rows.
  • Given an edge references a missing node, when validation runs, then commit is refused with bounded deterministic diagnostics.
  • Given cancellation during commit preparation, when the project reopens, then only the prior published graph is visible.

Non-goals

Remote object-store orchestration, domain-specific loaders, or distributed ingest.

Related issues

Canonical tracker: #735. Direct prerequisite: #737.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    coreCore source code changesenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions