diff --git a/AGENTS.md b/AGENTS.md index df9d5daa7..8d0700d3b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,7 +23,7 @@ Rust owns behavior. Python and Node are thin bindings—never fallback engines. - Cypher: `graphforge-cypher → graphforge-ir → graphforge-rel → graphforge-exec`. - Public API: `graphforge-api`. - Storage: `graphforge-storage`. -- Results are Arrow; graph data is Parquet; metadata is JSON. +- Tabular/data-bearing results are Arrow; control/metadata/lifecycle/explanation/construction may return scalars, collections, unit, or handles; graph data is Parquet; metadata is JSON. - Analyst verbs bypass the Cypher parser. - Runtime catalog IDs and ontology IDs are distinct. Never substitute one for the other. - Logical plans and wrapper tests are not end-to-end proof. diff --git a/docs/book/architecture/execution-model.md b/docs/book/architecture/execution-model.md index 7efb49fa1..898e9a364 100644 --- a/docs/book/architecture/execution-model.md +++ b/docs/book/architecture/execution-model.md @@ -250,8 +250,12 @@ Full algorithm catalog: [Algorithm Verbs](algorithms.md). ## Arrow Result Contract -All results are returned as **Arrow RecordBatch streams**. The schema for query results -carries GraphForge metadata: +Data-bearing execution results — Cypher `execute`, analyst verbs, and other +tabular algorithm or inspection tables — are returned as **Arrow RecordBatch +streams**. Control, metadata, lifecycle, explanation, and construction surfaces +may return scalars, collections, unit, or handles instead; see +[Arrow as the Data Contract](overview.md#arrow-as-the-data-contract). The schema +for query results carries GraphForge metadata: ```rust fn result_schema(fields: Vec, query_id: &str, ontology_ver: &str) -> Schema { diff --git a/docs/book/architecture/overview.md b/docs/book/architecture/overview.md index 0ca2673b6..643214f5c 100644 --- a/docs/book/architecture/overview.md +++ b/docs/book/architecture/overview.md @@ -1,7 +1,7 @@ # GraphForge Architecture Overview **Status:** v0.5.0 — Rust core shipped -**Last Updated:** 2026-07-27 +**Last Updated:** 2026-08-14 > **Implementation status legend** (used across architecture docs): **Shipped** = implemented and > tested on `main`; **Partially built** = some paths real, others stubbed; **Designed** = specified, @@ -34,10 +34,10 @@ the stable in-memory and FFI contract, Parquet for durable graph data). ## Architecture Principles -1. **Arrow is the wire contract** — results cross language boundaries as Arrow RecordBatch streams; no GraphForge-specific buffer protocol +1. **Arrow is the data-plane wire contract** — Cypher, analyst verbs, and other tabular/data-bearing results cross language boundaries as Arrow RecordBatch streams; no GraphForge-specific buffer protocol for those results 2. **GraphForge owns the semantics** — the Cypher compiler, ontology, and Graph IR live in GraphForge-owned Rust crates; no storage provider or binding becomes the semantic owner 3. **DataFusion is the execution backbone** — GraphForge extends DataFusion with custom graph operators rather than writing a full executor from scratch -4. **Unified result contract** — all methods return Arrow Tables; no surface returns a bespoke result type +4. **Scoped result contract** — data-returning operations use Arrow in and Arrow out; control, metadata, lifecycle, explanation, and construction surfaces may return scalars, collections, unit, or construction handles (not binding-owned graph result objects) 5. **Correctness over performance** — strict openCypher TCK compliance remains the primary constraint 6. **Ontology is progressive, not required** — GraphForge supports three modes: `exploratory` (no ontology required, all labels accepted), `advisory` (ontology present, violations are warnings), and `strict` (ontology enforced, violations are errors). Exploratory analysis is a first-class workflow. See [ADR 0003](../../adr/0003-progressive-ontology.md). 7. **Three layers, clean boundaries** — graph concerns, knowledge concerns, and workbench concerns are separated; the graph layer stays graph-native and never absorbs the others. See the next section and [ADR 0005](../../adr/0005-layered-architecture.md). @@ -193,7 +193,11 @@ See [AST & Planning](ast-and-planning.md) for the full compiler pipeline. ## Arrow as the Data Contract -All execution results cross language boundaries as **Arrow RecordBatch streams**. Arrow provides: +**Data-plane** results — Cypher `execute` / streaming sinks, analyst verbs +(`rank` / `cluster` / `paths` / `analyze` / `similar` / `find`), tabular +inspection such as `schema()`, bulk-construction receipts, and other +data-bearing algorithm or knowledge tables — cross language boundaries as +**Arrow RecordBatch streams**. Arrow provides: - A stable, language-independent columnar memory format - Zero-copy in-process exchange via the C Data Interface @@ -211,7 +215,31 @@ graphforge.query_id = "01J..." ``` These annotations survive IPC serialization and Parquet round-trips, which is why Arrow is -the correct contract rather than a Polars or Python-specific result type. +the correct contract for tabular results rather than a Polars or Python-specific result type. + +### Control and construction plane (intentional non-Arrow returns) + +Not every public method is a tabular data operation. The Rust facade +(`graphforge-api`) intentionally returns non-Arrow values for control, +metadata, lifecycle, explanation, and construction. Python and Node mirror the +same categories as thin projections — they do **not** execute graph logic or +rebuild tabular engine results into binding-owned objects. + +| Category | Typical returns | Examples (Rust → Python / Node) | +| -------- | --------------- | -------------------------------- | +| **Metadata / inspection** | string collections, scalars | `labels()` / `relationship_types()` → `Vec` / `list[str]`; `node_count()` → `u64` / `int` | +| **Explanation** | plain text | `explain()` → `String` / `str` | +| **Lifecycle / control** | unit (`()` / `None`) | `index(...)`, `load_ontology(...)`, `adopt_ontology(...)`, `clear_ontology(...)`, `execute_to_parquet(...)`, embedding publish helpers | +| **Construction handles** | instance-bound handles | `add_node(...)` → `NodeHandle`; `add_edge(...)` → `EdgeHandle` | + +**Construction handles vs metadata/control:** a `NodeHandle` / `EdgeHandle` is a +Rust-owned, instance-bound identity token (stable UUID plus label or relationship +metadata) so callers can wire subsequent construction or selectors. It is not a +tabular query result and not a binding-side graph object model. Metadata and +control returns (`Vec`, `u64`, `String`, `()`) answer inspection, +planning, or lifecycle questions without columnar payloads. Bulk construction +and Cypher/analyst paths remain on the Arrow data plane (including Arrow +receipts for atomic bulk publish). --- @@ -219,14 +247,15 @@ the correct contract rather than a Polars or Python-specific result type. | Language | Mechanism | Crate / Package | Result contract | v0.5.0 | | ---------- | ---------------- | ----------------------------------------- | -------------------------------------------- | -------- | -| **Rust** | Native crate API | `graphforge-api` / `graphforge-core` | Arrow batches via the facade | **Shipped** | -| **Python** | PyO3 + maturin | `graphforge-bindings-py` | `pyarrow.Table` or `RecordBatchReader` | **Shipped** (thin) | -| **Node** | napi-rs | `graphforge-bindings-node` | Arrow IPC `Buffer` → `tableFromIPC(buf)` | **Shipped** (thin) | -| **Swift** | UniFFI (planned) | deferred | Arrow IPC | **Deferred** (v0.5.1) | -| **Kotlin** | UniFFI (planned) | deferred | Arrow IPC | **Deferred** (v0.5.1) | +| **Rust** | Native crate API | `graphforge-api` / `graphforge-core` | Arrow for data-bearing results; scalars / collections / unit / handles elsewhere | **Shipped** | +| **Python** | PyO3 + maturin | `graphforge-bindings-py` | `pyarrow.Table` (or reader) for tabular results; same non-Arrow categories as Rust | **Shipped** (thin) | +| **Node** | napi-rs | `graphforge-bindings-node` | Arrow IPC `Buffer` → `tableFromIPC(buf)` for tabular results; same non-Arrow categories as Rust | **Shipped** (thin) | +| **Swift** | UniFFI (planned) | deferred | Arrow IPC (data plane) | **Deferred** (v0.5.1) | +| **Kotlin** | UniFFI (planned) | deferred | Arrow IPC (data plane) | **Deferred** (v0.5.1) | The architectural rule: **never let a binding become the semantic owner**. Bindings project requests and results; the Rust core owns Cypher, verbs, storage, and knowledge semantics. +Bindings never reshape tabular engine results into binding-owned row/object graphs. See [ADR 0001](../../adr/0001-rust-core.md). --- @@ -242,7 +271,7 @@ Shipped v0.5.0 expects these surfaces to stay green on `main`: | Ontology runtime | Load/validate round-trips for progressive modes | | Data contract | Arrow/Parquet/IPC round-trips pass | | Storage | Parquet project generations with atomic publication / recovery | -| Bindings | Thin Python and Node projections execute and consume Arrow results | +| Bindings | Thin Python and Node projections; tabular results stay Arrow | | Knowledge | knowledge ledger + epistemic records attach by UUID without changing graph results | --- diff --git a/docs/book/architecture/refactor-v0.5.md b/docs/book/architecture/refactor-v0.5.md index d724caa58..e5ccdc37d 100644 --- a/docs/book/architecture/refactor-v0.5.md +++ b/docs/book/architecture/refactor-v0.5.md @@ -85,7 +85,9 @@ GraphForge is organised into three layers with strict boundaries [ADR 0006](../../adr/0006-epistemic-model.md)). Lives in `provenance/` + `knowledge/`. Attaches to graph objects **by UUID reference only**. - **Workbench layer** — the analyst verbs, hybrid search, workflows, exploration, project envelope. - Consumes the layers below and returns Arrow; holds no graph-semantic state. + Consumes the layers below; data-bearing verb results are Arrow. Holds no + graph-semantic state. Control/lifecycle/construction surfaces may return + scalars, collections, unit, or handles. **Boundary rule:** knowledge attaches by UUID reference, never by embedding columns on graph tables. Graph-native query results never depend on knowledge-layer data (a tested invariant). This is what diff --git a/docs/development/contributing.md b/docs/development/contributing.md index c6093e775..4708f7a09 100644 --- a/docs/development/contributing.md +++ b/docs/development/contributing.md @@ -228,7 +228,7 @@ onboarding contract. ## Design Principles 1. **Spec-driven correctness** — openCypher semantics over performance -2. **Arrow as the wire contract** — results cross language boundaries as Arrow RecordBatch streams +2. **Arrow as the data-plane wire contract** — Cypher and analyst/data-bearing results cross language boundaries as Arrow RecordBatch streams; control/metadata/lifecycle/explanation/construction may return scalars, collections, unit, or handles 3. **GraphForge owns the semantics** — no binding or storage provider becomes the semantic owner 4. **Surfaces stay independent** — analyst verbs bypass the Cypher parser; they do not rewrite it 5. **Inspectable** — `explain` at every compiler stage; structured errors with spans diff --git a/docs/engineering/ARCHITECTURE.md b/docs/engineering/ARCHITECTURE.md index 8c992c92a..7fecc715c 100644 --- a/docs/engineering/ARCHITECTURE.md +++ b/docs/engineering/ARCHITECTURE.md @@ -45,7 +45,10 @@ flowchart LR - **Ontology** — optional progressive model (`exploratory` / `advisory` / `strict`). - **Knowledge records** — provenance, confidence, epistemic assertions attached by UUID, never as graph-table columns that alter Cypher results. -- **Results** — Arrow tables/batches as the cross-language contract. +- **Results (data plane)** — Arrow tables/batches for Cypher, analyst verbs, and + other tabular/data-bearing surfaces. Control, metadata, lifecycle, explanation, + and construction may return scalars, collections, unit, or handles; see + [`../book/architecture/overview.md`](../book/architecture/overview.md#arrow-as-the-data-contract). - **On disk** — Parquet for graph data; JSON for metadata and contracts. Schemas and frozen inventories also live under [`../contracts/`](https://github.com/CurateLabs/graphforge/tree/main/docs/contracts) and diff --git a/docs/guide/graph-construction.md b/docs/guide/graph-construction.md index 3dae16909..e177460f2 100644 --- a/docs/guide/graph-construction.md +++ b/docs/guide/graph-construction.md @@ -2,7 +2,9 @@ Build graphs with the Python API or openCypher. This page is the everyday construction path for v0.5.0: scalar nodes and edges first, then atomic bulk -batches. Results and receipts are Apache Arrow tables. +batches. Scalar `add_node` / `add_edge` return construction handles; atomic bulk +publish and Cypher/analyst paths return Apache Arrow tables (including bulk +receipts). For deeper architecture (project generations, Rust-owned validation, binding parity), see the [Book](../book/README.md). diff --git a/docs/guide/overview.md b/docs/guide/overview.md index a97e19b1c..e3f552ade 100644 --- a/docs/guide/overview.md +++ b/docs/guide/overview.md @@ -94,10 +94,17 @@ This pattern matches two Person nodes connected by a KNOWS relationship. ## Result Types -**v0.5.0:** ALL methods return a PyArrow `Table` — `execute`, `rank`, `cluster`, `find`, -and `schema`. There are no `CypherValue` wrappers and no `SearchHit` objects. Access values -via `.as_py()` or pass the table directly to pandas, Polars, or NetworkX, which all accept -Arrow as input. +**v0.5.0 data plane:** Cypher `execute`, analyst verbs (`rank`, `cluster`, +`paths`, `analyze`, `similar`, `find`), and tabular helpers such as `schema()` +return a PyArrow `Table`. There are no `CypherValue` wrappers and no +`SearchHit` objects for those results. Access values via `.as_py()` or pass the +table directly to pandas, Polars, or NetworkX. + +**Control / construction plane:** methods such as `labels()`, +`relationship_types()`, `node_count()`, `explain()`, ontology lifecycle helpers, +and scalar `add_node` / `add_edge` return lists, integers, strings, `None`, or +construction handles — not Arrow tables. See the +[architecture overview](../book/architecture/overview.md#arrow-as-the-data-contract). ```python table = forge.execute("MATCH (p:Person) RETURN p.name, p.age") diff --git a/docs/guide/visualization.md b/docs/guide/visualization.md index 5d7e43fcf..87346ebf6 100644 --- a/docs/guide/visualization.md +++ b/docs/guide/visualization.md @@ -1,6 +1,7 @@ # Visualization examples -GraphForge returns Arrow tables from its public Python and Node APIs. This guide +GraphForge returns Arrow tables from its public Python and Node **data-plane** +APIs (Cypher `execute`, analyst verbs, and other tabular results). This guide shows how to take **one shared real-data projection** and render it with common ecosystem libraries in both Python and Node.js—without adding visualization behavior to GraphForge Core. diff --git a/docs/index.md b/docs/index.md index f35de8349..74c896eff 100644 --- a/docs/index.md +++ b/docs/index.md @@ -138,7 +138,7 @@ lifecycle commands: ```text Python (PyO3, thin) ─┐ -Node (N-API, thin) ──┼──> graphforge-api ──> Arrow results +Node (N-API, thin) ──┼──> graphforge-api ──> Arrow (data plane) + thin control/handle returns CLI (thin launcher) ─┘ Cypher: graphforge-cypher ──> graphforge-ir ──> graphforge-rel ──> graphforge-exec diff --git a/docs/reference/api.md b/docs/reference/api.md index 11f3a604c..455156f55 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -8,9 +8,18 @@ > [`graphforge-checkpoint-api/1`](../contracts/checkpoint-api-v1.json). !!! note "v0.5.0 unified API" -This page documents the **v0.5.0 API**. All methods return Arrow Tables — there are no -`CypherValue` wrappers or `SearchHit` objects. The conventional instance name is `forge`, -not `db`. The Rust crate API is documented separately below. +This page documents the **v0.5.0 API**. **Data-returning** operations — +Cypher `execute`, analyst verbs, `schema()`, bulk-construction receipts, and +other tabular/data-bearing surfaces — return Arrow Tables. There are no +`CypherValue` wrappers or `SearchHit` objects for those results. Control, +metadata, lifecycle, explanation, and scalar construction surfaces return +scalars, collections, unit, or construction handles instead (for example +`labels()` → `list[str]`, `node_count()` → `int`, `explain()` → `str`, +`add_node()` → `NodeHandle`). See the +[architecture overview](../book/architecture/overview.md#arrow-as-the-data-contract) +for the data-plane versus control/construction-plane contract. The conventional +instance name is `forge`, not `db`. The Rust crate API is documented separately +below. UUID normalization and canonical Arrow/result fingerprints are identical across Rust, Python, and Node because all three surfaces use the versioned @@ -3663,6 +3672,10 @@ registered again after reopen before its space can refresh. ## Graph Inspection +These methods are **metadata / control plane**, not tabular query results +(except `schema()`, which remains an Arrow table). Return types match the Rust +facade and the thin Python/Node projections. + ### `schema()` → `pyarrow.Table` Columns: `label`, `node_count`, `rel_type`, `rel_count`.