Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Rust owns behavior. Python and Node are thin bindings—never fallback engines.
- Cypher: `graphforge-cypher → graphforge-ir → graphforge-rel → graphforge-exec`.
- Public API: `graphforge-api`.
- Storage: `graphforge-storage`.
- Results are Arrow; graph data is Parquet; metadata is JSON.
- Tabular/data-bearing results are Arrow; control/metadata/lifecycle/explanation/construction may return scalars, collections, unit, or handles; graph data is Parquet; metadata is JSON.
- Analyst verbs bypass the Cypher parser.
- Runtime catalog IDs and ontology IDs are distinct. Never substitute one for the other.
- Logical plans and wrapper tests are not end-to-end proof.
Expand Down
8 changes: 6 additions & 2 deletions docs/book/architecture/execution-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,8 +250,12 @@ Full algorithm catalog: [Algorithm Verbs](algorithms.md).

## Arrow Result Contract

All results are returned as **Arrow RecordBatch streams**. The schema for query results
carries GraphForge metadata:
Data-bearing execution results — Cypher `execute`, analyst verbs, and other
tabular algorithm or inspection tables — are returned as **Arrow RecordBatch
streams**. Control, metadata, lifecycle, explanation, and construction surfaces
may return scalars, collections, unit, or handles instead; see
[Arrow as the Data Contract](overview.md#arrow-as-the-data-contract). The schema
for query results carries GraphForge metadata:

```rust
fn result_schema(fields: Vec<Field>, query_id: &str, ontology_ver: &str) -> Schema {
Expand Down
51 changes: 40 additions & 11 deletions docs/book/architecture/overview.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# GraphForge Architecture Overview

**Status:** v0.5.0 — Rust core shipped
**Last Updated:** 2026-07-27
**Last Updated:** 2026-08-14

> **Implementation status legend** (used across architecture docs): **Shipped** = implemented and
> tested on `main`; **Partially built** = some paths real, others stubbed; **Designed** = specified,
Expand Down Expand Up @@ -34,10 +34,10 @@ the stable in-memory and FFI contract, Parquet for durable graph data).

## Architecture Principles

1. **Arrow is the wire contract** — results cross language boundaries as Arrow RecordBatch streams; no GraphForge-specific buffer protocol
1. **Arrow is the data-plane wire contract** — Cypher, analyst verbs, and other tabular/data-bearing results cross language boundaries as Arrow RecordBatch streams; no GraphForge-specific buffer protocol for those results
2. **GraphForge owns the semantics** — the Cypher compiler, ontology, and Graph IR live in GraphForge-owned Rust crates; no storage provider or binding becomes the semantic owner
3. **DataFusion is the execution backbone** — GraphForge extends DataFusion with custom graph operators rather than writing a full executor from scratch
4. **Unified result contract** — all methods return Arrow Tables; no surface returns a bespoke result type
4. **Scoped result contract** — data-returning operations use Arrow in and Arrow out; control, metadata, lifecycle, explanation, and construction surfaces may return scalars, collections, unit, or construction handles (not binding-owned graph result objects)
5. **Correctness over performance** — strict openCypher TCK compliance remains the primary constraint
6. **Ontology is progressive, not required** — GraphForge supports three modes: `exploratory` (no ontology required, all labels accepted), `advisory` (ontology present, violations are warnings), and `strict` (ontology enforced, violations are errors). Exploratory analysis is a first-class workflow. See [ADR 0003](../../adr/0003-progressive-ontology.md).
7. **Three layers, clean boundaries** — graph concerns, knowledge concerns, and workbench concerns are separated; the graph layer stays graph-native and never absorbs the others. See the next section and [ADR 0005](../../adr/0005-layered-architecture.md).
Expand Down Expand Up @@ -193,7 +193,11 @@ See [AST & Planning](ast-and-planning.md) for the full compiler pipeline.

## Arrow as the Data Contract

All execution results cross language boundaries as **Arrow RecordBatch streams**. Arrow provides:
**Data-plane** results — Cypher `execute` / streaming sinks, analyst verbs
(`rank` / `cluster` / `paths` / `analyze` / `similar` / `find`), tabular
inspection such as `schema()`, bulk-construction receipts, and other
data-bearing algorithm or knowledge tables — cross language boundaries as
**Arrow RecordBatch streams**. Arrow provides:

- A stable, language-independent columnar memory format
- Zero-copy in-process exchange via the C Data Interface
Expand All @@ -211,22 +215,47 @@ graphforge.query_id = "01J..."
```

These annotations survive IPC serialization and Parquet round-trips, which is why Arrow is
the correct contract rather than a Polars or Python-specific result type.
the correct contract for tabular results rather than a Polars or Python-specific result type.

### Control and construction plane (intentional non-Arrow returns)

Not every public method is a tabular data operation. The Rust facade
(`graphforge-api`) intentionally returns non-Arrow values for control,
metadata, lifecycle, explanation, and construction. Python and Node mirror the
same categories as thin projections — they do **not** execute graph logic or
rebuild tabular engine results into binding-owned objects.

| Category | Typical returns | Examples (Rust → Python / Node) |
| -------- | --------------- | -------------------------------- |
| **Metadata / inspection** | string collections, scalars | `labels()` / `relationship_types()` → `Vec<String>` / `list[str]`; `node_count()` → `u64` / `int` |
| **Explanation** | plain text | `explain()` → `String` / `str` |
| **Lifecycle / control** | unit (`()` / `None`) | `index(...)`, `load_ontology(...)`, `adopt_ontology(...)`, `clear_ontology(...)`, `execute_to_parquet(...)`, embedding publish helpers |
| **Construction handles** | instance-bound handles | `add_node(...)` → `NodeHandle`; `add_edge(...)` → `EdgeHandle` |

**Construction handles vs metadata/control:** a `NodeHandle` / `EdgeHandle` is a
Rust-owned, instance-bound identity token (stable UUID plus label or relationship
metadata) so callers can wire subsequent construction or selectors. It is not a
tabular query result and not a binding-side graph object model. Metadata and
control returns (`Vec<String>`, `u64`, `String`, `()`) answer inspection,
planning, or lifecycle questions without columnar payloads. Bulk construction
and Cypher/analyst paths remain on the Arrow data plane (including Arrow
receipts for atomic bulk publish).

---

## Multi-Language Bindings

| Language | Mechanism | Crate / Package | Result contract | v0.5.0 |
| ---------- | ---------------- | ----------------------------------------- | -------------------------------------------- | -------- |
| **Rust** | Native crate API | `graphforge-api` / `graphforge-core` | Arrow batches via the facade | **Shipped** |
| **Python** | PyO3 + maturin | `graphforge-bindings-py` | `pyarrow.Table` or `RecordBatchReader` | **Shipped** (thin) |
| **Node** | napi-rs | `graphforge-bindings-node` | Arrow IPC `Buffer` → `tableFromIPC(buf)` | **Shipped** (thin) |
| **Swift** | UniFFI (planned) | deferred | Arrow IPC | **Deferred** (v0.5.1) |
| **Kotlin** | UniFFI (planned) | deferred | Arrow IPC | **Deferred** (v0.5.1) |
| **Rust** | Native crate API | `graphforge-api` / `graphforge-core` | Arrow for data-bearing results; scalars / collections / unit / handles elsewhere | **Shipped** |
| **Python** | PyO3 + maturin | `graphforge-bindings-py` | `pyarrow.Table` (or reader) for tabular results; same non-Arrow categories as Rust | **Shipped** (thin) |
| **Node** | napi-rs | `graphforge-bindings-node` | Arrow IPC `Buffer` → `tableFromIPC(buf)` for tabular results; same non-Arrow categories as Rust | **Shipped** (thin) |
| **Swift** | UniFFI (planned) | deferred | Arrow IPC (data plane) | **Deferred** (v0.5.1) |
| **Kotlin** | UniFFI (planned) | deferred | Arrow IPC (data plane) | **Deferred** (v0.5.1) |

The architectural rule: **never let a binding become the semantic owner**. Bindings project
requests and results; the Rust core owns Cypher, verbs, storage, and knowledge semantics.
Bindings never reshape tabular engine results into binding-owned row/object graphs.
See [ADR 0001](../../adr/0001-rust-core.md).

---
Expand All @@ -242,7 +271,7 @@ Shipped v0.5.0 expects these surfaces to stay green on `main`:
| Ontology runtime | Load/validate round-trips for progressive modes |
| Data contract | Arrow/Parquet/IPC round-trips pass |
| Storage | Parquet project generations with atomic publication / recovery |
| Bindings | Thin Python and Node projections execute and consume Arrow results |
| Bindings | Thin Python and Node projections; tabular results stay Arrow |
| Knowledge | knowledge ledger + epistemic records attach by UUID without changing graph results |
---

Expand Down
4 changes: 3 additions & 1 deletion docs/book/architecture/refactor-v0.5.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ GraphForge is organised into three layers with strict boundaries
[ADR 0006](../../adr/0006-epistemic-model.md)). Lives in `provenance/` + `knowledge/`. Attaches to
graph objects **by UUID reference only**.
- **Workbench layer** — the analyst verbs, hybrid search, workflows, exploration, project envelope.
Consumes the layers below and returns Arrow; holds no graph-semantic state.
Consumes the layers below; data-bearing verb results are Arrow. Holds no
graph-semantic state. Control/lifecycle/construction surfaces may return
scalars, collections, unit, or handles.

**Boundary rule:** knowledge attaches by UUID reference, never by embedding columns on graph tables.
Graph-native query results never depend on knowledge-layer data (a tested invariant). This is what
Expand Down
2 changes: 1 addition & 1 deletion docs/development/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ onboarding contract.
## Design Principles

1. **Spec-driven correctness** — openCypher semantics over performance
2. **Arrow as the wire contract** — results cross language boundaries as Arrow RecordBatch streams
2. **Arrow as the data-plane wire contract** — Cypher and analyst/data-bearing results cross language boundaries as Arrow RecordBatch streams; control/metadata/lifecycle/explanation/construction may return scalars, collections, unit, or handles
3. **GraphForge owns the semantics** — no binding or storage provider becomes the semantic owner
4. **Surfaces stay independent** — analyst verbs bypass the Cypher parser; they do not rewrite it
5. **Inspectable** — `explain` at every compiler stage; structured errors with spans
Expand Down
5 changes: 4 additions & 1 deletion docs/engineering/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,10 @@ flowchart LR
- **Ontology** — optional progressive model (`exploratory` / `advisory` / `strict`).
- **Knowledge records** — provenance, confidence, epistemic assertions attached by UUID,
never as graph-table columns that alter Cypher results.
- **Results** — Arrow tables/batches as the cross-language contract.
- **Results (data plane)** — Arrow tables/batches for Cypher, analyst verbs, and
other tabular/data-bearing surfaces. Control, metadata, lifecycle, explanation,
and construction may return scalars, collections, unit, or handles; see
[`../book/architecture/overview.md`](../book/architecture/overview.md#arrow-as-the-data-contract).
- **On disk** — Parquet for graph data; JSON for metadata and contracts.

Schemas and frozen inventories also live under [`../contracts/`](https://github.com/CurateLabs/graphforge/tree/main/docs/contracts) and
Expand Down
4 changes: 3 additions & 1 deletion docs/guide/graph-construction.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

Build graphs with the Python API or openCypher. This page is the everyday
construction path for v0.5.0: scalar nodes and edges first, then atomic bulk
batches. Results and receipts are Apache Arrow tables.
batches. Scalar `add_node` / `add_edge` return construction handles; atomic bulk
publish and Cypher/analyst paths return Apache Arrow tables (including bulk
receipts).

For deeper architecture (project generations, Rust-owned validation, binding
parity), see the [Book](../book/README.md).
Expand Down
15 changes: 11 additions & 4 deletions docs/guide/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,10 +94,17 @@ This pattern matches two Person nodes connected by a KNOWS relationship.

## Result Types

**v0.5.0:** ALL methods return a PyArrow `Table` — `execute`, `rank`, `cluster`, `find`,
and `schema`. There are no `CypherValue` wrappers and no `SearchHit` objects. Access values
via `.as_py()` or pass the table directly to pandas, Polars, or NetworkX, which all accept
Arrow as input.
**v0.5.0 data plane:** Cypher `execute`, analyst verbs (`rank`, `cluster`,
`paths`, `analyze`, `similar`, `find`), and tabular helpers such as `schema()`
return a PyArrow `Table`. There are no `CypherValue` wrappers and no
`SearchHit` objects for those results. Access values via `.as_py()` or pass the
table directly to pandas, Polars, or NetworkX.

**Control / construction plane:** methods such as `labels()`,
`relationship_types()`, `node_count()`, `explain()`, ontology lifecycle helpers,
and scalar `add_node` / `add_edge` return lists, integers, strings, `None`, or
construction handles — not Arrow tables. See the
[architecture overview](../book/architecture/overview.md#arrow-as-the-data-contract).

```python
table = forge.execute("MATCH (p:Person) RETURN p.name, p.age")
Expand Down
3 changes: 2 additions & 1 deletion docs/guide/visualization.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Visualization examples

GraphForge returns Arrow tables from its public Python and Node APIs. This guide
GraphForge returns Arrow tables from its public Python and Node **data-plane**
APIs (Cypher `execute`, analyst verbs, and other tabular results). This guide
shows how to take **one shared real-data projection** and render it with common
ecosystem libraries in both Python and Node.js—without adding visualization
behavior to GraphForge Core.
Expand Down
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ lifecycle commands:

```text
Python (PyO3, thin) ─┐
Node (N-API, thin) ──┼──> graphforge-api ──> Arrow results
Node (N-API, thin) ──┼──> graphforge-api ──> Arrow (data plane) + thin control/handle returns
CLI (thin launcher) ─┘

Cypher: graphforge-cypher ──> graphforge-ir ──> graphforge-rel ──> graphforge-exec
Expand Down
19 changes: 16 additions & 3 deletions docs/reference/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,18 @@
> [`graphforge-checkpoint-api/1`](../contracts/checkpoint-api-v1.json).

!!! note "v0.5.0 unified API"
This page documents the **v0.5.0 API**. All methods return Arrow Tables — there are no
`CypherValue` wrappers or `SearchHit` objects. The conventional instance name is `forge`,
not `db`. The Rust crate API is documented separately below.
This page documents the **v0.5.0 API**. **Data-returning** operations —
Cypher `execute`, analyst verbs, `schema()`, bulk-construction receipts, and
other tabular/data-bearing surfaces — return Arrow Tables. There are no
`CypherValue` wrappers or `SearchHit` objects for those results. Control,
metadata, lifecycle, explanation, and scalar construction surfaces return
scalars, collections, unit, or construction handles instead (for example
`labels()` → `list[str]`, `node_count()` → `int`, `explain()` → `str`,
`add_node()` → `NodeHandle`). See the
[architecture overview](../book/architecture/overview.md#arrow-as-the-data-contract)
for the data-plane versus control/construction-plane contract. The conventional
instance name is `forge`, not `db`. The Rust crate API is documented separately
below.

UUID normalization and canonical Arrow/result fingerprints are identical
across Rust, Python, and Node because all three surfaces use the versioned
Expand Down Expand Up @@ -3663,6 +3672,10 @@ registered again after reopen before its space can refresh.

## Graph Inspection

These methods are **metadata / control plane**, not tabular query results
(except `schema()`, which remains an Arrow table). Return types match the Rust
facade and the thin Python/Node projections.

### `schema()` → `pyarrow.Table`

Columns: `label`, `node_count`, `rel_type`, `rel_count`.
Expand Down