feat(api): define a bounded embedded execution resource policy #337

Problem

GraphForge has no single bounded execution-resource contract. The public facade builds a long-lived Tokio runtime with exactly two workers, while DataFusion independently chooses default target partitions from machine parallelism. Graph algorithms do not yet have a GraphForge-owned CPU pool, and memory, spill, I/O concurrency, and concurrent heavy-query limits are not coordinated through one per-instance policy.

This makes performance configuration incomplete and can create oversubscription, hardware-dependent behavior, or an inability to execute the M4 thread-parity matrix without test-only implementation seams.

Objective

Define and implement one embedded, per-GraphForge execution resource policy that bounds runtime workers, query partitions, memory, spill, I/O concurrency, and heavy-query concurrency while preserving deterministic results and CPU-only operation on every supported machine.

Debt / regime

• Debt type: architecture and test/proof
• Quality regime: A compute

Requirements

1. Add one Rust-owned per-instance resource policy with explicit and automatic modes. It must cover compute threads, Tokio runtime workers, DataFusion target partitions and batch size, memory budget, bounded spill configuration, I/O concurrency, and maximum concurrent heavy queries.
2. The policy must be constructed and validated before the long-lived runtime and execution sessions are created. Invalid, unavailable, or unsafe settings must return structured errors rather than being ignored or partially applied.
3. Automatic mode must derive a bounded configuration from available parallelism and declared memory without assuming that maximum partitions are optimal for every workload. Small workloads must retain a low-overhead serial or minimally partitioned path.
4. Use instance-owned resources rather than process-global configuration. A future deterministic CPU-kernel child must be able to consume a private bounded Rayon pool without using Rayon's global pool, but this issue does not parallelize graph algorithms.
5. Apply the accepted policy consistently to DataFusion sessions, runtime scheduling, memory accounting, spill, and admission of concurrent heavy work. Total concurrency must not exceed the declared instance budget merely because different subsystems own different pools.
6. Preserve canonical Arrow schemas, row ordering, values and fingerprints, cancellation, resource limits, structured errors, atomic publication, and recovery behavior at every supported thread configuration.
7. Execute the deferred test(performance): establish the M4 embedded baseline and entry gate #334 parity matrix for 1, 2, 4, 8, and automatic when the machine supports those configurations. Record unavailable configurations explicitly without fabricating results.
8. Keep GraphForge embedded and CPU-complete. No daemon, network authority, GPU, Python engine, Node engine, or foreign graph runtime is introduced.

Acceptance Criteria

• One documented public Rust policy configures all in-scope execution resources before a GraphForge instance begins work.
• The same declared compute budget is observable in the facade runtime and every DataFusion execution session, with no unbounded or process-global worker pool introduced.
• Memory exhaustion, unsafe spill configuration, unsupported thread counts, and heavy-query admission failures return documented structured outcomes without partial results.
• The test(performance): establish the M4 embedded baseline and entry gate #334 matrix executes for every supported 1/2/4/8/automatic configuration and proves identical schemas, ordering, result fingerprints, errors, cancellation, and limits.
• Automatic mode records the selected policy and retains a low-overhead path for small workloads; timing observations remain hardware-specific evidence rather than flaky CI thresholds.
• Existing fixed-hop demand/cancellation behavior, concurrency gates, publication/recovery tests, and applicable public Rust/Python/Node acceptance remain green.
• Resource-policy documentation names defaults, validation, auto-selection, spill safety, observability, and the distinction between configuration and future kernel acceleration.

BDD Completion Scenarios

Scenario: One embedded instance receives one bounded compute policy

Given a caller constructs GraphForge with an explicit supported resource policy
When the facade creates its runtime and executes relational work
Then runtime workers, query partitions, memory, spill, I/O, and heavy-query admission reflect that policy
And no subsystem silently creates an unbounded competing pool.

Scenario: Thread configurations preserve the public contract

Given one committed graph and invocation from the #334 matrix
When it runs with each supported thread configuration
Then schema, ordering, deterministic result fingerprint, errors, cancellation, and resource limits are identical
And only recorded performance and resource observations may differ.

Scenario: Unsupported resource configuration fails closed

Given a thread, memory, spill, or concurrency setting that cannot be honored safely
When GraphForge validates or applies the policy
Then it returns the documented structured error before producing results
And no partial runtime, spill artifact, or execution state becomes authoritative.

Implementation Notes

Likely surfaces include crates/graphforge-api/src/lib.rs, public construction/configuration types, crates/graphforge-exec/src/lib.rs, DataFusion SessionConfig/runtime environment construction, binding option adapters, concurrency tests, and the #334 evidence harness.

Prefer one immutable normalized policy shared by subsystem adapters. Avoid environment-only configuration and avoid separate knobs whose combined concurrency can exceed the declared process budget.

Observability

Expose safe aggregate diagnostics for requested and normalized policy, active/queued heavy work, DataFusion partitions, memory/spill totals, and selected automatic mode. Do not record query text, parameters, UUIDs, properties, graph contents, local project paths, or sensitive environment details.

Security And Privacy

No network or authorization surface is added. Spill must be disabled or constrained to an approved contained directory, reject traversal and links, enforce declared limits, and clean temporary artifacts after success, failure, or cancellation. Diagnostics must not expose graph data.

Testing

• Unit-test normalization, unavailable counts, overflow, unsafe spill paths, automatic selection, and stable structured errors.
• Integration-test policy projection into Tokio and DataFusion, heavy-query admission, cancellation, memory exhaustion, and spill cleanup.
• Execute the test(performance): establish the M4 embedded baseline and entry gate #334 1/2/4/8/automatic parity matrix through the public Rust facade; retain short deterministic CI coverage and larger manual/scheduled evidence.
• Map Scenario 1 to Rust integration tests, Scenario 2 to the test(performance): establish the M4 embedded baseline and entry gate #334 evidence matrix, and Scenario 3 to resource and recovery tests.
• Run formatting, targeted clippy/tests, and repository-required exact-head CI for every changed surface.

Documentation

Document the public policy, defaults and auto-selection, memory/spill behavior, concurrency semantics, binding projection, metric meanings, and reproducible thread-parity commands.

Non-Goals

• Parallelizing any graph algorithm or adding Rayon work to algorithm kernels.
• Implementing GPU selection or a GPU backend.
• Adding distributed scheduling, a service, or a remote execution authority.
• Making wall-clock thresholds required on noisy shared CI runners.

Related Issues

• test(performance): establish the M4 embedded baseline and entry gate #334 — M4 embedded baseline and entry gate; blocks this issue.
• epic(performance): close M4 embedded performance and scale #335 — canonical M4 closure tracker; this issue blocks it.
• fix(storage): stream adjacency index builds past the 134,217,727-edge Arrow limit #336 — bounded adjacency builder; consumes the resource policy after this issue lands.