[Kaap10]

Closes #80

Summary

Implements an incremental, in-memory space-partitioned vector indexing prototype (SPFreshHotIndex) and dynamic rebalancer (SPFreshRebalancer) following the SPFresh architecture (Zhang et al., NeurIPS 2023), alongside an architectural design document and a comprehensive test suite.

Acceptance Criteria Checklist

• Design note: Added docs/designs/spfresh_hot_index.md documenting mathematical formulation, spherical 2-means split protocol ($N_{max}$), nearest-neighbor merge protocol ($N_{min}$), normalized centroid drift tracking ($\Delta_{drift}$), per-partition locking hierarchy, and hot-to-cold tier persistence lifecycle.
• Prototype: Implemented src/dynavec/spfresh.py (Partition, SPFreshHotIndex, SPFreshRebalancer, SPFreshConfig) in pure Python + NumPy with zero external dependencies.

Key Technical Details

1. Dynamic Bipartitioning: Fast spherical 2-means split with unit-norm projection and farthest-first seed initialization when partitions exceed max_partition_size.
2. Dynamic Merging: Deferred merge to nearest neighbor centroid when partitions drop below min_partition_size with overflow protection.
3. Centroid Drift & Migration: Normalized drift tracking with automated boundary vector migration during maintenance cycles.
4. Fine-Grained Concurrency: Per-partition RLock enabling non-blocking query snapshots alongside active rebalancing.

Test Coverage

• tests/test_spfresh.py: 27 unit, stress, recall benchmark, and concurrency tests (all passing).
• Full test suite: 213 passed, 2 skipped (100% pass rate).
• Linting: ruff check passes cleanly with 0 errors.

---

Please let me know if any changes are needed.

[codeforstartups]

Verified locally (213 passed, ruff clean). Substantial but fully additive SPFresh hot-index module (Partition / SPFreshHotIndex / SPFreshRebalancer + config validation) with a thorough 660-line test suite and a design doc — no changes to existing modules beyond the __init__ export, so zero regression risk. Implements the SPFresh-style incremental rebalancing roadmap item #80. Great work @Kaap10!