Vectorizer

High-performance vector database and search engine in Rust for semantic search, document indexing, and AI applications. Ships as a Cargo workspace (5 crates) with binary RPC + HTTP transports, a React dashboard, and native SDKs for Rust, Python, TypeScript, Go, and C#.

✨ Key Features

Transport & API

• VectorizerRPC (default, port 15503) — binary MessagePack over TCP, multiplexed connection pool. See wire spec.
• REST API (port 15002) — universal HTTP fallback, powers the dashboard and any caller that doesn't speak raw TCP.
• gRPC — Qdrant-compatible service.
• GraphQL — full REST parity with async-graphql + GraphiQL playground.
• MCP — 31 focused tools for AI model integration (Cursor, Claude Desktop, etc.).
• UMICP Protocol — native JSON types + tool discovery endpoint.

Performance

• SIMD acceleration — AVX2-optimized vector ops with runtime CPU detection (5-10x faster).
• Metal GPU — macOS Apple Silicon via hive-gpu 0.2; logs render real device name, driver, VRAM.
• Sub-3ms search (CPU) / <1ms (GPU) via HNSW indexing.
• 4-5x faster than Qdrant in head-to-head benchmarks (0.16-0.23ms vs 0.80-0.87ms avg latency).

Storage

• .vecdb unified format — 20-30% space savings, automatic snapshots.
• Memory-mapped storage — datasets larger than RAM, efficient OS paging.
• Product Quantization — 64x memory reduction with minimal accuracy loss.
• Scalar Quantization + cache hit ratio metrics.

High Availability & Scaling

• Raft consensus via openraft (pinned =0.10.0-alpha.30) — automatic leader election in 1-5s, write-redirect via HTTP 307, WAL-backed durable replication, DNS discovery for Kubernetes headless services. See the HA on Kubernetes runbook.
• Master-Replica — TCP streaming replication with full/partial sync, exponential reconnect backoff (5s→60s).
• Distributed sharding — horizontal scaling with automatic routing; distributed hybrid search via RemoteHybridSearch RPC with dense-only fallback for mixed-version clusters.
• HiveHub cluster mode — multi-tenant with quotas, usage tracking, tenant isolation, mandatory MMap storage, 1GB cache cap.

Search

• Semantic similarity — Cosine, Euclidean, Dot Product.
• Hybrid search — Dense + Sparse with Reciprocal Rank Fusion (RRF).
• Intelligent search — query expansion, semantic reranking.
• Multi-collection search across projects.
• Graph relationships — automatic edge discovery, neighbor exploration, shortest-path finding.

Embeddings & Docs

• Built-in providers — TF-IDF, BM25, FastEmbed, BERT, MiniLM, custom models. embedding_provider (on POST /collections) and model (on POST /embed) are honoured contracts as of v3.4.0 (issue #306) — unknown providers / models return 400 unsupported_provider / 400 unsupported_model with the available list; no more silent BM25-512 coercion. Discover registered providers via GET /stats.providers or the list_providers MCP tool. See docs/users/guides/EMBEDDINGS.md.
• Multilingual & synonym search — with the -fastembed image, embedding.additional_models: ["fastembed:multilingual-e5-small"] registers the E5 model next to BM25, and collections created with "embedding_provider": "fastembed:multilingual-e5-small" (384 dims) match by meaning across languages ("automóvel" finds "carro"); E5 query: / passage: prefixes are applied automatically. See docs/users/guides/EMBEDDINGS.md.
• Document conversion — PDF, DOCX, XLSX, PPTX, HTML, XML, images (14 formats).
• Qdrant API compatibility — Snapshots, Sharding, Cluster Management, Query (with prefetch), Search Groups, Matrix, Named Vectors (partial), PQ/Binary quantization config.
• Summarization — extractive, keyword, sentence, abstractive (OpenAI GPT).

Security

• JWT + API Key authentication with RBAC, scoped API keys (per-collection permissions), atomic key rotation with grace window, RFC 7662 token introspection, admin audit log (in-memory ring + daily-rotated JSONL).
• API key usage metrics — per-key usage_count (atomic, lock-free) plus a 30-day per-day ring buffer surfaced via GET /auth/keys/{id}/usage.
• Permission update without rotation — PUT /auth/keys/{id}/permissions swaps permissions/scopes while keeping key_hash/id/user_id/created_at immutable.
• Hardened dashboard cookies + CSRF — login/refresh emit vectorizer_session (HttpOnly; Secure; SameSite=Strict) plus a sibling XSRF-TOKEN cookie; require_csrf_middleware guards every mutating /auth/* and /admin/* request. auth.cookies.insecure_dev opt-out for plain-HTTP loopback dev (boot rejects it on 0.0.0.0).
• Loopback dev-mode auth bypass — auth.dev_mode_skip_loopback short-circuits credential validation as local-dev-admin and stamps X-Vectorizer-Dev-Mode: true on every response. Boot fails on any non-loopback host.
• JWT secret is mandatory — boot refuses to start with empty / default / <32 char secrets when auth is enabled.
• First-run root credentials written to {data_dir}/.root_credentials (0o600), never logged.
• Payload encryption — optional ECC-P256 + AES-256-GCM, zero-knowledge, per-collection policies (docs).
• TLS 1.2/1.3 with mTLS, configurable cipher suites, ALPN.
• Per-API-key rate limiting with tiers + overrides.
• Path-traversal guard on file discovery; canonicalized base, symlink-escape refusal.

UI

• Web Dashboard — React + TypeScript; JWT login, graph CRUD (edges, neighbors, paths), collection management, API sandbox, setup wizard with glassmorphism design. Embedded in the binary (~26MB, no external assets needed).
• Desktop GUI — Electron + vis-network for visual database management.

🎉 Latest Release: v3.8.2

The -fastembed image is now published, with a multilingual model baked in — ghcr.io/hivellm/vectorizer:3.8.2-fastembed, alongside the unchanged default ghcr.io/hivellm/vectorizer:3.8.2. See CHANGELOG.md for the full breakdown and upgrade notes.

Added — -fastembed image published on GHCR

• ghcr.io/hivellm/vectorizer:3.8.2-fastembed, public, no pull secret, built by the same GitHub Actions workflow with GitHub credentials only — no Docker Hub account needed. Its runtime moved off the Docker Hardened Images base to public gcr.io/distroless/cc-debian13:nonroot (glibc, pinned by digest). See HA runbook §11.

Changed — -fastembed image bakes multilingual-e5-small, outside /data

• The image bakes fastembed:multilingual-e5-small at /vectorizer/models/fastembed and points the new VECTORIZER_FASTEMBED_CACHE_DIR env var at it. Living outside /data means a Kubernetes PVC mounted there no longer hides the model, so pods no longer download it at boot.

Changed — collections default to the provider's native dimension

• POST /collections and RPC collections.create without a dimension now use the resolved embedding provider's native width (e.g. 384 for fastembed:multilingual-e5-small) instead of always defaulting to 512.

---

v3.8.1 and v3.7.x highlights (previous releases)

• 3.8.1 — the 3.8.0 code made publishable (ghcr.io/hivellm/vectorizer:3.8.1; no 3.8.0 image was ever pushed); multilingual semantic search via the fastembed provider; collections embed with their own provider; HA followers converge on the leader and survive restarts.
• 3.7.2 — HA replication no longer stops after a pod regains Raft leadership (Address in use on :7001); rustls 0.23.45 (RUSTSEC-2026-0285).
• 3.7.1 — the default image is FROM scratch: 30 base-OS CVEs to zero, and no shell inside (docker exec … sh no longer works); the healthcheck probes /ready.
• 3.7.0 — embedding_provider: "none" for pre-computed vectors of any width; BREAKING: embedding_provider is nullable on collection responses.

Server-side at v3.8.2. The Rust SDK tracks server versioning; TypeScript, Python, Go, and C# SDKs are also on v3.8.2.

---

For prior releases (v3.6.1 and earlier) see CHANGELOG.md.

🚀 Quick Start

Install Script (Linux/macOS)

curl -fsSL https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.sh | bash

Installs CLI + systemd service. Commands: sudo systemctl {status|restart|stop} vectorizer, sudo journalctl -u vectorizer -f.

Install Script (Windows)

powershell -c "irm https://raw.githubusercontent.com/hivellm/vectorizer/main/scripts/install.ps1 | iex"

Installs CLI + Windows Service (requires Admin). Commands: Get-Service Vectorizer, {Start|Stop|Restart}-Service Vectorizer.

Docker

docker run -d \
  --name vectorizer \
  -p 15002:15002 -p 15503:15503 \
  -v vec-data:/data \
  -e VECTORIZER_AUTH_ENABLED=true \
  -e VECTORIZER_ADMIN_USERNAME=admin \
  -e VECTORIZER_ADMIN_PASSWORD=your-secure-password \
  -e VECTORIZER_JWT_SECRET=$(openssl rand -hex 64) \
  --restart unless-stopped \
  ghcr.io/hivellm/vectorizer:3.8.2

ghcr.io/hivellm/vectorizer:3.8.2 is the default image (FROM scratch, BM25 only, no shell), public — no registry login needed. The -fastembed variant (glibc, ONNX Runtime, for dense and multilingual models) is published as ghcr.io/hivellm/vectorizer:3.8.2-fastembed, also public — see the HA runbook §11 for configuration.

Starting in hivehub/vectorizer:3.4.0 the image defaults VECTORIZER_DATA_DIR=/data, so a single --volume vec-data:/data mount captures every collection, auth key, JWT secret, and snapshot. docker compose up -d --force-recreate vectorizer is now safe; see docs/users/configuration/DATA_DIRECTORY.md for the 3.3.0 migration runbook (issue #300).

Docker Compose with profiles:

cp config/.env.example .env
# Edit .env with your credentials
docker compose --profile default up -d          # standalone
docker compose --profile dev up -d              # dev overlay
docker compose --profile ha up -d               # Raft cluster
docker compose --profile hub up -d              # multi-tenant

Profiles are mutually exclusive on host port 15002.

Images: GHCR (primary) · Docker Hub (mirror)

High Availability on Kubernetes

A 3-pod Raft cluster (automatic leader election, leader-to-follower replication, safe rolling updates) runs from the manifests in deploy/k8s/ with ghcr.io/hivellm/vectorizer:3.8.2. Writes go to the leader — followers answer them with HTTP 307 and the leader's address; reads are served by any pod. Follow the HA on Kubernetes runbook for install, validation, failover and upgrades (clusters on ≤ 3.7.2 need one all-pods restart when moving to 3.8.0).

Build from Source

git clone https://github.com/hivellm/vectorizer.git
cd vectorizer

cargo build --release                          # Basic
cargo build --release --features hive-gpu      # macOS Metal
cargo build --release --features full          # All features
./target/release/vectorizer

Keeping target/ bounded. Cargo never GCs target/ — stale rlibs accumulate until you nuke them. This repo already pins [profile.dev] debug = "line-tables-only" + [profile.release] strip = true + CARGO_INCREMENTAL=0 on CI; on a developer box, run bash scripts/sweep-target.sh (or pwsh scripts/sweep-target.ps1 on Windows) weekly to drop stale artifacts without losing the incremental hot set. Full runbook + scheduler examples in docs/development/rust-target-hygiene.md (issue #320).

Access Points

Surface	URL	Notes
VectorizerRPC (primary)	vectorizer://localhost:15503	Binary MessagePack over TCP — see operator guide
REST API	http://localhost:15002	Universal HTTP fallback
Web Dashboard	http://localhost:15002/dashboard/	React UI, embedded in binary
MCP Server	http://localhost:15002/mcp	31 tools for AI agents
GraphQL	http://localhost:15002/graphql	GraphiQL at /graphql
UMICP Discovery	http://localhost:15002/umicp/discover
Health Check	http://localhost:15002/health

Upgrading from v2.x? RPC is now on by default on port 15503. REST is unchanged. If you can't expose the new port, set rpc.enabled: false. See v3.x migration guide.

Configuration

Configs live under config/:

config/
├── config.yml             # Base config (your deployment)
├── config.example.yml     # Reference
├── modes/
│   ├── dev.yml            # Layered override: verbose logs, loopback, watcher on
│   └── production.yml     # Layered override: warn logs, larger threads/cache, zstd, scheduled snapshots
└── presets/               # Standalone full configs (legacy style)
    ├── production.yml
    ├── cluster.yml
    ├── hub.yml
    └── development.yml

Layered loader (recommended):

VECTORIZER_MODE=production ./target/release/vectorizer

Merges config/modes/production.yml over config/config.yml. Typos in the mode override fail fast at boot.

Authentication

Auth is enabled by default in Docker. Default creds — change in production.

# Login
curl -X POST http://localhost:15002/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","password":"admin"}'

# JWT in requests
curl http://localhost:15002/collections \
  -H "Authorization: Bearer YOUR_JWT_TOKEN"

# Create API key (JWT required)
curl -X POST http://localhost:15002/auth/keys \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_JWT_TOKEN" \
  -d '{"name":"Production","permissions":["read","write"],"expires_in_days":90}'

# API key in requests (NO Bearer prefix)
curl http://localhost:15002/collections \
  -H "Authorization: YOUR_API_KEY"

Method	Header	Use case
JWT	Authorization: Bearer <token>	Dashboard, short-lived sessions
API Key	Authorization: <key>	MCP, CLI, long-lived integrations

Production must set:

• VECTORIZER_JWT_SECRET — ≥32 chars, not the historical default. Boot aborts otherwise.
• VECTORIZER_ADMIN_PASSWORD — strong, ≥32 chars.

First-run root credentials are written to {data_dir}/.root_credentials (0o600), never printed to stdout. Read and delete after first login.

See Docker Authentication Guide and Security Policy.

📊 Performance

Metric	Value
Search latency (CPU)	< 3ms
Search latency (Metal GPU)	< 1ms
Throughput	4,400-6,000 QPS (vs Qdrant 1,100-1,300)
Storage reduction	20-30% (.vecdb) + PQ 64x
MCP tools	31
Document formats	14

Benchmark vs Qdrant

• Search: 4-5x faster (0.16-0.23ms vs 0.80-0.87ms avg latency).
• Insert: Fire-and-forget pattern, configurable batch / body limits, background processing.
• Scenarios: Small (1K) / Medium (5K) / Large (10K) vectors × dimensions 384 / 512 / 768.

See Benchmark Documentation.

🔄 Feature Comparison

Feature	Vectorizer	Qdrant	pgvector	Pinecone	Weaviate	Milvus	Chroma
Core
Language	Rust	Rust	C	C++/Go	Go	C++/Go	Python
License	Apache 2.0	Apache 2.0	PostgreSQL	Proprietary	BSD	Apache 2.0	Apache 2.0
APIs
REST	✅	✅	via PG	✅	✅	✅	✅
gRPC (Qdrant-compat)	✅	✅	❌	✅	✅	✅	❌
GraphQL	✅ + GraphiQL	❌	❌	❌	✅	❌	❌
MCP	✅ 31 tools	❌	❌	❌	❌	❌	❌
Binary RPC	✅ MessagePack	❌	❌	❌	❌	❌	❌
SDKs	Rust, Python, TS, Go, C#	All	All	Most	Most	Most	Python
Performance
Search latency	< 3ms CPU / < 1ms GPU	1-5ms	5-50ms	50-100ms	10-50ms	5-20ms	10-100ms
SIMD	✅ AVX2	✅	✅	✅	❌	✅	❌
GPU	✅ Metal	✅ CUDA	❌	✅ Cloud	❌	✅ CUDA	❌
Storage
HNSW	✅	✅	✅	✅	✅	✅	✅
PQ (64x)	✅	✅	❌	✅	❌	✅	❌
Scalar Quantization	✅	✅	❌	✅	❌	✅	❌
MMap	✅	✅	✅	❌	✅	✅	❌
Advanced
Graph Relationships	✅ auto + GUI	❌	❌	❌	✅	❌	❌
Document Processing	✅ 14 formats	❌	❌	❌	✅	❌	✅
Hybrid Search	✅	✅	✅	✅	✅	✅	❌
Query Expansion	✅	❌	❌	❌	❌	❌	❌
Qdrant API compat	✅ + migration	N/A	❌	❌	❌	❌	❌
Scaling
Sharding	✅	✅	via PG	✅ Cloud	✅	✅	❌
Replication	✅ Raft + Master-Replica	✅	via PG	✅ Cloud	✅	✅	❌
Management
Dashboard	✅ React + graph GUI	✅ basic	pgAdmin	✅ Cloud	✅	✅	✅ basic
Desktop GUI	✅ Electron	❌	❌	❌	❌	❌	❌
Security
JWT + API Keys	✅	✅	via PG	✅ Cloud	✅	✅	✅
Payload Encryption	✅ ECC-P256 + AES-GCM	❌	via PG	✅ Cloud	❌	❌	❌

Key Differentiators

• MCP integration (31 tools) — native AI-agent protocol.
• Graph relationships — auto-discovery + full GUI (edges, path-finding, neighbor exploration).
• GraphQL — full REST parity + GraphiQL.
• Document processing — 14 formats built in.
• Qdrant compatibility — full API + migration tools.
• Performance — 4-5x faster than Qdrant in benchmarks.
• Binary RPC default — MessagePack over TCP on port 15503 for low-overhead client traffic.
• Complete SDK coverage — Rust, Python, TypeScript (+JS), Go, C# — all on v3.5.0.

Best fit: AI apps needing MCP, document ingestion, graph relationships, and sub-ms search with an embedded dashboard.

🎯 Use Cases

• RAG systems — semantic search with automatic document conversion.
• Document search — PDFs, Office, web content.
• Code analysis — semantic code navigation.
• Knowledge bases — enterprise multi-format search.

🔧 MCP Integration

Cursor / Claude Desktop config:

{
  "mcpServers": {
    "vectorizer": {
      "url": "http://localhost:15002/mcp",
      "type": "streamablehttp"
    }
  }
}

Available Tools (31)

Core operations (9) list_collections · create_collection · get_collection_info · insert_text · get_vector · update_vector · delete_vector · search · multi_collection_search

Advanced search (4) search_intelligent (query expansion) · search_semantic (reranking) · search_extra (combined) · search_hybrid (dense + sparse RRF)

Discovery & files (7) filter_collections · expand_queries · get_file_content · list_files · get_file_chunks · get_project_outline · get_related_files

Graph (8) graph_list_nodes · graph_get_neighbors · graph_find_related · graph_find_path · graph_create_edge · graph_delete_edge · graph_discover_edges · graph_discover_status

Maintenance (3) list_empty_collections · cleanup_empty_collections · get_collection_stats

Cluster-management operations are REST-only for security.

📦 Client SDKs

Server-side at v3.8.2. The Rust SDK tracks server versioning; the TypeScript, Python, Go, and C# SDKs are also on v3.8.2. The TypeScript SDK ships compiled CJS + ESM — usable from plain JavaScript, no separate JS package needed.

SDK	Install
Python	pip install vectorizer-sdk
TypeScript / JS	npm install @hivehub/vectorizer-sdk
Rust	cargo add vectorizer-sdk
C#	dotnet add package Vectorizer.Sdk (REST) · Vectorizer.Sdk.Rpc (RPC)
Go	go get github.com/hivellm/vectorizer-sdk-go

Every SDK accepts both vectorizer://host[:port] (RPC, default port 15503) and http(s)://host[:port] (REST) URLs through the same endpoint parser.

🔄 Qdrant Migration

• Config migration — parse Qdrant YAML/JSON → Vectorizer format.
• Data migration — export from Qdrant, import into Vectorizer.
• Validation — integrity + compatibility checks.
• REST compatibility — full Qdrant API at /qdrant/*.

use vectorizer::migration::qdrant::{QdrantDataExporter, QdrantDataImporter};

let exported = QdrantDataExporter::export_collection(
    "http://localhost:6333",
    "my_collection"
).await?;

let result = QdrantDataImporter::import_collection(&store, &exported).await?;

See Qdrant Migration Guide.

☁️ HiveHub Cloud

Multi-tenant cluster mode integration with HiveHub.Cloud.

• Tenant isolation — owner-scoped collections.
• Quota enforcement — collections / vectors / storage per tenant.
• Usage tracking — automatic reporting.
• User-scoped backups.

hub:
  enabled: true
  api_url: "https://api.hivehub.cloud"
  tenant_isolation: "collection"
  usage_report_interval: 300

export HIVEHUB_SERVICE_API_KEY="your-service-api-key"

Cluster-mode requirements (enforced at boot):

Requirement	Default
MMap storage (Memory storage rejected)	Enforced
Max cache memory across all caches	1 GB
File watcher	Disabled
Strict config validation	Enabled

cluster:
  enabled: true
  node_id: "node-1"
  memory:
    max_cache_memory_bytes: 1073741824
    enforce_mmap_storage: true
    disable_file_watcher: true
    strict_validation: true

See HiveHub Integration and Cluster Memory Limits.

🏗️ Workspace Layout

crates/
├── vectorizer-core/       # Foundation: error, codec, quantization, simd, compression, paths
├── vectorizer-grpc/       # tonic-generated gRPC types (first-party, cluster, Qdrant-compatible)
├── vectorizer/            # Engine (umbrella): db, embedding, models, cache, persistence, search, ...
├── vectorizer-server/     # Transport: HTTP / gRPC / MCP / RPC + binary
└── vectorizer-cli/        # CLI binaries
sdks/rust/                 # Rust SDK — RPC transport via the shared `thunder-rpc` crate

Runtime directories resolve to platform-standard locations (~/.local/share/vectorizer/ on Linux, ~/Library/Application Support/vectorizer/ on macOS, %APPDATA%\vectorizer\ on Windows), overridable via VECTORIZER_DATA_DIR / VECTORIZER_LOGS_DIR.

📚 Documentation

• User Documentation — install + tutorials
• API Reference — REST
• VectorizerRPC Spec — wire protocol
• RPC Operator Guide
• Configuration — layered loader
• Bulk-upsert Backpressure Runbook — 429 / Retry-After, vocab-build cap, ops metrics (#263)
• v3.x Migration — RPC-default rollout
• Dashboard Integration
• Qdrant Compatibility
• HiveHub Integration
• Cluster Memory Limits
• MCP Guide
• Encryption
• Technical Specs — architecture, performance, implementation

📄 License

Apache License 2.0 — see LICENSE.

🤝 Contributing

See CONTRIBUTING.md.