[stephenhungg]

Summary

gbrain embed --all currently processes pages sequentially, making one embedBatch call at a time. On real-world corpora this is painfully slow — on a 20k-chunk personal knowledge base I measured ~2 chunks/sec, which meant embedding from scratch took ~2.5 hours.

This PR parallelizes the loop with a sliding worker pool. Same workload now runs at ~60 chunks/sec, finishing in ~8 minutes. That's a ~30x speedup, entirely bottlenecked on OpenAI's embedding API rather than on the client.

Design

• N workers share a mutable nextIdx queue; each worker pulls the next page as soon as it finishes its current one.
• Default concurrency: 20. Sits safely under OpenAI's tier-1 embedding RPM limits (3000+/min ≈ 50/sec) and is gentle on postgres connection pools.
• Tunable via GBRAIN_EMBED_CONCURRENCY env var for users on higher tiers or tighter infra.
• Error handling unchanged: per-page errors are logged and don't abort the run.
• Progress counter (processed/total pages, embedded chunks) still works — writes are just interleaved across workers.

Why a sliding pool over fixed-window Promise.all?

Per-page chunk counts are uneven (some pages have 1 chunk, others 50). A fixed window of Promise.all chunks of size N leaves fast workers idling while the slowest in the window finishes. A sliding pool keeps every worker saturated, which matters more the more skewed the distribution is.

Benchmarks

Tested on a real 19,507-page / 20,693-chunk gbrain import (life-wiki):

metric	before	after
throughput	~2 chunks/sec	~60 chunks/sec
wall time (full embed)	~2.5 hours	~8 minutes
concurrency	1	20 (default)

No observed rate-limit errors from OpenAI at concurrency=20 on a tier-1 key.

Tests

Adds test/embed.test.ts with mocked embedBatch + Proxy-based mock engine (matching the pattern in test/import-file.test.ts):

• concurrent execution — asserts max concurrent embedBatch calls is > 1 and <= CONCURRENCY limit
• serial fallback — GBRAIN_EMBED_CONCURRENCY=1 keeps max concurrent at exactly 1
• --stale skip — pages whose chunks are already embedded don't trigger an embedBatch call

$ bun test test/embed.test.ts
 3 pass
 0 fail
 6 expect() calls

Full suite still green: 259 pass, 0 fail.

Test plan

• bun test — all existing tests pass
• bun test test/embed.test.ts — new concurrency tests pass
• Manual run against real 20k-chunk corpus — completed in ~8 min, no errors, 100% coverage verified
• Verified hybrid search still returns correct results after parallel embed

[garrytan]

Included in fix wave PR #65 (v0.9.1). 30x embedAll speedup landed with the sliding worker pool. Thanks stephenhungg! 🙏