feat: migrate all DO SQLite raw SQL to Drizzle ORM by iscekic · Pull Request #684 · Kilo-Org/cloud

iscekic · 2026-02-28T01:45:42Z

Summary

Migrates ~140 raw sql.exec() queries across 8 Durable Objects in 7 workers to use drizzle-orm with the durable-sqlite driver, as specified in docs/migrate-do-sqlite-to-drizzle.md.

Net result: -3818 / +3634 lines (net -184)

Workers migrated

Worker	DO(s)	Tables	Queries	Key changes
`cloudflare-o11y`	AlertConfigDO	2	~10	Simple CRUD → Drizzle select/insert/delete
`cloudflare-session-ingest`	SessionIngestDO, SessionAccessCacheDO	3	~16	ON CONFLICT upserts → `.onConflictDoUpdate()`
`cloudflare-ai-attribution`	AttributionTrackerDO	3	~25	INNER JOIN + RETURNING → `.innerJoin()` + `.returning()`
`cloudflare-webhook-agent-ingest`	TriggerDO	2	~20	Dynamic ALTER TABLE migration → Drizzle migrations
`cloud-agent`	CloudAgentSession	3	~25	Dynamic WHERE builder → Drizzle `and()`/`inArray()`/`gt()`
`cloud-agent-next`	CloudAgentSession	3	~20	Same pattern as cloud-agent
`cloudflare-app-builder`	GitRepositoryDO/SqliteFS	1	~20	Tagged template SQL → Drizzle query builder

Per-worker changes

Each worker gets:

drizzle.config.ts — Drizzle Kit config with dialect: 'sqlite', driver: 'durable-sqlite'
src/db/sqlite-schema.ts — sqliteTable() definitions with indexes
drizzle/ — Generated migration SQL + journal

Each worker loses:

Custom table.ts interpolator utility (where applicable)
Zod-based table definitions in src/db/tables/ (where applicable)
Custom migration systems (migrations.ts rewritten to use Drizzle migrator)
sql-helpers.ts dynamic WHERE clause builders (replaced by Drizzle operators)

Backward compatibility

Generated migration SQL uses CREATE TABLE IF NOT EXISTS / CREATE INDEX IF NOT EXISTS so existing DOs (which already have tables but lack __drizzle_migrations) won't fail when Drizzle runs the initial migration for the first time.

Typecheck

pnpm typecheck passes. All per-worker typechecks show only pre-existing errors in worker-utils/encryption.ts.

Migrate ~140 raw sql.exec() queries across 8 Durable Objects in 7 workers to use drizzle-orm with the durable-sqlite driver. Each worker gets a Drizzle schema (sqlite-schema.ts), generated migrations, and type-safe query builder calls replacing hand-rolled SQL strings. Workers migrated: - cloudflare-o11y (AlertConfigDO) - cloudflare-session-ingest (SessionIngestDO, SessionAccessCacheDO) - cloudflare-ai-attribution (AttributionTrackerDO) - cloudflare-webhook-agent-ingest (TriggerDO) - cloud-agent (events, leases, command-queue) - cloud-agent-next (events, leases) - cloudflare-app-builder (GitRepositoryDO/SqliteFS) Deleted per-worker table.ts interpolator utils, Zod-based table definitions in db/tables/, custom migration systems, and sql-helpers. Added drizzle-kit to the pnpm catalog.

Without this, existing DO instances (which already have tables from the old raw DDL but lack __drizzle_migrations) would fail when Drizzle's migrator tries to run the initial migration.

cloud-agent-next/src/session/queries/events.ts

docs/do-sqlite-drizzle.md

kilo-code-bot · 2026-02-28T01:57:26Z

Code Review Summary

Status: 2 Issues Remaining | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	0

Previous Issues (Resolved ✅)

The following issues from earlier review rounds have been confirmed fixed:

✅ Unused sql import in cloud-agent-next/src/session/queries/events.ts
✅ as cast → satisfies in findByFilters
✅ Materialize-then-yield → yield* rows pattern
✅ Two-query pattern (SELECT COUNT + DELETE) → single DELETE with .returning()
✅ Incorrect migration output paths in docs/do-sqlite-drizzle.md
✅ process_status column missing Drizzle enum option
✅ as ProcessStatus cast removed
✅ Record<string, unknown> → Partial<typeof requestsTable.$inferInsert>
✅ Date.now() called separately in .values() and .set()
✅ Missing blockConcurrencyWhile + migrate() in GitRepositoryDO
✅ Missing void prefix on floating promise
✅ Dead code runMigrations wrapper deleted
✅ Missing .sql rules in wrangler.jsonc for 4 workers

Remaining Issues (click to expand)

WARNING

File	Line	Issue
`cloud-agent/src/session/queries/events.ts`	94	`as StoredEvent` cast in `iterateByFilters` violates project coding style (STRONGLY AVOID `as`). The raw `SqlStorageCursor` returns untyped rows. Consider a runtime validation or Zod parse.
`cloud-agent-next/src/session/queries/events.ts`	112	Same `as StoredEvent` cast issue as `cloud-agent`.

Files Reviewed (117 files)

Config & Rules:

.kilocode/rules/coding-style.md - Updated DO SQLite rule
pnpm-workspace.yaml - Added drizzle-kit to catalog

Documentation:

docs/do-sqlite-drizzle.md - New Drizzle migration workflow doc ✅
docs/migrate-do-sqlite-to-drizzle.md - New migration plan doc ✅

cloud-agent-next (22 files):

drizzle.config.ts - New ✅
src/db/sqlite-schema.ts - New Drizzle schema ✅
src/persistence/CloudAgentSession.ts - Drizzle migration in constructor ✅
src/persistence/migrations.ts - Deleted ✅
src/session/queries/events.ts - Rewritten with Drizzle - 1 remaining issue
src/session/queries/executions.ts - New (KV-based, not SQLite) ✅
src/session/queries/index.ts - Updated exports ✅
src/session/queries/leases.ts - Rewritten with Drizzle ✅
src/utils/sql-helpers.ts - Deleted ✅
src/utils/table.ts - Deleted ✅
src/db/tables/* - Deleted ✅
drizzle/* - Generated migration artifacts ✅
test/integration/session/events.test.ts - Updated ✅
tsconfig.json, wrangler.jsonc, wrangler.test.jsonc - Config updates ✅

cloud-agent (22 files):

Same pattern as cloud-agent-next ✅
src/session/queries/command-queue.ts - New Drizzle query module ✅
1 remaining issue in events.ts

cloudflare-ai-attribution (12 files):

Full migration from table interpolator pattern to Drizzle ✅
src/dos/AttributionTracker.do.ts - Clean rewrite ✅
src/db/sqlite-schema.ts - New schema with enum + check constraints ✅
Deleted src/util/table.ts and src/db/tables/* ✅

cloudflare-app-builder (10 files):

src/git-repository-do.ts - Drizzle migration in constructor ✅
src/git/fs-adapter.ts - Full rewrite from custom tagged template to Drizzle ✅
src/db/sqlite-schema.ts - New schema ✅

cloudflare-o11y (6 files):

src/alerting/AlertConfigDO.ts - Clean Drizzle migration ✅
src/db/sqlite-schema.ts - New schema ✅

cloudflare-session-ingest (8 files):

src/dos/SessionIngestDO.ts - Drizzle migration ✅
src/dos/SessionAccessCacheDO.ts - Drizzle migration ✅
src/db/sqlite-schema.ts - New schema ✅

cloudflare-webhook-agent-ingest (10 files):

src/dos/TriggerDO.ts - Full rewrite from table interpolator to Drizzle ✅
src/db/sqlite-schema.ts - New schema with enum + check constraint ✅
src/db/types.ts - New type definitions ✅
Deleted src/util/table.ts and src/db/tables/* ✅

Generated/Lock files: pnpm-lock.yaml, migration SQL, snapshots, journals - Skipped

Overall Assessment: This is a well-executed, large-scale migration (~140 raw SQL queries across 8 DOs in 7 workers) to Drizzle ORM. The migration is consistent across all workers, follows the documented pattern, and previous review feedback has been addressed. The only remaining issues are the two as StoredEvent casts in the cursor-based iteration paths, which are a pragmatic trade-off for lazy iteration performance.

Fix these issues in Kilo Cloud

…igrate-do-sqlite-to-drizzle # Conflicts: # cloudflare-session-ingest/package.json # pnpm-lock.yaml

…asts, use single-query deletes, fix doc paths

cloudflare-webhook-agent-ingest/src/db/sqlite-schema.ts

cloudflare-webhook-agent-ingest/src/dos/TriggerDO.ts

cloudflare-app-builder/src/git/fs-adapter.ts

cloudflare-app-builder/src/git-repository-do.ts

… consistent migrations

cloudflare-app-builder/src/git-repository-do.ts

Co-authored-by: kilo-code-bot[bot] <240665456+kilo-code-bot[bot]@users.noreply.github.com>

cloud-agent/src/persistence/migrations.ts

cloud-agent/wrangler.jsonc

…configs

Drizzle's durable-sqlite driver doesn't expose a lazy cursor API, so .all() was loading all matching rows into memory. Replace with batched keyset pagination (500 rows/batch) to bound memory usage while still allowing callers to break early from the generator.

The table query interpolator pattern was removed by the Drizzle ORM migration. Replace with a reference to the Drizzle conventions doc.

Use Drizzle's toSQL() to build type-safe queries, then execute via the raw SqlStorageCursor API. This gives us: - rowsWritten for delete counts (no row materialization) - True lazy cursor iteration in iterateByFilters (no .all()) Each factory function now accepts a rawSql parameter alongside db.

…zzle

cloud-agent/src/session/queries/events.ts

…zzle

cloud-agent-next/src/session/queries/events.ts

…zzle

iscekic · 2026-03-01T21:54:49Z

Split into base + 8 worker PRs.

iscekic self-assigned this Feb 28, 2026

iscekic added 2 commits February 28, 2026 02:48

docs: add DO SQLite Drizzle migration workflow guide

0c2154b

fix: add IF NOT EXISTS to all Drizzle migration SQL

49fc9ae

Without this, existing DO instances (which already have tables from the old raw DDL but lack __drizzle_migrations) would fail when Drizzle's migrator tries to run the initial migration.

kilo-code-bot bot reviewed Feb 28, 2026

View reviewed changes

iscekic added 2 commits February 28, 2026 03:03

Merge remote-tracking branch 'origin/chore/worker-utils' into chore/m…

22de879

…igrate-do-sqlite-to-drizzle # Conflicts: # cloudflare-session-ingest/package.json # pnpm-lock.yaml

fix: address PR review comments - remove unused imports, replace as c…

12d2ec0

…asts, use single-query deletes, fix doc paths

kilo-code-bot bot reviewed Feb 28, 2026

View reviewed changes

fix: address remaining PR review comments - enum schema, type safety,…

8f698e8

… consistent migrations

kilo-code-bot bot reviewed Mar 1, 2026

View reviewed changes

cloudflare-app-builder/src/git-repository-do.ts Outdated Show resolved Hide resolved

chore: add void operator to match other DOs

9928981

Co-authored-by: kilo-code-bot[bot] <240665456+kilo-code-bot[bot]@users.noreply.github.com>

kilo-code-bot bot reviewed Mar 1, 2026

View reviewed changes

cloud-agent/src/persistence/migrations.ts Outdated Show resolved Hide resolved

cloud-agent/wrangler.jsonc Show resolved Hide resolved

iscekic added 8 commits March 1, 2026 14:53

fix: delete dead migrations wrappers, add .sql rules to all wrangler …

0751e59

…configs

chore: remove unrelated planning docs from this PR

ede8119

chore: update coding-style.md to replace obsolete SQL interpolator rules

1e4f4e9

The table query interpolator pattern was removed by the Drizzle ORM migration. Replace with a reference to the Drizzle conventions doc.

chore: standardize getLatestEventId alias to maxId across workers

e1ff6ef

test: update events.test.ts to use new (db, rawSql) factory signature

47511d2

Merge branch 'chore/worker-utils' into chore/migrate-do-sqlite-to-dri…

55e421c

…zzle

kilo-code-bot bot reviewed Mar 1, 2026

View reviewed changes

cloud-agent/src/session/queries/events.ts Show resolved Hide resolved

Merge branch 'chore/worker-utils' into chore/migrate-do-sqlite-to-dri…

a0d7da2

…zzle

kilo-code-bot bot reviewed Mar 1, 2026

View reviewed changes

cloud-agent-next/src/session/queries/events.ts Show resolved Hide resolved

Merge branch 'chore/worker-utils' into chore/migrate-do-sqlite-to-dri…

8697e1e

…zzle

Base automatically changed from chore/worker-utils to main March 1, 2026 21:10

iscekic closed this Mar 1, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: migrate all DO SQLite raw SQL to Drizzle ORM#684

feat: migrate all DO SQLite raw SQL to Drizzle ORM#684
iscekic wants to merge 17 commits intomainfrom
chore/migrate-do-sqlite-to-drizzle

iscekic commented Feb 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kilo-code-bot bot commented Feb 28, 2026 •

edited

Loading

WARNING

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iscekic commented Mar 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

iscekic commented Feb 28, 2026

Summary

Workers migrated

Per-worker changes

Backward compatibility

Typecheck

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kilo-code-bot bot commented Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Summary

Overview

Previous Issues (Resolved ✅)

WARNING

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

iscekic commented Mar 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

kilo-code-bot bot commented Feb 28, 2026 •

edited

Loading