Preliminary "foot in the door" SQLite support

Here, add a partial implementation for SQLite, specifically the driver
operations for `river_migration`, `river_leader`, `river_queue`, and
most of `river_job` (currently missing the two worst queries). Getting
to a fully functional SQLite driver has turned out to be a huge project,
so I'm kind of hoping this change can get our foot in the door so to
speak, and then let me iterate to a fully functional SQLite driver
incrementally.

The main reason this turned out to be quite an effort is that SQLite,
although nominally supporting a lot of the same syntax Postgres does,
just has a lot of unexpected limitations and quirks that all need to be
tracked down separately and a workaround added. For example:

* SQLite doesn't really have data types. Everything is either an
  integer, real, text, or blob, and anything more sophisticated just has
  to be shoehorned into one of these types. For example, there's no
  boolean. Despite the fanfare recently, there isn't even a jsonb. Jsonb
  gets pushed into a blob.

* The most annoying missing data type is a timestamp. Date/times are
  stored as either Unix integers or strings, and SQLite provides a
  number of obnoxious, error-prone, built-ins to work with them.

* No arrays.

* No modification operations in CTEs. (Although counting my lucky stars
  that CTEs are supported at least.)

* No listen/notify.

Aside from that though, we had to get some testing infrastructure in
place, and our testing has traditionally been quite affixed to Postgres,
which would've made a lot of changes necessary. However, the schema
additions in #848 do help to make this quite a bit easier with the
driver-based `TestSchema` and `TestTx` functions, and because they raise
schema automatically, it doesn't complicate test setup instructions in
any way by adding extra steps.

A few caveats:

* I've found that the combination of SQLite's reduced capabilities +
  sqlc's bugginess [1] make bit batch insert/update operations basically
  impossible (I tried every workaround I could possibly thin of), so
  these are done in loops of individual operations instead. I *think*
  this is okay for now though. For one, sqlc will hopefully get these
  limitations fixed eventually, and for two, SQLite databases will often
  be running locally, meaning the round trip cost per operation is much
  lower than what we'd see in a hosted Postgres somewhere.

* It's not totally clear that having a SQLite implementation will be
  "worth it enough" in the long run given that it will add some
  difficulty to non-trivial future database-related operations. My hope
  is that as it's in a prerelease state, we can gauge how bad it is to
  keep up to date. If it feels like way more effort than it's worth, we
  can still axe it before it ever becomes a functional driver.

[1] sqlc-dev/sqlc#3802 (comment)

Author: brandur

.gitignore

@@ -2,3 +2,4 @@
 /river
 /internal/cmd/riverbench/riverbench
 /internal/cmd/testdbman/testdbman
+/sqlite/

CHANGELOG.md

@@ -7,10 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Preliminary River driver for SQLite (`riverdriver/riversqlite`). This driver is not yet complete and no attempt should be made to use it yet because it won't work. Implementation is ongoing. [PR #870](https://github.com/riverqueue/river/pull/870).
+
 ### Changed
 
 - Job kinds must comply to a format of `\A[\w][\w\-\[\]<>\/.·:+]+\z`, mainly in an attempt to eliminate commas and spaces to make format more predictable for an upcoming search UI. This check can be disabled for now using `Config.SkipJobKindValidation`, but this option will likely be removed in a future version of River. [PR #879](https://github.com/riverqueue/river/pull/879).
 
+### Fixed
+
+- Resuming an already unpaused queue is now fully an no-op, and won't touch the row's `updated_at` like it (unintentionally) did before. [PR #870](https://github.com/riverqueue/river/pull/870).
+
 ## [0.21.0] - 2025-05-02
 
 ⚠️ Internal APIs used for communication between River and River Pro have changed. If using River Pro, make sure to update River and River Pro to latest at the same time to get compatible versions. River v0.21.0 is compatible with River Pro v0.13.0.

Makefile

@@ -28,6 +28,7 @@ generate/migrations: ## Sync changes of pgxv5 migrations to database/sql
 generate/sqlc: ## Generate sqlc
 	cd riverdriver/riverdatabasesql/internal/dbsqlc && sqlc generate
 	cd riverdriver/riverpgxv5/internal/dbsqlc && sqlc generate
+	cd riverdriver/riversqlite/internal/dbsqlc && sqlc generate
 
 # Looks at comments using ## on targets and uses them to produce a help output.
 .PHONY: help

docs/development.md

@@ -51,6 +51,7 @@ To run programs locally outside of tests, create and raise a development databas
     git tag riverdriver/$VERSION -m "release riverdriver/$VERSION"
     git tag riverdriver/riverpgxv5/$VERSION -m "release riverdriver/riverpgxv5/$VERSION"
     git tag riverdriver/riverdatabasesql/$VERSION -m "release riverdriver/riverdatabasesql/$VERSION"
+    git tag riverdriver/riversqlite/$VERSION -m "release riverdriver/riversqlite/$VERSION"
     git tag rivershared/$VERSION -m "release rivershared/$VERSION"
     git tag rivertype/$VERSION -m "release rivertype/$VERSION"
     git tag $VERSION

driver_test.go

@@ -11,82 +11,143 @@ import (
 	"github.com/jackc/pgx/v5"
 	"github.com/jackc/pgx/v5/stdlib"
 	"github.com/stretchr/testify/require"
+	_ "modernc.org/sqlite"
 
 	"github.com/riverqueue/river/internal/rivercommon"
 	"github.com/riverqueue/river/internal/riverinternaltest/riverdrivertest"
 	"github.com/riverqueue/river/riverdbtest"
 	"github.com/riverqueue/river/riverdriver"
 	"github.com/riverqueue/river/riverdriver/riverdatabasesql"
 	"github.com/riverqueue/river/riverdriver/riverpgxv5"
+	"github.com/riverqueue/river/riverdriver/riversqlite"
 	"github.com/riverqueue/river/rivershared/riversharedtest"
 	"github.com/riverqueue/river/rivertype"
 )
 
-func TestDriverDatabaseSQL(t *testing.T) {
+func TestDriverRiverDatabaseSQL(t *testing.T) {
 	t.Parallel()
 
 	var (
 		ctx     = context.Background()
-		dbPool  = riversharedtest.DBPool(ctx, t) // TODO
+		dbPool  = riversharedtest.DBPool(ctx, t)
 		stdPool = stdlib.OpenDBFromPool(dbPool)
 		driver  = riverdatabasesql.New(stdPool)
 	)
 	t.Cleanup(func() { require.NoError(t, stdPool.Close()) })
 
-	// Make sure to use the same schema as Pgx test transactions use or else
-	// operations without a schema included will clash with each other due to
-	// different `river_state` types with the error "ERROR: cached plan must not
-	// change result type (SQLSTATE 0A000)".
-	//
-	// Alternatively, we could switch dbPool to use a DBPoolClone so it's got a
-	// different statement cache.
-	var testTxSchema string
-	tx := riverdbtest.TestTxPgx(ctx, t)
-	require.NoError(t, tx.QueryRow(ctx, "SELECT current_schema()").Scan(&testTxSchema))
-
 	riverdrivertest.Exercise(ctx, t,
-		func(ctx context.Context, t *testing.T) (riverdriver.Driver[*sql.Tx], string) {
+		func(ctx context.Context, t *testing.T, opts *riverdbtest.TestSchemaOpts) (riverdriver.Driver[*sql.Tx], string) {
 			t.Helper()
 
-			return driver, riverdbtest.TestSchema(ctx, t, driver, nil)
+			return driver, riverdbtest.TestSchema(ctx, t, driver, opts)
 		},
-		func(ctx context.Context, t *testing.T) riverdriver.Executor {
+		func(ctx context.Context, t *testing.T) (riverdriver.Executor, riverdriver.Driver[*sql.Tx], string) {
 			t.Helper()
 
-			tx, err := stdPool.BeginTx(ctx, nil)
-			require.NoError(t, err)
-			t.Cleanup(func() { _ = tx.Rollback() })
+			tx, schema := riverdbtest.TestTx(ctx, t, driver, nil)
 
 			// The same thing as the built-in riversharedtest.TestTx does.
-			_, err = tx.ExecContext(ctx, "SET search_path TO '"+testTxSchema+"'")
+			_, err := tx.ExecContext(ctx, "SET search_path TO '"+schema+"'")
 			require.NoError(t, err)
 
-			return riverdatabasesql.New(nil).UnwrapExecutor(tx)
+			return driver.UnwrapExecutor(tx), driver, schema
 		})
 }
 
 func TestDriverRiverPgxV5(t *testing.T) {
 	t.Parallel()
 
+	var (
+		ctx    = context.Background()
+		dbPool = riversharedtest.DBPool(ctx, t)
+		driver = riverpgxv5.New(dbPool)
+	)
+
+	riverdrivertest.Exercise(ctx, t,
+		func(ctx context.Context, t *testing.T, opts *riverdbtest.TestSchemaOpts) (riverdriver.Driver[pgx.Tx], string) {
+			t.Helper()
+
+			return driver, riverdbtest.TestSchema(ctx, t, driver, opts)
+		},
+		func(ctx context.Context, t *testing.T) (riverdriver.Executor, riverdriver.Driver[pgx.Tx], string) {
+			t.Helper()
+
+			tx := riverdbtest.TestTxPgx(ctx, t)
+			return riverpgxv5.New(nil).UnwrapExecutor(tx), driver, "" // no schema necessary because `search_path` is set
+		})
+}
+
+func TestDriverRiverSQLite(t *testing.T) {
+	t.Parallel()
+
 	ctx := context.Background()
 
+	openPool := func(t *testing.T) *riversqlite.Driver {
+		t.Helper()
+
+		dbPool, err := sql.Open("sqlite", "sqlite/river_test.sqlite3")
+		require.NoError(t, err)
+		t.Cleanup(func() { require.NoError(t, dbPool.Close()) })
+
+		// An unfortunate aspect of how SQLite does schemas (which work via an
+		// `ATTACH DATABASE` command) is that they're attached per database. If
+		// we raise a test schema on one connection, attach it, but then our
+		// database pool starts another connection, that test schema won't be
+		// available over there. So here we set maximum pool connections to one,
+		// and then unlike Postgres, use individual connection pools for each
+		// separate test schema. Unlike Postgres which has a postmaster that has
+		// to coordinate everything, each pool should be able to operate quite
+		// independently anyway, so it's not likely to create much of a
+		// contention problem. In fact, it's probably faster.
+		//
+		// I've seen some broad recommendations that it might be better to
+		// always set maximum connections for a single SQLite database to 1
+		// anyway. See for example:
+		//
+		// https://news.ycombinator.com/item?id=30369095
+		dbPool.SetMaxOpenConns(1)
+
+		return riversqlite.New(dbPool)
+	}
+
 	riverdrivertest.Exercise(ctx, t,
-		func(ctx context.Context, t *testing.T) (riverdriver.Driver[pgx.Tx], string) {
+		func(ctx context.Context, t *testing.T, opts *riverdbtest.TestSchemaOpts) (riverdriver.Driver[*sql.Tx], string) {
 			t.Helper()
 
-			var (
-				dbPool = riversharedtest.DBPool(ctx, t)
-				driver = riverpgxv5.New(dbPool)
-				schema = riverdbtest.TestSchema(ctx, t, driver, nil)
-			)
+			// Unlike other drivers, we MUST use a per-schema pool here because
+			// the technique we use to create "schemas" (SQLite `ATTACH
+			// DATABASE` because it doesn't have schemas) operates on a per
+			// connection basis. An attached database is only available to the
+			// connection that attached it. We set maximum connections for each
+			// pool to one so we don't have to worry about unattached databases.
+			driver := openPool(t)
 
-			return driver, schema
+			return driver, riverdbtest.TestSchema(ctx, t, driver, opts)
 		},
-		func(ctx context.Context, t *testing.T) riverdriver.Executor {
+		func(ctx context.Context, t *testing.T) (riverdriver.Executor, riverdriver.Driver[*sql.Tx], string) {
 			t.Helper()
 
-			tx := riverdbtest.TestTxPgx(ctx, t)
-			return riverpgxv5.New(nil).UnwrapExecutor(tx)
+			// Don't reuse this for non-transaction test cases. Databases are
+			// attached on a per-connection database, so it's only safe to share a
+			// pool when a connection is guaranteed to be affixed through a
+			// transaction.
+			testTxDriver := openPool(t)
+
+			tx, schema := riverdbtest.TestTx(ctx, t, testTxDriver, &riverdbtest.TestTxOpts{
+				// Unfortunately, the normal test transaction schema sharing has
+				// to be disabled for SQLite. When enabled, there's too much
+				// contention on the shared test databases and operations fail
+				// with `database is locked (5) (SQLITE_BUSY)`, which is a
+				// common concurrency error in SQLite whose recommended
+				// remediation is a backoff and retry. I tried various
+				// techniques like journal_mode=WAL, but it didn't seem to help
+				// enough. SQLite databases are just local files anyway, and
+				// test transactions can still reuse schemas freed by other
+				// tests through TestSchema, so this should be okay performance
+				// wise.
+				DisableSchemaSharing: true,
+			})
+			return testTxDriver.UnwrapExecutor(tx), testTxDriver, schema
 		})
 }
 

go.mod

@@ -5,13 +5,13 @@ go 1.23.0
 toolchain go1.24.1
 
 require (
-	github.com/davecgh/go-spew v1.1.1
 	github.com/jackc/pgerrcode v0.0.0-20240316143900-6e2875d9b438
 	github.com/jackc/pgx/v5 v5.7.4
 	github.com/jackc/puddle/v2 v2.2.2
 	github.com/riverqueue/river/riverdriver v0.21.0
 	github.com/riverqueue/river/riverdriver/riverdatabasesql v0.21.0
 	github.com/riverqueue/river/riverdriver/riverpgxv5 v0.21.0
+	github.com/riverqueue/river/riverdriver/riversqlite v0.21.0
 	github.com/riverqueue/river/rivershared v0.21.0
 	github.com/riverqueue/river/rivertype v0.21.0
 	github.com/robfig/cron/v3 v3.0.1
@@ -21,15 +21,32 @@ require (
 	go.uber.org/goleak v1.3.0
 	golang.org/x/sync v0.13.0
 	golang.org/x/text v0.24.0
+	modernc.org/sqlite v1.37.0
 )
 
 require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/google/uuid v1.6.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
 	github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
 	github.com/lib/pq v1.10.9 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/ncruces/go-strftime v0.1.9 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
+	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/tidwall/match v1.1.1 // indirect
 	github.com/tidwall/pretty v1.2.1 // indirect
-	golang.org/x/crypto v0.31.0 // indirect
+	golang.org/x/crypto v0.37.0 // indirect
+	golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
+	golang.org/x/sys v0.32.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
+	modernc.org/libc v1.64.0 // indirect
+	modernc.org/mathutil v1.7.1 // indirect
+	modernc.org/memory v1.10.0 // indirect
 )
+
+// TODO(brandur): Remove this before first release including SQLite driver. It's
+// needed temporarily because there's no riversqlite tag to target (the one
+// referenced above is fake and does not exist).
+replace github.com/riverqueue/river/riverdriver/riversqlite => ../river/riverdriver/riversqlite