Repository navigation
feat(database)!: baseline schema for new SQLite databases and 1.70.0 minimum upgrade version - #977
Merged
Merged
Conversation
…ved devices api_tokens.scope defaulted to 'full', which is not a valid scope, and the model's GORM default tag filled an empty scope with it on create. Migration m20261013000000 drops the column default (ALTER COLUMN on PostgreSQL; on SQLite a rebuild through the driver's AlterColumn, with the table's indexes recreated from their saved definitions) and the model tag is removed. Existing rows holding 'full' are left unchanged: they are already rejected at authentication, and rewriting them to a valid scope would grant access nobody approved. Migration m20261013000001 adds devices.auto_archived (default false). The missed-ping monitor now archives through AutoArchiveDevice, which sets it; manual device and host archive or unarchive clear it. Unarchiving on the next successful metrics upload is not part of this change. Refs #965
An empty SQLite database now gets the baseline schema and default settings through the gormigrate InitSchema hook, and every historical migration ID is recorded as applied, so a new install never runs the 65-step history. PostgreSQL already worked this way. The empty check is explicit because gormigrate would also treat a database with tables but no recorded migrations as new. Before migrating, classifySQLiteDatabase reads the migrations table and refuses, without writing anything, a Scrutiny database that has Staros migrations but not m20260823000000, the last schema migration in 1.70.0. A database whose history holds only AnalogJ/scrutiny migration IDs keeps running the full history, which the AnalogJ import path needs, so no migration code is deleted. The baseline did not match the history on SQLite: attribute_overrides lacked its empty-string defaults and NOT NULL device_id, settings made setting_key_name NOT NULL, and notify_urls lacked the heartbeat_enabled default of true. Create-only structs in schema_baseline.go now build those three tables; the runtime models are unchanged. gormigrate.DefaultOptions is a shared pointer. The nested migration that builds the reference settings set UseTransaction=false on it, so the outer SQLite run never committed. Each run now copies the options. Tests: TestBaselineSchemaMatchesMigrationChainExactly compares column types, NOT NULL, defaults, primary keys, indexes, and foreign keys of a baseline database against the full history. Real v1.70.0, v1.69.0, and AnalogJ v0.9.5 databases in testdata cover upgrade, refusal (file hash unchanged), and the AnalogJ path. BREAKING CHANGE: a SQLite database last migrated by a Scrutiny release older than 1.70.0 is refused at startup. Run the latest 1.76.x release on it once, then start 2.0. AnalogJ/scrutiny databases are not affected. Refs #965
…fore writing Review follow-ups for the baseline change: - PostgreSQL databases again get their baseline from schemaModels, as 1.76.x created them. The SQLite-only create structs had given 2.0-created PostgreSQL databases a nullable settings.setting_key_name, a NOT NULL attribute_overrides.device_id, and a heartbeat_enabled default that 1.76-created ones lack, with no migration to converge them. TestPostgres_SchemaDoesNotDependOnCreationVersion builds a database the 1.76 way and compares columns, indexes, and constraints after upgrade. - NewScrutinyRepository now classifies a SQLite database over a read-only connection before the normal connection sets journal_mode=WAL and before InfluxDB setup, so a refused database is byte-for-byte unchanged. Migrate still classifies on its own connection; its message now says no schema or data was changed, which is what it can promise there. - Only migration IDs this fork added count as Staros history. An unknown ID, such as one from an unreleased AnalogJ/scrutiny build, no longer gets an AnalogJ database refused. The message names the newest Staros migration. Refs #965
…ews in schema parity The baseline equivalence test now also compares, per table, whether the DDL uses AUTOINCREMENT, CHECK, COLLATE, or WITHOUT ROWID, and lists every trigger and view. No difference was found: both sides create the same eight AUTOINCREMENT tables. Column order stays out of the comparison. It differs between a new and an upgraded database by construction (ADD COLUMN appends, AutoMigrate follows struct order), no SQL in the backend reads or writes columns by position, and TESTING.md now asks migrations to keep naming columns. Refs #965
3 of 14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WP-K (migration baseline) for the 2.0.0 release. New SQLite databases are created directly from the baseline schema instead of replaying about 65 migrations, SQLite databases older than 1.70.0 are refused at startup without being modified, and two database follow-ups land: the legacy
fulltoken scope default is dropped and devices record whether they were auto-archived.Type of Change
Related Issues
Related to #965 and #973.
Changes Made
schema_baseline.go, recordSCHEMA_INIT, and mark every historical migration ID applied. Databases with migration history run the remaining chain unchanged. No migration code was deleted.m20260823000000(the last migration in v1.70.0; v1.69.2 has the same set) is refused before any write, with a message to run the latest 1.76.x first. Postgres support began in 1.76.0, so every Postgres database is past the cutoff. Databases holding only AnalogJ migration IDs, and AnalogJ databases with tables but no history, keep running the full chain.attribute_overridesdefaults anddevice_idNOT NULL,settings.setting_key_namenullability,notify_urls.heartbeat_enableddefault). Create-only structs inschema_baseline.gocorrect it; runtime models are unchanged.gormigrate.DefaultOptionsis a shared pointer, and a nested migration flippedUseTransactionon it, leaving the outer run's transaction open on new databases. Each run now copies the options.api_tokens.scopeno longer defaults to the legacyfull(forward migration: PostgresDROP DEFAULT, SQLite table rebuild with indexes recreated). Existingfullrows are left unchanged; they are already rejected at authentication, and converting them would grant access nobody approved.devices.auto_archivedcolumn (default false), set by the missed-ping auto-archive and cleared by any manual archive or unarchive of the device or host.docs/UPGRADE_2.0.mdminimum upgrade-from section;TESTING.md.Testing
go build ./...,go vet ./...,go test ./...(also run by the builder against local Postgres 16 and pgbouncer as CI does: 40 packages ok);golangci-lint run --new-from-rev=origin/beta(v2.14.0, linux/amd64) 0 issues. Safety tests:TestBaselineSchemaMatchesMigrationChainExactly: compares column types, NOT NULL, defaults, primary keys, indexes, and foreign keys between a chain-built and a baseline-built database. It caught the drift above.TestMigrateFreshSQLiteInstallUsesBaseline: schema, settings rows, and row counts equal the chain.TestMigrateRefusesDatabaseOlderThanMinimumWithoutChangingIt: a real v1.69.0 database keeps its SHA-256 and gets no journal or WAL file.TestMigrateUpgradesMinimumVersionDatabase: a real v1.70.0 database ends with a schema equal to a fresh install and keeps its data.TestMigrateRunsFullHistoryForAnalogJDatabase: a real AnalogJ v0.9.5 database.TestClassifySQLiteDatabase,TestPostgres_BaselineMatchesSQLiteHistory,TestPostgres_ScopeDefaultAndAutoArchivedMigrationsRun,TestAutoArchivedFlag,TestAPITokenWithoutScopeIsNotGivenLegacyFull.Fixtures (
testdata/scrutiny-v1.70.0.db.gz,scrutiny-v1.69.0.db.gz,scrutiny-analogj-v0.9.5.db.gz, 2 to 7.5 KB) were produced by running each release's ownMigrate.TestAnalogJMigrationIntegrationneeds an InfluxDB service and runs in CI only.Checklist
Notes
Review: required (database migration; hard to undo)