feat(migrate): versioned schema migrations for the on-disk KB

docs/migrations.md

@@ -0,0 +1,75 @@
+# Schema migrations — `vouch migrate`
+
+The `.vouch/` layout is the durable contract: yaml claims, markdown pages with
+frontmatter, json sessions, the jsonl audit log. As the pydantic models in
+`src/vouch/models.py` evolve, a KB created against an older model would become a
+load-time error. `vouch migrate` gives that evolution a versioned, reversible,
+audit-logged upgrade path so a schema change never silently breaks a KB in the
+wild.
+
+## Two version axes
+
+vouch tracks two independent versions, and `vouch migrate` covers both:
+
+| | Stamp | What it governs | Reached by |
+|---|---|---|---|
+| **Format** (integer) | `config.yaml` `version` | the `.vouch/` directory layout (subdirs, `.gitignore`) | `vouch migrate` *(no subcommand)* |
+| **Schema** (semver) | `.vouch/schema_version` | the model schema of each artifact | `vouch migrate <subcommand>` |
+
+A KB with no `.vouch/schema_version` file is treated as the baseline (`0.1.0`),
+so existing KBs keep loading until their first migrate. `vouch init` stamps the
+current version on bootstrap.
+
+## Commands
+
+```bash
+vouch migrate status                 # current schema version, target, pending migrations
+vouch migrate plan                   # dry-run: every file each pending migration would change
+vouch migrate plan --to 0.3.0        # plan against a specific target
+vouch migrate apply                  # apply pending migrations (audit-logged, atomic)
+vouch migrate apply --yes            # skip the confirmation prompt (CI)
+vouch migrate rollback               # reverse the most recently applied migration
+vouch migrate verify                 # parse-load every artifact under the current version
+```
+
+## Manifests
+
+Migrations are **data, not code**: yaml manifests in the repo-root `migrations/`
+directory, one consecutive version step each. See
+[`migrations/README.md`](../migrations/README.md) for the full format and the
+transform verbs (`rename`, `default`, `drop`, `split`, `merge`). Only consecutive
+manifests apply — a `0.1 → 0.5` KB walks through `0.2`, `0.3`, `0.4` in order.
+
+## Safety model
+
+- **Atomic per file.** Each artifact is rewritten to a temp file, `fsync`-ed,
+  then `os.replace`-d into place. A file is always its old bytes or its new
+  bytes — never a torn mix.
+- **Reversible.** Before any rewrite, the prior content of every file the step
+  touches is journalled to `.vouch/migrations/rollback-<id>.jsonl`. `vouch
+  migrate rollback` replays it, restoring the exact prior bytes — so
+  apply → rollback is byte-equivalent (modulo the audit-log entries recording
+  the round trip).
+- **Crash-safe.** The journal is written and fsynced *before* the first rewrite,
+  and `.vouch/schema_version` is bumped *last*. An interrupted apply therefore
+  leaves the KB reporting its prior version, with a journal to recover from.
+- **Precondition.** Apply refuses if the KB has pending proposals — it won't
+  rewrite a reviewer's in-flight queue out from under them. Resolve them with
+  `vouch list-pending` first.
+- **`state.db` is disposable.** The runner does not migrate the FTS5 / embedding
+  cache; the CLI rebuilds it after a successful apply.
+
+## Audit trail
+
+Each applied manifest logs one `kb.migrate.apply` event (manifest id, file count,
+rollback-journal id); a rollback logs `kb.migrate.rollback`. The legacy format
+migration continues to log `kb.migrate`.
+
+## Out of scope (v1)
+
+- Cross-major jumps in a single manifest (consecutive only).
+- `custom: path/to/migration.py` script transforms (lands once the verb
+  taxonomy stabilises).
+- Migrating `state.db` (disposable; rebuilt by `vouch index`).
+- Bundle version embedding / refusing mismatched imports — a follow-up.
+- Concurrent multi-process migrate runs (single-writer assumption).

migrations/README.md

@@ -0,0 +1,57 @@
+# Schema migration manifests
+
+Each file here is a **migration manifest**: a single, consecutive
+model-schema version step, declared as data. Adding one is a contributor-friendly,
+single-file PR — the same shape as `adapters/`. The runner
+(`vouch.migrations.runner`) chains them from a KB's current
+`.vouch/schema_version` toward the target.
+
+Provenance is derived; migrations are not. A manifest rewrites the durable
+source-of-truth files under `.vouch/` (yaml claims, markdown page frontmatter,
+etc.). The `state.db` cache is disposable and rebuilt afterward by `vouch index`.
+
+## File naming
+
+`NNNN-short-slug.yaml`, e.g. `0001-add-scope-spec.yaml`. Files are applied in
+filename order; exactly one manifest may migrate *from* any given version.
+
+## Format
+
+```yaml
+from_version: "0.1.0"          # must equal the previous manifest's to_version
+to_version:   "0.2.0"          # strictly greater than from_version
+artifact:     claims           # claims | pages | entities | relations | evidence | sessions
+description:  "rename confidence -> certainty; default scope=project"
+transforms:                    # applied in order to each artifact's dict
+  - rename:  {from: confidence, to: certainty}
+  - default: {field: scope, value: project}
+  - drop:    {field: legacy_flag}
+  - split:   {field: name, into: [first, last], on: " "}
+  - merge:   {fields: [first, last], into: name, with: " "}
+reverse:                        # documents the inverse (rollback is journal-based)
+  - rename:  {from: certainty, to: confidence}
+  - drop:    {field: scope}
+```
+
+### Transform verbs
+
+| Verb | Spec | Effect |
+|------|------|--------|
+| `rename`  | `{from, to}` | move a field's value to a new key |
+| `default` | `{field, value}` | set a field only if absent |
+| `drop`    | `{field}` | remove a field |
+| `split`   | `{field, into: [..], on}` | split a string field into several |
+| `merge`   | `{fields: [..], into, with}` | join several fields into one |
+
+## Apply / rollback
+
+`vouch migrate apply` rewrites each artifact atomically (temp + fsync +
+rename) and journals the prior content to `.vouch/migrations/rollback-<id>.jsonl`
+*before* writing, then bumps `.vouch/schema_version` last. `vouch migrate
+rollback` replays the newest journal to restore the exact prior bytes. An
+interrupted apply therefore always leaves the KB at its prior version with a
+journal to recover from.
+
+> No real manifests ship yet — the first lands with ROADMAP 0.2's multi-dim
+> scopes (#2). This directory documents the convention and is exercised by the
+> synthetic manifests in `tests/fixtures/migrations/`.

src/vouch/cli.py

@@ -374,18 +374,28 @@ def fsck() -> None:
     sys.exit(0 if report.ok else 1)
 
 
-@cli.command()
-@click.option("--check", "check_only", is_flag=True, help="Only check whether migration is needed.")
-@click.option("--dry-run", is_flag=True, help="Show planned changes without writing.")
-@click.option("--to-version", type=int, default=None, help="Target KB format version.")
+@cli.group(invoke_without_command=True)
+@click.option("--check", "check_only", is_flag=True, help="Only check if a migration is needed.")
+@click.option("--dry-run", is_flag=True, help="Show planned format changes without writing.")
+@click.option("--to-version", type=int, default=None, help="Target KB format (integer) version.")
 @click.option("--json", "as_json", is_flag=True, help="Emit JSON instead of text.")
+@click.pass_context
 def migrate(
+    ctx: click.Context,
     check_only: bool,
     dry_run: bool,
     to_version: int | None,
     as_json: bool,
 ) -> None:
-    """Upgrade the on-disk .vouch/ layout to the supported format."""
+    """Migrate the KB.
+
+    With no subcommand, runs the legacy integer *format* migration of the
+    .vouch/ directory layout. The subcommands (status / plan / apply / rollback
+    / verify) drive the semver model-schema migrations keyed off
+    .vouch/schema_version.
+    """
+    if ctx.invoked_subcommand is not None:
+        return
     if check_only and dry_run:
         raise click.ClickException("--check and --dry-run are mutually exclusive")
 
@@ -430,6 +440,128 @@ def migrate(
         sys.exit(1)
 
 
+@migrate.command("status")
+@click.option("--json", "as_json", is_flag=True, help="Emit JSON instead of text.")
+def migrate_status(as_json: bool) -> None:
+    """Show the KB schema version, target, and pending migrations."""
+    store = _load_store()
+    with _cli_errors():
+        result = migrations_mod.schema_status(store)
+    if as_json:
+        _emit_json(result)
+        return
+    click.echo(f"schema: {result['schema_version']} -> {result['target_version']}")
+    if result["up_to_date"]:
+        click.echo("up to date")
+    else:
+        click.echo(f"pending: {', '.join(result['pending'])}")
+
+
+@migrate.command("plan")
+@click.option("--to", "to_version", default=None, help="Target schema version (semver).")
+@click.option("--json", "as_json", is_flag=True, help="Emit JSON instead of text.")
+def migrate_plan(to_version: str | None, as_json: bool) -> None:
+    """Dry-run: list every file each pending migration would change."""
+    store = _load_store()
+    with _cli_errors():
+        result = migrations_mod.schema_plan(store, to_version=to_version)
+    if as_json:
+        _emit_json(result)
+        return
+    if not result["needed"]:
+        click.echo(f"schema: {result['current_version']} (up to date)")
+        return
+    click.echo(f"schema: {result['current_version']} -> {result['target_version']}")
+    for step in result["steps"]:
+        click.echo(
+            f"- {step['manifest_id']}: {step['from_version']} -> {step['to_version']} "
+            f"({step['artifact']}, {step['file_count']} file(s))"
+        )
+        for rel in step["changed"]:
+            click.echo(f"  * {rel}")
+    click.echo(f"total: {result['total_files']} file(s)")
+
+
+@migrate.command("apply")
+@click.option("--to", "to_version", default=None, help="Target schema version (semver).")
+@click.option("--yes", is_flag=True, help="Skip the confirmation prompt (CI).")
+@click.option("--json", "as_json", is_flag=True, help="Emit JSON instead of text.")
+def migrate_apply(to_version: str | None, yes: bool, as_json: bool) -> None:
+    """Apply pending schema migrations (audit-logged, atomic, reversible)."""
+    store = _load_store()
+    with _cli_errors():
+        preview = migrations_mod.schema_plan(store, to_version=to_version)
+        if not preview["needed"]:
+            if as_json:
+                _emit_json(
+                    {
+                        "applied": False,
+                        "from_version": preview["current_version"],
+                        "to_version": preview["current_version"],
+                        "manifests": [],
+                        "files": 0,
+                    }
+                )
+            else:
+                click.echo(f"schema: {preview['current_version']} (up to date)")
+            return
+        if not yes:
+            click.confirm(
+                f"Apply {len(preview['steps'])} migration(s) "
+                f"({preview['current_version']} -> {preview['target_version']}, "
+                f"{preview['total_files']} file(s))?",
+                abort=True,
+            )
+        result = migrations_mod.schema_apply(store, to_version=to_version, actor=_whoami())
+        # state.db is derived; rebuild it under the new layout.
+        health.rebuild_index(store)
+    if as_json:
+        _emit_json(result)
+    else:
+        click.echo(
+            f"schema: {result['from_version']} -> {result['to_version']} "
+            f"({result['files']} file(s), {len(result['manifests'])} manifest(s))"
+        )
+
+
+@migrate.command("rollback")
+@click.option("--json", "as_json", is_flag=True, help="Emit JSON instead of text.")
+def migrate_rollback(as_json: bool) -> None:
+    """Reverse the most recently applied schema migration."""
+    store = _load_store()
+    with _cli_errors():
+        result = migrations_mod.schema_rollback(store, actor=_whoami())
+        health.rebuild_index(store)
+    if as_json:
+        _emit_json(result)
+    else:
+        click.echo(
+            f"schema: {result['from_version']} -> {result['to_version']} "
+            f"(rolled back {result['files']} file(s))"
+        )
+
+
+@migrate.command("verify")
+@click.option("--json", "as_json", is_flag=True, help="Emit JSON instead of text.")
+def migrate_verify(as_json: bool) -> None:
+    """Parse-load every artifact under the current schema version."""
+    store = _load_store()
+    with _cli_errors():
+        result = migrations_mod.schema_verify(store)
+    if as_json:
+        _emit_json(result)
+    elif result["ok"]:
+        click.echo(
+            f"verified {result['checked']} artifact(s) at schema {result['schema_version']}"
+        )
+    else:
+        click.echo(f"FAILED: {len(result['errors'])} of {result['checked']} artifact(s)")
+        for err in result["errors"]:
+            click.echo(f"  ✗ {err['path']}: {err['error']}")
+    if not result["ok"]:
+        sys.exit(1)
+
+
 # --- metrics --------------------------------------------------------------
 
 

src/vouch/migrations/__init__.py

@@ -0,0 +1,78 @@
+"""KB migrations.
+
+Two layers live here, by design:
+
+* **Legacy integer "format" migrations** (`_legacy`) — the ``config.yaml``
+  ``version`` / ``KB_FORMAT_VERSION`` directory-layout bump, reached by
+  ``vouch migrate`` with no subcommand. Preserved verbatim.
+* **Semver model-schema migrations** (`runner` + `manifest` + `rewriter` +
+  `journal` + `schema`) — the versioned plan/apply/rollback/verify flow keyed off
+  ``.vouch/schema_version`` and driven by yaml manifests. Reached by
+  ``vouch migrate status|plan|apply|rollback|verify``.
+
+Both are re-exported here so importers keep using ``vouch.migrations`` unchanged.
+"""
+
+from __future__ import annotations
+
+from ._legacy import (
+    MigrationError,
+    MigrationPlan,
+    MigrationResult,
+    MigrationStep,
+    build_plan,
+    detect_version,
+    migrate,
+    read_config,
+    write_config,
+)
+from .manifest import (
+    Manifest,
+    ManifestError,
+    default_manifests_dir,
+    load_manifests,
+    parse_manifest,
+)
+from .runner import (
+    SchemaPlan,
+    SchemaPlanStep,
+    build_schema_plan,
+)
+from .runner import apply as schema_apply
+from .runner import plan as schema_plan
+from .runner import rollback as schema_rollback
+from .runner import status as schema_status
+from .runner import verify as schema_verify
+from .schema import (
+    BASELINE_SCHEMA_VERSION,
+    read_schema_version,
+    write_schema_version,
+)
+
+__all__ = [
+    "BASELINE_SCHEMA_VERSION",
+    "Manifest",
+    "ManifestError",
+    "MigrationError",
+    "MigrationPlan",
+    "MigrationResult",
+    "MigrationStep",
+    "SchemaPlan",
+    "SchemaPlanStep",
+    "build_plan",
+    "build_schema_plan",
+    "default_manifests_dir",
+    "detect_version",
+    "load_manifests",
+    "migrate",
+    "parse_manifest",
+    "read_config",
+    "read_schema_version",
+    "schema_apply",
+    "schema_plan",
+    "schema_rollback",
+    "schema_status",
+    "schema_verify",
+    "write_config",
+    "write_schema_version",
+]

src/vouch/migrations.py → src/vouch/migrations/_legacy.py

@@ -1,8 +1,10 @@
-"""On-disk KB format migrations.
+"""Legacy integer "KB format" migrations.
 
-The migration layer only changes source-of-truth files under `.vouch/`.
-Derived caches such as state.db are rebuilt by the CLI after a successful
-apply.
+This predates the semver model-schema runner (see :mod:`.runner`) and governs the
+*directory layout* version stamped in ``config.yaml`` (``KB_FORMAT_VERSION``) —
+making sure the ``.vouch/`` subdir tree and ``.gitignore`` exist. It is reached
+by ``vouch migrate`` with no subcommand and is preserved verbatim; the semver
+runner is a separate, additive concern keyed off ``.vouch/schema_version``.
 """
 
 from __future__ import annotations
@@ -13,7 +15,7 @@
 
 import yaml
 
-from .storage import KB_FORMAT_VERSION, SUBDIRS, KBStore, _starter_config
+from ..storage import KB_FORMAT_VERSION, SUBDIRS, KBStore, _starter_config
 
 
 class MigrationError(RuntimeError):