From b65e7629cb7d9a4f0ce08b3b7ca8f6a035bef4f8 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Sat, 21 Feb 2026 02:46:40 +0100 Subject: [PATCH 1/5] docs(quick-019): add complete metadata schema reference - Document all 10 metadata objects with field tables, encryption details, storage locations, and source file cross-references - Cover FolderMetadata v2, FolderChild, FolderEntry, FilePointer, FileMetadata v1, VersionEntry, EncryptedVaultKeys, DeviceRegistry, DeviceEntry, and wire format - Include encryption hierarchy, cross-implementation parity matrix, IPNS key derivation summary, and version history per schema Co-Authored-By: Claude Opus 4.6 Entire-Checkpoint: dfd95a3736bf --- docs/METADATA_SCHEMAS.md | 425 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 425 insertions(+) create mode 100644 docs/METADATA_SCHEMAS.md diff --git a/docs/METADATA_SCHEMAS.md b/docs/METADATA_SCHEMAS.md new file mode 100644 index 000000000..b97d46d50 --- /dev/null +++ b/docs/METADATA_SCHEMAS.md @@ -0,0 +1,425 @@ +# CipherBox Metadata Schema Reference + +**Version:** 1.0 +**Last Updated:** 2026-02-21 +**Status:** Stable + +## Table of Contents + +1. [Overview](#1-overview) +2. [Encryption Hierarchy](#2-encryption-hierarchy) +3. [Wire Format](#3-wire-format) +4. [FolderMetadata (v2)](#4-foldermetadata-v2) +5. [FolderChild (Union)](#5-folderchild-union) +6. [FolderEntry](#6-folderentry) +7. [FilePointer](#7-filepointer) +8. [FileMetadata (v1)](#8-filemetadata-v1) +9. [VersionEntry](#9-versionentry) +10. [EncryptedVaultKeys](#10-encryptedvaultkeys) +11. [DeviceRegistry (v1)](#11-deviceregistry-v1) +12. [DeviceEntry](#12-deviceentry) +13. [Cross-Implementation Parity](#13-cross-implementation-parity) +14. [IPNS Key Derivation Summary](#14-ipns-key-derivation-summary) + +--- + +## 1. Overview + +CipherBox stores all metadata encrypted client-side using AES-256-GCM or ECIES before persisting to IPFS or the server database. The server is zero-knowledge -- it never sees plaintext metadata or unencrypted keys. + +Metadata exists in two implementations that must produce byte-identical JSON (camelCase field names): + +- **TypeScript** -- `packages/crypto/src/` (web app, shared crypto library) +- **Rust** -- `apps/desktop/src-tauri/src/crypto/` (desktop app, uses `serde(rename_all = "camelCase")`) + +This document defines the canonical schema for every metadata object in the system. For rules governing how these schemas evolve over time, see [METADATA_EVOLUTION_PROTOCOL.md](METADATA_EVOLUTION_PROTOCOL.md). + +--- + +## 2. Encryption Hierarchy + +Each metadata type uses a specific encryption scheme and storage location. + +| Metadata Type | Encrypted By | Algorithm | Storage | IPNS Addressing | +| ------------------ | ---------------------------- | -------------------------- | --------- | ------------------------- | +| FolderMetadata | Folder's own `folderKey` | AES-256-GCM | IPFS blob | Folder's IPNS name | +| FileMetadata | Parent folder's `folderKey` | AES-256-GCM | IPFS blob | File's own IPNS name | +| EncryptedVaultKeys | User's secp256k1 `publicKey` | ECIES | Server DB | N/A | +| DeviceRegistry | User's secp256k1 `publicKey` | ECIES | IPFS blob | Registry's IPNS name | +| File content | Per-file random `fileKey` | AES-256-GCM or AES-256-CTR | IPFS blob | N/A (CID in FileMetadata) | + +**Key principle:** Access to a folder's `folderKey` grants access to all children (subfolders via ECIES-wrapped keys, files via the parent's `folderKey` encrypting their metadata). + +--- + +## 3. Wire Format + +Folder metadata and file metadata share the same encrypted envelope format for IPFS storage. + +```json +{ + "iv": "", + "data": "" +} +``` + +| Field | Type | Encoding | Description | +| ------ | ------ | -------- | --------------------------------------------------------------- | +| `iv` | string | hex | 12-byte AES-256-GCM initialization vector (24 hex characters) | +| `data` | string | base64 | AES-256-GCM ciphertext with appended 16-byte authentication tag | + +**Source files:** + +- TS folder: `packages/crypto/src/folder/types.ts:53-58` (`EncryptedFolderMetadata`) +- TS file: `packages/crypto/src/file/types.ts:75-80` (`EncryptedFileMetadata`) +- Rust: inline struct in `apps/desktop/src-tauri/src/fuse/operations.rs` (defined locally) + +**Encoding note:** The `iv` field is hex-encoded. The `data` field uses standard base64 (not base64url, not hex). Mixing up encodings causes decryption failures. + +--- + +## 4. FolderMetadata (v2) + +The top-level metadata object for each folder. Contains an array of children (subfolders and file pointers). + +**Current version:** `v2` + +| Field | Type | Required | Description | +| ---------- | --------------- | -------- | --------------------------------------------- | +| `version` | `'v2'` | Yes | Schema version (literal string `"v2"`) | +| `children` | `FolderChild[]` | Yes | Array of FolderEntry and FilePointer children | + +**Encryption:** AES-256-GCM with this folder's 32-byte `folderKey`. + +**Storage:** IPFS (as JSON envelope `{iv, data}`), addressed via the folder's IPNS name. + +**Source files:** + +- TS types: `packages/crypto/src/folder/types.ts:15-20` +- TS validator: `packages/crypto/src/folder/metadata.ts:32-78` +- Rust: `apps/desktop/src-tauri/src/crypto/folder.rs:78-84` + +**Version history:** + +| Version | Phase | Change | +| ------- | ------- | -------------------------------------------------------------------------------------------------------- | +| v1 | Initial | Children contained inline `FileEntry` with `cid`, `fileKeyEncrypted`, `fileIv`, `size`, `encryptionMode` | +| v2 | 12.6 | Children use `FilePointer` (slim IPNS reference). Per-file metadata moved to dedicated IPNS records | + +v1 was completely removed in Phase 11.2. The validator rejects v1 data with a `CryptoError`. This was a clean-break migration (pre-production vault wipe). + +--- + +## 5. FolderChild (Union) + +Discriminated union type for children within a folder. + +| Discriminant | Type | Description | +| ---------------- | ------------- | ---------------------------------------- | +| `type: 'folder'` | `FolderEntry` | Subfolder with ECIES-wrapped keys | +| `type: 'file'` | `FilePointer` | Slim reference to a per-file IPNS record | + +**Source files:** + +- TS: `packages/crypto/src/folder/types.ts:25` +- Rust: `apps/desktop/src-tauri/src/crypto/folder.rs:66-73` (serde internally tagged enum with `#[serde(tag = "type", rename_all = "lowercase")]`) + +--- + +## 6. FolderEntry + +A subfolder child within folder metadata. Contains ECIES-wrapped keys for accessing the subfolder's own metadata. + +| Field | Type | Encoding | Required | Description | +| ------------------------- | ------ | --------- | -------- | ------------------------------------------------------------------- | +| `type` | string | -- | Yes | Always `"folder"` | +| `id` | string | UUID | Yes | Unique identifier for this subfolder | +| `name` | string | -- | Yes | Folder name (plaintext; entire metadata blob is encrypted) | +| `ipnsName` | string | base32/36 | Yes | IPNS name for resolving this subfolder's metadata | +| `ipnsPrivateKeyEncrypted` | string | hex | Yes | ECIES-wrapped 64-byte Ed25519 IPNS key (161 bytes, 322 hex chars) | +| `folderKeyEncrypted` | string | hex | Yes | ECIES-wrapped 32-byte AES-256 folder key (129 bytes, 258 hex chars) | +| `createdAt` | number | -- | Yes | Unix timestamp in milliseconds | +| `modifiedAt` | number | -- | Yes | Unix timestamp in milliseconds | + +**Not independently encrypted** -- lives inside the parent `FolderMetadata` blob. + +**ECIES wrapping:** Both `folderKeyEncrypted` and `ipnsPrivateKeyEncrypted` are encrypted to the vault owner's secp256k1 `publicKey`. See [VAULT_EXPORT_FORMAT.md](VAULT_EXPORT_FORMAT.md) Section 4 for the ECIES ciphertext binary format. + +**Subfolder IPNS keys:** Randomly generated Ed25519 keypairs (not HKDF-derived). This is different from root vault and per-file IPNS keys which are deterministically derived. + +**Source files:** + +- TS: `packages/crypto/src/folder/types.ts:31-47` +- Rust: `apps/desktop/src-tauri/src/crypto/folder.rs:28-45` + +**Version history:** Unchanged since initial design. + +--- + +## 7. FilePointer + +A slim file reference within v2 folder metadata. Points to a file's own IPNS record instead of embedding file data inline. + +| Field | Type | Encoding | Required | Description | +| ------------------ | ------ | -------- | -------- | -------------------------------------------------------------- | +| `type` | string | -- | Yes | Always `"file"` | +| `id` | string | UUID | Yes | Unique file identifier (used in HKDF derivation for file IPNS) | +| `name` | string | -- | Yes | File name (plaintext; entire metadata blob is encrypted) | +| `fileMetaIpnsName` | string | base36 | Yes | IPNS name of the file's own metadata record | +| `createdAt` | number | -- | Yes | Unix timestamp in milliseconds | +| `modifiedAt` | number | -- | Yes | Unix timestamp in milliseconds | + +**Not independently encrypted** -- lives inside the parent `FolderMetadata` blob. + +**Key distinction from v1 FileEntry:** FilePointer does not contain `cid`, `fileKeyEncrypted`, `fileIv`, `encryptionMode`, or `size`. All file crypto material is in the per-file `FileMetadata` record, enabling file content updates without touching folder metadata. + +**Source files:** + +- TS: `packages/crypto/src/file/types.ts:57-69` +- Rust: `apps/desktop/src-tauri/src/crypto/folder.rs:50-63` + +**Version history:** Added in Phase 12.6 (replaced inline `FileEntry` in v2 folders). + +--- + +## 8. FileMetadata (v1) + +Per-file metadata stored in a file's own IPNS record. Contains all crypto material needed to decrypt the file content. + +**Current version:** `v1` + +| Field | Type | Encoding | Required | Default | Description | +| ------------------ | ------------------ | -------- | -------- | ------- | ------------------------------------------------------ | +| `version` | `'v1'` | -- | Yes | -- | Schema version (literal string `"v1"`) | +| `cid` | string | CIDv1 | Yes | -- | IPFS content identifier of the encrypted file | +| `fileKeyEncrypted` | string | hex | Yes | -- | ECIES-wrapped 32-byte AES-256 file key (258 hex chars) | +| `fileIv` | string | hex | Yes | -- | 12-byte IV used for file encryption (24 hex chars) | +| `size` | number | -- | Yes | -- | Original unencrypted file size in bytes | +| `mimeType` | string | -- | Yes | -- | MIME type of the original file | +| `encryptionMode` | `'GCM'` \| `'CTR'` | -- | No | `'GCM'` | Encryption algorithm used for file content | +| `createdAt` | number | -- | Yes | -- | Unix timestamp in milliseconds | +| `modifiedAt` | number | -- | Yes | -- | Unix timestamp in milliseconds | +| `versions` | `VersionEntry[]` | -- | No | omitted | Past versions of this file (newest first) | + +**Encryption:** AES-256-GCM with the parent folder's `folderKey` (not the file's own key). This means anyone who can read the folder can also read file metadata and decrypt the file. + +**Storage:** IPFS (as JSON envelope `{iv, data}`), addressed via the file's own IPNS name. The IPNS name is deterministically derived from the user's `privateKey` + `fileId` via HKDF (see [Section 14](#14-ipns-key-derivation-summary)). + +**Source files:** + +- TS types: `packages/crypto/src/file/types.ts:30-51` +- TS validator: `packages/crypto/src/file/metadata.ts:93-188` +- Rust: `apps/desktop/src-tauri/src/crypto/folder.rs:166-192` + +**Version history:** + +| Version | Phase | Change | Version Bumped? | +| --------------------- | --------- | ---------------------------------------------------------------------- | --------------- | +| v1 (initial) | 12.6 | Initial per-file IPNS schema | -- | +| v1 + `encryptionMode` | 12.6/12.1 | Optional field added, defaults to `'GCM'`. Supports AES-CTR streaming. | No | +| v1 + `versions` | 13 | Optional `VersionEntry[]` array. Omitted when empty. | No | + +Both additions were additive optional fields with sensible defaults -- the version field was not bumped. This informal pattern is formalized in [METADATA_EVOLUTION_PROTOCOL.md](METADATA_EVOLUTION_PROTOCOL.md). + +**Rust serde annotations for optional fields:** + +```rust +#[serde(default = "default_encryption_mode")] // defaults to "GCM" +pub encryption_mode: String, + +#[serde(skip_serializing_if = "Option::is_none")] +#[serde(default)] +pub versions: Option>, +``` + +--- + +## 9. VersionEntry + +A single past version of a file. Embedded in the `versions` array of `FileMetadata`. Each entry contains the full crypto context needed to independently decrypt that version's content. + +| Field | Type | Encoding | Required | Description | +| ------------------ | ------------------ | -------- | -------- | ------------------------------------------------------------------ | +| `cid` | string | CIDv1 | Yes | IPFS content identifier of the encrypted file for this version | +| `fileKeyEncrypted` | string | hex | Yes | ECIES-wrapped 32-byte AES-256 key for this version (258 hex chars) | +| `fileIv` | string | hex | Yes | 12-byte IV used for this version's encryption (24 hex chars) | +| `size` | number | -- | Yes | Original unencrypted file size in bytes | +| `timestamp` | number | -- | Yes | When this version was created (Unix ms) | +| `encryptionMode` | `'GCM'` \| `'CTR'` | -- | Yes | Encryption mode used for this version | + +**Not independently encrypted** -- embedded in the parent `FileMetadata` blob. + +**Key difference from FileMetadata:** `encryptionMode` is **required** (not optional) because past versions always record the explicit encryption mode used at the time of creation. + +**Constraints:** + +- Maximum 10 versions per file +- 15-minute cooldown between version creation + +**Source files:** + +- TS types: `packages/crypto/src/file/types.ts:10-23` +- TS validator: `packages/crypto/src/file/metadata.ts:32-87` +- Rust: `apps/desktop/src-tauri/src/crypto/folder.rs:145-160` + +**Version history:** Added in Phase 13 (File Versioning). + +--- + +## 10. EncryptedVaultKeys + +Root-level keys for the user's vault, encrypted with ECIES for server-side storage. + +| Field | Type | Encoding | Required | Description | +| ------------------------- | ------------ | -------- | -------- | ---------------------------------------------------------- | +| `encryptedRootFolderKey` | `Uint8Array` | raw | Yes | ECIES-wrapped 32-byte AES-256 root folder key (129 bytes) | +| `encryptedIpnsPrivateKey` | `Uint8Array` | raw | Yes | ECIES-wrapped 64-byte Ed25519 IPNS private key (161 bytes) | + +**Encryption:** ECIES with the user's secp256k1 `publicKey`. + +**Storage:** Server database (API `users` table). Transmitted as raw `Uint8Array` in TypeScript, not hex-encoded strings. + +**Source files:** + +- TS: `packages/crypto/src/vault/types.ts:32-37` + +**No Rust equivalent.** The desktop app retrieves vault keys from the API response and decrypts them using the webview's TypeScript crypto. + +**Version history:** + +| Change | Phase | Description | +| --------------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `rootIpnsPublicKey` removed | 12.3.1 | Was previously stored alongside private key. Removed because it is derivable from the private key via deterministic Ed25519 derivation. Eliminates redundancy and potential inconsistency. | + +**No version field.** Changes to this structure should be paired with a new vault export format version (see [VAULT_EXPORT_FORMAT.md](VAULT_EXPORT_FORMAT.md)). + +--- + +## 11. DeviceRegistry (v1) + +The encrypted device registry tracking all authenticated devices for a user. + +**Current version:** `v1` + +| Field | Type | Required | Description | +| ---------------- | --------------- | -------- | --------------------------------------------------------- | +| `version` | `'v1'` | Yes | Schema version (literal string `"v1"`) | +| `sequenceNumber` | number | Yes | Monotonically increasing counter for IPNS ordering | +| `devices` | `DeviceEntry[]` | Yes | Array of all device entries (including revoked for audit) | + +**Encryption:** ECIES with the user's secp256k1 `publicKey`. The entire registry is encrypted as a single blob (not AES-GCM like folder/file metadata). + +**Storage:** IPFS (raw ECIES ciphertext blob, not the `{iv, data}` JSON envelope), addressed via the device registry IPNS name (HKDF-derived). + +**`sequenceNumber` usage:** Each update increments the sequence number. Used by the IPNS publisher to set the record's sequence field, ensuring newer records supersede older ones across the DHT. + +**Validator constraints:** + +- `sequenceNumber` must be a non-negative integer +- `devices` must be an array (may be empty) +- Generic error messages (`'Invalid registry format'`) to avoid leaking schema details to attackers + +**Source files:** + +- TS types: `packages/crypto/src/registry/types.ts:54-61` +- TS validator: `packages/crypto/src/registry/schema.ts:26-58` + +**No Rust equivalent.** The desktop app uses the webview's TypeScript crypto for device registry operations. + +**Version history:** Added in Phase 12.2 (Encrypted Device Registry). + +--- + +## 12. DeviceEntry + +An individual device record within the `DeviceRegistry`. Tracks authentication status, platform information, and revocation state. + +| Field | Type | Encoding | Required | Description | +| ------------- | ------------------------------------------------ | -------- | -------- | ---------------------------------------------------------- | +| `deviceId` | string | hex | Yes | SHA-256 hash of device's Ed25519 public key (64 hex chars) | +| `publicKey` | string | hex | Yes | Device's Ed25519 public key, uncompressed (130 hex chars) | +| `name` | string | -- | Yes | Human-readable device name (max 200 chars) | +| `platform` | `'web'` \| `'macos'` \| `'linux'` \| `'windows'` | -- | Yes | Platform identifier | +| `appVersion` | string | -- | Yes | Application version string (max 50 chars) | +| `deviceModel` | string | -- | Yes | Device model or OS version (max 200 chars) | +| `ipHash` | string | hex | Yes | SHA-256 hash of IP address at registration (64 hex chars) | +| `status` | `'pending'` \| `'authorized'` \| `'revoked'` | -- | Yes | Authorization status | +| `createdAt` | number | -- | Yes | When device was first registered (Unix ms) | +| `lastSeenAt` | number | -- | Yes | Last time device synced with registry (Unix ms) | +| `revokedAt` | number \| null | -- | Yes | When device was revoked (Unix ms), `null` if not revoked | +| `revokedBy` | string \| null | -- | Yes | Device ID of the revoking device, `null` if not revoked | + +**Not independently encrypted** -- embedded in the parent `DeviceRegistry` blob. + +**Validator constraints:** + +- `deviceId`: exactly 64 hex characters (SHA-256 output) +- `publicKey`: exactly 130 hex characters (uncompressed secp256k1 point: `04` prefix + 64 bytes) +- `ipHash`: exactly 64 hex characters (SHA-256 output) +- `name`: max 200 characters +- `appVersion`: max 50 characters +- `deviceModel`: max 200 characters +- `platform`: must be one of the four valid values +- `status`: must be one of the three valid values +- `revokedAt` and `revokedBy`: must both be null or both be non-null + +**Source files:** + +- TS types: `packages/crypto/src/registry/types.ts:21-46` +- TS validator: `packages/crypto/src/registry/schema.ts:63-136` + +**Version history:** Added in Phase 12.2 (Encrypted Device Registry). + +--- + +## 13. Cross-Implementation Parity + +TypeScript and Rust implementations must produce identical JSON for the same logical data. + +| Schema | TypeScript | Rust | Notes | +| ------------------ | ------------------- | -------------------------------------- | --------------- | +| FolderMetadata | `folder/types.ts` | `crypto/folder.rs` | Both platforms | +| FolderChild | `folder/types.ts` | `crypto/folder.rs` (serde tagged enum) | Both platforms | +| FolderEntry | `folder/types.ts` | `crypto/folder.rs` | Both platforms | +| FilePointer | `file/types.ts` | `crypto/folder.rs` | Both platforms | +| FileMetadata | `file/types.ts` | `crypto/folder.rs` | Both platforms | +| VersionEntry | `file/types.ts` | `crypto/folder.rs` | Both platforms | +| EncryptedVaultKeys | `vault/types.ts` | -- | TypeScript only | +| DeviceRegistry | `registry/types.ts` | -- | TypeScript only | +| DeviceEntry | `registry/types.ts` | -- | TypeScript only | + +**Rust serialization strategy:** All Rust structs use `#[serde(rename_all = "camelCase")]` to produce camelCase JSON field names matching the TypeScript convention. The `FolderChild` enum uses `#[serde(tag = "type", rename_all = "lowercase")]` for internally tagged union serialization. + +**Types without Rust implementations** (EncryptedVaultKeys, DeviceRegistry, DeviceEntry) are handled entirely by the TypeScript crypto library running in the desktop app's webview. The Rust backend does not need to serialize or deserialize these types directly. + +--- + +## 14. IPNS Key Derivation Summary + +CipherBox uses HKDF-SHA256 to derive deterministic Ed25519 IPNS keypairs from the user's secp256k1 private key. All derivations share the same salt but use different HKDF info strings for domain separation. + +| Purpose | Salt | HKDF Info | Source File | +| -------------------- | -------------- | ----------------------------------- | --------------------------------------------- | +| Root vault IPNS | `CipherBox-v1` | `cipherbox-vault-ipns-v1` | `packages/crypto/src/vault/derive-ipns.ts` | +| Device registry IPNS | `CipherBox-v1` | `cipherbox-device-registry-ipns-v1` | `packages/crypto/src/registry/derive-ipns.ts` | +| Per-file IPNS | `CipherBox-v1` | `cipherbox-file-ipns-v1:{fileId}` | `packages/crypto/src/file/derive-ipns.ts` | + +**Derivation path:** + +```text +secp256k1 privateKey (32 bytes) + -> HKDF-SHA256(salt, info) -> 32-byte Ed25519 seed + -> Ed25519 keypair (@noble/ed25519) + -> IPNS name (CIDv1 with libp2p-key codec + identity multihash) +``` + +**Subfolder IPNS keys:** Subfolders use randomly generated Ed25519 keypairs, not HKDF-derived. The keypair is ECIES-wrapped with the vault owner's public key and stored in the parent folder's `FolderEntry.ipnsPrivateKeyEncrypted` field. + +**Per-file IPNS validation:** The `fileId` used in the HKDF info string must be at least 10 characters (enforced by the derivation function). This ensures UUID-length identifiers and prevents accidental short strings in the derivation material. + +--- + +_Document version: 1.0_ +_Last updated: 2026-02-21_ +_See also: [METADATA_EVOLUTION_PROTOCOL.md](METADATA_EVOLUTION_PROTOCOL.md) for schema change rules_ +_See also: [VAULT_EXPORT_FORMAT.md](VAULT_EXPORT_FORMAT.md) for recovery and crypto format details_ From dcb49e1bcb5ba0244cece637f770302600ddf6b9 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Sat, 21 Feb 2026 02:48:34 +0100 Subject: [PATCH 2/5] docs(quick-019): add metadata schema evolution protocol - Define formal rules for additive vs breaking metadata changes - Include dual-platform checklist (TypeScript + Rust) for every schema change - Document version field convention and version bump decision criteria - Add backward compatibility test patterns and cross-platform round-trip testing - Cover recovery tool compatibility matrix and dangerous gray areas Co-Authored-By: Claude Opus 4.6 Entire-Checkpoint: 7507d6e5e4c9 --- docs/METADATA_EVOLUTION_PROTOCOL.md | 276 ++++++++++++++++++++++++++++ 1 file changed, 276 insertions(+) create mode 100644 docs/METADATA_EVOLUTION_PROTOCOL.md diff --git a/docs/METADATA_EVOLUTION_PROTOCOL.md b/docs/METADATA_EVOLUTION_PROTOCOL.md new file mode 100644 index 000000000..1dad08c02 --- /dev/null +++ b/docs/METADATA_EVOLUTION_PROTOCOL.md @@ -0,0 +1,276 @@ +# CipherBox Metadata Schema Evolution Protocol + +**Version:** 1.0 +**Last Updated:** 2026-02-21 +**Status:** Active + +## Table of Contents + +1. [Purpose](#1-purpose) +2. [Guiding Principles](#2-guiding-principles) +3. [Change Classification](#3-change-classification) +4. [Evolution Checklist](#4-evolution-checklist) +5. [Version Field Convention](#5-version-field-convention) +6. [Testing Requirements](#6-testing-requirements) +7. [Recovery Tool Compatibility Matrix](#7-recovery-tool-compatibility-matrix) +8. [References](#8-references) + +--- + +## 1. Purpose + +CipherBox metadata is encrypted client-side and stored on IPFS. Once written, records may persist indefinitely -- IPFS content is immutable and old CIDs remain valid even after IPNS updates. This has three consequences: + +1. **Old records are permanent.** A new client version must be able to read metadata written by any prior version. +2. **Clients of different versions coexist.** A user may have the desktop app on v1.3 and the web app on v1.4. Both must read and write metadata the other can understand. +3. **Two implementations must agree.** The TypeScript crypto library (`packages/crypto/`) and the Rust desktop implementation (`apps/desktop/src-tauri/src/crypto/`) must produce byte-identical JSON for the same logical data. + +This protocol establishes formal rules for evolving metadata schemas. It replaces the informal pattern of adding optional fields without documentation that was used through Phase 13. + +--- + +## 2. Guiding Principles + +1. **Backward compatibility is mandatory.** New clients must read old data. Old clients must not crash on new data (graceful degradation via ignoring unknown fields). + +2. **Forward compatibility via optional fields.** Unknown JSON fields are ignored by both `JSON.parse()` and serde (CipherBox uses manual validators, not strict schema validation). This is safe only if new fields are optional. + +3. **Version bumps are breaking changes.** Incrementing the `version` field (e.g., `'v1'` to `'v2'`) signals that old clients cannot fully understand the data. This is a last resort. + +4. **Both platforms move together.** A schema change is not complete until both TypeScript and Rust implementations are updated. A partially-shipped change creates cross-device data corruption risk. + +5. **Validators are the source of truth.** If the validator does not check a field, that field does not exist from a compatibility perspective. Type definitions document intent; validators enforce it. + +--- + +## 3. Change Classification + +### 3.1 Additive (Non-Breaking) Changes + +Adding a new optional field with a sensible default. The metadata version field is NOT bumped. + +**Examples of additive changes:** + +- Adding a new optional field to an existing object +- Adding a new enum variant to a string union field (only if consumers ignore unknown variants) +- Adding new properties to an existing nested object field + +**Rules for additive changes:** + +- The field MUST be optional in TypeScript (`field?: Type`) +- The field MUST have `#[serde(default)]` in Rust (or `#[serde(default = "default_fn")]` if the default is not the zero value) +- The field MUST have `#[serde(skip_serializing_if = "Option::is_none")]` in Rust (or `skip_serializing_if = "Vec::is_empty"` for arrays) +- The validator MUST accept data without the new field +- The validator MUST NOT reject unknown fields (already true -- CipherBox uses manual validation, not strict schema validation) +- The default value MUST preserve existing behavior. If there is no default that preserves existing behavior, this is a breaking change. +- The version field is NOT bumped + +**Historical examples:** + +| Field | Schema | Phase | Default | Rationale | +| ----------------------------- | ------------ | --------- | --------------------- | -------------------------------------------------------------- | +| `FileMetadata.encryptionMode` | FileMetadata | 12.6/12.1 | `'GCM'` | All pre-CTR files used GCM; default preserves behavior | +| `FileMetadata.versions` | FileMetadata | 13 | `undefined` (omitted) | No versions existed before Phase 13; omission means no history | + +### 3.2 Breaking Changes (Version Bump Required) + +Changes that alter the meaning or structure of existing data. The metadata version field MUST be bumped. + +**What counts as a breaking change:** + +- Removing a field +- Changing a field's type (e.g., `string` to `number`) +- Making an optional field required +- Changing the semantics of an existing field (same type, different meaning) +- Restructuring a nested object or array format +- Changing a default value (this is a breaking change in disguise -- see Section 3.3) + +**Rules for breaking changes:** + +- The version field MUST be bumped (e.g., `'v1'` to `'v2'`) +- Choose one migration strategy: + - **Migration path:** Write a converter that upgrades old format to new format. Preferred for data at rest. + - **Dual support:** Read both versions, write only new version. Temporary measure -- converge within 2 releases. + - **Clean break:** Reject old format entirely. Only valid when ALL existing data can be wiped (e.g., pre-production). +- Both TypeScript and Rust validators must be updated simultaneously +- The recovery tool (`apps/web/public/recovery.html`) must handle both old and new versions unless a clean break was chosen + +**Historical example:** + +| Change | Schema | Phase | Strategy | Details | +| -------- | -------------- | --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| v1 to v2 | FolderMetadata | 12.6/11.2 | Clean break | Changed file children from inline FileEntry to slim FilePointer. Pre-production vault wipe. v1 later removed entirely in Phase 11.2. | + +### 3.3 Dangerous Gray Areas + +These changes look safe but can cause subtle failures. Treat them with extra scrutiny. + +**Changing default values:** If `encryptionMode` default changed from `'GCM'` to `'CTR'`, old files without the field would be decrypted with the wrong algorithm. Changing defaults is a breaking change. + +**Widening a string union type:** Adding `'CTR'` to the `encryptionMode` union is safe only if old clients handle unknown modes gracefully. In practice, they do not -- an unknown mode passed to Web Crypto API causes a hard failure. New enum variants MUST be paired with the optional-field pattern (old clients never see the new value because the field is absent from old data). + +**Reordering array elements:** The `versions` array order (newest-first) is semantic. All version display code assumes index 0 is newest. Changing to oldest-first would silently break version numbering UI without any deserialization error. + +**Renaming a field:** Even if the type and semantics are identical, a rename is a breaking change. The old field name will be missing from the JSON, causing validator failures. Use serde aliases (`#[serde(alias = "oldName")]`) if a rename is absolutely necessary, and only as a dual-support transition. + +--- + +## 4. Evolution Checklist + +Complete this checklist for every metadata schema change. This is the most important section of this document -- it is what developers use during implementation. + +### 4.1 Before Implementation + +- [ ] Classify the change: Additive (Section 3.1) or Breaking (Section 3.2)? +- [ ] If additive: What is the default value? Does it preserve existing behavior for all existing data? +- [ ] If breaking: What migration strategy? (migration path / dual support / clean break) +- [ ] Document the planned change in `docs/METADATA_SCHEMAS.md` version history section for the affected schema +- [ ] Check: Does the recovery tool (`apps/web/public/recovery.html`) need updating? +- [ ] Check: Does the vault export format (`docs/VAULT_EXPORT_FORMAT.md`) need updating? + +### 4.2 TypeScript Implementation + +- [ ] Update type definition in `packages/crypto/src/{domain}/types.ts` +- [ ] If optional: use `field?: Type` syntax +- [ ] Update validator in `packages/crypto/src/{domain}/metadata.ts` or `schema.ts` +- [ ] If optional: validator accepts `undefined`/missing field +- [ ] If optional: validator applies default value when constructing the return object +- [ ] If breaking: validator checks `version` field and branches accordingly +- [ ] Add or update unit tests in `packages/crypto/` for the new or changed field + +### 4.3 Rust Implementation + +- [ ] Update struct in `apps/desktop/src-tauri/src/crypto/folder.rs` (or relevant file) +- [ ] If optional: use `Option` with `#[serde(default)]` and `#[serde(skip_serializing_if = "Option::is_none")]` +- [ ] If new enum variant: ensure `#[serde(rename_all = "...")]` handles casing correctly +- [ ] Verify JSON round-trip: serialize then deserialize produces identical output +- [ ] Add or update cargo tests + +### 4.4 Cross-Platform Verification + +- [ ] Generate test JSON from TypeScript, deserialize in Rust (or vice versa) +- [ ] Verify old-format JSON (without new field) deserializes correctly in both implementations +- [ ] Verify new-format JSON (with new field) serializes identically in both implementations +- [ ] Run: `pnpm test` (TypeScript) and `cargo test --features fuse` (Rust) + +### 4.5 Downstream Updates + +- [ ] Recovery tool (`apps/web/public/recovery.html`): handles new field or version +- [ ] Vault export format (`docs/VAULT_EXPORT_FORMAT.md`): update if export-related schemas changed +- [ ] API client regeneration: run `pnpm api:generate` if API DTOs changed +- [ ] `docs/METADATA_SCHEMAS.md`: add entry to version history for the affected schema + +--- + +## 5. Version Field Convention + +All versioned metadata objects use a string `version` field with values like `'v1'`, `'v2'`, etc. + +| Schema | Current Version | Has Version Field | Evolves Via | +| ------------------ | --------------- | ----------------- | ----------------------------- | +| FolderMetadata | `v2` | Yes | Own version field | +| FileMetadata | `v1` | Yes | Own version field | +| DeviceRegistry | `v1` | Yes | Own version field | +| EncryptedVaultKeys | -- | No | Vault export format version | +| FolderEntry | -- | No | Parent FolderMetadata version | +| FilePointer | -- | No | Parent FolderMetadata version | +| VersionEntry | -- | No | Parent FileMetadata version | +| DeviceEntry | -- | No | Parent DeviceRegistry version | + +**Objects without version fields** (FolderEntry, FilePointer, VersionEntry, DeviceEntry) evolve only through their parent's version. For example, adding a field to FilePointer requires bumping FolderMetadata's version (unless the new field is optional with a sensible default, in which case the parent version is not bumped). + +**EncryptedVaultKeys** has no version field. Changes to its structure should be paired with a new vault export format version (see `docs/VAULT_EXPORT_FORMAT.md`). + +--- + +## 6. Testing Requirements + +### 6.1 Backward Compatibility Test Pattern + +For every schema change, add a test with a hardcoded JSON string representing the OLD format (before the change). Verify the new validator/deserializer accepts it and produces correct output. + +**TypeScript example:** + +```typescript +// Test: FileMetadata without versions field (pre-Phase 13 format) +it('accepts old format without versions field', () => { + const oldFormatJson = JSON.parse( + '{"version":"v1","cid":"bafybeig...","fileKeyEncrypted":"04ab...",' + + '"fileIv":"aabbccdd11223344eeff5566","size":1024,"mimeType":"text/plain",' + + '"createdAt":1705268100000,"modifiedAt":1705268100000}' + ); + const result = validateFileMetadata(oldFormatJson); + expect(result.versions).toBeUndefined(); + expect(result.encryptionMode).toBe('GCM'); // default applied +}); +``` + +**Rust example:** + +```rust +#[test] +fn accepts_old_format_without_versions() { + let old_json = r#"{ + "version": "v1", "cid": "bafybeig...", "fileKeyEncrypted": "04ab...", + "fileIv": "aabbccdd11223344eeff5566", "size": 1024, + "mimeType": "text/plain", "encryptionMode": "GCM", + "createdAt": 1705268100000, "modifiedAt": 1705268100000 + }"#; + let meta: FileMetadata = serde_json::from_str(old_json).unwrap(); + assert!(meta.versions.is_none()); + assert_eq!(meta.encryption_mode, "GCM"); +} +``` + +### 6.2 Cross-Platform Round-Trip Test + +Produce a JSON string from TypeScript, feed it to the Rust deserializer (via a hardcoded string in a cargo test or a generated fixture file). Verify all fields parse correctly and no data is lost. + +The reverse direction (Rust-produced JSON verified in TypeScript) is equally valuable, particularly for fields where serde and manual TypeScript serialization might diverge (e.g., `skip_serializing_if` behavior for empty arrays vs `undefined`). + +### 6.3 Unknown Field Resilience Test + +Add a JSON string with an extra field that does not exist in the current schema. Verify both TypeScript and Rust deserializers accept the data without error and ignore the unknown field. This confirms forward compatibility. + +--- + +## 7. Recovery Tool Compatibility Matrix + +The recovery tool (`apps/web/public/recovery.html`) is a standalone HTML file that operates independently of the CipherBox web app. It has its own inline implementations of crypto operations and metadata parsing. + +**When the recovery tool MUST be updated:** + +- Any change that affects how files are discovered (IPNS resolution, folder traversal, child type detection) +- Any change that affects how files are decrypted (encryption algorithm, key format, IV format) +- Any breaking change (version bump) to FolderMetadata, FileMetadata, or EncryptedVaultKeys +- Adding a new encryption mode (the recovery tool must know how to decrypt it) + +**When the recovery tool does NOT need updating:** + +- Additive metadata fields that do not affect crypto operations (e.g., a hypothetical `tags` field on FilePointer) +- Changes to DeviceRegistry or DeviceEntry (the recovery tool does not use device data) +- UI-only changes or validator tightening that does not reject valid data + +**Current recovery tool capabilities:** + +- Reads FolderMetadata v2 (v1 also supported as legacy) +- Reads FileMetadata v1 (with optional `encryptionMode` and `versions` fields) +- Decrypts AES-256-GCM and AES-256-CTR content +- Resolves IPNS names via delegated routing API +- Traverses folder hierarchy recursively +- Handles per-file IPNS (v2 FilePointer) and inline file data (v1 FileEntry) + +--- + +## 8. References + +- **Metadata Schema Reference:** [docs/METADATA_SCHEMAS.md](METADATA_SCHEMAS.md) -- field tables, encryption, storage, and source file cross-references for all 10 metadata objects +- **Vault Export Format:** [docs/VAULT_EXPORT_FORMAT.md](VAULT_EXPORT_FORMAT.md) -- recovery procedure and ECIES ciphertext format +- **Technical Architecture:** `00-Preliminary-R&D/Documentation/TECHNICAL_ARCHITECTURE.md` -- encryption hierarchy and key management design +- **Data Flows:** `00-Preliminary-R&D/Documentation/DATA_FLOWS.md` -- sequence diagrams and test vectors + +--- + +_Protocol version: 1.0_ +_Last updated: 2026-02-21_ +_Applies to: All metadata objects in `packages/crypto/` and `apps/desktop/src-tauri/src/crypto/`_ From 6479198e8c1699f062f8cd45d342be6f9f9935de Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Sat, 21 Feb 2026 02:50:33 +0100 Subject: [PATCH 3/5] docs(quick-019): complete metadata schema evolution protocol plan Tasks completed: 3/3 - Create METADATA_SCHEMAS.md (all 10 metadata objects) - Create METADATA_EVOLUTION_PROTOCOL.md (formal evolution rules) - Update Claude memory with documentation references SUMMARY: .planning/quick/019-metadata-schema-evolution-protocol/019-SUMMARY.md Co-Authored-By: Claude Opus 4.6 Entire-Checkpoint: 74ee8c0674c8 --- .planning/STATE.md | 9 +- .../019-SUMMARY.md | 96 +++++++++++++++++++ 2 files changed, 101 insertions(+), 4 deletions(-) create mode 100644 .planning/quick/019-metadata-schema-evolution-protocol/019-SUMMARY.md diff --git a/.planning/STATE.md b/.planning/STATE.md index 50bbde213..a53912cba 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -12,7 +12,7 @@ See: .planning/PROJECT.md (updated 2026-02-11) Phase: 13 (File Versioning) Plan: 5 of 5 Status: Phase complete -Last activity: 2026-02-19 -- Completed 13-05-PLAN.md (Recovery Tool + Final Verification) +Last activity: 2026-02-21 -- Completed quick task 019 (Metadata Schema Evolution Protocol) Progress: [#########################] (M1 complete, M2 Phase 12 complete, Phase 12.2 complete, Phase 12.3 complete, Phase 12.3.1 complete, Phase 12.4 complete, Phase 12.5 complete, Phase 12.6 complete, Phase 12.1 complete, Phase 11.1: 7/7 COMPLETE, Phase 11.2: 3/3 COMPLETE, Phase 13: 5/5 COMPLETE) @@ -203,6 +203,7 @@ Recent decisions affecting current work: | 016 | Refine wallet and MFA UI elements | 2026-02-16 | d004eb0 | [016-refine-wallet-and-mfa-ui-elements](./quick/016-refine-wallet-and-mfa-ui-elements/) | | 017 | Desktop binary staging release | 2026-02-19 | 8351fd2 | [017-desktop-binary-staging-release](./quick/017-desktop-binary-staging-release/) | | 018 | E2E versioning tests | 2026-02-19 | 3fd131e | [018-e2e-versioning-tests](./quick/018-e2e-versioning-tests/) | +| 019 | Metadata schema evolution protocol | 2026-02-21 | dcb49e1 | [019-metadata-schema-evolution-protocol](./quick/019-metadata-schema-evolution-protocol/) | ### Research Flags @@ -222,12 +223,12 @@ Recent decisions affecting current work: ## Session Continuity -Last session: 2026-02-19 -Stopped at: Completed quick task 018: E2E versioning tests +Last session: 2026-02-21 +Stopped at: Completed quick task 019: Metadata Schema Evolution Protocol Resume file: None Next: Run /gsd:discuss-phase 14 or /gsd:plan-phase 14 for User-to-User Sharing. --- _State initialized: 2026-01-20_ -_Last updated: 2026-02-19 after completing quick task 018 (E2E versioning tests)_ +_Last updated: 2026-02-21 after completing quick task 019 (Metadata Schema Evolution Protocol)_ diff --git a/.planning/quick/019-metadata-schema-evolution-protocol/019-SUMMARY.md b/.planning/quick/019-metadata-schema-evolution-protocol/019-SUMMARY.md new file mode 100644 index 000000000..62924e88f --- /dev/null +++ b/.planning/quick/019-metadata-schema-evolution-protocol/019-SUMMARY.md @@ -0,0 +1,96 @@ +--- +phase: quick-019 +plan: 01 +subsystem: documentation +tags: [metadata, schema, evolution, protocol, documentation] +dependency-graph: + requires: [] + provides: [metadata-schema-reference, schema-evolution-protocol] + affects: [phase-14-sharing, any-metadata-changes] +tech-stack: + added: [] + patterns: [additive-field-evolution, version-bump-breaking-changes] +key-files: + created: + - docs/METADATA_SCHEMAS.md + - docs/METADATA_EVOLUTION_PROTOCOL.md + modified: [] +decisions: + - id: q019-d01 + description: 'All 10 metadata objects documented with field tables, encryption, storage, and source references' + rationale: 'Ground truth before Phase 14 adds shared folder metadata' + - id: q019-d02 + description: 'Formal additive vs breaking change classification with dual-platform checklist' + rationale: 'Prevents ad-hoc field additions without considering cross-platform compat' +metrics: + duration: 5m + completed: 2026-02-21 +--- + +# Quick Task 019: Metadata Schema Evolution Protocol Summary + +**One-liner:** Complete metadata schema reference (10 objects) and formal evolution protocol with dual-platform checklist for additive vs breaking changes. + +## What Was Done + +### Task 1: METADATA_SCHEMAS.md -- Complete Metadata Reference + +Created `docs/METADATA_SCHEMAS.md` documenting all 10 metadata objects in the system: + +1. **FolderMetadata (v2)** -- top-level folder with children array +2. **FolderChild** -- discriminated union (folder | file) +3. **FolderEntry** -- subfolder with ECIES-wrapped keys +4. **FilePointer** -- slim per-file IPNS reference +5. **FileMetadata (v1)** -- per-file crypto material with optional encryptionMode and versions +6. **VersionEntry** -- past file version with full crypto context +7. **EncryptedVaultKeys** -- ECIES-wrapped root keys for server storage +8. **DeviceRegistry (v1)** -- encrypted device list on IPFS +9. **DeviceEntry** -- individual device record with auth status +10. **Wire Format** -- shared `{iv, data}` JSON envelope for IPFS storage + +Each object includes: + +- Field table with types, encoding, and required/optional +- Encryption algorithm and key used +- Storage location and IPNS addressing +- Source file cross-references (TypeScript and Rust with line numbers) +- Version history documenting when fields were added and whether version was bumped + +Also includes encryption hierarchy table, cross-implementation parity matrix (TS vs Rust), and IPNS key derivation summary (HKDF salt/info for vault, registry, and per-file). + +### Task 2: METADATA_EVOLUTION_PROTOCOL.md -- Formal Evolution Rules + +Created `docs/METADATA_EVOLUTION_PROTOCOL.md` with: + +- **Change classification:** Additive (optional field + sensible default, no version bump) vs Breaking (version bump required) +- **Dangerous gray areas:** Changing defaults, widening unions, reordering arrays, renaming fields +- **Evolution checklist:** 5-section checklist covering before-implementation, TypeScript, Rust, cross-platform verification, and downstream updates +- **Version field convention:** Table showing which schemas have version fields and how versionless schemas (FolderEntry, FilePointer, etc.) evolve through their parent +- **Testing requirements:** Backward compatibility test patterns, cross-platform round-trip tests, unknown field resilience tests +- **Recovery tool compatibility matrix:** When the recovery tool must vs need not be updated + +### Task 3: Claude Memory Reference + +Added "Metadata Schema Documentation" section to MEMORY.md with references to both docs and key rules for future sessions. + +## Decisions Made + +1. **All 10 metadata objects get dedicated sections** -- not just the versioned ones. Wire format, union types, and embedded objects (VersionEntry, DeviceEntry) are documented with the same rigor. +2. **Evolution protocol formalizes the existing informal pattern** -- optional fields with serde defaults are now explicitly classified as "additive non-breaking changes" with documented rules. +3. **Checklist is the central deliverable** -- Section 4 of the protocol is designed for copy-paste into tickets and PRs. + +## Deviations from Plan + +None -- plan executed exactly as written. + +## Commits + +| Task | Commit | Description | +| ---- | ----------- | -------------------------------------- | +| 1 | `b65e7629c` | Add complete metadata schema reference | +| 2 | `dcb49e1bc` | Add metadata schema evolution protocol | +| 3 | -- | MEMORY.md update (not in git repo) | + +## Next Phase Readiness + +These docs are ready for Phase 14 (Sharing) which will add shared folder metadata. The evolution protocol and checklist should be followed when designing the sharing metadata schema. From 41eaea5c03274c76f17cd50edd7a8be382176af0 Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Sat, 21 Feb 2026 02:52:55 +0100 Subject: [PATCH 4/5] docs(quick-019): metadata schema evolution protocol Quick task completed: documented all 10 metadata schemas and created formal evolution protocol with dual-platform checklist. Co-Authored-By: Claude Opus 4.6 Entire-Checkpoint: 969f55a217fb --- .../019-PLAN.md | 461 ++++++++++++++++++ 1 file changed, 461 insertions(+) create mode 100644 .planning/quick/019-metadata-schema-evolution-protocol/019-PLAN.md diff --git a/.planning/quick/019-metadata-schema-evolution-protocol/019-PLAN.md b/.planning/quick/019-metadata-schema-evolution-protocol/019-PLAN.md new file mode 100644 index 000000000..bc5a4f0d7 --- /dev/null +++ b/.planning/quick/019-metadata-schema-evolution-protocol/019-PLAN.md @@ -0,0 +1,461 @@ +--- +phase: quick-019 +plan: 01 +type: execute +wave: 1 +depends_on: [] +files_modified: + - docs/METADATA_SCHEMAS.md + - docs/METADATA_EVOLUTION_PROTOCOL.md +autonomous: true + +must_haves: + truths: + - 'Every metadata object in the system is documented with field types, encryption, storage, and source file references' + - 'A formal evolution protocol exists with clear rules for additive vs breaking changes' + - 'Future tickets that touch metadata can reference these docs for backward compatibility rules' + artifacts: + - path: 'docs/METADATA_SCHEMAS.md' + provides: 'Complete metadata schema reference' + contains: 'FolderMetadata' + - path: 'docs/METADATA_EVOLUTION_PROTOCOL.md' + provides: 'Schema evolution rules and checklist' + contains: 'Breaking Change' + key_links: + - from: 'docs/METADATA_EVOLUTION_PROTOCOL.md' + to: 'docs/METADATA_SCHEMAS.md' + via: 'cross-reference' + pattern: 'METADATA_SCHEMAS.md' +--- + + +Create formal documentation for all CipherBox metadata schemas and a schema evolution protocol. + +Purpose: CipherBox has 10 metadata objects across TypeScript and Rust with no formal evolution rules. Fields have been added informally (optional fields, serde defaults) without version bumps. This documentation establishes the ground truth and future rules before Phase 14 (Sharing) adds shared folder metadata. + +Output: Two docs/ files (METADATA_SCHEMAS.md, METADATA_EVOLUTION_PROTOCOL.md) plus a Claude memory reference. + + + +@./.claude/get-shit-done/workflows/execute-plan.md +@./.claude/get-shit-done/templates/summary.md + + + +@.planning/PROJECT.md +@.planning/STATE.md + +Source files for all metadata types: +@packages/crypto/src/folder/types.ts +@packages/crypto/src/file/types.ts +@packages/crypto/src/vault/types.ts +@packages/crypto/src/registry/types.ts +@packages/crypto/src/folder/metadata.ts +@packages/crypto/src/file/metadata.ts +@packages/crypto/src/registry/schema.ts +@apps/desktop/src-tauri/src/crypto/folder.rs + +Existing docs for style reference: +@docs/VAULT_EXPORT_FORMAT.md + + + + + + Task 1: Create METADATA_SCHEMAS.md -- Complete metadata reference + docs/METADATA_SCHEMAS.md + +Create `docs/METADATA_SCHEMAS.md` documenting all 10 metadata objects. Follow the style and formatting conventions of `docs/VAULT_EXPORT_FORMAT.md` (tables, code blocks, clear field descriptions). + +Structure: + + + +# CipherBox Metadata Schema Reference + +**Version:** 1.0 +**Last Updated:** 2026-02-21 +**Status:** Stable + +## Table of Contents + +(numbered sections) + +## 1. Overview + +- CipherBox stores all metadata encrypted client-side +- Metadata exists in TypeScript (packages/crypto/) and Rust (apps/desktop/src-tauri/src/crypto/) +- Both implementations must produce byte-identical JSON (camelCase field names) + +## 2. Encryption Hierarchy + +Diagram/table showing: + +- FolderMetadata: AES-256-GCM with folderKey -> IPFS (via IPNS) +- FileMetadata: AES-256-GCM with parent's folderKey -> IPFS (per-file IPNS) +- EncryptedVaultKeys: ECIES with user's secp256k1 publicKey -> Server DB +- DeviceRegistry: ECIES with user's secp256k1 publicKey -> IPFS (via IPNS) + +## 3. Wire Format + +Document the EncryptedFolderMetadata / EncryptedFileMetadata envelope: +{ iv: string (hex, 24 chars = 12 bytes), data: string (base64, AES-GCM ciphertext + 16-byte tag) } +Note: Both folder and file metadata use the same envelope format. + +## 4. FolderMetadata (v2) + +- Current version: v2 +- Type definition (TypeScript syntax) +- Field table with: Field | Type | Required | Encoding | Description +- Encryption: AES-256-GCM with this folder's folderKey +- Storage: IPFS, addressed via folder's IPNS name +- Source files: + - TS types: packages/crypto/src/folder/types.ts:15-20 + - TS validator: packages/crypto/src/folder/metadata.ts:32-78 + - Rust: apps/desktop/src-tauri/src/crypto/folder.rs:78-84 +- Version history: + - v1 (initial): children contained inline FileEntry with cid/fileKeyEncrypted/fileIv/size/encryptionMode + - v2 (Phase 12.6): children use FilePointer (slim IPNS reference). v1 completely removed in Phase 11.2. + +## 5. FolderChild (union) + +- Discriminated union on `type` field: 'folder' | 'file' +- TS: FolderEntry | FilePointer +- Rust: serde tagged enum FolderChild { Folder(FolderEntry), File(FilePointer) } +- Source: folder/types.ts:25, folder.rs:66-73 + +## 6. FolderEntry + +- Full field table +- Note: ipnsPrivateKeyEncrypted and folderKeyEncrypted are ECIES-wrapped with parent folder owner's publicKey +- Source: folder/types.ts:31-47, folder.rs:28-45 +- No version history (unchanged since initial design) + +## 7. FilePointer + +- Full field table +- Note: Slim reference -- no crypto material. All file data is in the per-file IPNS record. +- Source: file/types.ts:57-69, folder.rs:50-63 +- Added in Phase 12.6 (replaced inline FileEntry) + +## 8. FileMetadata (v1) + +- Current version: v1 +- Full field table +- Note encryptionMode is optional (defaults to 'GCM') -- added in Phase 12.6 for CTR support +- Note versions is optional (omitted when empty) -- added in Phase 13 +- Encryption: AES-256-GCM with PARENT folder's folderKey (NOT the file's own key) +- Storage: IPFS, addressed via file's own IPNS name (derived via HKDF from user privateKey + fileId) +- Source files: + - TS types: packages/crypto/src/file/types.ts:30-51 + - TS validator: packages/crypto/src/file/metadata.ts:93-188 + - Rust: apps/desktop/src-tauri/src/crypto/folder.rs:166-192 +- Version history: + - v1 (Phase 12.6): Initial per-file IPNS schema + - v1 + encryptionMode (Phase 12.6/12.1): Optional field, defaults to 'GCM'. NOT a version bump. + - v1 + versions (Phase 13): Optional VersionEntry array. NOT a version bump. + - NOTE: Version was NOT bumped for either addition. This is the motivation for the evolution protocol. + +## 9. VersionEntry + +- Full field table +- Note: encryptionMode is REQUIRED (not optional like in FileMetadata) because past versions always record explicit mode +- Source: file/types.ts:10-23, folder.rs:145-160 +- TS validator: file/metadata.ts:32-87 +- Added in Phase 13 + +## 10. EncryptedVaultKeys + +- Full field table (2 fields: encryptedRootFolderKey, encryptedIpnsPrivateKey) +- Note: Uint8Array in TS (not hex strings like other encrypted fields) +- Encryption: ECIES with user's secp256k1 publicKey +- Storage: Server database (users table) +- Source: packages/crypto/src/vault/types.ts:32-37 +- No Rust equivalent (desktop gets vault keys from API response) +- Version history: rootIpnsPublicKey removed in Phase 12.3.1 (derivable from private key) + +## 11. DeviceRegistry (v1) + +- Current version: v1 +- Full field table +- sequenceNumber: monotonically increasing, used for IPNS sequence ordering +- Encryption: ECIES with user's secp256k1 publicKey (entire registry as one blob) +- Storage: IPFS, addressed via device registry IPNS name (derived via HKDF) +- Source: + - TS types: packages/crypto/src/registry/types.ts:54-61 + - TS validator: packages/crypto/src/registry/schema.ts:26-58 +- No Rust equivalent (desktop uses TS types via API) +- Added in Phase 12.2 + +## 12. DeviceEntry + +- Full field table (12 fields) +- Note validation constraints: deviceId = 64 hex chars (SHA-256), publicKey = 130 hex chars (uncompressed secp256k1), ipHash = 64 hex chars +- Note max lengths: name 200, appVersion 50, deviceModel 200 +- Source: registry/types.ts:21-46, validator: registry/schema.ts:63-136 +- Added in Phase 12.2 + +## 13. Cross-Implementation Parity + +Table comparing TS vs Rust for each type: +| Schema | TypeScript | Rust | Notes | + +- FolderMetadata: both +- FolderEntry: both +- FilePointer: both +- FolderChild: both +- FileMetadata: both +- VersionEntry: both +- EncryptedVaultKeys: TS only +- DeviceRegistry: TS only +- DeviceEntry: TS only + +## 14. IPNS Key Derivation Summary + +Table of all HKDF-derived IPNS keys: +| Purpose | Salt | Info | Source | +| Vault root IPNS | CipherBox-v1 | cipherbox-vault-ipns-v1 | vault crypto | +| Device registry IPNS | CipherBox-v1 | cipherbox-device-registry-ipns-v1 | registry crypto | +| Per-file IPNS | CipherBox-v1 | cipherbox-file-ipns-v1:{fileId} | file crypto | +Note: Folder IPNS keys are randomly generated (not derived) and stored ECIES-wrapped in parent folder metadata. + +Do NOT copy-paste type definitions verbatim. Summarize them in field tables (like VAULT_EXPORT_FORMAT.md does). Cross-reference source file paths with line numbers. + + +File exists at docs/METADATA_SCHEMAS.md. Manual review: all 10 metadata objects documented, each has field table + encryption + storage + source refs + version history where applicable. + +Complete metadata reference covering all 10 objects with field tables, encryption details, storage locations, source file cross-references, and version history. + + + + Task 2: Create METADATA_EVOLUTION_PROTOCOL.md -- Formal evolution rules + docs/METADATA_EVOLUTION_PROTOCOL.md + +Create `docs/METADATA_EVOLUTION_PROTOCOL.md` with formal rules for evolving metadata schemas. This is the key deliverable -- it prevents ad-hoc field additions without considering cross-platform compatibility. + +Structure: + + + +# CipherBox Metadata Schema Evolution Protocol + +**Version:** 1.0 +**Last Updated:** 2026-02-21 +**Status:** Active + +## 1. Purpose + +Why this protocol exists: metadata is encrypted and stored on IPFS. Once written, old records may persist indefinitely. Clients of different versions must coexist. Desktop (Rust) and web (TypeScript) must agree on schemas. + +## 2. Guiding Principles + +1. **Backward compatibility is mandatory.** New clients must read old data. Old clients must not crash on new data (graceful degradation). +2. **Forward compatibility via optional fields.** Unknown fields are ignored (JSON.parse/serde default behavior). +3. **Version bumps are breaking changes.** Incrementing the version field signals old clients cannot fully understand the data. This is a last resort. +4. **Both platforms move together.** A schema change is not complete until both TS and Rust implementations are updated. +5. **Validators are the source of truth.** If the validator doesn't check it, the field doesn't exist from a compatibility perspective. + +## 3. Change Classification + +### 3.1 Additive (Non-Breaking) Changes + +- Adding a NEW optional field with a sensible default +- Adding a new enum variant to a string union field (if consumers ignore unknown variants) +- Adding new properties to an existing object field + +Rules: + +- Field MUST be optional in TypeScript (`field?: Type`) +- Field MUST have `#[serde(default)]` in Rust +- Field MUST have `#[serde(skip_serializing_if = "Option::is_none")]` in Rust (or equivalent for Vec with is_empty) +- Validator MUST accept data without the new field +- Validator MUST NOT reject unknown fields (already true -- we use manual validation, not strict schema validation) +- Default value MUST preserve existing behavior (e.g., encryptionMode defaults to 'GCM' because all pre-CTR files were GCM) +- Version field is NOT bumped + +Examples from history: + +- FileMetadata.encryptionMode (Phase 12.6): optional, defaults to 'GCM' +- FileMetadata.versions (Phase 13): optional, omitted when empty + +### 3.2 Breaking Changes (Version Bump Required) + +- Removing a field +- Changing a field's type +- Making an optional field required +- Changing the semantics of an existing field +- Restructuring the children array format + +Rules: + +- Version field MUST be bumped (e.g., 'v1' -> 'v2') +- Old version support: decide one of: + a) Migration path: write converter that upgrades old format to new (preferred for data at rest) + b) Dual support: read both versions, write only new (temporary -- converge within 2 releases) + c) Clean break: reject old format entirely (only when ALL data can be wiped, e.g., pre-production) +- Both TS and Rust validators must be updated simultaneously +- Recovery tool (recovery.html) must handle both old and new versions unless clean break + +Examples from history: + +- FolderMetadata v1 -> v2 (Phase 12.6): Changed file children from inline FileEntry to slim FilePointer. Clean break chosen (pre-production, vault wipe). v1 later removed entirely (Phase 11.2). + +### 3.3 Dangerous Gray Areas + +- Changing default values: This is a breaking change in disguise. If encryptionMode default changed from 'GCM' to 'CTR', old files would be decrypted with wrong mode. +- Widening a union type: Adding 'CTR' to encryptionMode union is safe only if old clients handle unknown modes gracefully (they don't -- they'd pass unknown mode to Web Crypto API and fail). MUST be paired with the optional-field pattern. +- Reordering array elements: versions array order (newest-first) is semantic. Changing to oldest-first would break all version display code. + +## 4. Evolution Checklist + +For EVERY metadata schema change, complete this checklist: + +### 4.1 Before Implementation + +- [ ] Classify change: Additive or Breaking? +- [ ] If additive: What is the default value? Does it preserve existing behavior? +- [ ] If breaking: What version bump strategy? (migration / dual / clean break) +- [ ] Document the change in METADATA_SCHEMAS.md version history section +- [ ] Check: Does the recovery tool (recovery.html) need updating? + +### 4.2 TypeScript Implementation + +- [ ] Update type definition in packages/crypto/src/{domain}/types.ts +- [ ] If optional: use `field?: Type` syntax +- [ ] Update validator in packages/crypto/src/{domain}/metadata.ts or schema.ts +- [ ] If optional: validator accepts undefined/missing field +- [ ] If optional: validator applies default when constructing return object +- [ ] If breaking: validator checks version field and branches +- [ ] Add/update unit tests in packages/crypto for the new/changed field + +### 4.3 Rust Implementation + +- [ ] Update struct in apps/desktop/src-tauri/src/crypto/folder.rs (or relevant file) +- [ ] If optional: use `Option` with `#[serde(default)]` and `#[serde(skip_serializing_if = "Option::is_none")]` +- [ ] If new enum variant: ensure `#[serde(rename_all = "...")]` handles casing +- [ ] Verify JSON round-trip: serialize -> deserialize produces identical output +- [ ] Add/update cargo tests + +### 4.4 Cross-Platform Verification + +- [ ] Generate test JSON from TypeScript, deserialize in Rust (or vice versa) +- [ ] Verify old-format JSON (without new field) deserializes correctly in both +- [ ] Verify new-format JSON (with new field) serializes identically in both +- [ ] Run: `pnpm test` (TS) + `cargo test --features fuse` (Rust) + +### 4.5 Downstream Updates + +- [ ] Recovery tool (recovery.html): handles new field/version +- [ ] Vault export format (docs/VAULT_EXPORT_FORMAT.md): update if export format changes +- [ ] API client regeneration: `pnpm api:generate` if API DTOs changed +- [ ] METADATA_SCHEMAS.md: add entry to version history for affected schema + +## 5. Version Field Convention + +All versioned metadata objects use a string version field: 'v1', 'v2', etc. + +| Schema | Current Version | Has Version Field | +| FolderMetadata | v2 | Yes | +| FileMetadata | v1 | Yes | +| DeviceRegistry | v1 | Yes | +| EncryptedVaultKeys | (none) | No | +| FolderEntry | (none) | No | +| FilePointer | (none) | No | +| VersionEntry | (none) | No | +| DeviceEntry | (none) | No | + +Objects without version fields (FolderEntry, FilePointer, VersionEntry, DeviceEntry) evolve only via their parent's version. For example, changing FilePointer fields requires bumping FolderMetadata version. + +EncryptedVaultKeys has no version field. Changes to its structure should be paired with a new vault export format version. + +## 6. Testing Requirements + +### 6.1 Backward Compatibility Test Pattern + +For every schema change, add a test with a hardcoded JSON string representing the OLD format (before the change). Verify the new validator/deserializer accepts it and produces correct output. + +```typescript +// Example: FileMetadata without versions field (pre-Phase 13) +const oldFormatJson = + '{"version":"v1","cid":"bafy...","fileKeyEncrypted":"04...","fileIv":"aabb...","size":1024,"mimeType":"text/plain","createdAt":1705268100000,"modifiedAt":1705268100000}'; +const result = validateFileMetadata(JSON.parse(oldFormatJson)); +expect(result.versions).toBeUndefined(); +expect(result.encryptionMode).toBe('GCM'); // default applied +``` + +### 6.2 Cross-Platform Round-Trip Test + +Produce JSON from TypeScript, feed to Rust deserializer (via file or inline string in cargo test). Verify all fields parse correctly. + +## 7. Recovery Tool Compatibility Matrix + +The recovery tool (apps/web/public/recovery.html) is a standalone HTML file that must work independently. It has its own inline implementations of crypto operations. + +When updating schemas: + +- The recovery tool MUST be updated for any change that affects how files are discovered or decrypted +- Additive metadata fields that don't affect crypto operations (e.g., a hypothetical `tags` field) do NOT require recovery tool updates +- Breaking changes (version bumps) ALWAYS require recovery tool updates + +## 8. References + +- Metadata Schema Reference: docs/METADATA_SCHEMAS.md +- Vault Export Format: docs/VAULT_EXPORT_FORMAT.md +- Technical Architecture: 00-Preliminary-R&D/Documentation/TECHNICAL_ARCHITECTURE.md +- Data Flows: 00-Preliminary-R&D/Documentation/DATA_FLOWS.md + +Write clean, specific prose. Do NOT use vague language. Every rule must be actionable. The checklist in Section 4 is the most important part -- it's what developers will actually use. + + +File exists at docs/METADATA_EVOLUTION_PROTOCOL.md. Manual review: contains change classification (additive vs breaking), checklist covering TS/Rust/recovery/tests, version field convention table, and backward compatibility test pattern. + +Formal evolution protocol with classification rules, dual-platform checklist, version convention, testing requirements, and recovery tool compatibility matrix. + + + + Task 3: Update Claude memory with metadata documentation reference + + +Add the following entry to the MEMORY.md file (under a new "## Metadata Schema Documentation" heading, placed after the "## Patterns and Lessons" section): + +```markdown +## Metadata Schema Documentation + +- **Schema reference:** `docs/METADATA_SCHEMAS.md` -- documents all 10 metadata objects (FolderMetadata, FileMetadata, DeviceRegistry, etc.) with field tables, encryption, storage, source file cross-references +- **Evolution protocol:** `docs/METADATA_EVOLUTION_PROTOCOL.md` -- formal rules for evolving metadata schemas (additive vs breaking changes, version bump rules, dual-platform checklist) +- **ALWAYS consult these docs** before any ticket that adds/removes/changes fields on metadata objects +- Evolution checklist (Section 4 of protocol) must be completed for every schema change +- Key rule: optional fields with sensible defaults = no version bump; removing/changing fields = version bump required +``` + +This ensures future Claude sessions are aware of these docs when working on metadata-related tickets. + +NOTE: The MEMORY.md file is at `/Users/michael/.claude/projects/-Users-michael-Code-cipher-box/memory/MEMORY.md`. Read it first, then append the new section. + + +MEMORY.md contains the new "Metadata Schema Documentation" section with references to both docs. + +Claude memory updated with metadata documentation references for future ticket awareness. + + + + + +1. `docs/METADATA_SCHEMAS.md` exists and documents all 10 metadata objects +2. `docs/METADATA_EVOLUTION_PROTOCOL.md` exists with formal evolution rules +3. Both files follow the style conventions of existing `docs/VAULT_EXPORT_FORMAT.md` +4. Claude MEMORY.md references both new documents +5. No source code was modified (documentation only) + + + + +- All 10 metadata objects documented with field tables, encryption keys, storage locations, and source file references +- Evolution protocol contains actionable classification rules and a checklist covering both TypeScript and Rust +- Version history for each schema captures when fields were added and whether version was bumped +- Claude memory references these docs for future metadata-related tickets + + + +After completion, create `.planning/quick/019-metadata-schema-evolution-protocol/019-SUMMARY.md` + From 67b80dd8465847e5e2b123b4e1912b1d0a41f3bc Mon Sep 17 00:00:00 2001 From: Michael Yankelev Date: Sat, 21 Feb 2026 03:03:36 +0100 Subject: [PATCH 5/5] docs(learnings): quick-019 - metadata schema evolution protocol Co-Authored-By: Claude Opus 4.6 Entire-Checkpoint: e672595ddac3 --- ...2-21-metadata-schema-evolution-protocol.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 .learnings/2026-02-21-metadata-schema-evolution-protocol.md diff --git a/.learnings/2026-02-21-metadata-schema-evolution-protocol.md b/.learnings/2026-02-21-metadata-schema-evolution-protocol.md new file mode 100644 index 000000000..07f64c291 --- /dev/null +++ b/.learnings/2026-02-21-metadata-schema-evolution-protocol.md @@ -0,0 +1,33 @@ +# Metadata Schema Evolution Protocol + +**Date:** 2026-02-21 + +## Original Prompt + +> Create a formal metadata schema evolution protocol for all metadata objects created by the system. Ensure that this documentation is referenced in claude memory for future tickets that make changes to the metadata. + +## What I Learned + +- CipherBox has 10 metadata objects across TypeScript and Rust that must produce byte-identical JSON +- Fields were added informally (optional + serde defaults) without version bumps through Phase 13 -- no protocol existed +- The `FileMetadata.version` field stayed at `'v1'` despite two additive changes (`encryptionMode`, `versions`) -- this is defensible but means the version field is useless for feature detection +- `FolderMetadata` had a clean-break v1->v2 migration (pre-production vault wipe) -- this strategy is only valid when all data can be wiped +- Changing a default value is a breaking change in disguise (e.g., changing `encryptionMode` default from `'GCM'` to `'CTR'` would silently decrypt old files with the wrong algorithm) +- Objects without their own version field (FolderEntry, FilePointer, VersionEntry, DeviceEntry) evolve through their parent's version +- The recovery tool (`apps/web/public/recovery.html`) has its own inline crypto implementations and must be updated independently for any change affecting file discovery or decryption + +## What Would Have Helped + +- A formal protocol from Phase 12.6 onward (when per-file IPNS was introduced) would have prevented the informal pattern +- Cross-platform round-trip tests (TS -> Rust -> TS) should be standard for every schema change + +## Key Files + +- `docs/METADATA_SCHEMAS.md` -- complete reference for all 10 metadata objects +- `docs/METADATA_EVOLUTION_PROTOCOL.md` -- formal evolution rules and checklist (Section 4 is the actionable checklist) +- `packages/crypto/src/file/types.ts` -- FileMetadata, VersionEntry, FilePointer types +- `packages/crypto/src/folder/types.ts` -- FolderMetadata, FolderEntry, FolderChild types +- `packages/crypto/src/registry/types.ts` -- DeviceRegistry, DeviceEntry types +- `packages/crypto/src/vault/types.ts` -- EncryptedVaultKeys, VaultInit types +- `apps/desktop/src-tauri/src/crypto/folder.rs` -- Rust equivalents for all FUSE-relevant types +- `apps/web/public/recovery.html` -- standalone recovery tool with inline metadata parsing