diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc new file mode 100644 index 00000000..18d87145 --- /dev/null +++ b/.markdownlint-cli2.jsonc @@ -0,0 +1,6 @@ +{ + "config": { + "MD013": false, + }, + "ignores": ["node_modules/**", "packages/**/node_modules/**", "target/**"], +} diff --git a/CHANGELOG.md b/CHANGELOG.md index 18f2c25f..f0667ec1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -587,6 +587,10 @@ Applied, Rejected, Obstructed}` with receipt evidence and typed contract ### Fixed +- `echo-file-aperture` now normalizes `HostFileSnapshot` material at the + aperture boundary so caller-forged snapshot metadata or fingerprints cannot + bind a basis, observation receipt, or materialization verification to bytes + different from the observed host bytes. - `scripts/verify-local.sh` stamp storage now resolves the real gitdir via `git rev-parse --git-dir`, so pre-commit and pre-push hooks work in linked worktrees where `.git` is a file rather than a directory. Stamps are diff --git a/Cargo.lock b/Cargo.lock index 835479f9..c400fa40 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -614,6 +614,14 @@ dependencies = [ "warp-core", ] +[[package]] +name = "echo-file-aperture" +version = "0.1.0" +dependencies = [ + "blake3", + "thiserror 2.0.17", +] + [[package]] name = "echo-graph" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index ce8bfa2f..433ffdc2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,6 +11,7 @@ members = [ "crates/warp-geom", "crates/warp-benches", "crates/echo-app-core", + "crates/echo-file-aperture", "crates/echo-config-fs", "crates/echo-session-proto", "crates/echo-graph", @@ -42,6 +43,7 @@ echo-cas = { version = "0.1.0", path = "crates/echo-cas" } echo-config-fs = { version = "0.1.0", path = "crates/echo-config-fs" } echo-dind-tests = { version = "0.1.0", path = "crates/echo-dind-tests" } echo-dry-tests = { version = "0.1.0", path = "crates/echo-dry-tests" } +echo-file-aperture = { version = "0.1.0", path = "crates/echo-file-aperture" } echo-graph = { version = "0.1.0", path = "crates/echo-graph" } echo-runtime-schema = { version = "0.1.0", path = "crates/echo-runtime-schema", default-features = false } echo-registry-api = { version = "0.1.0", path = "crates/echo-registry-api" } diff --git a/METHOD.md b/METHOD.md index 890fb8eb..962c503d 100644 --- a/METHOD.md +++ b/METHOD.md @@ -24,42 +24,58 @@ The Echo work doctrine: A backlog, a loop, and honest bookkeeping. | **`CONTINUUM.md`** | Platform memo: hot/cold split and shared contracts. | | **`METHOD.md`** | Repo work doctrine (this document). | -## Backlog Lanes +## Backlog -| Lane | Purpose | -| :---------------- | :---------------------------------------------------------- | -| **`asap/`** | Imminent work; pull into the next cycle. | -| **`v0.1.0/`** | Release-bar blockers for the local contract-host milestone. | -| **`up-next/`** | Queued after `asap/`, outside the release-bar lane. | -| **`cool-ideas/`** | Uncommitted experiments. | -| **`bad-code/`** | Technical debt that must be addressed. | -| **`inbox/`** | Raw ideas. | +GitHub Issues are the live backlog. Labels are the index. -`v0.1.0/` is a milestone lane, not an automatic priority override. Move a -release-blocking card into `asap/` only when it becomes the current pull. +| Label | Purpose | +| :-------------------- | :----------------------------- | +| **`lane:inbox`** | Unprocessed work or raw ideas. | +| **`lane:asap`** | Pull into a cycle soon. | +| **`lane:up-next`** | Queued after `asap`. | +| **`lane:cool-ideas`** | Uncommitted experiments. | +| **`lane:bad-code`** | Technical debt. | +| **`lane:release`** | Release-bar work. | + +Historical filesystem backlog cards are archived under +`docs/method/graveyard/github-issue-migration/`. The `docs/method/backlog/` +directory remains only as a compatibility marker for legacy `cargo xtask +method ...` workspace discovery. Do not add new live work cards there. ## The Cycle Loop ```mermaid stateDiagram-v2 direction LR - [*] --> Pull: asap/ + [*] --> Pull: issue Pull --> Branch: cycle/ - Branch --> Red: failing tests + Branch --> Design: template + Design --> Red: failing tests Red --> Green: passing tests Green --> Retro: findings/debt Retro --> Ship: PR to main Ship --> [*] ``` -1. **Pull**: Move an item from `backlog/asap/` to `docs/design/`. -2. **Branch**: Create `cycle/-`. -3. **Red**: Write failing tests. Playback questions become specs. -4. **Green**: Make them pass. Fix determinism drift (DIND). -5. **Retro**: Document findings and follow-on debt in the cycle doc. -6. **Ship**: Open a PR to `main`. Update `BEARING.md` and `CHANGELOG.md` after merge. +1. **Pull**: Choose a GitHub Issue with Method labels. Create or update a + design doc under `docs/design/` and link it to the issue. +2. **Branch**: Create a branch named from the issue title, or + `cycle/-` when the work is explicitly cycle-shaped. +3. **Design**: Use `docs/method/design-template.md`. +4. **Red**: Write failing tests from the design's playback questions. +5. **Green**: Make them pass. Fix determinism drift with DIND when relevant. +6. **Witness**: Record the reproducible proof. +7. **Retro**: Document drift and follow-on debt. +8. **Ship**: Open a PR to `main`. + +Design docs may define intent, but they do not prove implementation. Runtime +and product work must include at least one executable behavior witness such as +Rust API behavior, CLI output, WASM ABI behavior, schema validation, WAL/WSC +recovery, DIND determinism, or generated contract behavior. ## Naming Convention -Backlog and cycle files follow: `-.md` or `-.md` -Example: `RE-017-byte-pipeline.md` +Issues use readable titles and Method labels. Branches use lowercase slugs of +the issue title unless a cycle id is explicitly assigned. + +Design docs use the Echo template at `docs/method/design-template.md`. diff --git a/crates/echo-file-aperture/Cargo.toml b/crates/echo-file-aperture/Cargo.toml new file mode 100644 index 00000000..5f1b5285 --- /dev/null +++ b/crates/echo-file-aperture/Cargo.toml @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: Apache-2.0 +# © James Ross Ω FLYING•ROBOTS +[package] +name = "echo-file-aperture" +version = "0.1.0" +edition = "2021" +license.workspace = true +repository.workspace = true +rust-version.workspace = true +description = "Echo-owned host file aperture contract" +readme = "README.md" +keywords = ["echo", "files", "causal", "aperture"] +categories = ["filesystem", "data-structures"] + +[dependencies] +blake3 = "1.5" +thiserror = "2" + +[lints] +workspace = true diff --git a/crates/echo-file-aperture/README.md b/crates/echo-file-aperture/README.md new file mode 100644 index 00000000..d40fdffe --- /dev/null +++ b/crates/echo-file-aperture/README.md @@ -0,0 +1,26 @@ + + + +# echo-file-aperture + +`echo-file-aperture` is the Echo-owned standard artifact for host-file +observation, basis tracking, content admission, and materialization +verification. + +It is not a filesystem runtime and it is not a FUSE adapter. Callers provide +host capabilities such as reading bytes, writing bytes, and collecting path +evidence. This crate owns the deterministic contract that turns those host +facts into file sites, fingerprints, basis tokens, projections, receipts, and +obstructions. + +File-site identity is resolved by authority tier: + +- platform identity derives a `PlatformStable` `FileSiteId`; +- path-only evidence derives a weaker `PathBound` `FileSiteId`. + +Path evidence is observation evidence, not durable authority. `FileSiteId` +binds local host-aperture observations; it is not `WorldlineId` and must not be +treated as portable WSC causal identity. + +The first slice is intentionally in-memory. Later slices should attach these +receipts and retained materials to Echo WAL/WSC recovery surfaces. diff --git a/crates/echo-file-aperture/src/lib.rs b/crates/echo-file-aperture/src/lib.rs new file mode 100644 index 00000000..9b6abbc8 --- /dev/null +++ b/crates/echo-file-aperture/src/lib.rs @@ -0,0 +1,782 @@ +// SPDX-License-Identifier: Apache-2.0 +// © James Ross Ω FLYING•ROBOTS +//! Echo-owned host file aperture contract. +//! +//! A host file is not Echo state. It is an observed boundary artifact and a +//! materialization target. This crate owns the deterministic file-aperture +//! contract used by Echo consumers such as editors and projection adapters. +//! +//! The first implementation slice is in-memory: callers supply host bytes and +//! path evidence, while the aperture records file-site basis, admitted content, +//! observation posture, and materialization verification posture. Later slices +//! should bind these records to WAL/WSC retention and Echo scheduler receipts. + +#![forbid(unsafe_code)] + +use std::collections::BTreeMap; +use std::fmt; + +/// Number of bytes in Echo file-aperture digests. +const DIGEST_LEN: usize = 32; + +/// Domain prefix for platform-stable file-site identity. +const FILE_SITE_PLATFORM_DOMAIN: &[u8] = b"echo.file-site.v2.platform"; +/// Domain prefix for path-bound file-site identity. +const FILE_SITE_PATH_BOUND_DOMAIN: &[u8] = b"echo.file-site.v2.path-bound"; +/// Domain prefix for host metadata fingerprints. +const HOST_METADATA_DOMAIN: &[u8] = b"echo:file-aperture:metadata:v1"; +/// Domain prefix for basis tokens. +const BASIS_DOMAIN: &[u8] = b"echo:file-aperture:basis:v1"; + +/// Stable Echo identity for a file-like host artifact. +#[repr(transparent)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct FileSiteId([u8; DIGEST_LEN]); + +impl FileSiteId { + /// Derives a file site id from the strongest available identity tier. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if identity evidence is + /// too large to length-prefix deterministically. + pub fn for_identity(identity: &HostFileIdentity) -> Result { + Ok(identity.site_resolution()?.file_site_id) + } + + /// Derives a platform-stable file site id. + /// + /// Platform identity should include any host or aperture namespace needed to + /// make the bytes meaningful within the local filesystem aperture. This id + /// is host-aperture evidence, not portable WSC causal identity. + /// + /// # Errors + /// + /// Returns [`FileApertureError::EmptyPlatformIdentity`] when + /// `platform_identity` is empty, or [`FileApertureError::LengthOverflow`] + /// when it is too large to length-prefix deterministically. + pub fn from_platform_identity(platform_identity: &[u8]) -> Result { + if platform_identity.is_empty() { + return Err(FileApertureError::EmptyPlatformIdentity); + } + let mut hasher = blake3::Hasher::new(); + hasher.update(FILE_SITE_PLATFORM_DOMAIN); + update_len_prefixed(&mut hasher, platform_identity)?; + Ok(Self(*hasher.finalize().as_bytes())) + } + + /// Derives a path-bound file site id. + /// + /// Path-bound identity is weaker than platform-stable identity. It is useful + /// when no stronger host-local identity exists, but a rename cannot be + /// proven as the same file site from path evidence alone. + /// + /// # Errors + /// + /// Returns [`FileApertureError::EmptyPathEvidence`] when `path_evidence` is + /// empty, or [`FileApertureError::LengthOverflow`] when it is too large to + /// length-prefix deterministically. + pub fn from_path_bound_evidence(path_evidence: &[u8]) -> Result { + if path_evidence.is_empty() { + return Err(FileApertureError::EmptyPathEvidence); + } + let mut hasher = blake3::Hasher::new(); + hasher.update(FILE_SITE_PATH_BOUND_DOMAIN); + update_len_prefixed(&mut hasher, path_evidence)?; + Ok(Self(*hasher.finalize().as_bytes())) + } + + /// Constructs a file site id from raw digest bytes. + #[must_use] + pub const fn from_bytes(bytes: [u8; DIGEST_LEN]) -> Self { + Self(bytes) + } + + /// Returns raw digest bytes. + #[must_use] + pub const fn as_bytes(&self) -> &[u8; DIGEST_LEN] { + &self.0 + } +} + +impl fmt::Display for FileSiteId { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write_hex(f, &self.0) + } +} + +/// Identity posture for an Echo file site resolution. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum FileSiteIdentityPosture { + /// Site id was derived from stable host-local platform identity evidence. + PlatformStable, + /// Site id was derived only from path evidence and is rename-sensitive. + PathBound, +} + +/// Resolved host identity for a file-like local aperture artifact. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct FileSiteResolution { + /// Resolved host-aperture file site id. + pub file_site_id: FileSiteId, + /// Identity strength that produced `file_site_id`. + pub posture: FileSiteIdentityPosture, +} + +/// Content digest for a file projection. +#[repr(transparent)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct FileContentDigest([u8; DIGEST_LEN]); + +impl FileContentDigest { + /// Computes a content digest from exact file bytes. + #[must_use] + pub fn for_bytes(bytes: &[u8]) -> Self { + Self(*blake3::hash(bytes).as_bytes()) + } + + /// Constructs a content digest from raw digest bytes. + #[must_use] + pub const fn from_bytes(bytes: [u8; DIGEST_LEN]) -> Self { + Self(bytes) + } + + /// Returns raw digest bytes. + #[must_use] + pub const fn as_bytes(&self) -> &[u8; DIGEST_LEN] { + &self.0 + } +} + +impl fmt::Display for FileContentDigest { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write_hex(f, &self.0) + } +} + +/// Echo-owned token naming the causal basis for a file projection. +#[repr(transparent)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct FileBasisToken([u8; DIGEST_LEN]); + +impl FileBasisToken { + /// Derives a basis token from a site id, generation, and content digest. + #[must_use] + pub fn derive(site_id: FileSiteId, generation: u64, digest: FileContentDigest) -> Self { + let mut hasher = blake3::Hasher::new(); + hasher.update(BASIS_DOMAIN); + hasher.update(site_id.as_bytes()); + hasher.update(&generation.to_le_bytes()); + hasher.update(digest.as_bytes()); + Self(*hasher.finalize().as_bytes()) + } + + /// Constructs a basis token from raw digest bytes. + #[must_use] + pub const fn from_bytes(bytes: [u8; DIGEST_LEN]) -> Self { + Self(bytes) + } + + /// Returns raw digest bytes. + #[must_use] + pub const fn as_bytes(&self) -> &[u8; DIGEST_LEN] { + &self.0 + } +} + +impl fmt::Display for FileBasisToken { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write_hex(f, &self.0) + } +} + +/// Host-supplied evidence used to locate and identify file material. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct HostFileIdentity { + /// Host path bytes or path-like evidence supplied by the caller. + pub path_evidence: Vec, + /// Optional platform file identity, such as namespaced device/inode or file + /// id bytes. + pub platform_identity: Option>, +} + +impl HostFileIdentity { + /// Builds host file identity evidence. + /// + /// Path evidence is required because it is the user's visible observation + /// coordinate even when a stronger platform identity is available. + /// + /// # Errors + /// + /// Returns [`FileApertureError::EmptyPathEvidence`] when `path_evidence` + /// is empty, [`FileApertureError::EmptyPlatformIdentity`] when + /// `platform_identity` is present but empty, or + /// [`FileApertureError::LengthOverflow`] when evidence is too large to + /// length-prefix deterministically. + pub fn new( + path_evidence: impl Into>, + platform_identity: Option>, + ) -> Result { + let path_evidence = path_evidence.into(); + if path_evidence.is_empty() { + return Err(FileApertureError::EmptyPathEvidence); + } + ensure_len_fits(path_evidence.len())?; + if let Some(platform_identity) = &platform_identity { + if platform_identity.is_empty() { + return Err(FileApertureError::EmptyPlatformIdentity); + } + ensure_len_fits(platform_identity.len())?; + } + Ok(Self { + path_evidence, + platform_identity, + }) + } + + /// Resolves the Echo file site id and identity posture for this host + /// identity. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if identity evidence is + /// too large to length-prefix deterministically. + pub fn site_resolution(&self) -> Result { + if let Some(platform_identity) = &self.platform_identity { + return Ok(FileSiteResolution { + file_site_id: FileSiteId::from_platform_identity(platform_identity)?, + posture: FileSiteIdentityPosture::PlatformStable, + }); + } + + Ok(FileSiteResolution { + file_site_id: FileSiteId::from_path_bound_evidence(&self.path_evidence)?, + posture: FileSiteIdentityPosture::PathBound, + }) + } + + /// Derives the Echo file site id for this host identity. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if identity evidence is + /// too large to length-prefix deterministically. + pub fn site_id(&self) -> Result { + Ok(self.site_resolution()?.file_site_id) + } +} + +/// Deterministic metadata retained with a host observation. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct HostFileMetadata { + /// Exact observed byte length. + pub byte_len: u64, +} + +impl HostFileMetadata { + /// Builds metadata for exact file bytes. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if the byte length cannot + /// be represented in Echo's deterministic metadata encoding. + pub fn for_bytes(bytes: &[u8]) -> Result { + Ok(Self { + byte_len: ensure_len_fits(bytes.len())?, + }) + } + + /// Computes a deterministic metadata digest. + #[must_use] + pub fn digest(&self) -> [u8; DIGEST_LEN] { + let mut hasher = blake3::Hasher::new(); + hasher.update(HOST_METADATA_DOMAIN); + hasher.update(&self.byte_len.to_le_bytes()); + *hasher.finalize().as_bytes() + } +} + +/// Fingerprint of observed host file content and metadata. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct HostFileFingerprint { + /// Digest of exact file bytes. + pub content_digest: FileContentDigest, + /// Exact observed byte length. + pub byte_len: u64, + /// Digest of deterministic host metadata retained by this slice. + pub metadata_digest: [u8; DIGEST_LEN], +} + +impl HostFileFingerprint { + /// Builds a host fingerprint from exact bytes and metadata. + #[must_use] + pub fn from_parts(bytes: &[u8], metadata: HostFileMetadata) -> Self { + Self { + content_digest: FileContentDigest::for_bytes(bytes), + byte_len: metadata.byte_len, + metadata_digest: metadata.digest(), + } + } +} + +/// Exact host file material supplied to the aperture. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct HostFileSnapshot { + /// Host identity evidence for the observed material. + pub identity: HostFileIdentity, + /// Exact bytes observed from the host. + pub bytes: Vec, + /// Deterministic metadata for the observed material. + pub metadata: HostFileMetadata, + /// Content and metadata fingerprint for the observation. + pub fingerprint: HostFileFingerprint, +} + +impl HostFileSnapshot { + /// Builds a host file snapshot from identity evidence and exact bytes. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if the bytes are too long + /// for deterministic metadata encoding. + pub fn new( + identity: HostFileIdentity, + bytes: impl Into>, + ) -> Result { + let bytes = bytes.into(); + let metadata = HostFileMetadata::for_bytes(&bytes)?; + let fingerprint = HostFileFingerprint::from_parts(&bytes, metadata); + Ok(Self { + identity, + bytes, + metadata, + fingerprint, + }) + } + + fn into_observed_material(self) -> Result { + let metadata = HostFileMetadata::for_bytes(&self.bytes)?; + let fingerprint = HostFileFingerprint::from_parts(&self.bytes, metadata); + Ok(ObservedHostFileMaterial { + identity: self.identity, + bytes: self.bytes, + fingerprint, + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +struct ObservedHostFileMaterial { + identity: HostFileIdentity, + bytes: Vec, + fingerprint: HostFileFingerprint, +} + +/// Bounded Echo projection of file content at a basis. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FileContentProjection { + /// Echo file site being projected. + pub site_id: FileSiteId, + /// Identity strength for the projected file site. + pub site_identity_posture: FileSiteIdentityPosture, + /// Basis token for this exact projection. + pub basis: FileBasisToken, + /// Content digest for the projected bytes. + pub content_digest: FileContentDigest, + /// Exact projected byte length. + pub byte_len: u64, + /// Exact projected bytes. + pub bytes: Vec, +} + +/// Posture assigned to a host observation. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum HostObservationPosture { + /// The file site had no prior causal basis and this observation imported it. + InitialImport, + /// The observed bytes matched the current Echo basis. + Unchanged, + /// The host bytes differed from the current Echo basis and were admitted + /// as external drift. + ExternalChange, +} + +/// Receipt returned after accepting host file material into the aperture. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct HostFileObservationReceipt { + /// Deterministic per-aperture observation sequence. + pub observation_id: u64, + /// File site named by the observation. + pub site_id: FileSiteId, + /// Identity strength used to resolve `site_id`. + pub site_identity_posture: FileSiteIdentityPosture, + /// Observation posture. + pub posture: HostObservationPosture, + /// Host fingerprint observed by this receipt. + pub fingerprint: HostFileFingerprint, + /// Projection returned to the caller after reconciliation. + pub projection: FileContentProjection, +} + +/// Proposed file content against an explicit Echo basis. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FileContentProposal { + /// Echo file site the proposal targets. + pub site_id: FileSiteId, + /// Basis token the caller claims to have edited from. + pub basis: FileBasisToken, + /// Desired file bytes. + pub bytes: Vec, +} + +impl FileContentProposal { + /// Builds a file content proposal. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if the desired bytes are + /// too long for deterministic metadata encoding. + pub fn new( + site_id: FileSiteId, + basis: FileBasisToken, + bytes: impl Into>, + ) -> Result { + let bytes = bytes.into(); + ensure_len_fits(bytes.len())?; + Ok(Self { + site_id, + basis, + bytes, + }) + } +} + +/// Posture assigned to an admitted content proposal. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ContentAdmissionPosture { + /// Proposal matched the current basis and did not advance content. + Unchanged, + /// Proposal changed content and advanced the file basis. + AdmittedChange, +} + +/// Receipt returned after admitting desired file content. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FileContentIntentReceipt { + /// File site targeted by the content intent. + pub site_id: FileSiteId, + /// Content admission posture. + pub posture: ContentAdmissionPosture, + /// Basis token supplied by the caller. + pub previous_basis: FileBasisToken, + /// Projection after admission. + pub projection: FileContentProjection, +} + +/// Posture assigned to host materialization verification. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum MaterializationVerificationPosture { + /// Host material matched the admitted projection digest. + Verified, + /// Host material did not match the admitted projection digest. + DigestMismatch, +} + +/// Receipt returned after checking materialized host bytes. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct FileMaterializationReceipt { + /// File site whose materialization was checked. + pub site_id: FileSiteId, + /// Basis token expected by the verification. + pub basis: FileBasisToken, + /// Expected content digest from Echo's admitted projection. + pub expected_digest: FileContentDigest, + /// Observed host content digest. + pub observed_digest: FileContentDigest, + /// Materialization verification posture. + pub posture: MaterializationVerificationPosture, +} + +/// Errors returned by the file aperture contract. +#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)] +pub enum FileApertureError { + /// Host identity did not include path evidence. + #[error("host file identity path evidence is empty")] + EmptyPathEvidence, + /// Host identity carried an empty platform identity. + #[error("host file identity platform identity is empty")] + EmptyPlatformIdentity, + /// Input length cannot be represented in deterministic file aperture + /// encodings. + #[error("file aperture input length overflows deterministic encoding")] + LengthOverflow, + /// The requested file site has no admitted basis. + #[error("unknown file site {site_id}")] + UnknownFileSite { + /// Unknown file site id. + site_id: FileSiteId, + }, + /// A caller supplied an old or unrelated basis token. + #[error("stale file basis for {site_id}: expected {expected}, got {actual}")] + StaleBasis { + /// File site whose basis check failed. + site_id: FileSiteId, + /// Current Echo basis token. + expected: FileBasisToken, + /// Caller-supplied basis token. + actual: FileBasisToken, + }, + /// Host materialization evidence named a different file site. + #[error("host file identity mapped to {observed_site_id}, expected {expected_site_id}")] + SiteIdentityMismatch { + /// Expected file site id. + expected_site_id: FileSiteId, + /// Observed file site id. + observed_site_id: FileSiteId, + }, +} + +/// In-memory implementation of the Echo file aperture contract. +#[derive(Default, Debug)] +pub struct InMemoryFileAperture { + next_observation_id: u64, + sites: BTreeMap, +} + +impl InMemoryFileAperture { + /// Accepts host file material and returns a reconciled Echo projection. + /// + /// # Errors + /// + /// Returns [`FileApertureError::LengthOverflow`] if host identity evidence + /// cannot be encoded deterministically. + pub fn observe( + &mut self, + snapshot: HostFileSnapshot, + ) -> Result { + let snapshot = snapshot.into_observed_material()?; + let site_resolution = snapshot.identity.site_resolution()?; + let site_id = site_resolution.file_site_id; + let observation_id = self.next_observation_id; + self.next_observation_id = self + .next_observation_id + .checked_add(1) + .ok_or(FileApertureError::LengthOverflow)?; + + let fingerprint = snapshot.fingerprint; + let posture = match self.sites.get_mut(&site_id) { + Some(state) if state.content_digest == fingerprint.content_digest => { + HostObservationPosture::Unchanged + } + Some(state) => { + state.advance(fingerprint.content_digest, snapshot.bytes)?; + HostObservationPosture::ExternalChange + } + None => { + self.sites.insert( + site_id, + FileState::new( + fingerprint.content_digest, + snapshot.bytes, + site_resolution.posture, + ), + ); + HostObservationPosture::InitialImport + } + }; + + let projection = self.projection(site_id)?; + Ok(HostFileObservationReceipt { + observation_id, + site_id, + site_identity_posture: site_resolution.posture, + posture, + fingerprint, + projection, + }) + } + + /// Returns the current projection for a file site. + /// + /// # Errors + /// + /// Returns [`FileApertureError::UnknownFileSite`] when `site_id` has no + /// admitted basis. + pub fn projection( + &self, + site_id: FileSiteId, + ) -> Result { + let state = self + .sites + .get(&site_id) + .ok_or(FileApertureError::UnknownFileSite { site_id })?; + state.projection(site_id) + } + + /// Admits desired content against the current file basis. + /// + /// This is the causal content-intent step before any host filesystem write. + /// + /// # Errors + /// + /// Returns [`FileApertureError::UnknownFileSite`] when the file site is not + /// known, or [`FileApertureError::StaleBasis`] when the proposal basis does + /// not match the current Echo basis. + pub fn propose_content( + &mut self, + proposal: FileContentProposal, + ) -> Result { + let state = + self.sites + .get_mut(&proposal.site_id) + .ok_or(FileApertureError::UnknownFileSite { + site_id: proposal.site_id, + })?; + let current_basis = state.basis(proposal.site_id); + if current_basis != proposal.basis { + return Err(FileApertureError::StaleBasis { + site_id: proposal.site_id, + expected: current_basis, + actual: proposal.basis, + }); + } + + let proposed_digest = FileContentDigest::for_bytes(&proposal.bytes); + let posture = if proposed_digest == state.content_digest { + ContentAdmissionPosture::Unchanged + } else { + state.advance(proposed_digest, proposal.bytes)?; + ContentAdmissionPosture::AdmittedChange + }; + let projection = state.projection(proposal.site_id)?; + Ok(FileContentIntentReceipt { + site_id: proposal.site_id, + posture, + previous_basis: current_basis, + projection, + }) + } + + /// Verifies that host material matches an admitted Echo projection. + /// + /// # Errors + /// + /// Returns [`FileApertureError::UnknownFileSite`] when the site is not + /// known, [`FileApertureError::StaleBasis`] when `basis` is not current, or + /// [`FileApertureError::SiteIdentityMismatch`] when `snapshot` names a + /// different file site. + pub fn verify_materialization( + &self, + site_id: FileSiteId, + basis: FileBasisToken, + snapshot: HostFileSnapshot, + ) -> Result { + let snapshot = snapshot.into_observed_material()?; + let observed_site_id = snapshot.identity.site_id()?; + if observed_site_id != site_id { + return Err(FileApertureError::SiteIdentityMismatch { + expected_site_id: site_id, + observed_site_id, + }); + } + + let state = self + .sites + .get(&site_id) + .ok_or(FileApertureError::UnknownFileSite { site_id })?; + let current_basis = state.basis(site_id); + if current_basis != basis { + return Err(FileApertureError::StaleBasis { + site_id, + expected: current_basis, + actual: basis, + }); + } + + let expected_digest = state.content_digest; + let observed_digest = snapshot.fingerprint.content_digest; + let posture = if expected_digest == observed_digest { + MaterializationVerificationPosture::Verified + } else { + MaterializationVerificationPosture::DigestMismatch + }; + Ok(FileMaterializationReceipt { + site_id, + basis, + expected_digest, + observed_digest, + posture, + }) + } +} + +#[derive(Clone, Debug)] +struct FileState { + generation: u64, + content_digest: FileContentDigest, + identity_posture: FileSiteIdentityPosture, + bytes: Vec, +} + +impl FileState { + fn new( + content_digest: FileContentDigest, + bytes: Vec, + identity_posture: FileSiteIdentityPosture, + ) -> Self { + Self { + generation: 0, + content_digest, + identity_posture, + bytes, + } + } + + fn advance( + &mut self, + content_digest: FileContentDigest, + bytes: Vec, + ) -> Result<(), FileApertureError> { + ensure_len_fits(bytes.len())?; + self.generation = self + .generation + .checked_add(1) + .ok_or(FileApertureError::LengthOverflow)?; + self.content_digest = content_digest; + self.bytes = bytes; + Ok(()) + } + + fn basis(&self, site_id: FileSiteId) -> FileBasisToken { + FileBasisToken::derive(site_id, self.generation, self.content_digest) + } + + fn projection(&self, site_id: FileSiteId) -> Result { + Ok(FileContentProjection { + site_id, + site_identity_posture: self.identity_posture, + basis: self.basis(site_id), + content_digest: self.content_digest, + byte_len: ensure_len_fits(self.bytes.len())?, + bytes: self.bytes.clone(), + }) + } +} + +fn ensure_len_fits(len: usize) -> Result { + u64::try_from(len).map_err(|_error| FileApertureError::LengthOverflow) +} + +fn update_len_prefixed(hasher: &mut blake3::Hasher, bytes: &[u8]) -> Result<(), FileApertureError> { + let len = ensure_len_fits(bytes.len())?; + hasher.update(&len.to_le_bytes()); + hasher.update(bytes); + Ok(()) +} + +fn write_hex(f: &mut fmt::Formatter<'_>, bytes: &[u8; DIGEST_LEN]) -> fmt::Result { + for byte in bytes { + write!(f, "{byte:02x}")?; + } + Ok(()) +} diff --git a/crates/echo-file-aperture/tests/file_aperture_tests.rs b/crates/echo-file-aperture/tests/file_aperture_tests.rs new file mode 100644 index 00000000..22e9eb39 --- /dev/null +++ b/crates/echo-file-aperture/tests/file_aperture_tests.rs @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: Apache-2.0 +// © James Ross Ω FLYING•ROBOTS +//! Contract tests for the Echo-owned file aperture. + +use echo_file_aperture::{ + ContentAdmissionPosture, FileApertureError, FileContentDigest, FileContentProposal, FileSiteId, + FileSiteIdentityPosture, HostFileFingerprint, HostFileIdentity, HostFileSnapshot, + HostObservationPosture, InMemoryFileAperture, MaterializationVerificationPosture, +}; + +fn snapshot(path: &str, bytes: &[u8]) -> Result { + HostFileSnapshot::new( + HostFileIdentity::new(path.as_bytes(), None)?, + bytes.to_vec(), + ) +} + +fn identity_with_platform( + path: &str, + platform_identity: &[u8], +) -> Result { + HostFileIdentity::new(path.as_bytes(), Some(platform_identity.to_vec())) +} + +#[test] +fn platform_identity_keeps_file_site_stable_across_path_move() -> Result<(), FileApertureError> { + let before_move = identity_with_platform("/tmp/a/demo.txt", b"host-a:dev-1:inode-42")?; + let after_move = identity_with_platform("/tmp/b/demo.txt", b"host-a:dev-1:inode-42")?; + let before_resolution = before_move.site_resolution()?; + let after_resolution = after_move.site_resolution()?; + + assert_eq!( + before_resolution.file_site_id, + after_resolution.file_site_id + ); + assert_eq!( + before_resolution.posture, + FileSiteIdentityPosture::PlatformStable + ); + assert_eq!( + after_resolution.posture, + FileSiteIdentityPosture::PlatformStable + ); + Ok(()) +} + +#[test] +fn different_platform_identity_wins_over_same_path() -> Result<(), FileApertureError> { + let first = identity_with_platform("/tmp/demo.txt", b"host-a:dev-1:inode-42")?; + let second = identity_with_platform("/tmp/demo.txt", b"host-a:dev-1:inode-43")?; + + assert_ne!(first.site_id()?, second.site_id()?); + assert_eq!( + first.site_resolution()?.posture, + FileSiteIdentityPosture::PlatformStable + ); + Ok(()) +} + +#[test] +fn path_only_identity_is_explicitly_path_bound() -> Result<(), FileApertureError> { + let first = HostFileIdentity::new(b"/tmp/demo.txt", None)?; + let second = HostFileIdentity::new(b"/tmp/demo.txt", None)?; + let resolution = first.site_resolution()?; + + assert_eq!(resolution.file_site_id, second.site_id()?); + assert_eq!(resolution.posture, FileSiteIdentityPosture::PathBound); + Ok(()) +} + +#[test] +fn path_only_move_creates_distinct_path_bound_sites() -> Result<(), FileApertureError> { + let before_move = HostFileIdentity::new(b"/tmp/a/demo.txt", None)?; + let after_move = HostFileIdentity::new(b"/tmp/b/demo.txt", None)?; + + assert_ne!(before_move.site_id()?, after_move.site_id()?); + assert_eq!( + before_move.site_resolution()?.posture, + FileSiteIdentityPosture::PathBound + ); + assert_eq!( + after_move.site_resolution()?.posture, + FileSiteIdentityPosture::PathBound + ); + Ok(()) +} + +#[test] +fn platform_site_derivation_does_not_include_path_bytes() -> Result<(), FileApertureError> { + let identity = identity_with_platform("/tmp/demo.txt", b"host-a:dev-1:inode-42")?; + let platform_site = FileSiteId::from_platform_identity(b"host-a:dev-1:inode-42")?; + let path_bound_site = FileSiteId::from_path_bound_evidence(b"/tmp/demo.txt")?; + + assert_eq!(identity.site_id()?, platform_site); + assert_ne!(identity.site_id()?, path_bound_site); + Ok(()) +} + +#[test] +fn host_observation_receipt_exposes_identity_posture() -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + let snapshot = HostFileSnapshot::new( + identity_with_platform("/tmp/demo.txt", b"host-a:dev-1:inode-42")?, + b"one".to_vec(), + )?; + + let receipt = aperture.observe(snapshot)?; + + assert_eq!( + receipt.site_identity_posture, + FileSiteIdentityPosture::PlatformStable + ); + assert_eq!( + receipt.projection.site_identity_posture, + FileSiteIdentityPosture::PlatformStable + ); + Ok(()) +} + +#[test] +fn empty_platform_identity_is_invalid() { + let error = HostFileIdentity::new(b"/tmp/demo.txt", Some(Vec::new())); + + assert!(matches!( + error, + Err(FileApertureError::EmptyPlatformIdentity) + )); +} + +#[test] +fn unknown_host_file_admits_initial_import_before_projection() -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + + let receipt = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + + assert_eq!(receipt.observation_id, 0); + assert_eq!(receipt.posture, HostObservationPosture::InitialImport); + assert_eq!( + receipt.site_identity_posture, + FileSiteIdentityPosture::PathBound + ); + assert_eq!( + receipt.projection.site_identity_posture, + FileSiteIdentityPosture::PathBound + ); + assert_eq!(receipt.projection.bytes, b"one"); + assert_eq!( + receipt.projection.content_digest, + receipt.fingerprint.content_digest + ); + Ok(()) +} + +#[test] +fn forged_snapshot_fingerprint_cannot_override_observed_bytes() -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + let identity = HostFileIdentity::new(b"/tmp/demo.txt", None)?; + let forged_metadata = echo_file_aperture::HostFileMetadata { byte_len: 3 }; + let forged_snapshot = HostFileSnapshot { + identity, + bytes: b"one".to_vec(), + metadata: forged_metadata, + fingerprint: HostFileFingerprint::from_parts(b"two", forged_metadata), + }; + + let receipt = aperture.observe(forged_snapshot)?; + + assert_eq!(receipt.projection.bytes, b"one"); + assert_eq!( + receipt.projection.content_digest, + FileContentDigest::for_bytes(b"one") + ); + assert_eq!( + receipt.fingerprint.content_digest, + FileContentDigest::for_bytes(b"one") + ); + Ok(()) +} + +#[test] +fn unchanged_host_file_records_no_change_projection() -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + let first = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + + let second = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + + assert_eq!(second.observation_id, 1); + assert_eq!(second.posture, HostObservationPosture::Unchanged); + assert_eq!(second.projection.basis, first.projection.basis); + assert_eq!(second.projection.bytes, b"one"); + Ok(()) +} + +#[test] +fn changed_host_file_admits_external_change_transition() -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + let first = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + + let second = aperture.observe(snapshot("/tmp/demo.txt", b"two")?)?; + + assert_eq!(second.posture, HostObservationPosture::ExternalChange); + assert_ne!(second.projection.basis, first.projection.basis); + assert_eq!(second.projection.bytes, b"two"); + Ok(()) +} + +#[test] +fn save_current_basis_admits_content_and_verifies_materialization() -> Result<(), FileApertureError> +{ + let mut aperture = InMemoryFileAperture::default(); + let initial = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + let proposal = + FileContentProposal::new(initial.site_id, initial.projection.basis, b"three".to_vec())?; + + let admitted = aperture.propose_content(proposal)?; + let materialized = aperture.verify_materialization( + admitted.site_id, + admitted.projection.basis, + snapshot("/tmp/demo.txt", b"three")?, + )?; + + assert_eq!(admitted.posture, ContentAdmissionPosture::AdmittedChange); + assert_eq!(admitted.projection.bytes, b"three"); + assert_eq!( + materialized.posture, + MaterializationVerificationPosture::Verified + ); + assert_eq!(materialized.expected_digest, materialized.observed_digest); + Ok(()) +} + +#[test] +fn stale_basis_obstructs_save_without_changing_projection() -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + let initial = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + let changed = aperture.observe(snapshot("/tmp/demo.txt", b"two")?)?; + let proposal = + FileContentProposal::new(initial.site_id, initial.projection.basis, b"three".to_vec())?; + + let error = aperture.propose_content(proposal); + let projection = aperture.projection(changed.site_id)?; + + assert!(matches!(error, Err(FileApertureError::StaleBasis { .. }))); + assert_eq!(projection.basis, changed.projection.basis); + assert_eq!(projection.bytes, b"two"); + Ok(()) +} + +#[test] +fn verification_digest_mismatch_returns_materialization_obstruction( +) -> Result<(), FileApertureError> { + let mut aperture = InMemoryFileAperture::default(); + let initial = aperture.observe(snapshot("/tmp/demo.txt", b"one")?)?; + let proposal = + FileContentProposal::new(initial.site_id, initial.projection.basis, b"three".to_vec())?; + let admitted = aperture.propose_content(proposal)?; + + let materialized = aperture.verify_materialization( + admitted.site_id, + admitted.projection.basis, + snapshot("/tmp/demo.txt", b"not-three")?, + )?; + + assert_eq!( + materialized.posture, + MaterializationVerificationPosture::DigestMismatch + ); + assert_ne!(materialized.expected_digest, materialized.observed_digest); + Ok(()) +} diff --git a/docs/BEARING.md b/docs/BEARING.md index 6638adf3..3d158efe 100644 --- a/docs/BEARING.md +++ b/docs/BEARING.md @@ -3,7 +3,7 @@ # BEARING -Last updated: 2026-05-25. +Last updated: 2026-06-03. This signpost summarizes current direction. It does not create commitments or replace backlog items, design docs, retros, or CLI status. If it disagrees with @@ -24,6 +24,9 @@ Trusted runtime-control history is defined in The causal WAL doctrine and recovery design is defined in `docs/design/causal-wal-end-to-end.md`. +The Echo-owned file aperture design is defined in +`docs/design/echo-owned-file-aperture.md`. + The WAL/WSC storage relationship is tracked in `docs/method/backlog/v0.1.0/PLATFORM_wal-wsc-storage-relationship.md`. diff --git a/docs/design/echo-owned-file-aperture.md b/docs/design/echo-owned-file-aperture.md new file mode 100644 index 00000000..77584f9a --- /dev/null +++ b/docs/design/echo-owned-file-aperture.md @@ -0,0 +1,516 @@ + + + + + +--- +title: "PLATFORM-0533 - Echo-Owned File Aperture" +legend: "PLATFORM" +lane: "design" +issue: "https://github.com/flyingrobots/echo/issues/533" +status: "active" +owners: + - "@flyingrobots" +created: "2026-06-06" +updated: "2026-06-06" +--- + + + +# PLATFORM-0533 - Echo-Owned File Aperture + +## Linked Issue + +- [Issue #533](https://github.com/flyingrobots/echo/issues/533) + +## Decision Summary + +Echo owns the standard host-file aperture contract used by Jedit, WARP DRIVE, +and other local-file consumers. The aperture observes host bytes, resolves +host-local file-site identity posture, admits external drift, accepts content +proposals against explicit basis tokens, and verifies materialization without +letting applications maintain private causal file ledgers. + +## Sponsored Human + +A Jedit user wants to open any file from disk, see exactly the bytes there, +edit normally, and save normally so that the editor feels ordinary, without +having to understand Echo observations, basis tokens, drift receipts, or +materialization verification. + +## Sponsored Agent + +An agent needs a machine-readable file aperture contract so it can inspect why +a file has particular bytes, replay or reconstruct a retained coordinate, and +distinguish host observations from materializations, without inferring state +from Jedit UI caches, WARP DRIVE FUSE internals, or private app logs. + +## Hill + +By the end of this cycle, Echo exposes a reusable Rust file aperture crate that +can observe host file material, resolve file-site identity posture, project +bytes at a basis, obstruct stale proposals, and verify materialized host bytes, +and the repo proves it with focused `echo-file-aperture` contract tests. + +## Current Truth + +Before this branch, Echo already had the runtime ingredients for witnessed +admission, scheduler-owned receipt correlation, QueryView readings, retained +evidence posture, and app-noun-clean contract hosting: +[`docs/BEARING.md#52:d5945169511a5ab937824fd6e49adf06f4c19ae9`](https://github.com/flyingrobots/echo/blob/d5945169511a5ab937824fd6e49adf06f4c19ae9/docs/BEARING.md#L52) +and +[`docs/BEARING.md#66:d5945169511a5ab937824fd6e49adf06f4c19ae9`](https://github.com/flyingrobots/echo/blob/d5945169511a5ab937824fd6e49adf06f4c19ae9/docs/BEARING.md#L66). + +Before this branch, the workspace did not contain an `echo-file-aperture` +member; the merge-target workspace member list moved directly from +`echo-app-core` to `echo-config-fs`: +[`Cargo.toml#13:d5945169511a5ab937824fd6e49adf06f4c19ae9`](https://github.com/flyingrobots/echo/blob/d5945169511a5ab937824fd6e49adf06f4c19ae9/Cargo.toml#L13). + +Before this branch, Method used filesystem backlog cards as the primary pull +source and did not yet require every design packet to link a GitHub Issue: +[`METHOD.md#55:d5945169511a5ab937824fd6e49adf06f4c19ae9`](https://github.com/flyingrobots/echo/blob/d5945169511a5ab937824fd6e49adf06f4c19ae9/METHOD.md#L55). + +The implementation added by this branch is intentionally in-memory. It does +not yet bind observations, proposals, or materialization receipts to WAL/WSC +durability. + +## Problem + +Host files sit at an awkward boundary. Users call opening a file a read and +saving a file a write, but at the Echo boundary both actions can create causal +history. Opening can import unknown bytes or admit external drift. Saving must +admit desired content against an explicit basis before any host write can be +called successful. + +The first implementation also exposed a real contract bug: `FileSiteId` was +originally derived from both `path_evidence` and `platform_identity`, making it +path-sensitive even when stable platform identity existed. That poisoned the +rename story because the same host-local file moved from `/a/demo.txt` to +`/b/demo.txt` became two different file sites. + +## Scope + +This cycle includes: + +- adding the `echo-file-aperture` crate as a workspace member; +- defining host snapshot, file site, basis, content proposal, observation, and + materialization verification types; +- resolving file-site identity by authority tier instead of hashing all + evidence together; +- exposing `PlatformStable` versus `PathBound` identity posture; +- proving path moves preserve `FileSiteId` only when platform identity is + available; +- documenting that `FileSiteId` is host-aperture evidence, not portable WSC + causal identity. + +## Non-Goals + +This cycle does not include: + +- implementing a FUSE mount in Echo; +- making Echo core a filesystem runtime; +- adding Jedit editor nouns to Echo production core; +- using WARP DRIVE inode or fixture-tree scaffolding as the shared contract; +- wiring file aperture records into scheduler-owned ticks, WAL, WSC, or CAS; +- guaranteeing reconstruction after Echo retention policy redacts, prunes, + encrypts, or obstructs required support material. + +## User Experience / Product Shape + +The happy-path product shape remains ordinary: + +```text +open file -> contents appear +edit -> text changes +save -> saved +external edit -> current disk bytes appear or a clear obstruction is reported +``` + +The Echo-facing open flow is: + +```mermaid +flowchart TD + Open[User selects path] --> HostRead[Host bytes and identity evidence] + HostRead --> Resolve[Resolve FileSiteId and identity posture] + Resolve --> Compare{Matches current Echo basis?} + Compare -->|unknown| Import[Initial import] + Compare -->|changed| Drift[External-change admission] + Compare -->|same| Unchanged[No-change observation] + Import --> Projection[FileContentProjection] + Drift --> Projection + Unchanged --> Projection + Projection --> Render[Client renders exact bytes] +``` + +The Echo-facing save flow is: + +```mermaid +flowchart TD + Save[User saves] --> Proposal[FileContentProposal with explicit basis] + Proposal --> Basis{Basis current?} + Basis -->|no| Stale[StaleBasis obstruction] + Basis -->|yes| Admit[Admit content proposal] + Admit --> Materialize[Host writes admitted bytes] + Materialize --> Verify[Verify final host digest] + Verify --> Receipt[FileMaterializationReceipt] +``` + +### Accessibility Considerations + +Rendered clients should not expose Echo jargon during successful open and save. +When an obstruction affects the user, clients should translate the structured +posture into clear product language while preserving machine-readable receipts. + +## Runtime / API Contract + +The contract is the `echo-file-aperture` Rust crate. + +Relevant exported types: + +- `HostFileIdentity` +- `HostFileSnapshot` +- `FileSiteId` +- `FileSiteIdentityPosture` +- `FileSiteResolution` +- `FileContentDigest` +- `FileBasisToken` +- `FileContentProjection` +- `HostFileObservationReceipt` +- `FileContentProposal` +- `FileContentIntentReceipt` +- `FileMaterializationReceipt` +- `FileApertureError` +- `InMemoryFileAperture` + +Identity rules: + +- path evidence is observation evidence; +- platform identity is stronger host-local file evidence; +- `FileSiteId` is host-aperture evidence used to bind local file observations; +- `FileSiteId` is not `WorldlineId`; +- `WorldlineId` remains portable logical document identity; +- `ContentRef` or equivalent retained material refs name immutable bytes; +- future `BindingRef` or `AnchorId` machinery should express rebinding between + host file sites and causal document worldlines. + +Derivation rules: + +```text +if platform_identity exists: + FileSiteId = H("echo.file-site.v2.platform", platform_identity) + posture = PlatformStable +else: + FileSiteId = H("echo.file-site.v2.path-bound", path_evidence) + posture = PathBound +``` + +Platform identity bytes should include any host or aperture namespace needed to +make the bytes meaningful for that local filesystem aperture. Echo must not +treat platform identity as universal WSC identity across machines. + +## Lower Modes + +This cycle is a Rust API and test-surface change, not a rendered UI. The lower +mode is the deterministic contract test output from `cargo test -p +echo-file-aperture`. Future CLI or pipe surfaces must expose posture as +structured data, not English-only text. + +## Data / State Model + +| Category | Description | +| ------------------------- | --------------------------------------------------------------------------- | +| Source of truth | In this slice, `InMemoryFileAperture` state; future slices bind to WAL/WSC. | +| Derived state | `FileSiteId`, `FileBasisToken`, content digest, metadata digest. | +| Invalid states | Empty path evidence, empty platform identity, stale basis, site mismatch. | +| Reset behavior | Dropping the in-memory aperture drops state; this is intentional for now. | +| Serialization | Stable byte preimages use explicit domain prefixes and length prefixes. | +| Deterministic assumptions | `BTreeMap` ordering and BLAKE3 digests provide deterministic local results. | + +```mermaid +stateDiagram-v2 + [*] --> Unknown + Unknown --> InitialImport: observe unknown host bytes + InitialImport --> Unchanged: observe same digest + InitialImport --> ExternalChange: observe changed host digest + ExternalChange --> Proposed: propose content at current basis + Proposed --> StaleBasis: basis mismatch + Proposed --> Admitted: basis current + Admitted --> Verified: host digest matches projection + Admitted --> DigestMismatch: host digest differs +``` + +## Echo Authority Boundary + +Echo owns: + +- file-site resolution posture; +- basis-token derivation; +- content admission; +- stale-basis obstruction; +- materialization verification posture; +- future WAL/WSC retention for file aperture evidence. + +Applications and host adapters provide: + +- host path evidence; +- platform identity evidence when available; +- exact observed bytes; +- host read/write capabilities; +- product-specific UI and localized copy. + +Applications must not keep private causal file logs as fallback authority. If +Echo lacks retained evidence because of policy, redaction, corruption, or +missing material, the information is unavailable through Echo. + +## Determinism / DIND Posture + +This slice touches hashing and canonical identity. Hash preimages are +domain-separated and length-prefixed. File aperture state uses `BTreeMap` so +iteration order remains deterministic. No clocks, randomness, or wall-time +cadence are involved. + +The focused determinism witness is the contract test matrix in +`crates/echo-file-aperture/tests/file_aperture_tests.rs`. Full DIND is deferred +until file aperture records bind into scheduler-owned ticks or WAL/WSC replay. + +## WAL / WSC / Retention Posture + +This slice is explicitly in-memory. It defines the contract shape and +deterministic identity behavior but does not claim crash-recoverable file +history. + +Future WAL/WSC work must retain enough evidence to answer: + +- which host observation was accepted; +- which basis was current; +- whether a proposal was admitted or obstructed; +- which materialization was authorized; +- whether verification matched the admitted projection; +- why reconstruction is unavailable when support material is missing. + +## Accessibility Posture + +| Concern | Posture | +| --------------------------------- | -------------------------------------------------------------------- | +| Semantic labels or facts | Receipts expose posture enums and digests. | +| Focus order or ownership | Not applicable; this is not a rendered surface. | +| Hidden or visual-only information | Not applicable; the contract is inspectable without UI scraping. | +| Keyboard behavior | Not applicable. | +| Secret or redaction behavior | Future retention must report missing/redacted material structurally. | + +## Localization / Directionality Posture + +No user-visible strings are added to product surfaces. Error messages are Rust +diagnostics, not localized UI copy. Clients such as Jedit and WARP DRIVE own +localized wording for user-facing file-open, file-save, and obstruction flows. + +## Agent Inspectability / Explainability Posture + +Agents can inspect: + +- `FileSiteId`; +- `FileSiteIdentityPosture`; +- `HostFileObservationReceipt`; +- `FileBasisToken`; +- `ContentAdmissionPosture`; +- `MaterializationVerificationPosture`; +- expected and observed digests; +- typed `FileApertureError` variants. + +No agent has to scrape pixels, terminal prose, or app-local caches to tell +whether identity was platform-stable or path-bound. + +## Linked Invariants + +- Tests are executable spec. +- Echo owns causality; applications own presentation. +- Paths are evidence, not authority. +- `FileSiteId` is not `WorldlineId`. +- Missing retained evidence obstructs; apps do not invent fallback truth. +- Echo core remains app-noun-clean. + +## Design Alternatives Considered + +### Option A: Hash all identity evidence together + +Pros: + +- Simple implementation. +- Every changed input changes the digest. + +Cons: + +- Path moves create new file sites even when stable platform identity exists. +- The preimage hides authority precedence behind field order. +- It violates the path-as-evidence invariant. + +### Option B: Select one authority tier + +Pros: + +- Platform-stable identity survives rename. +- Path-only identity is honestly weaker and visibly `PathBound`. +- Domain-separated preimages make authority tier explicit. + +Cons: + +- Callers must inspect posture. +- Future cross-machine rebinding needs explicit `WorldlineId` or binding + machinery instead of pretending host IDs are portable. + +## Decision + +Choose Option B. `FileSiteId` derivation selects a single identity authority +tier. Platform identity wins when present and path evidence remains observation +evidence. Path-only identity remains supported but is explicitly path-bound. + +## Implementation Slices + +- [x] Slice 1: Record Echo-owned file aperture design. +- [x] Slice 2: Add `echo-file-aperture` crate with in-memory observations, + proposals, basis obstruction, and materialization verification. +- [x] Slice 3: Fix `FileSiteId` derivation so platform identity does not include + path bytes and expose identity posture. +- [ ] Slice 4: Bind file aperture observations and receipts to scheduler-owned + ticks. +- [ ] Slice 5: Retain file aperture evidence through WAL/WSC/CAS. +- [ ] Slice 6: Add client conformance fixtures for Jedit and WARP DRIVE. + +## Tests To Write First + +Behavior tests required: + +- [x] Unknown host file admits initial import before returning projection. +- [x] Known unchanged host file records no-change posture. +- [x] Known changed host file admits external-change transition. +- [x] Current-basis save admits content and verifies materialization. +- [x] Stale-basis save obstructs without changing projection. +- [x] Digest mismatch returns materialization obstruction. +- [x] Same platform identity with different path evidence keeps one + `FileSiteId`. +- [x] Different platform identity with same path evidence produces different + `FileSiteId`. +- [x] Path-only identity is explicitly `PathBound`. +- [x] Platform identity derivation does not include path bytes. + +Documentation and process tests: + +- [x] Design packet links GitHub Issue #533. +- [x] Design packet names real validation commands. + +## Acceptance Criteria + +The work is done when: + +- [x] Behavior tests prove the in-memory file aperture contract. +- [x] Runtime API exposes `PlatformStable` versus `PathBound` posture. +- [x] Path moves preserve `FileSiteId` only when stable platform identity is + available. +- [x] Docs state that `FileSiteId` is not `WorldlineId`. +- [x] Issue and design packet are linked. +- [ ] WAL/WSC durability is proven in a later cycle. +- [ ] Jedit and WARP DRIVE consume the shared contract in later cycles. + +## Validation Plan + +Commands expected before PR: + +```bash +docs=( + docs/design/echo-owned-file-aperture.md + crates/echo-file-aperture/README.md + docs/method/design-template.md + docs/method/README.md + docs/technical-teardown.md + METHOD.md + docs/BEARING.md +) + +cargo fmt --check -p echo-file-aperture +cargo test -p echo-file-aperture +cargo clippy -p echo-file-aperture --all-targets -- -D warnings +cargo check --workspace +./scripts/check-no-app-nouns-in-core.sh +pnpm exec markdownlint-cli2 "${docs[@]}" +pnpm exec prettier --check "${docs[@]}" +dead_ref_args=() +for doc in "${docs[@]}"; do + dead_ref_args+=(--file "$doc") +done +cargo xtask lint-dead-refs "${dead_ref_args[@]}" +git diff --check origin/main...HEAD +``` + +Full `cargo xtask lint-dead-refs` is currently blocked by repository baseline +dead links from the historical backlog migration. Use the touched-file scoped +command for this cycle unless that baseline is repaired separately. + +## Playback / Witness + +Run: + +```bash +cargo test -p echo-file-aperture +``` + +Review the identity-specific tests: + +- `platform_identity_keeps_file_site_stable_across_path_move` +- `different_platform_identity_wins_over_same_path` +- `path_only_identity_is_explicitly_path_bound` +- `path_only_move_creates_distinct_path_bound_sites` +- `platform_site_derivation_does_not_include_path_bytes` + +These prove the corrected authority-tier behavior. + +## Risks + +Known risks: + +- Platform identity bytes from two host apertures can collide if callers omit + host namespace evidence. +- Path-bound IDs can still be mistaken for durable document identity. +- Whole-file content proposals are simple but may be wasteful. +- In-memory receipts can be mistaken for durability. + +Mitigations: + +- Document that platform identity should include host/aperture namespace. +- Expose `FileSiteIdentityPosture` everywhere callers need to inspect site + strength. +- Keep `FileSiteId` separate from `WorldlineId`. +- Leave WAL/WSC integration as explicit follow-on work. + +## Follow-On Debt + +- Create WAL/WSC integration issues when the next file aperture cycle begins. +- Add shared conformance fixtures for Jedit and WARP DRIVE. +- Add an explanation surface equivalent to WARP DRIVE's proposed + `/.warp/why/` and Jedit's history drawer. +- Decide whether future APIs need explicit `BindingRef` or `AnchorId` types for + local file site to worldline rebinding. + +## Retrospective + +What changed from the design: + +- The review found a P1 identity bug. `FileSiteId` now selects one authority + tier instead of hashing path and platform evidence together. +- The design packet now follows the Echo template and links Issue #533. + +What the tests proved: + +- Platform-stable identity survives path moves. +- Path-only identity is explicit and rename-sensitive. +- Platform-derived `FileSiteId` preimages do not include path bytes. +- Stale-basis and materialization-obstruction behavior still works. + +What remains open: + +- WAL/WSC durability. +- Scheduler-owned admission for file aperture events. +- Jedit and WARP DRIVE client integration. + +PR: + +- Not opened yet for this branch. diff --git a/docs/method/README.md b/docs/method/README.md index 48a13bd5..408579a4 100644 --- a/docs/method/README.md +++ b/docs/method/README.md @@ -183,8 +183,8 @@ Same loop regardless: - **Feature** — design, test, build, ship. - **Design** — the deliverable is docs, not code. -- **Debt** — pull from `bad-code/`. The hill is "this no longer - bothers us." +- **Debt** — pull from GitHub Issues labeled `lane:bad-code`. The hill is + "this no longer bothers us." --- @@ -231,25 +231,55 @@ in one sentence, the cycle is too big. Split it. 1. **Design** — write a design doc from the template at `docs/method/design-template.md`. Required sections: - - **Title and legend** — cycle number, name, legend link. - - **Why this cycle exists** — motivation and context. - - **Depends on** — explicit dependency chain (or "nothing"). - - **Human users / jobs / hills** — who benefits, what they do, - one-sentence hill from the human perspective. - - **Agent users / jobs / hills** — same, from the agent - perspective. - - **Human playback** — concrete walk-through scenario proving - the human hill. - - **Agent playback** — concrete walk-through scenario proving - the agent hill. - - **Implementation outline** — numbered steps of what the code - (or docs) will do. - - **Tests to write first** — the RED phase, named in the design. - - **Risks / unknowns** — what might go wrong. - - **Postures** — accessibility, localization, agent - inspectability. If not relevant, say so explicitly. Silence - is not a position. - - **Non-goals** — what this cycle will not do. + - **Linked Issue** — public tracker for the pulled work. + - **Decision Summary** — the specific boundary and behavior this + cycle will create or change. + - **Sponsored Human / Sponsored Agent / Hill** — who benefits and + the observable outcome they need. + - **Current Truth** — factual evidence from files, commands, APIs, + issues, PRs, and known tests. Strong claims cite permalinks with + fully qualified commit SHAs. + - **Problem, Scope, Non-Goals** — the defect or gap, what this + cycle includes, and what it explicitly excludes. + - **User Experience / Product Shape** — required for rendered UI, + visible CLI, docs, or user-facing behavior. + - **Runtime / API Contract** — required for code-facing changes. + - **Lower Modes** — required for user-visible, agent-visible, CLI, + JSON, fixture, dry-run, DIND, or partial-evidence behavior. + - **Data / State Model** — source of truth, derived state, invalid + states, reset behavior, serialization, and deterministic + assumptions. + - **Echo Authority Boundary** — who owns causality, admission, + receipts, readings, materialization, retention, recovery, and + host capabilities. + - **Determinism / DIND Posture** — required when runtime, + scheduling, math, replay, storage ordering, host observation, + clocks, randomness, hashing, or canonicalization are involved. + - **WAL / WSC / Retention Posture** — required when causal + history, durable recovery, retained material, receipts, + readings, snapshots, checkpoints, materialization, or missing + evidence are involved. + - **Accessibility, Localization, Agent Inspectability** — state + the posture or say "Not applicable" with a reason. Silence is + not a position. + - **App-Noun Boundary** — required when work is motivated by + Jedit, WARP DRIVE, git-warp, Graft, Bijou, or another external + consumer. + - **Linked Invariants, Alternatives, Decision** — the governing + repo invariants, rejected options, and chosen approach. + - **Implementation Slices, Tests To Write First, Acceptance + Criteria, Validation Plan, Playback / Witness, Risks, + Follow-On Debt, Retrospective** — how the cycle will be proven + and closed. + + A design doc may define intent, but it does not prove + implementation. For implementation work, at least one required test + must exercise the actual software surface: Rust API, runtime + behavior, CLI output, WASM ABI, schema validation, WAL/WSC recovery, + DIND determinism, generated contract behavior, or another executable + surface. Documentation assertions are allowed as evidence-ledger + checks, but they cannot be the only acceptance test for runtime or + product work. 2. **RED** — write failing tests. Playback questions become specs. Default to agent surface first. @@ -269,8 +299,8 @@ in one sentence, the cycle is too big. Split it. 5. **Close** — write the retro and witness packet on the branch. - Drift check (mandatory). Undocumented drift is the only true failure mode. - - New debt to `bad-code/`. - - Cool ideas to `cool-ideas/`. + - New debt becomes a GitHub Issue labeled `lane:bad-code`. + - Cool ideas become GitHub Issues labeled `lane:cool-ideas`. - Backlog maintenance. Closing the cycle packet does not mean `main` has accepted it yet. @@ -409,7 +439,7 @@ The following commands are planned but **not yet implemented**: | Command | Purpose | | ---------------------------------- | ---------------------------------------------------------------- | -| `cargo xtask method pull ` | Promote a backlog item into the next numbered cycle. | +| `cargo xtask method pull ` | Promote a legacy filesystem card into the next numbered cycle. | | `cargo xtask method close [cycle]` | Write a retro and create its `witness/` directory. | | `cargo xtask method drift [cycle]` | Check active cycle playback questions against test descriptions. | @@ -430,10 +460,10 @@ The issue tracker is tiered by lane labels. Choice within a lane is judgment at ## Naming conventions -| Convention | Example | When | -| -------------------- | -------------------------------- | -------------------------- | -| `ALL_CAPS.md` | `BEARING.md` | Signpost — root or `docs/` | -| `lowercase.md` | `guide.md` | Everything else | -| `_.md` | `KERNEL_writer-head-registry.md` | Backlog with legend | -| `.md` | `debt-scheduler-god-module.md` | Backlog without legend | -| `/` | `0001-docs-audit/` | Cycle directory | +| Convention | Example | When | +| -------------------- | -------------------------------- | ----------------------------- | +| `ALL_CAPS.md` | `BEARING.md` | Signpost — root or `docs/` | +| `lowercase.md` | `guide.md` | Everything else | +| `_.md` | `KERNEL_writer-head-registry.md` | Legacy backlog with legend | +| `.md` | `debt-scheduler-god-module.md` | Legacy backlog without legend | +| `/` | `0001-docs-audit/` | Cycle directory | diff --git a/docs/method/design-template.md b/docs/method/design-template.md index f5d9884f..05c903e2 100644 --- a/docs/method/design-template.md +++ b/docs/method/design-template.md @@ -1,87 +1,465 @@ -# NNNN — Cycle title + + +--- +title: "{LEGEND}-{ID} - {Short Title}" +legend: "{KERNEL|MATH|PLATFORM|DOCS}" +lane: "design" +issue: "https://github.com/flyingrobots/echo/issues/{number}" +status: "draft|active|landed|superseded" +owners: + - "@flyingrobots" +created: "YYYY-MM-DD" +updated: "YYYY-MM-DD" +--- + + -_One-line description of what this cycle delivers._ +# {LEGEND}-{ID} - {Short Title} -Legend: `LEGEND` (for example, [PLATFORM](./legends/PLATFORM.md)) +## Linked Issue -Depends on: +- [Issue #{number}](https://github.com/flyingrobots/echo/issues/{number}) -- `NNNN — dependency` (or "nothing") +## Decision Summary -## Why this cycle exists +One short paragraph describing the decision this document is making. -What problem exists today. What is missing, broken, or dishonest. -Why now and not later. +This must be specific enough that a reviewer can say whether the +implementation matches the design. Avoid roadmap language. Say what will +exist, what it will do, and what boundary it owns. -## Human users / jobs / hills +## Sponsored Human -### Primary human users +A {type of user} wants {capability/outcome} so that {reason}, without having +to {current pain or unsafe workaround}. -Who benefits from this work (roles, not names). +## Sponsored Agent -### Human jobs +An agent needs {inspectable contract/tool/surface} so it can {operation}, +without inferring {unstable/private-runtime/visual-only state}. -1. First thing the human needs to do. -2. Second thing. +## Hill -### Human hill +By the end of this cycle, {user/agent} can {observable outcome} through +{surface/API/command}, and the repo proves it with {tests/witnesses}. -One sentence. A human can \_\_\_ without \_\_\_. +## Current Truth -## Agent users / jobs / hills +Describe what exists today. This section is factual, not aspirational. -### Primary agent users +Include concrete anchors: -Who benefits from this work on the agent side. +- files; +- commands; +- exported APIs; +- current docs and dogfood surfaces; +- current failure mode; +- relevant GitHub issues or PRs; +- known test coverage. -### Agent jobs +Provide links to evidence supporting strong claims, including source code or +test results at specific, fully qualified git commit SHAs. -1. First thing the agent needs to do. -2. Second thing. +Citation format: -### Agent hill +```md +[{repo-relative-path}#{line-number}:{fully-qualified-commit-sha}](https://github.com/flyingrobots/echo/blob/{fully-qualified-commit-sha}/{repo-relative-path}#L{line-number}) +``` -One sentence. An agent can \_\_\_ and programmatically determine \_\_\_. +Use the merge-target SHA for pre-change truth. Use the PR or landed SHA in the +retrospective for implemented truth. -## Human playback +## Problem -1. The human does X. -2. The output shows Y. -3. The human can now answer Z without opening any files. +State the actual problem. -## Agent playback +Good: -1. The agent runs X. -2. The output contains Y. -3. The agent determines Z. +- "The retained reading recovery path can report a reading id while its payload + material is missing, so clients cannot distinguish a recoverable reading from + an unavailable one." -## Implementation outline +Bad: -1. First step. -2. Second step. -3. Third step. +- "Retention should be better." -## Tests to write first +## Scope -- test proving X -- test proving Y -- test proving Z +This cycle includes: -## Risks / unknowns +- ... -- Risk one and how it might manifest. -- Unknown one and what we'd do if it bites. +## Non-Goals -## Postures +This cycle does not include: -- **Accessibility:** State the posture or say "not applicable" with - a reason. -- **Localization:** Same. -- **Agent inspectability:** Same. +- ... -## Non-goals +Non-goals prevent the design from silently expanding while the PR is in flight. -- What this cycle will not do, to defend scope. +## User Experience / Product Shape + +Use this section for rendered UI, visible CLI, docs, or user-facing behavior. +If the work is not rendered or user-facing, say "Not applicable" and explain +why. + +Answer: + +- What is the user trying to do? +- What are the user's goals? +- What are the user's pain points? +- What communicates system state and actions? +- What communicates success or failure? +- Can the user undo or retry? +- What animations or transitions exist? +- How does this behave in left-to-right and right-to-left locales? +- How does this behave for keyboard and screen-reader users? + +### User Journey + +```mermaid +flowchart TD + Start[User starts task] --> Action[User takes action] + Action --> Success[System communicates success] + Action --> Failure[System communicates failure] + Failure --> Retry[User can retry or exit] +``` + +### Wide UI Mockup + +Embed a wide UI mockup image when the work changes rendered UI. + +- Use an SVG in the design packet when static structure is enough. +- Use a generated or captured bitmap only when visual fidelity matters. +- State terminal dimensions and theme assumptions. + +### Narrow UI Mockup + +Embed a narrow UI mockup image when the work changes rendered UI. + +- Include small terminal geometry and wrapping behavior. +- Include panel collapse, focus ownership, and footer behavior. + +### Accessibility Considerations + +{screen-reader/agent-friendly summary of state, actions, and facts} + +If the work is not a rendered surface, say "Not applicable" and explain why. + +## Runtime / API Contract + +Name the software contract. + +Include only relevant subsections: + +- exported functions and types; +- command intents; +- schema input and output; +- facts emitted; +- state transitions; +- layout, focus, and input boundaries; +- Echo authority boundary; +- Wesley or generated contract artifact boundary; +- Continuum, WASM, CLI, or other ABI boundary; +- host capability boundary; +- error behavior; +- compatibility aliases or migration behavior. + +This is the section tests should be able to compile against. + +## Lower Modes + +Required when the work is user-visible, agent-visible, or has a non-visual +operation mode. + +Describe: + +- terminal size constraints; +- no-color or reduced-color output; +- text, JSON, pipe, fixture, or dry-run output; +- keyboard-only operation; +- behavior when optional adapters are unavailable; +- behavior when Echo, filesystem, network, or retained evidence is partial; +- DIND, local-only, or no-network behavior. + +If no lower mode applies, say "Not applicable" and explain why. + +## Data / State Model + +Describe state that persists, mutates, or crosses boundaries. + +| Category | Description | +| ------------------------- | ----------- | +| Source of truth | ... | +| Derived state | ... | +| Invalid states | ... | +| Reset behavior | ... | +| Serialization | ... | +| Deterministic assumptions | ... | + +Use Mermaid diagrams for complex state, entity relationships, or data flow. + +```mermaid +stateDiagram-v2 + [*] --> Draft + Draft --> Active + Active --> Landed + Active --> Superseded +``` + +## Echo Authority Boundary + +Name which side owns causality, admission, receipts, readings, +materialization, retention, recovery, and host capabilities. + +State what applications, generated contracts, host adapters, or external tools +may provide. Explicitly reject hidden fallback authority when the work touches +causal history, retained evidence, or host material. + +## Determinism / DIND Posture + +Required when the work touches runtime behavior, scheduling, math, replay, +storage ordering, host observation, clocks, randomness, hashing, or +canonicalization. + +State: + +- ordering assumptions; +- clock or cadence assumptions; +- random seed policy, if any; +- filesystem or map iteration ordering; +- hash and canonical encoding policy; +- scheduler or replay expectations; +- whether `cargo xtask dind run` or a narrower determinism witness applies. + +If determinism is not relevant, say "Not applicable" and explain why. + +## WAL / WSC / Retention Posture + +Required when the work touches causal history, durable recovery, submissions, +receipts, reading envelopes, retained material, snapshots, checkpoints, +materialization, or missing evidence. + +State: + +- what must survive restart; +- what is retained by digest, coordinate, envelope, receipt, or witness; +- how missing, redacted, encrypted-unavailable, corrupt, or obstructed material + is reported; +- whether WAL, WSC, CAS, or in-memory retention is involved; +- what reconstruction or replay can and cannot claim. + +If not relevant, say "Not applicable" and explain why. + +## Accessibility Posture + +State how accessibility is preserved. + +| Concern | Posture | +| --------------------------------- | ------- | +| Semantic labels or facts | ... | +| Focus order or ownership | ... | +| Hidden or visual-only information | ... | +| Keyboard behavior | ... | +| Secret or redaction behavior | ... | + +If the work is not a rendered surface, still describe any agent-readable or +machine-readable posture that replaces visual affordances. + +## Localization / Directionality Posture + +Required when new visible strings are added or changed. + +| Concern | Posture | +| -------------------------- | ------- | +| User-visible strings | ... | +| Catalog keys | ... | +| Supported locales updated | ... | +| Directionality assumptions | ... | +| Validation command | ... | + +If no strings change, say "Not applicable" and explain why. + +## Agent Inspectability / Explainability Posture + +Describe how an agent can inspect the result without scraping pixels, +terminal prose, private app caches, or hidden runtime state. + +Examples: + +- stable ids; +- metadata fields; +- emitted facts; +- deterministic text or JSON output; +- registry entries; +- schema descriptions; +- command ids; +- machine-readable witness output. + +## App-Noun Boundary + +Required for any work motivated by Jedit, WARP DRIVE, git-warp, Graft, Bijou, +or another external consumer. + +State which names are allowed only in docs, fixtures, generated contract +artifacts, tests, or external adapters. Confirm that production Echo core does +not import application nouns or implementation details such as editor buffers, +FUSE inodes, panes, project files, product UI panels, or private app caches. + +If no external-consumer vocabulary is involved, say "Not applicable" and +explain why. + +## Linked Invariants + +List repo invariants this work must preserve. + +Examples: + +- Tests are executable spec. +- Runtime truth beats type theater. +- Design should become repo truth. +- Echo owns causality; clients own presentation and host affordances. +- Applications define vocabulary; Echo owns admission, receipts, readings, and + retention posture. +- Retained evidence is explicit; missing support obstructs rather than falling + back. +- Production Echo core stays app-noun-clean. +- Determinism is binary. + +## Design Alternatives Considered + +### Option A: {name} + +Pros: + +- ... + +Cons: + +- ... + +### Option B: {name} + +Pros: + +- ... + +Cons: + +- ... + +## Decision + +State the chosen option and why. + +If the decision is temporary, name the expiration or migration window. + +## Implementation Slices + +- [ ] Slice 1: {smallest testable slice and intended commit message} +- [ ] Slice 2: {next slice} +- [ ] Slice 3: {next slice} + +Each slice should be small enough to commit or review independently. Each slice +should correspond to one test case, user story, or inspectable process change. + +If the right slices are unclear, budget a spike to understand the requirements +and slice appropriately. + +## Tests To Write First + +Behavior tests required: + +- [ ] {package/runtime/render test that fails before implementation} +- [ ] {integration test that exercises user-visible behavior} +- [ ] {lower-mode or pipe/accessibility test, if relevant} +- [ ] {regression test for the specific bug or risk} + +Documentation and process tests, only if relevant: + +- [ ] {design/changelog/backlog/process assertion} + +Rule: documentation tests cannot be the only proof for implementation work. + +## Acceptance Criteria + +The work is done when: + +- [ ] Behavior test proves {contract} +- [ ] Rendered output or runtime API proves {user-visible outcome} +- [ ] Lower modes are covered, if relevant +- [ ] New strings have supported translations, if relevant +- [ ] Docs, changelog, or dogfood are updated when behavior or direction + changes +- [ ] Issue and PR are linked correctly +- [ ] CI and local validation are green + +## Validation Plan + +Commands expected before PR: + +```bash +cargo fmt --check +cargo clippy -p {crate} --all-targets -- -D warnings +cargo test -p {crate} {focused tests} +cargo xtask lint-dead-refs --file {touched-doc}.md +git diff --check +``` + +Trim commands that do not apply. Add package-specific commands when needed. +Use `cargo xtask dind run` when determinism is part of the hill. + +## Playback / Witness + +Describe what a reviewer can run or inspect. + +Examples: + +```bash +cargo test -p warp-core --test {cycle_test} +cargo xtask dind run +cargo xtask lint-dead-refs --file docs/design/{cycle}/{doc}.md +``` + +If there is a visual, CLI, WASM, or TUI result, include the route, command, +key sequence, terminal size, theme, locale, feature flags, and fixture needed +to reproduce it. + +## Risks + +Known risks: + +- ... + +Mitigations: + +- ... + +## Follow-On Debt + +Create GitHub issues for anything deferred. + +Do not hide future work in prose. If it matters, it gets an issue. + +## Retrospective + +Fill this in after implementation. + +What changed from the design: + +- ... + +What the tests proved: + +- ... + +What remains open: + +- ... + +PR: + +- [PR #{number}](https://github.com/flyingrobots/echo/pull/{number}) diff --git a/docs/technical-teardown.md b/docs/technical-teardown.md index 9a7a6099..3beff0b0 100644 --- a/docs/technical-teardown.md +++ b/docs/technical-teardown.md @@ -33,6 +33,9 @@ - [16.3 Design Critiques: Assumptions and Risk Hotspots](#163-design-critiques-assumptions-and-risk-hotspots) - [16.4 Typed Pseudo-Definitions for Core Runtime Types](#164-typed-pseudo-definitions-for-core-runtime-types) - [17. Deep Dives: Technical Feats and Trade-Offs](#17-deep-dives-technical-feats-and-trade-offs) + - [17.6 Echo Tick: From GraphQL to BTR](#176-echo-tick-from-graphql-to-btr) + - [17.7 Forks, Strands, and Braids](#177-forks-strands-and-braids) + - [17.8 File Materialization and Ingest](#178-file-materialization-and-ingest) - [18. Entity-Relationship View](#18-entity-relationship-view) - [Appendix A: Type-Intent to Data Layout Mapping](#appendix-a-type-intent-to-data-layout-mapping) - [Appendix B: Reference Command Paths](#appendix-b-reference-command-paths) @@ -69,6 +72,15 @@ mindmap Parallel Executor Snapshot Ledger + Contract Ingress + GraphQL Contract + Wesley Artifacts + EINT Envelope + Witnessed Submission + Ticketed Runtime Ingress + Scheduler-Owned Tick + Tick Receipt + BTR Suffix Graph Model Node Edge @@ -76,10 +88,27 @@ mindmap Attachment Value Scope Hashes State Root + Worldline Geometry + Fork + Strand + Support Pin + Braid + Settlement + File Aperture + Host Observation + File Site + File Projection + Content Intent + Host Materialization + Verification Receipt Data Stores On-disk WSC snapshot In-memory GraphStore + Causal WAL + CAS / Retained Material Tick History + Reading Envelopes + Materialization Outbox Policy + Telemetry Supporting Systems clap CLI @@ -95,46 +124,79 @@ This teardown is written for a reader who has not seen this codebase before. It ## Glossary: Domain Dictionary -| Term | Definition | Why it matters | -| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -| WSC (Warp Snapshot Container) | Binary snapshot format storing one or more _warps_ (stateful graph partitions) with nodes, edges, and attachments. | It is the transport and persistence substrate for the runtime’s canonical data. | -| Warp | A deterministic unit of graph state used by the execution model. | It sets the boundary for hash computation, validation, and execution traversal. | -| Node | A vertex in the graph. In runtime terms, nodes are the state-bearing objects that rules read from and mutate. | Core object that gets read and mutated by rules. | -| Edge | A directed relation between nodes (for example parent/child, dependency, or domain-specific arcs). | Encodes topology and traversal semantics. | -| Attachment | Binary metadata associated with nodes or edges; often a typed payload. | Captures payload-bearing semantics without changing the row shape. | -| Blob | Raw binary payload storage referenced by attachments. | Separates large payload bytes from structured row tables via offsets/lengths. | -| Engine | The execution engine that accepts intents, schedules rewrites, applies rules, updates graph state, and produces snapshots. | The authoritative coordinator for deterministic execution. | -| Intent | A unit of requested work submitted into a transaction. | Source object that becomes pending rewrites in a transaction. | -| Tick | A unit-of-progress marker in state evolution. | Enables historical progression and snapshot indexing. | -| Tick Receipt | The engine artifact that says whether a transaction committed cleanly, plus conflict details. | Exposes acceptance/rejection and blocker details for diagnostics. | -| Ledger | Sequence of execution history entries recording root state progression. | Provides auditability and time-travel context. | -| Scope Hash | A digest derived from rule inputs/metadata used during conflict planning. | Helps arbitration and conflict checks remain deterministic. | -| Ingestion (`ingest_intent`) | Canonicalized intake path for incoming intents into runtime graph form. | Enforces idempotent behavior and graph materialization of submissions. | -| Dispatcher / Scheduler | Internal subsystem managing pending transactions, intents, and queued rewrite commands. | Controls ordering and fairness of execution work. | -| Parallel Work Unit | Chunk of deterministic rewrite operations split across worker shards. | Supports throughput scaling while preserving deterministic merge semantics. | +| Term | Definition | Why it matters | +| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| WSC (Warp Snapshot Container) | Binary snapshot/evidence format storing graph partitions and, in newer store paths, causal-history envelopes. | It carries durable verification, retention, and recovery evidence around the causal authority. | +| Warp | A deterministic unit of graph state used by the execution model. | It sets the boundary for hash computation, validation, and execution traversal. | +| Node | A vertex in the graph. In runtime terms, nodes are the state-bearing objects that rules read from and mutate. | Core object that gets read and mutated by rules. | +| Edge | A directed relation between nodes (for example parent/child, dependency, or domain-specific arcs). | Encodes topology and traversal semantics. | +| Attachment | Binary metadata associated with nodes or edges; often a typed payload. | Captures payload-bearing semantics without changing the row shape. | +| Blob | Raw binary payload storage referenced by attachments. | Separates large payload bytes from structured row tables via offsets/lengths. | +| Engine | The execution engine that accepts intents, schedules rewrites, applies rules, updates graph state, and produces snapshots. | The authoritative coordinator for deterministic execution. | +| Intent | A unit of requested work submitted into a transaction. | Source object that becomes pending rewrites in a transaction. | +| Tick | A unit-of-progress marker in state evolution. | Enables historical progression and snapshot indexing. | +| Tick Receipt | The engine artifact that says whether a transaction committed cleanly, plus conflict details. | Exposes acceptance/rejection and blocker details for diagnostics. | +| Ledger | Sequence of execution history entries recording root state progression. | Provides auditability and time-travel context. | +| Scope Hash | A digest derived from rule inputs/metadata used during conflict planning. | Helps arbitration and conflict checks remain deterministic. | +| Ingestion (`ingest_intent`) | Canonicalized intake path for incoming intents into runtime graph form. | Enforces idempotent behavior and graph materialization of submissions. | +| GraphQL Contract | Authored application schema that names domain types, operations, and footprint claims before Wesley compiles them. | Keeps application nouns above the Echo runtime kernel. | +| Wesley | Contract compiler/generator that emits helpers, codecs, registry metadata, and ABI-facing artifacts from authored schemas. | Bridges authored domain semantics into Echo's generic runtime boundary. | +| EINT | Canonical Echo intent envelope used at ABI/runtime ingress. | Gives submitted operation bytes stable structure before scheduler admission. | +| BTR | Boundary Transition Record: a contiguous provenance segment with input/output boundary hashes and validated entries. | Packages a witnessed suffix for validation, replay, or causal exchange. | +| Fork | A copied worldline prefix at a precise tick, usually used to create a speculative child lane. | Gives time-travel and counterfactual work an exact causal basis. | +| Strand | A named relation over a child worldline derived from a source lane at a fork basis. | Makes speculative work inspectable instead of anonymous branch state. | +| Braid | Read-only plural geometry over one observed lane plus support-pinned lanes. | Lets observation include multiple exact coordinates without settlement. | +| File Aperture | Echo-owned contract for observing host file bytes, admitting drift, and materializing lawful writes. | Prevents apps from maintaining a shadow causal history for files. | +| Dispatcher / Scheduler | Internal subsystem managing pending transactions, intents, and queued rewrite commands. | Controls ordering and fairness of execution work. | +| Parallel Work Unit | Chunk of deterministic rewrite operations split across worker shards. | Supports throughput scaling while preserving deterministic merge semantics. | ## High-Level Mental Model -Echo is split into two cooperating layers: - -1. A command-line interface layer (`warp-cli`) that lets you validate, inspect, and benchmark artifacts. -2. A runtime core (`warp-core`) that owns state representation, scheduling, deterministic execution, and snapshot/hash logic. - -The CLI is largely a thin orchestration layer: it reads inputs, invokes validation or runtime helpers, then formats output (text or JSON). The heavy technical behavior sits in the core modules. +Echo is a deterministic WARP runtime over witnessed causal history. The useful +mental model is no longer "a CLI wrapped around a graph engine." That was a +good early entry point, but the current system has five cooperating surfaces: + +1. **Diagnostic tools** (`warp-cli`) validate, inspect, benchmark, and report + WAL or recovery posture without becoming the authority. +2. **Runtime core** (`warp-core`) owns scheduling, deterministic execution, + receipt production, installed contract dispatch, QueryView routing, + retained evidence posture, and crash-recoverable causal history. +3. **Contract ingress** lets authored GraphQL/Wesley artifacts become canonical + EINT envelopes, witnessed submissions, ticketed runtime ingress, and + scheduler-owned ticks. +4. **Observation and retention** make reads first-class through + `ReadingEnvelope`/QueryView outputs, retained material references, WAL/WSC + evidence, and explicit obstruction posture. +5. **Standard apertures** such as `echo-file-aperture` define reusable boundary + contracts for host artifacts without smuggling application nouns into the + kernel. + +The CLI remains a thin orchestration layer: it reads inputs, invokes validation +or runtime helpers, then formats text or JSON. The deeper behavior sits in +runtime crates and standard aperture crates. ```mermaid flowchart TD - A[CLI Entrypoint: main.rs] --> B{Command parsed by clap} - B -->|verify| C[WSC validation + per-warp canonical hash] - B -->|inspect| D[Decode and summarize graph topology] - B -->|wal doctor| E[Read WAL metadata and classify posture] - B -->|wal submission-posture| F[Inspect submission-level posture] - B -->|bench| G[Run criterion benchmarks and diff baseline] - C --> H[Output sink: Text / JSON] - D --> H - E --> H - F --> H - G --> H + CLI[warp-cli diagnostics] --> FORMAT[Text / JSON output] + CLI --> VERIFY[WSC verify / inspect] + CLI --> WAL[WAL doctor / submission posture] + CLI --> BENCH[Bench report] + + CONTRACT[GraphQL + Wesley artifacts] --> EINT[Canonical EINT envelope] + EINT --> SUB[Witnessed submission] + SUB --> HOST[Trusted host ticketed ingress] + HOST --> CORE[warp-core scheduler-owned tick] + + CORE --> RECEIPT[TickReceipt / obstruction] + CORE --> READING[QueryView ReadingEnvelope] + CORE --> RETAIN[Retained material refs] + CORE --> HISTORY[WAL / WSC causal evidence] + + APERTURE[echo-file-aperture] --> HOSTOBS[Host file observation] + APERTURE --> HOSTMAT[Host materialization verification] + HOSTOBS --> SUB + RECEIPT --> FORMAT + READING --> FORMAT ``` ## 1. Entry Point @@ -567,21 +629,38 @@ Hashing order and encoding order are deliberate: deterministic ordering eliminat ```mermaid flowchart TD - S1[On-disk WSC file] -->|open + validate| S2[WscFile] - S2 -->|warp_view slices| S3[Reader views zero-copy] - S3 -->|reified| S4[GraphStore in memory] - S4 -->|apply_to_state| S5[Ledger + Snapshot history] - S5 -->|snapshot and jump_to_tick| S6[Persistence boundaries for checkpoints] - S4 -->|dispatch| S7[Parallel executor] - S7 -->|validated deltas| S5 + A[Witnessed submissions] --> B[Ticketed runtime ingress] + B --> C[Scheduler-owned ticks] + C --> D[Receipts + provenance entries] + D --> E[WAL / WSC causal evidence] + D --> F[Retained material refs] + D --> G[Reading envelopes] + G --> H[Product readings and diagnostics] + F --> H + E --> I[Recovery / replay posture] + E --> J[Snapshot or suffix export] + K[On-disk WSC snapshot] -->|open + validate| L[WscFile] + L -->|warp_view slices| M[Reader views] + M -->|reified| N[GraphStore in memory] + N -->|apply_to_state| C + O[Host file bytes] --> P[File aperture observation] + P --> A + D --> Q[Materialization intent] + Q --> R[Host file verification] ``` ### Practical implications -- **On disk**: long-lived source of truth for transport/verification. -- **In memory**: fast execution working state. -- **Snapshot history**: auditability and time-travel checkpoints. -- **WAL (causal)**: externalized execution posture and submission diagnostics. +- **Causal history**: witnessed submissions, ticketed ingress, ticks, receipts, + and retained evidence are the authority. +- **On disk**: WSC/WAL artifacts are durable evidence and recovery material, not + a shortcut around admission law. +- **In memory**: runtime state is fast working material derived from causal + history and checkpoint/snapshot inputs. +- **Readings**: `ReadingEnvelope` and QueryView outputs are bounded + observations with explicit basis and retained-evidence posture. +- **Files**: host files are observed boundary artifacts and materialization + targets. They are not Echo state. ## 10. Concurrency and Asynchronous Flows @@ -810,7 +889,10 @@ timeline ## 16.1 Project Progress and Future Use Cases -This project is at an unusually useful middle point: the core runtime primitives exist, operational tooling is wired, but a lot of external integration scaffolding remains intentionally lightweight. +This project is at an unusually useful middle point: the runtime authority +model is much more real than the early CLI-only teardown implied, but the +product-facing integration surface is still being hardened under `jedit` +pressure. ### Where it is today @@ -819,41 +901,66 @@ This project is at an unusually useful middle point: the core runtime primitives - bounded-parallel rewrite planning, - conflict-aware reservation, - and checkpointed history/snapshot mechanics. +- Installed contract packages can dispatch mutation handlers through + witnessed, ticketed, scheduler-owned ticks. +- QueryView routes can reach installed query observers and return + `ReadingEnvelope` posture with authored observer identity and retained + evidence coordinates. +- WAL-backed runtime ACK/recovery work has moved accepted submissions, tick + outcomes, receipt correlations, retained refs, and recovery posture toward + crash-recoverable evidence instead of process memory. +- WSC store contracts and envelope tests cover accepted submissions, receipt + correlations, retained material, reading refs, commit markers, and corrupted + history obstruction posture. +- `echo-file-aperture` exists as the first Echo-owned standard file aperture + slice. It deterministically identifies host file sites, records observations, + reconciles basis drift, proposes content, obstructs stale writes, and verifies + materialized output in memory. - Runtime and CLI boundaries are well separated: command tooling does not mutate core scheduler state except through explicit paths. -- Observability is mostly artifact-driven (text/JSON output + WAL posture maps), not yet full telemetry streaming. +- Observability is mostly artifact-driven: text/JSON output, WAL posture maps, + WSC envelope checks, reading envelopes, and focused witnesses. It is not yet + a polished telemetry streaming product. - There is already a strong testability story from determinism (hashes, canonical ordering, strict validation), which is ideal for replay and bisecting regressions. +- The file aperture is not yet bound end-to-end to the scheduler, WAL/WSC + retention, and host materialization outbox. That is the next integration hill, + not completed truth. ### Where it is heading Based on the present design, the roadmap is likely to continue toward: -1. **Richer command surfaces** - - REST/HTTP adapters that call the same engine primitives. - - Programmatic entry points that reuse `verify`/`inspect` logic rather than reimplementing it. -2. **Stronger production hardening** - - WAL posture hooks becoming first-class monitoring outputs, - - explicit SLA/error budget handling around benchmark drift, - - safer defaults around concurrency tuning and worker saturation. -3. **Incremental persistence improvements** - - smarter snapshot compaction around tick deltas, - - optional lazy loading for very large WSC payloads, - - stronger migration paths if schema version evolves. +1. **File aperture integration** + - bind `echo-file-aperture` observations to witnessed submissions, + - admit external drift as causal history, + - form deterministic write/content intents from proposed host bytes, + - authorize materialization only after durable causal commit. +2. **Product release gate with jedit** + - keep `jedit` nouns in `jedit` contracts and adapters, + - make ordinary open/edit/save/recover/export behavior consume Echo + authority, + - prove recovery without a jedit-owned shadow causal ledger. +3. **Stronger production hardening** + - make WAL/WSC posture first-class monitoring output, + - stabilize contract-aware obstruction taxonomy, + - harden retained evidence recovery across restart and replay. 4. **Tooling and introspection maturity** - richer JSON schema for inspection output, - - machine-parseable receipts for dashboards, - - CLI output modes optimized for CI (CSV/NDJSON variants). + - machine-parseable receipts and readings for dashboards, + - CLI output modes optimized for CI and agent witnesses. ```mermaid flowchart TD - P[Current state] --> C1[Expose typed APIs] - P --> C2[Improve telemetry] - P --> C3[Improve persistence ergonomics] - C1 --> H1[Graph services / RPC layer] - C2 --> H2[Continuous health dashboards] - C3 --> H3[Faster snapshots + lower memory] - H1 --> F[Future use cases] + P[Current state] --> C1[File aperture integration] + P --> C2[jedit release gate] + P --> C3[WAL / WSC recovery hardening] + C1 --> H1[Host observation + drift admission] + C1 --> H2[Authorized materialization] + C2 --> H3[Open / edit / save / recover] + C3 --> H4[Retained evidence replay] + H1 --> F[External consumers] H2 --> F H3 --> F + H4 --> F ``` ### Cool ideas @@ -861,6 +968,9 @@ flowchart TD - Use the deterministic hash as a cross-language conformance gate in polyglot agent pipelines. - Export `inspect` output into a governance compliance feed (e.g., graph anomaly reports by edge/attachment histograms). - Build a `chaos mode` for `commit_with_receipt` that intentionally injects conflict conditions for testing scheduler robustness. +- Add a file-aperture witness that opens arbitrary host bytes, admits drift, + authorizes materialization, and verifies the final artifact from retained + evidence. - Add pluggable policy modules for alternative sharding strategies (e.g., topology-aware or latency-aware sharding). - Create an educational “execution tracer” mode that emits Mermaid-ready spans from sequence/state traces already implied by this teardown. @@ -1196,6 +1306,342 @@ flowchart TD BM9 --> BM10[mark status with comparison marker] ``` +### 17.6 Echo Tick: From GraphQL to BTR + +The most useful way to understand Echo's contract-hosting path is to follow one +operation all the way from authored schema to exportable causal suffix. + +The path is: + +1. A developer authors a GraphQL contract. +2. Wesley compiles that contract into operation ids, codecs, registry metadata, + and footprint-shaped runtime artifacts. +3. A host packs a canonical EINT envelope for the chosen operation. +4. Echo records the submission as witnessed ingress material. +5. A trusted runtime boundary tickets and stages that envelope into runtime + ingress. +6. The scheduler owns the tick that dispatches the handler and applies any + admitted rewrite. +7. The tick emits receipt evidence and advances provenance. +8. A contiguous provenance segment can be packaged as a BTR. + +The important point is that only the scheduler-owned tick mutates runtime +state. GraphQL authors vocabulary. Wesley compiles it. EINT carries canonical +bytes. Ticketed ingress stages work. The tick is where lawful execution becomes +history. + +```mermaid +sequenceDiagram + participant Dev + participant Wesley + participant Host + participant Echo + participant Scheduler + participant Prov + + Dev->>Wesley: GraphQL contract + directives + Wesley-->>Host: op ids, codecs, registry metadata, footprints + Host->>Echo: EINT envelope bytes + Echo-->>Host: witnessed submission id + Host->>Echo: admission ticket + envelope + Echo-->>Scheduler: ticketed runtime ingress + Scheduler->>Scheduler: scheduler-owned tick + Scheduler->>Scheduler: decode op + vars, check footprint, execute handler + Scheduler-->>Echo: TickReceipt + state patch + Echo->>Prov: append provenance entry + Prov-->>Echo: state root + commit hash + Host->>Prov: build_btr(worldline, start, end) + Prov-->>Host: BoundaryTransitionRecord +``` + +That sequence has several authority boundaries: + +| Boundary | Owns | Does not own | +| -------------------- | -------------------------------------------------------- | ---------------------------------- | +| GraphQL contract | domain nouns, operation shape, declared footprint | runtime scheduling or tick cadence | +| Wesley artifacts | generated codecs, ids, registry metadata, ABI helpers | mutable runtime state | +| EINT envelope | canonical operation bytes | admission or execution | +| witnessed submission | durable ingress evidence | scheduler-visible work | +| ticketed ingress | trusted staging into a writer head inbox | handler execution | +| scheduler tick | admission, dispatch, conflict checks, receipt production | host-side materialization | +| BTR | contiguous witnessed suffix export | new state mutation | + +In code terms, the runtime distinguishes submission evidence from executable +work. App-facing submission records canonical ingress material but does not +tick, stage runtime ingress, dispatch handlers, or mutate application state. A +trusted boundary later stages ticketed runtime ingress. The scheduler then +correlates the eventual receipt back to the witnessed submission and ticketed +ingress ids. + +```mermaid +flowchart TD + GQL[GraphQL contract] --> WES[Wesley generated artifacts] + WES --> EINT[EINT canonical envelope] + EINT --> SUB[Witnessed submission] + SUB --> TICKET[Admission ticket] + TICKET --> STAGE[Ticketed runtime ingress] + STAGE --> TICK[Scheduler-owned tick] + TICK --> RECEIPT[Tick receipt correlation] + TICK --> PROV[Provenance entry] + PROV --> BTR[Boundary Transition Record] +``` + +A BTR is not a checkpoint and not a replay shortcut by itself. It is a +validated contiguous segment: + +- one `worldline_id`; +- one `u0_ref`; +- input boundary hash before the segment; +- output boundary hash after the segment; +- ordered provenance entries for the selected tick range; +- logical counter and auth tag material for the transport or authority layer. + +Validation checks the selected range against registered provenance. It rejects +unknown worldlines, mismatched `u0_ref`, wrong input/output boundary hashes, +non-contiguous ticks, mixed worldlines, and entries that do not exactly match +stored history. + +The practical consequence: + +- a GraphQL mutation is not "called" in the usual app-framework sense; +- it is compiled into canonical runtime material; +- the runtime admits and ticks it under Echo law; +- the resulting history can be exported as a witnessed suffix. + +### 17.7 Forks, Strands, and Braids + +Echo uses three related but distinct ideas for counterfactual and plural +history work: + +| Concept | What it is | What it is not | +| ------- | ----------------------------------------------------------- | ------------------------------------------- | +| Fork | A copied worldline prefix at one precise tick. | A semantic relationship by itself. | +| Strand | A named relation over a forked child worldline. | A separate substrate or private scheduler. | +| Braid | Read-only support geometry across exact strand coordinates. | Settlement, import, or conflict resolution. | + +The distinction matters because each concept answers a different question. + +- A **fork** answers: "what state did this child lane start from?" +- A **strand** answers: "what is the named speculative relation between this + child lane and its basis?" +- A **braid** answers: "which exact support lanes participate in this local + reading?" + +```mermaid +flowchart TD + P[Parent worldline] -->|fork at tick N| C[Child worldline] + C --> S[Strand relation] + S --> B[ForkBasisRef
source lane + tick + commit + boundary] + S --> H[Writer heads
ordinary runtime control] + S --> PINS[Support pins] + PINS --> SUP1[Support strand at pinned tick] + PINS --> SUP2[Support strand at pinned tick] + SUP1 --> SITE[Observed braided site] + SUP2 --> SITE + S --> SITE +``` + +A fork copies enough provenance to create a child worldline at the requested +basis. Runtime forking is failure-atomic: if provenance copy, replay, +worldline registration, writer-head registration, or strand registration fails, +Echo restores runtime and provenance to their pre-fork state. The fork must not +leave partial truth behind. + +A strand then makes the relationship inspectable. The strand records: + +- stable `strand_id`; +- immutable `fork_basis_ref`; +- `child_worldline_id`; +- writer heads authorized for the child lane; +- optional read-only support pins. + +The fork basis is deliberately redundant: + +- source lane id; +- fork tick; +- commit hash at that tick; +- output boundary hash at that tick; +- provenance ref for native lookup. + +Those fields must all name the same provenance coordinate. If they disagree, +strand construction is invalid. + +Braids are narrower than settlement. A support pin says: + +```text +when reading this strand's local site, +also include that support strand at this exact pinned tick +``` + +It does not copy the support lane, authorize writes through it, merge it, +settle it, or create a new worldline. It only changes the observation geometry +for a bounded reading. + +```mermaid +stateDiagram-v2 + [*] --> ParentWorldline + ParentWorldline --> ForkedChild : fork(source, tick) + ForkedChild --> LiveStrand : register Strand + LiveStrand --> BraidedSite : add support pins + BraidedSite --> LiveStrand : unpin support + LiveStrand --> Dropped : drop strand + Dropped --> [*] +``` + +Settlement is the next layer. It compares a strand suffix against its basis, +decides whether a suffix can become history on another lane, and produces +conflict artifacts when it cannot. Braid geometry should feed settlement, but +it should not be collapsed into settlement. + +For debugger and observer surfaces, the useful rule is: + +```text +Forks establish basis. +Strands name speculative lanes. +Braids publish plural local sites. +Settlement decides what can become history. +``` + +### 17.8 File Materialization and Ingest + +File support sits at an awkward boundary because users think in ordinary files +while Echo thinks in witnessed causal history. + +The architectural rule is: + +```text +A host file is not Echo state. +It is an observed boundary artifact and a materialization target. +``` + +That means opening a file and saving a file can both create causal events. A +read can discover external drift. A write can authorize external +materialization. Echo should own the causal record for both; applications such +as Jedit or WARP-drive should not maintain a parallel causal ledger. + +#### Ingest: host bytes into causal history + +When a host file is opened, the user-visible guarantee is simple: + +```text +open file -> see the exact bytes currently on disk +``` + +The Echo-facing flow is more explicit: + +1. Resolve a host-aperture `FileSiteId` and identity posture from the host path + and available platform file identity. +2. Read bytes and relevant metadata through a host capability. +3. Compute canonical content and metadata digests. +4. Compare those digests with the latest retained Echo basis for that + coordinate. +5. If Echo has no basis, admit the host bytes as an observed boundary artifact. +6. If disk has drifted from the retained basis, admit that drift as an external + observation. +7. Return a reading envelope containing exact bytes, basis, digest, and + evidence posture. + +```mermaid +flowchart TD + Open[User opens path] --> Coord[Resolve FileCoordinate] + Coord --> HostRead[Read host bytes + metadata] + HostRead --> Digest[Canonical digests] + Digest --> Compare{Matches Echo basis?} + Compare -->|unknown| Observe[Admit host observation] + Compare -->|drifted| Drift[Admit external drift observation] + Compare -->|yes| Existing[Use retained basis] + Observe --> Reading[File reading envelope] + Drift --> Reading + Existing --> Reading + Reading --> App[App renders exact file contents] +``` + +The resolution rule is tiered. When stable platform identity exists, Echo +derives `FileSiteId` from that platform identity and reports +`PlatformStable`. The path remains observation evidence, so a rename keeps the +same file site. When platform identity is unavailable, Echo derives a +domain-separated path-bound id and reports `PathBound`; a rename cannot be +proven as the same host file from path evidence alone. + +`FileSiteId` is not `WorldlineId`. `FileSiteId` binds local host-aperture +observations. `WorldlineId` remains the portable logical document identity for +causal history. Future rebinding between them should be explicit through +binding or anchor records, not hidden by treating host file identity as WSC +identity. + +The application should not ask, "does my private sidecar know this file?" It +should ask Echo for the file aperture reading and render the bytes it receives. +If evidence is missing, redacted, encrypted-unavailable, or corrupt, Echo +should return an obstruction posture rather than silently delegating authority +back to the app. + +#### Materialization: causal history back to host files + +Saving a file is the inverse path. The editor or mount adapter proposes target +content. Echo compares it to the current basis, forms lawful write intents, and +authorizes materialization only after the causal transaction is durable. + +The WAL posture for external effects is already the right shape: + +```text +No external side effect may be performed before the causal transaction +authorizing that side effect is durably committed. +``` + +For files, that means: + +1. Form a deterministic write intent from old basis and new content. +2. Admit and execute the intent under scheduler-owned tick law. +3. Commit receipt, state delta, and materialization intent to durable history. +4. Publish an idempotent materialization effect token. +5. Write a temporary artifact. +6. `fsync` the temporary artifact. +7. Atomically rename it into place. +8. `fsync` the containing directory. +9. Verify final path digest and metadata. +10. Record `MaterializationEffectObserved`. + +```mermaid +sequenceDiagram + participant App + participant Echo + participant WAL + participant Outbox + participant Worker + participant FS as Host Filesystem + + App->>Echo: proposed file content + Echo->>Echo: diff against causal basis + Echo->>Echo: scheduler-owned tick + Echo->>WAL: commit receipt + materialization intent + WAL-->>Echo: durable commit + Echo->>Outbox: idempotent effect token + Worker->>Outbox: claim effect token + Worker->>FS: temp write + fsync + atomic rename + Worker->>FS: verify final digest + Worker->>WAL: MaterializationEffectObserved +``` + +The hard case is crash recovery: + +- if Echo crashes before WAL commit, the file change was never authorized; +- if Echo crashes after WAL commit but before materialization, recovery can + retry from the idempotent token; +- if Echo crashes after materialization but before the observation record, + recovery verifies the final artifact and records, retries, repairs, or + obstructs according to policy. + +This is why file ingest and file materialization belong in Echo's shared +aperture layer rather than in each consumer. WARP-drive may present a POSIX +mount. Jedit may present an editor buffer. Both are product surfaces over the +same lower truth: + +```text +Echo owns file observation, drift admission, write intent formation, +materialization authorization, and retained evidence posture. +Consumers own presentation and host affordances. +``` + ## 18. Entity-Relationship View The data model is best understood as a graph of stateful entities with constrained foreign keys and one canonical parent for each warp.