Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@
locator, writer epoch, and recovery certificate evidence; missing manifests or
unavailable locators produce typed projection obstructions instead of empty
success.
- `warp-core` now exposes `WalRecoveryPlan` records that bootstrap from a
projected WAL root or storage manifest, record checkpoint posture, committed
replay suffix, tail posture, recovered index roots, retained-material posture,
and projected evidence posture without requiring graph WAL nodes as input.
- `warp-core` can now materialize WAL projection records into deterministic
WARP graph facts with root, writer epoch, segment, commit-anchor, and recovery
certificate nodes plus typed graph edges suitable for WSC serialization. The
Expand Down Expand Up @@ -76,6 +80,9 @@
`retained_reading_missing_payload_is_not_empty_success` witness, locking the
app-safe missing-retention posture for reading payloads, reading envelopes,
and retained receipt support.
- `cargo xtask test-slice durability-release` now includes the exact
`recovery_plan_bootstraps_from_wal_root` witness, locking recovery plan
bootstrap posture without using Cargo's slow package-level name filter.
- `warp-core` trusted runtime hosts now configure runtime WAL through
`TrustedRuntimeWalConfig`, including in-memory and filesystem-backed
adapters. `TrustedRuntimeWalStoreKind` exposes the configured adapter kind as
Expand Down
223 changes: 223 additions & 0 deletions crates/warp-core/src/causal_wal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3200,6 +3200,229 @@ impl WalRecoveryProjection {
}
}

/// Bootstrap evidence used to start a recovery plan.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum WalRecoveryBootstrapSource {
/// Recovery started from an already projected WAL root.
WalRoot {
/// Projection root digest.
root_digest: Hash,
/// Stable identity digest of the projected root.
root_identity_digest: Hash,
},
/// Recovery started from storage manifest evidence.
StorageManifest {
/// Published manifest digest.
manifest_digest: Hash,
},
}

impl WalRecoveryBootstrapSource {
/// Builds bootstrap evidence from a projected WAL root.
#[must_use]
pub fn from_wal_root(root: &WalRoot) -> Self {
Self::WalRoot {
root_digest: root.root_digest,
root_identity_digest: root.identity_digest(),
}
}

/// Builds bootstrap evidence from a storage manifest.
#[must_use]
pub fn from_manifest(manifest: &WalManifest) -> Self {
Self::StorageManifest {
manifest_digest: manifest.manifest_digest,
}
}
}

/// Checkpoint selection posture for a recovery plan.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum WalRecoveryCheckpointPosture {
/// Recovery did not use a checkpoint base.
NotSelected,
/// Recovery selected a checkpoint and records its validation posture.
Selected {
/// Selected checkpoint digest.
checkpoint_digest: Hash,
/// Validation posture for the selected checkpoint.
validation: CheckpointValidationPosture,
},
}

/// Committed WAL suffix replayed by a recovery plan.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct WalRecoveryReplaySuffix {
/// First committed LSN replayed.
pub first_lsn: Option<Lsn>,
/// Last committed LSN replayed.
pub last_lsn: Option<Lsn>,
/// Number of committed transactions replayed.
pub committed_transactions: u64,
/// Final committed transaction digest.
pub final_commit_digest: Option<Hash>,
}

impl WalRecoveryReplaySuffix {
/// Builds replay suffix evidence from a recovery scan report.
#[must_use]
pub fn from_report(report: &RecoveryScanReport) -> Self {
Self {
first_lsn: report.first_committed_lsn(),
last_lsn: report.last_committed_lsn(),
committed_transactions: len_u64(report.transactions.len()),
final_commit_digest: report.last_commit_digest(),
}
}
}

/// Index roots produced by recovery.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct WalRecoveryIndexRoots {
/// Recovered frontier root.
pub recovered_frontier_root: Hash,
/// Recovered index root.
pub recovered_indexes_root: Hash,
}

impl WalRecoveryIndexRoots {
/// Builds index-root evidence from a recovery certificate.
#[must_use]
pub fn from_certificate(certificate: &RecoveryCertificate) -> Self {
Self {
recovered_frontier_root: certificate.recovered_frontier_root,
recovered_indexes_root: certificate.recovered_indexes_root,
}
}
}

/// Retained-material availability posture for recovery.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum WalRecoveryRetainedMaterialAvailability {
/// All retained material required by the recovered index is available.
Available,
/// At least one retained material reference is missing, corrupt, or obstructed.
Obstructed,
}

/// Retained-material posture recorded by a recovery plan.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct WalRecoveryRetainedMaterialPosture {
/// Overall retained-material availability.
pub availability: WalRecoveryRetainedMaterialAvailability,
/// Number of recovered retained material references.
pub material_count: u64,
/// Number of recovered reading references.
pub reading_count: u64,
/// Number of retained-material obstructions.
pub obstruction_count: u64,
}

impl WalRecoveryRetainedMaterialPosture {
/// Builds retained-material posture from recovered retained evidence.
#[must_use]
pub fn from_retention_index(
index: &RecoveredRetentionIndex,
available_material: &BTreeSet<Hash>,
) -> Self {
let obstruction_count =
len_u64(retained_material_obstructions(index, available_material).len());
Comment on lines +3328 to +3329

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude diagnostic losses from recovery-plan obstruction counts

When a recovered WAL has only missing or corrupt RetainedMaterialKind::Diagnostic material, retained_material_obstructions returns a DiagnosticLoss scope; the existing doctor path filters that scope because diagnostic loss does not block causal recovery. This new posture uses the unfiltered obstruction count, so the recovery plan reports Obstructed even though all recovery-required material is available, creating false obstruction evidence for diagnostic-only loss.

Useful? React with 👍 / 👎.

Self {
availability: if obstruction_count == 0 {
WalRecoveryRetainedMaterialAvailability::Available
} else {
WalRecoveryRetainedMaterialAvailability::Obstructed
},
Comment on lines +3331 to +3335

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Count missing reading refs in recovery posture

When the WAL contains a ReadingEnvelopeRetained record whose ReadingRefRecord.posture is Missing, Corrupt, or Obstructed (or whose payload/envelope digest has no matching retained-material record), retained_material_obstructions still returns zero because it only walks material_by_digest. The plan then reports Available even with unrecoverable reading evidence, hiding the app-safe missing-retention posture during recovery planning.

Useful? React with 👍 / 👎.

material_count: len_u64(index.material_by_digest.len()),
reading_count: len_u64(index.reading_by_id.len()),
obstruction_count,
}
}
}

/// Recovery plan/report shape tying bootstrap, replay, retention, and projection evidence.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct WalRecoveryPlan {
/// Evidence used to bootstrap the recovery plan.
pub bootstrap_source: WalRecoveryBootstrapSource,
/// Checkpoint selection and validation posture.
pub checkpoint_posture: WalRecoveryCheckpointPosture,
/// Committed suffix replayed after the bootstrap source.
pub replay_suffix: WalRecoveryReplaySuffix,
/// Tail posture observed during recovery.
pub tail_posture: RecoveryTailPosture,
/// Recovered frontier and index roots.
pub index_roots: WalRecoveryIndexRoots,
/// Retained-material availability posture.
pub retained_material_posture: WalRecoveryRetainedMaterialPosture,
/// Projection posture for graph-ready WAL evidence.
pub projected_evidence_posture: WalRecoveryProjectionPosture,
/// Number of typed projection obstructions.
pub projected_evidence_obstruction_count: u64,
}

impl WalRecoveryPlan {
/// Builds a recovery plan that bootstraps from a projected WAL root.
#[must_use]
pub fn from_wal_root(
root: &WalRoot,
report: &RecoveryScanReport,
checkpoint_posture: WalRecoveryCheckpointPosture,
certificate: &RecoveryCertificate,
retained_material_posture: WalRecoveryRetainedMaterialPosture,
projection: &WalRecoveryProjection,
) -> Self {
Self::from_bootstrap_source(
WalRecoveryBootstrapSource::from_wal_root(root),
report,
checkpoint_posture,
certificate,
retained_material_posture,
projection,
)
}

/// Builds a recovery plan that bootstraps from storage manifest evidence.
#[must_use]
pub fn from_manifest(
manifest: &WalManifest,
report: &RecoveryScanReport,
checkpoint_posture: WalRecoveryCheckpointPosture,
certificate: &RecoveryCertificate,
retained_material_posture: WalRecoveryRetainedMaterialPosture,
projection: &WalRecoveryProjection,
) -> Self {
Self::from_bootstrap_source(
WalRecoveryBootstrapSource::from_manifest(manifest),
report,
checkpoint_posture,
certificate,
retained_material_posture,
projection,
)
}

fn from_bootstrap_source(
bootstrap_source: WalRecoveryBootstrapSource,
report: &RecoveryScanReport,
checkpoint_posture: WalRecoveryCheckpointPosture,
certificate: &RecoveryCertificate,
retained_material_posture: WalRecoveryRetainedMaterialPosture,
projection: &WalRecoveryProjection,
) -> Self {
Self {
bootstrap_source,
checkpoint_posture,
replay_suffix: WalRecoveryReplaySuffix::from_report(report),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Report only the post-checkpoint replay suffix

When checkpoint_posture is Selected, recovery is supposed to use the checkpoint as the base and replay only the WAL tail after that checkpoint, but this always derives the suffix from the full scan report. A plan for a checkpointed recovery will therefore report the WAL's first committed LSN and all committed transactions, including history already covered by the checkpoint, which makes the recorded replay evidence inaccurate for checkpoint-backed recovery.

Useful? React with 👍 / 👎.

tail_posture: report.tail_posture,
index_roots: WalRecoveryIndexRoots::from_certificate(certificate),
retained_material_posture,
projected_evidence_posture: projection.posture,
projected_evidence_obstruction_count: len_u64(projection.obstructions.len()),
}
}
}

/// Materialized WARP graph facts for a projected WAL root.
///
/// The graph is a read-only fact projection. It carries typed nodes, edges, and
Expand Down
Loading
Loading