Migrated from Method backlog
This issue was created from a legacy filesystem backlog card. GitHub Issues are now the live work tracker; repository docs remain Method evidence.
Source backlog: docs/method/backlog/cool-ideas/PLATFORM_deep-storage-wire-protocol.md
Original lane: cool-ideas
Original legend: PLATFORM
Original backlog card
Milestone: Deep Storage | Priority: P2
Status: active cool idea. crates/echo-cas provides local memory-only
content-addressed storage; no deep-storage wire message types, codec golden
vectors, BlobExchange state machine, or network transfer path exists yet.
This remains operational as the network exchange follow-up after the local CAS
API and disk tier mature.
Wire Protocol
WANT/PROVIDE/FRAME message types for network-level blob exchange between peers.
T-5-3-1: Message type definitions and binary encoding
User Story: As a developer, I want a compact binary wire format for blob exchange so that peers can request and transfer blobs efficiently.
Requirements:
- R1: Define message types:
Want { hash: BlobHash }, Provide { hash: BlobHash, size: u64 }, Frame { hash: BlobHash, offset: u64, data: Vec<u8> }, Have { hash: BlobHash }, Error { code: u16, message: String }.
- R2: Binary encoding: 1-byte message tag + length-prefixed fields. All integers little-endian.
- R3: Implement
encode(&self) -> Vec<u8> and decode(bytes: &[u8]) -> Result<Message, WireError> for each type.
- R4: Maximum message size: 1MB (enforced in
decode).
Acceptance Criteria:
Definition of Done:
Scope: Message type definitions, encode/decode, golden vectors, error types.
Out of Scope: Transport layer (TCP/WebSocket). Connection management. Authentication.
Test Plan:
- Goldens: Hex-encoded golden vectors for each message type (Want, Provide, Frame, Have, Error).
- Failures: Unknown message tag. Truncated at every byte position. Oversized message. Invalid UTF-8 in Error message.
- Edges: Frame with 0-byte data. Want for zero-hash. Provide with size=0.
- Fuzz/Stress: Proptest: encode random messages, decode, assert equality. Decode 100,000 random byte slices (must not panic).
Blocked By: none
Blocking: T-5-3-2
Est. Hours: 5h
Expected Complexity: ~300 LoC
T-5-3-2: Request/response protocol and backpressure
User Story: As a developer, I want a protocol state machine that handles blob exchange with flow control so that network transfers do not overwhelm memory.
Requirements:
- R1:
BlobExchange struct manages the protocol state: pending wants, in-flight provides, received frames.
- R2: Outbound:
request(hash) enqueues a Want. When a Provide arrives, send Want again to confirm (pull model).
- R3: Inbound: on receiving
Want, check local store and send Provide (or Error::NotFound). On confirmed Want, stream Frame messages.
- R4: Backpressure: limit in-flight concurrent wants to a configurable
max_concurrent (default 16).
- R5: Frame reassembly: accumulate
Frame chunks for a hash until offset + data.len() == size, then call put_verified.
Acceptance Criteria:
Definition of Done:
Scope: Protocol state machine, backpressure, frame reassembly, put_verified integration.
Out of Scope: Transport binding (TCP/WS). Discovery/peer management. Encryption.
Test Plan:
- Goldens: N/A (stateful protocol -- tested behaviorally).
- Failures: Provide for a hash we never wanted (ignored). Frame for unknown hash (ignored). Frame with offset beyond declared size.
- Edges: 1-byte blob (single frame). Blob exactly at 1MB boundary. Duplicate Want for same hash.
- Fuzz/Stress: Simulate 100 concurrent blob transfers with random frame ordering; all must complete successfully.
Blocked By: T-5-3-1
Blocking: none
Est. Hours: 6h
Expected Complexity: ~350 LoC
Migrated from Method backlog
This issue was created from a legacy filesystem backlog card. GitHub Issues are now the live work tracker; repository docs remain Method evidence.
Source backlog:
docs/method/backlog/cool-ideas/PLATFORM_deep-storage-wire-protocol.mdOriginal lane:
cool-ideasOriginal legend:
PLATFORMOriginal backlog card
Wire Protocol
WANT/PROVIDE/FRAME message types for network-level blob exchange between peers.
T-5-3-1: Message type definitions and binary encoding
User Story: As a developer, I want a compact binary wire format for blob exchange so that peers can request and transfer blobs efficiently.
Requirements:
Want { hash: BlobHash },Provide { hash: BlobHash, size: u64 },Frame { hash: BlobHash, offset: u64, data: Vec<u8> },Have { hash: BlobHash },Error { code: u16, message: String }.encode(&self) -> Vec<u8>anddecode(bytes: &[u8]) -> Result<Message, WireError>for each type.decode).Acceptance Criteria:
decode(encode(msg)) == msgfor all message types.decodeof truncated bytes returnsWireError::Incomplete.decodeof a message exceeding 1MB returnsWireError::TooLarge.Definition of Done:
Scope: Message type definitions, encode/decode, golden vectors, error types.
Out of Scope: Transport layer (TCP/WebSocket). Connection management. Authentication.
Test Plan:
Blocked By: none
Blocking: T-5-3-2
Est. Hours: 5h
Expected Complexity: ~300 LoC
T-5-3-2: Request/response protocol and backpressure
User Story: As a developer, I want a protocol state machine that handles blob exchange with flow control so that network transfers do not overwhelm memory.
Requirements:
BlobExchangestruct manages the protocol state: pending wants, in-flight provides, received frames.request(hash)enqueues aWant. When aProvidearrives, sendWantagain to confirm (pull model).Want, check local store and sendProvide(orError::NotFound). On confirmedWant, streamFramemessages.max_concurrent(default 16).Framechunks for a hash untiloffset + data.len() == size, then callput_verified.Acceptance Criteria:
max_concurrent=16results in 16 active wants, 16 queued.CasError::HashMismatch.Definition of Done:
Scope: Protocol state machine, backpressure, frame reassembly,
put_verifiedintegration.Out of Scope: Transport binding (TCP/WS). Discovery/peer management. Encryption.
Test Plan:
Blocked By: T-5-3-1
Blocking: none
Est. Hours: 6h
Expected Complexity: ~350 LoC