PocketIC silently reloads canister state from stale checkpoint after macOS sleep

[myurr]

Bug: PocketIC silently reloads canister execution state from stale checkpoint after macOS sleep

Summary

When dfx start (v0.30.2, PocketIC-based) runs on macOS and the laptop sleeps for ≥ ~90 minutes, certain canisters have their Wasm heap reset and stable memory rolled back to a stale on-disk checkpoint — even though the PocketIC process and all sandbox processes remain alive throughout. In-memory page deltas that should still be in the process's address space are lost. This causes canister state to silently regress to an earlier point in time.

Environment

• dfx: 0.30.2
• pocket-ic: bundled with dfx 0.30.2 (running under Rosetta on Apple Silicon)
• macOS: Apple Silicon, macOS Sequoia (15.x)
• Rust canister toolchain: ic-stable-structures 0.7, ic-cdk 0.20, ic-cdk-timers 1.0, candid 0.10

Reproduction Steps

Minimal reproduction canister

A trivial canister (push_stress) with zero application logic — only ic-stable-structures MemoryManager + StableBTreeMap + StableCell:

# Cargo.toml
[dependencies]
candid = "0.10"
ic-cdk = "0.20"
ic-cdk-timers = "1.0"
ic-stable-structures = "0.7"

The canister:

1. Uses MemoryManager with a StableBTreeMap and two StableCells
2. Maintains a heap counter (heap_tick) incremented every 30 seconds via timer
3. Maintains a stable counter (push_count in a StableCell) incremented alongside heap_tick
4. Has stable memory grown to 2,177 pages (~134 MB) via Memory::grow()

Steps

1. Start dfx start on macOS
2. Deploy the canister and verify: heap_tick > 0, push_count > 0, stable_pages = 2177
3. Close the laptop lid for ≥ 90 minutes
4. Open the lid
5. Query get_status → heap_tick has reset to a small number (counting from 0), push_count continues from its pre-sleep value

heap_tick resetting proves the Wasm heap was re-initialized. push_count continuing from its StableCell value proves stable memory is partially restored (the StableCell's MemoryId region was recovered), but writes from the period before the sleep may be lost.

Key observations

• The PocketIC process (pocket-ic --port-file ... --ttl 2592000 --log-levels error) remains alive throughout — verified via ps -p <pid> -o lstart showing continuous uptime across multiple sleep events.
• All canister sandbox processes (pocket-ic --run-as-canister-sandbox ...) also remain alive with their original PIDs and start dates.
• macOS power log (pmset -g log) confirms normal Clamshell Sleep → DarkWake → Full Wake transitions, NOT process termination.
• Despite the process surviving, canister execution state is reloaded from a stale on-disk checkpoint.

Trigger Conditions

Through controlled experiments over 7 sleep/wake events (Apr 1–3, 2026), we identified the conditions that trigger the bug:

Canister	Stable pages	Uses MemoryManager	Affected?	Sleep events tested
Service map (production)	2,177	Yes	Always (7/7 events ≥ 90 min)	7
push_stress (test)	513	Yes	Never (0/5)	5
push_stress (test, grown)	2,177	Yes	Yes (1/1) — first event after growing	1
raw_stable_probe (test)	2,177	No (DefaultMemoryImpl)	Never (0/1)	1
stable_memory_probe (test)	385	No	Never (0/7)	7

Two conditions appear necessary:

1. Stable memory ≥ ~2,177 pages (~134 MB) — push_stress at 513 pages was immune across 5 events; at 2,177 pages it was affected on the first event.
2. MemoryManager usage — push_stress (MemoryManager, 2,177 pages) was affected; raw_stable_probe (DefaultMemoryImpl, 2,177 pages) was not. (Based on 1 event — needs more data to confirm.)

Sleep duration threshold: ~90 minutes. Sleeps < ~90 minutes do not trigger the issue. The PocketIC embedder config shows max_sandbox_idle_time: 30 seconds, which may be related — all sandboxes exceed this during any sleep.

Observed Behavior

Heap reset

All heap (Wasm linear memory) variables reset to their default values, as if the canister had been freshly installed. This is observed via:

• LAST_HEARTBEAT_NS (heap variable) resetting to 0
• heap_tick (heap counter) resetting to 0
• Ring buffer contents (stored in a MemoryManager-backed VM region) being wiped

Stable memory rollback

Stable memory is restored from an old on-disk checkpoint, not from the in-memory page deltas:

• The service map's B-tree registry shows tree_len values from an older checkpoint (e.g., 23 entries when it should have 44).
• Writes made between the last on-disk checkpoint and the sleep are lost. In one test, a deduplication operation successfully removed 21 entries from the B-tree; after sleep, all 21 removed entries reappeared.
• StableCell values (small, single-page structures) survive, suggesting that some stable memory regions are recovered while others are not.

Late reconciliation

Minutes after wake, the correct state sometimes becomes visible — old entries that were "missing" reappear. This creates duplicate entries when the canister has already created new entries based on the stale state.

Why This Shouldn't Happen

The PocketIC process is not killed by macOS sleep. All in-memory state — including page deltas accumulated since the last checkpoint — should be in the process's address space when it resumes. The IC execution controller's execute() method passes execution_state.wasm_memory and execution_state.stable_memory to sandboxes, which should contain all accumulated page deltas.

Something inside PocketIC's state management is discarding or bypassing the in-memory page deltas and reloading canister execution state from on-disk checkpoints during or after the sleep/wake cycle. Possible mechanisms:

1. Checkpoint cycle triggered by round catch-up: After waking, PocketIC processes many accumulated rounds. If this triggers a checkpoint cycle (tip_to_checkpoint_and_switch + reset_tip_to), and the checkpoint flush doesn't fully capture all canisters' page deltas, the subsequent tip reset would discard the missing deltas.

2. Sandbox eviction + stale execution state reload: The max_sandbox_idle_time: 30s would mark all sandboxes as idle during sleep. If the eviction path reloads execution state from on-disk checkpoints rather than the main process's canonical in-memory state, the stale checkpoint would be used.

3. heap_delta_estimate overflow: If the accumulated heap deltas across all canisters exceed a threshold during catch-up processing, the system might force a checkpoint that doesn't properly flush all deltas.

Impact

This is a local development only issue — it cannot affect mainnet (where consensus and replication ensure state consistency). However, it is highly disruptive for local development:

• Silent data loss — writes to stable memory are rolled back without any error or warning
• Canister heap resets break any canister that relies on heap-resident state (counters, caches, flags)
• The "late reconciliation" phase creates duplicate state that is difficult to detect and clean up
• Cleanup operations (like dedup) are themselves subject to rollback on the next sleep event

Diagnostic Data Available

We have extensive logs, overlay file histories, and canister state snapshots from 7 sleep/wake events. The full investigation log (500+ lines) is available if useful. We can also provide the push_stress canister source code as a minimal reproduction case.

[myurr]

Minimal reproduction canister: push_stress

This is the complete source of the test canister that reproduces the bug. It has zero application logic — only ic-stable-structures MemoryManager, StableBTreeMap, StableCell, and a 30-second timer.

Cargo.toml

[package]
name = "push_stress"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
candid = "0.10"
ic-cdk = "0.20"
ic-cdk-timers = "1.0"
ic-stable-structures = "0.7"

src/lib.rs

use candid::Principal;
use ic_stable_structures::memory_manager::{MemoryId, MemoryManager, VirtualMemory};
use ic_stable_structures::{DefaultMemoryImpl, Memory, StableBTreeMap, StableCell};
use std::cell::RefCell;
use std::time::Duration;

const SENTINEL: [u8; 8] = *b"PSHSTRES";
const PUSH_COUNT_MEM: MemoryId = MemoryId::new(100);
const REGISTRY_MEM: MemoryId = MemoryId::new(101);
const SENTINEL_MEM: MemoryId = MemoryId::new(102);

type Mem = VirtualMemory<DefaultMemoryImpl>;

thread_local! {
    static MEMORY_MANAGER: RefCell<MemoryManager<DefaultMemoryImpl>> =
        RefCell::new(MemoryManager::init(DefaultMemoryImpl::default()));

    static PROBE_PRINCIPAL: RefCell<Option<Principal>> = const { RefCell::new(None) };

    static PUSH_COUNTER: RefCell<StableCell<u64, Mem>> = RefCell::new(
        StableCell::init(
            MEMORY_MANAGER.with(|m| m.borrow().get(PUSH_COUNT_MEM)),
            0u64,
        )
    );

    static REGISTRY: RefCell<StableBTreeMap<Vec<u8>, Vec<u8>, Mem>> = RefCell::new(
        StableBTreeMap::init(
            MEMORY_MANAGER.with(|m| m.borrow().get(REGISTRY_MEM))
        )
    );

    static SENTINEL_CELL: RefCell<StableCell<[u8; 8], Mem>> = RefCell::new(
        StableCell::init(
            MEMORY_MANAGER.with(|m| m.borrow().get(SENTINEL_MEM)),
            [0u8; 8],
        )
    );

    static TICK_COUNT: RefCell<u64> = const { RefCell::new(0) };
}

#[ic_cdk::init]
fn init(probe_principal: Principal) {
    PROBE_PRINCIPAL.with(|p| *p.borrow_mut() = Some(probe_principal));
    write_sentinel();
    grow_stable_memory(2050); // grow to ~2177 pages to trigger the bug
    start_timer();
}

#[ic_cdk::post_upgrade]
fn post_upgrade(probe_principal: Principal) {
    PROBE_PRINCIPAL.with(|p| *p.borrow_mut() = Some(probe_principal));
    check_stable_memory("post_upgrade");
    start_timer();
}

fn start_timer() {
    ic_cdk_timers::set_timer_interval(Duration::from_secs(30), || async {
        let tick = TICK_COUNT.with(|c| { let mut c = c.borrow_mut(); *c += 1; *c });

        // Increment stable push counter
        PUSH_COUNTER.with(|c| { let mut c = c.borrow_mut(); let _ = c.set(c.get() + 1); });

        // Write bulk data to dirty pages
        write_bulk_data();

        check_stable_memory(&format!("tick#{}", tick));
    });
}

fn write_sentinel() {
    SENTINEL_CELL.with(|c| { let _ = c.borrow_mut().set(SENTINEL); });
}

fn write_bulk_data() {
    MEMORY_MANAGER.with(|m| {
        let vm = m.borrow().get(MemoryId::new(103));
        while vm.size() < 2 { vm.grow(1); }
        let ts = ic_cdk::api::time().to_le_bytes();
        let mut block = [0u8; 64];
        block[..8].copy_from_slice(&ts);
        for i in 0..128 {
            block[8..16].copy_from_slice(&(i as u64).to_le_bytes());
            vm.write((i * 64) as u64, &block);
        }
    });
}

fn check_stable_memory(context: &str) {
    let sentinel_ok = SENTINEL_CELL.with(|c| c.borrow().get().clone() == SENTINEL);
    let push_count = PUSH_COUNTER.with(|c| *c.borrow().get());
    let stable_pages = ic_cdk::api::stable_size();
    let tick = TICK_COUNT.with(|c| *c.borrow());

    ic_cdk::println!(
        "[push_stress] {} sentinel={} stable_pages={} push_count={} heap_tick={}",
        context, sentinel_ok, stable_pages, push_count, tick,
    );
}

fn grow_stable_memory(target_pages: u64) {
    MEMORY_MANAGER.with(|m| {
        let vm = m.borrow().get(MemoryId::new(103));
        while ic_cdk::api::stable_size() < target_pages {
            if vm.grow(1) == -1 { break; }
        }
    });
}

#[ic_cdk::query]
fn get_status() -> String {
    let sentinel_ok = SENTINEL_CELL.with(|c| c.borrow().get().clone() == SENTINEL);
    let push_count = PUSH_COUNTER.with(|c| *c.borrow().get());
    let stable_pages = ic_cdk::api::stable_size();
    let tick = TICK_COUNT.with(|c| *c.borrow());

    format!(
        "sentinel={} stable_pages={} push_count={} heap_tick={}",
        sentinel_ok, stable_pages, push_count, tick,
    )
}

ic_cdk::export_candid!();

How to observe the bug

1. Deploy with any principal as the init arg (it doesn't need to point to a real canister for the reproduction)
2. Wait for a few ticks: dfx canister call push_stress get_status should show heap_tick increasing and push_count increasing
3. Close laptop lid for ~90+ minutes
4. Open lid and query again: heap_tick will have reset to a small number (counting from 0 since the internal restart), while push_count continues from its pre-sleep stable value

The heap_tick reset proves the Wasm heap was re-initialized from a stale checkpoint. The process (pocket-ic) remains alive throughout — verified via ps.