perf(storage): one fsync per 255 KB written, because the cache-release window is divided by stream count

[DecisionNerd]

Summary

cache_release_window_for_streams (graphforge-filesystem/src/cache_io.rs:16) divides a fixed 64 MiB budget by the number of active streams. With 256 partitions plus one that is

64 MiB / 257 = 255 KB per stream

and DurableFileCacheWriter::write calls synchronize_pending on every completed window — a full File::sync_all() followed by fadvise(DONTNEED).

So every spill in every family pays a full fsync per 255 KB written, and the window shrinks as partition count rises. That is the mechanism behind the measured merge_fsync_operations scaling at roughly 62 fsyncs per partition and growing linearly with data at a fixed count.

Scale

At S20 the ingest writes roughly 713 B/edge over 16.8M edges — about 12 GB through 255 KB windows, so on the order of 46,000 fsyncs. At 0.5-2 ms each that is 23-92 seconds of a ~170 second ingest, entirely serial.

Measured supporting evidence: merge_fsync_operations was 35,928 on the redesigned engine at S20 against 10,545 on the baseline, and a 4.2M-edge ingest at 256 partitions recorded 18,193 fsyncs plus 18,844 fadvise calls with 49,490 context switches. Effective cores sit at 0.82-0.85, consistent with a single thread spending roughly 15% of wall blocked on its own durability barriers.

Why this is the throughput lever worth pricing

Ingest has zero intra-process parallelism — verified by grep across append, seal, shape, encode and publish. Adding threads is a large programme. This is a constant-factor serial cost already paid on every run, and it is set by one formula.

It is also the shared root of the RSS/fsync trade recorded in #1439: raising partition count to help memory shrinks the window and multiplies fsyncs; lowering it does the reverse. Decoupling the window from stream count removes that coupling entirely.

The question that has to be answered first

What must be durable before a spill is sealed?

The per-window sync_all is only necessary if a crash between windows must leave the partial spill recoverable. If the correctness bound is "durable before the source is retired" — which is what SpillWriter::seal already establishes, with its own sync_all, root sync and receipt — then the per-window syncs buy nothing and can be replaced by sync_file_range for writeback pacing, or dropped until seal.

This is a durability-policy decision, not a tuning knob. It should be settled explicitly before the formula changes.

Note also that fadvise(DONTNEED) on spills that are read back seconds later converts every logical re-read into a device read. The ladder gate measures process RSS (VmHWM), and no cgroup memory limit is actually enforced — --memlimit appears only in a test fixture, and a baseline rung recorded a 12.41 GB cgroup peak. So the page cache is unconstrained and the eviction may be costing reads for a bound nothing is enforcing.

Acceptance

• merge_fsync_operations after the change is proportional to spills plus outputs rather than to bytes written divided by a shrinking window; state the measured counts at two scales.
• Ingest wall time and effective cores at S20, before and after, on a quiet host.
• The crash and recovery suites pass unmodified — recovery/tests.rs, construction_lifecycle_tests.rs. Prove by fault injection that no spill is retired before it is durable.
• The four fixed-width digests unchanged.

Related

#1439 (the RSS/fsync trade this decouples), #1387, #1417.

[DecisionNerd]

Durability decision: settled, and the premise was wrong

The issue asked whether a crash between windows must leave a partial spill recoverable. Nothing ever reads a partial spill, and the per-window sync_all is not a durability barrier in the first place.

1. A crash deletes every partial spill. It never resumes from one.

recover_shape_intent (graph_construction/recovery.rs:266) is all-or-nothing at the shape level. When intent.complete is false it refuses if the intent claims any output, then unlinks every shape-scoped name:

pub(super) fn is_shape_scoped_name(name: &str) -> bool {
    name.starts_with("shaped-") || name.starts_with("staged-") || name.starts_with("part-")
}

part- is the per-partition spills. They are deleted and the phase is redone. So the bytes the per-window syncs protect are, on the only code path that runs after a crash, unconditionally discarded.

This is the "analyst bench, not a forensic system" line exactly: durability whose sole unique coverage is bytes nobody reads.

2. sync_all() is a whole-file barrier, so the intermediate syncs are redundant by construction

synchronize_file is file.sync_all() (cache_io.rs:520), and seal calls sync_all_and_release() before it builds the receipt. One whole-file fsync at seal already covers everything written before it. The intermediate syncs cannot add durability the final barrier does not already provide.

3. What they are actually for: they are a precondition for fadvise(DONTNEED)

fn synchronize_before_release<T>(
    synchronize: impl FnOnce() -> io::Result<()>,
    release: impl FnOnce() -> io::Result<T>,
) -> io::Result<T> {
    synchronize()?;
    release()
}

You must sync before dropping pages or you discard dirty ones. The fsync serves the cache release, not the data. So the real question is not durability at all -- it is whether evicting page cache is worth an fsync per 255 KB.

The fix: raise the budget, do not decouple from stream count

My original "decouple the window from stream count" framing was wrong and would have removed a real invariant. DEFAULT_CACHE_RELEASE_WINDOW_BYTES is a shared budget, and the code enforces window x streams <= budget in two places (cache_io.rs:30, :54) with a test at cache_io/tests.rs:105. That invariant bounds aggregate dirty page cache in flight, which is a genuine resource. Keep it.

The defect is that the budget is 64 MiB on a 125 GB host -- 0.05% of RAM, and 1.6% of the 4 GiB per-rung budget. It is priced for a machine we are not running on.

Raising it preserves every barrier, every refusal and the aggregate bound, and only stops splitting one small budget 257 ways:

budget	window at 257 streams	fsyncs at S20 (~12 GB)
64 MiB (today)	255 KB	~46,000
512 MiB	2.0 MiB	~5,700
1 GiB	4.1 MiB	~2,900, 16x fewer

1 GiB of aggregate dirty pages sits comfortably inside the 4 GiB rung budget and is 0.8% of host RAM.

This is the fast-but-not-reckless shape of the change: nothing is removed, no barrier is weakened, no refusal is relaxed. A budget that was set too small for the hardware gets set correctly, and the fsync count falls out of it.

What must not change, and why

• sync_all_and_release() at seal -- the point where an artifact becomes something another phase consumes. This is the barrier that matters.
• The window x streams <= budget invariant and its test.
• reject_existing_merge_artifacts, the file_link_count != 1 check, and the identity checks in recovery.
• Fail-closed publication and unsupported-format refusal.

One open question worth pricing separately

fadvise(DONTNEED) on spills that are read back seconds later turns every logical re-read into a device read. The ladder gate measures process RSS (VmHWM), and the 4 GiB cgroup limit is recorded but never enforced -- --memlimit appears only in local_admission_fixture.py:87, and a baseline rung recorded a 12.41 GB cgroup peak. So we may be paying device reads to evict cache for a bound nothing enforces. That is a separate change from the budget and should be measured on its own.

Acceptance, revised

• merge_fsync_operations falls ~16x at S20 with the budget at 1 GiB; report measured counts at two scales.
• Ingest wall time and effective cores at S20 before and after, on a quiet host.
• cache_io/tests.rs aggregate-bound tests pass unmodified -- the invariant is preserved, not relaxed.
• Crash and recovery suites pass unmodified (recovery/tests.rs, construction_lifecycle_tests.rs).
• The four fixed-width digests unchanged.