Store backup blobs in packs instead of one file each - #17
Open
chicagobuss wants to merge 1 commit into
Open
Conversation
The per-file pool was the wrong shape. Blobs here average 4.9 KB, so storing them individually cost 20 MB on disk for 11 MB of data — more in block overhead than the data occupies — and the file count grew ~190/day with no ceiling. That count, not the volume, is what forces directory sharding, incremental cursors, per-key presence checks and a garbage-collection keep-set. All of it exists to manage files, not bytes. Blobs are immutable and named by their own sha256, so append them to pack files instead. A pack seals at day boundary or BACKUP_PACK_MAX_BYTES, whichever comes first; a sealed pack never changes, so it is uploaded exactly once and read only during a restore. Sealing by size alone is not enough: the open pack is re-sent offsite every run, and a pack that takes months to reach the cap means months of re-uploading the same tens of megabytes hourly. Measured on this instance: 2372 blobs across 3 files instead of 2372; warm run 2.4s -> 1.4s; steady-state offsite traffic is the current day's pack rather than a per-blob delta; disk 20 MB -> 7.3 MB. Deleted rather than kept: gc-pool.sh and the keep-set union rule (at ~90 MB/year packed there is nothing worth collecting), and the pool commands in s3util.py. The INDEX is checksummed, since it is now the only record of which pack holds a blob. restore.sh detects pack, pool and embedded-blobs archives, so snapshots written before this keep restoring while they remain in retention. Proven end to end before landing: a clean-room trial on a copy of this data caught two defects that reasoning had not — corruption and missing packs were only detected after pg_restore had already run, leaving a database whose blobs were never uploaded. Restore now pre-flights every pack for presence and hash before touching the target. Restores verified from local packs and from R2 alone, each booted as a live instance with row counts, workspaces, served content and revision history compared against production, and every restored blob re-hashed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The per-file pool was the wrong shape, and the numbers say so plainly:
The problem was never volume, it was file count — and file count is what forces directory sharding, incremental cursors, per-key presence checks, and a GC keep-set. Every one of those exists to manage files, not bytes.
What changed
Blobs are immutable and named by their own sha256, so they are appended to pack files. A pack seals at a day boundary or
BACKUP_PACK_MAX_BYTES, whichever comes first; a sealed pack never changes, so it uploads exactly once and is read only during a restore.Sealing by size alone was not enough, and testing caught it: the open pack is re-sent offsite every run, so a pack that takes months to reach the cap means months of re-uploading the same tens of megabytes hourly. Daily rolling bounds that to one day's blobs.
What this deletes
scripts/gc-pool.shand the keep-set union rule — at ~90 MB/year packed there is nothing worth collectings3util.py(sync-pool,verify-pool,upload-from-pool,push-pool,pull-pool,gc-pool)Net fewer moving parts than what is in production today.
INDEXis checksummed — it is now the only record of which pack holds a blob, so silent truncation would strand blobs that are physically present.Compatibility
restore.shdetects three layouts: packs, the per-file pool, and archives that embed their ownblobs/. Snapshots written before this keep restoring while they remain in retention. The pool directory is left in place for now and can be deleted once the last snapshot referencing it ages out.Proof
Built and exercised first in a clean room — separate Postgres, bucket and instance, seeded from a real production snapshot — before touching anything real. That caught two defects reasoning had not:
pg_restorehad already run, leaving a database whose blobs were never uploadedFileNotFoundErrortraceback rather than a refusalRestore now pre-flights every pack for presence and hash before touching the target database. Re-tested: missing pack refuses (exit 1, DB untouched), corrupt pack refuses before
pg_restore(exit 1), healthy restore unaffected.Then verified on production data into scratch, and again from R2 alone with local state ignored. Each restore was booted as a live tracker v1.5.2 instance and compared against production: row counts identical across documents/revisions/events/workspaces, 9 workspaces, documents served byte-identical, revision history intact, and every restored blob re-hashed (2373/2373, 0 corrupt).
All scratch databases, buckets and containers torn down; production untouched throughout and still healthy on v1.5.2.