Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 31 additions & 8 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1505,12 +1505,34 @@ npm run deploy
not one on these. Do not re-open it on the size table alone: that table now
favours dcb, and it is the wrong axis.

One cost the byte comparison hides: `node:zlib`'s brotli has NO dictionary
parameter (MODE, QUALITY, LGWIN, LGBLOCK, DISABLE_LITERAL_CONTEXT_MODELING,
SIZE_HINT, LARGE_WINDOW, NPOSTFIX, NDIRECT is the whole list). Those dcb
figures came from shelling out to the `brotli` CLI with `-D`. Switching would
reintroduce a CLI dependency in the build path, which is precisely what moving
the deltas into the build deleted.
One cost the byte comparison used to hide is RETIRED, and it is worth reading
as a correction rather than a fact. This note said `node:zlib`'s brotli has NO
dictionary parameter, listed the nine params as the whole list, and concluded
that switching would reintroduce a `brotli` CLI dependency in the build path,
which is precisely what moving the deltas into the build deleted. Node 26 takes
a `dictionary` option (nodejs/node#61763, merged 2026-02-13), and
`.node-version` is 26, so CI has it too. Measured 2026-08-10 on node v26.7.0,
q11 over a 21KB target: 70 bytes with no dictionary against 18 with one. The
dcb figures above still came from the CLI with `-D`, because that was the only
door on the day they were taken.

The CLI-dependency argument is therefore gone while the CONCLUSION is
untouched, since dcz won on DECODE and neither engine's encoder was ever the
question. Re-open it only with a fresh decode measurement, never on the size
table (see above) and no longer on this paragraph.

**Probe an engine's dictionary support, never infer it.** Three runtimes
disagree on this one option. Node 26 honours it. workerd accepts it for zstd
and silently ignores it, measured 2026-08-05, which is why `/terminal` ships no
delta calculator and says so on the page. Bun does the same thing through
1.3.14, its current stable, with the fix landing 2026-07-18 in oven-sh/bun#34427
and verified here on `1.4.0-canary.1`. **The failure is silent in every case,
because a frame compressed WITHOUT the dictionary still decodes fine WITH it**,
so nothing throws and the only signal is a byte count that never shrank. The
control is four lines: compress one target with no dictionary, the right one,
and a wrong one. An engine that honours the option prints a smaller number for
the right dictionary alone, and an engine that ignores it prints the same
number three times.

**zstd above level 19 is dead weight here.** Levels 19, 20, 21, 22, 22+long-
distance-matching and 22+btultra2 produce BYTE-IDENTICAL output on all 9 shell
Expand All @@ -1527,8 +1549,9 @@ npm run deploy

Deltas are BUILD OUTPUT, generated by build.mjs with `node:zlib`'s zstd. An
earlier version of this note said dictionary compression was "unreachable from
Node" and shipped a workstation script with committed artifacts; that limit is
BROTLI's, and generalizing it to zstd was wrong. `zstdCompressSync` takes a
Node" and shipped a workstation script with committed artifacts; that limit was
BROTLI's, generalizing it to zstd was wrong, and node 26 has since retired the
brotli half of it too (see above). `zstdCompressSync` takes a
`dictionary` option, it beats shelling out (116 bytes where the zstd CLI gave
120), and the foreign `zstd -d -D` CLI decodes Node's output byte-exact,
skippable prefix included — the interop check that matters, since the real
Expand Down
6 changes: 4 additions & 2 deletions build.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -1595,8 +1595,10 @@ for (const [file, srcPath, marker] of SHELLS) {
// luna.css change: 116 bytes against 7,615.
//
// This used to be a workstation script with committed artifacts, on the belief that
// dictionary compression was unreachable from Node. That was true of BROTLI and I wrongly
// generalized it: node:zlib's zstd DOES take a `dictionary` option. It is also better
// dictionary compression was unreachable from Node. That was true of BROTLI AT THE TIME and I
// wrongly generalized it: node:zlib's zstd DOES take a `dictionary` option. (node 26 has since
// given brotli one too, nodejs/node#61763, which retires the limit rather than the choice —
// dcz won on decode speed, not on either encoder's availability. See gotcha 14.) It is also better
// than shelling out — 116 bytes where the zstd CLI produced 120 — and portable, verified
// by having the foreign `zstd -d -D` CLI decode Node's bytes byte-exact, skippable prefix
// and all. That interop check is the one that matters, because the real decoder is a
Expand Down
51 changes: 36 additions & 15 deletions holding/garage/compression.html
Original file line number Diff line number Diff line change
Expand Up @@ -379,24 +379,36 @@ <h2>Deltas, and why they ship as dcz</h2>
</p>
<p>
RFC 9842 defines two encodings for this, and Cloudflare passes both through identically, so
the choice is free and purely about engineering. Brotli is nominally smaller. zstd decodes
about twice as fast:
the choice is free and purely about engineering. Brotli compresses smaller. zstd decodes
much faster:
</p>
<table class="cmp">
<thead><tr><th>measure</th><th>dcb (brotli)</th><th>dcz (zstd)</th></tr></thead>
<tbody>
<tr><td>the real luna.css delta</td><td>79 bytes</td><td>80 bytes</td></tr>
<tr><td>source-level deltas, larger</td><td class="y">5-8% smaller</td><td>n/a</td></tr>
<tr><td>decode, bare 46KB asset</td><td>0.094 ms</td><td class="y">0.046 ms</td></tr>
<tr><td>throughput</td><td>471 MB/s</td><td class="y">954 MB/s</td></tr>
<tr><td>all 12 shipping deltas, total</td><td class="y">3,344 bytes</td><td>3,589 bytes</td></tr>
<tr><td>widest single gap (nav.js)</td><td class="y">685 bytes</td><td>729 bytes</td></tr>
<tr><td>decode, 47,615 B rebuilt</td><td>0.1368 ms</td><td class="y">0.0165 ms</td></tr>
<tr><td>throughput</td><td>332 MiB/s</td><td class="y">2,752 MiB/s</td></tr>
</tbody>
</table>
<p>
I argued for brotli on bytes and got talked out of it, correctly. Bytes are a proxy;
latency is the thing you actually want. And the byte case collapses on inspection: on the
pair this site really ships, brotli wins by <em>one byte</em>. The 5-8% edge only appears
on much larger deltas than a shell asset produces. Meanwhile the decode numbers came off a
fast laptop, so the gap widens on the phones that need it most.
I argued for brotli on bytes and got talked out of it, correctly, and then the numbers I
argued from turned out to be wrong in <em>both</em> directions. Re-measured against real
dictionaries rather than stand-ins, brotli's size edge is bigger than the one byte I had
(245 bytes across the set, 6.8%, winning 11 of 12), and zstd's decode edge is far bigger
than the 2x I had (8.3x). The call held anyway, which is the interesting part.
</p>
<p>
It held for a structural reason: <strong>decode scales with the reconstruction, not with
the delta.</strong> A 685-byte dcb delta still rebuilds 47,615 bytes, so making the delta
smaller never makes the decode gap smaller. Break-even at 9 Mbps is about 135 bytes saved
per asset and dcb saves 44, so dcz wins by roughly 3x. On a slow phone the margin widens,
because decode scales with the CPU while 44 bytes stays 44 bytes.
</p>
<p>
Worth saying plainly: on the old numbers this was a coin flip, and on these it is not. The
size table now favours brotli by more than it ever did, and that table is the wrong axis to
decide on.
</p>
<p>
dcz's framing is the tidier of the two as a bonus. The dictionary hash rides in a Zstandard
Expand All @@ -422,7 +434,7 @@ <h2>Deltas, and why they ship as dcz</h2>
<h2>One more over-generalization</h2>
<p>
Dictionary compression is unreachable from Node's zlib, so the deltas started life as a
workstation script with committed artifacts. That sentence is true of brotli. I applied it
workstation script with committed artifacts. That sentence was true of brotli. I applied it
to zstd without checking, and <code>zstdCompressSync</code> takes a <code>dictionary</code>
option. It also beats shelling out to the CLI: 116 bytes against 120.
</p>
Expand All @@ -433,6 +445,15 @@ <h2>One more over-generalization</h2>
interop: the foreign <code>zstd -d -D</code> CLI decodes Node's output byte-exact, prefix
included, because the real decoder is a browser and not Node.
</p>
<p>
<strong>Update, August 2026:</strong> the brotli half expired too. Node 26 added a
<code>dictionary</code> option to brotli
(<a href="https://github.com/nodejs/node/pull/61763" rel="external">nodejs/node#61763</a>),
so on v26.7.0 a quality-11 compress of a 21 KB target goes from 70 bytes with no dictionary
to 18 with one. The limit I over-generalized was accurate about brotli the day I wrote it
and is now accurate about neither encoder. So a runtime limit has a shelf life, and the
version you measured on belongs in the note next to the measurement.
</p>

<h2>What shipped</h2>
<table class="cmp">
Expand Down Expand Up @@ -511,17 +532,17 @@ <h2>The part that transfers</h2>
"q": "The deltas ship as dcz even though brotli measured smaller. Why?",
"options": [
{
"t": "zstd decodes about 2x faster, and on the pair this site actually ships brotli's advantage was one byte.",
"t": "Decode scales with the reconstruction rather than the delta, so brotli's smaller delta never shrinks zstd's 8.3x decode lead.",
"ok": true,
"why": "Bytes are a proxy for latency. The 5-8% brotli edge only shows up on deltas far larger than a shell asset produces, while the decode gap grows on slow CPUs."
"why": "Re-measured against real dictionaries, dcb is 245 bytes (6.8%) ahead across all 12 pairs, and dcz still wins: rebuilding 47,615 bytes costs 0.0165 ms against 0.1368. Break-even at 9 Mbps needs ~135 bytes saved per asset and dcb saves 44."
},
{
"t": "Cloudflare only passes dcz through; dcb is not supported at the edge.",
"why": "Both are passed through identically, on all plans. That is exactly why the choice was free to make on engineering grounds."
},
{
"t": "Because brotli cannot use a custom dictionary, so dcb was never actually available.",
"why": "Brotli does custom dictionaries fine via its CLI. What it lacks is a dictionary option in Node's zlib, which is a tooling gap rather than a format limit."
"why": "Brotli does custom dictionaries fine. What it lacked was a dictionary option in Node's zlib, which was a tooling gap rather than a format limit, and node 26 closed even that."
}
]
},
Expand Down
3 changes: 2 additions & 1 deletion scripts/roll-shell-dictionary.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
//
// This is all that remains of the old gen-shell-deltas.mjs. The deltas themselves moved
// into build.mjs once it turned out node:zlib's zstd takes a `dictionary` option (the
// "unreachable from Node" limit was BROTLI's, and I had over-generalized it), so they are
// "unreachable from Node" limit was BROTLI's at the time, I had over-generalized it, and
// node 26 has since given brotli one as well), so they are
// build output now: no zstd CLI, no committed artifacts, nothing to forget.
//
// Rolling stays a deliberate human step for one reason: it WRITES INTO THE SOURCE TREE, and
Expand Down
Loading