Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .github/workflows/dictionary-roll.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ name: Dictionary roll
# Nobody remembered. Measured 2026-08-12: the last real roll was #178 on
# 2026-07-30, 161 commits earlier, and every dictionary-carrying shell asset
# production was serving (nav, luna, hoist, tooltip, lens) was absent from
# www/a-dict. Returning visitors were taking plain brotli on the render-blocking
# src/dict/a-dict. Returning visitors were taking plain brotli on the render-blocking
# path: 13.7 KB where the deltas are 1.3 KB. `pnpm run dcz:check` reported PASS
# throughout, because its shell probe offers a COMMITTED candidate and so could
# only ever agree with itself. That probe now has a coverage assertion beside it,
Expand Down Expand Up @@ -97,12 +97,12 @@ jobs:
id: diff
run: |
set -euo pipefail
if [ -z "$(git status --porcelain www/a-dict www/p-dict)" ]; then
if [ -z "$(git status --porcelain src/dict/a-dict src/dict/p-dict)" ]; then
echo "changed=false" >> "$GITHUB_OUTPUT"
echo "Dictionaries already cover what production is serving; nothing to do."
else
echo "changed=true" >> "$GITHUB_OUTPUT"
git status --short www/a-dict www/p-dict | head -40
git status --short src/dict/a-dict src/dict/p-dict | head -40
fi

- name: Open or update the roll PR
Expand All @@ -115,14 +115,14 @@ jobs:
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"

ADOPTED=$(git status --porcelain www/a-dict www/p-dict | grep -c '^??' || true)
PRUNED=$(git status --porcelain www/a-dict www/p-dict | grep -c '^ D' || true)
ADOPTED=$(git status --porcelain src/dict/a-dict src/dict/p-dict | grep -c '^??' || true)
PRUNED=$(git status --porcelain src/dict/a-dict src/dict/p-dict | grep -c '^ D' || true)

# One long-lived branch, force-pushed. A fresh branch per night would leave a
# trail of superseded PRs, and the roll is not a series: only the newest one
# describes what production is serving now.
git checkout -B "$BRANCH"
git add www/a-dict www/p-dict
git add src/dict/a-dict src/dict/p-dict
git commit -m "chore(dcz): roll the dictionaries onto what production serves

Adopted $ADOPTED, pruned $PRUNED. Sourced from https://aadhar.sh over the
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/perf-diff.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ on:
- "!www/images/**"
- "!www/cars/**"
- "!www/og/**"
- "!www/p-dict/**"
- "!www/a-dict/**"
- "!src/dict/p-dict/**"
- "!src/dict/a-dict/**"
- "!**.md"

permissions:
Expand Down
2 changes: 1 addition & 1 deletion .oxlintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
// captured off the wire, and `pretext.lib.js` is @chenglou/pretext 0.0.7
// vendored same-origin so /garage/pretext can load it under `script-src
// 'self'`. Linting either one reports on somebody else's minifier.
"ignorePatterns": ["www/a-dict/**", "www/garage/pretext.lib.js", ".build/**", "tools/oxlint/anti-slop/**"],
"ignorePatterns": ["src/dict/a-dict/**", "www/garage/pretext.lib.js", ".build/**", "tools/oxlint/anti-slop/**"],

// NAMING ANY PLUGIN REPLACES THE DEFAULT SET, so this list has to restate the
// four that were on implicitly. Measured 2026-08-15: with `"plugins":
Expand Down
12 changes: 6 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2351,7 +2351,7 @@ pnpm run deploy:direct
step to forget, and no staleness tripwire needed, because a delta is a pure
function of bytes the build just produced.

What still has to be committed is `www/a-dict/`, the SHELL dictionary set,
What still has to be committed is `src/dict/a-dict/`, the SHELL dictionary set,
because an `/a/` asset is content-addressed: a change mints a new URL, so its
dictionary must be bytes the BROWSER already holds and no build can derive that
from source. `pnpm run shell:roll` adopts the current shell and prunes to 3 per
Expand All @@ -2378,7 +2378,7 @@ pnpm run deploy:direct
from the staged documents, ships it at an immutable `/a/page-family.<hash8>.dict`,
and every HTML response advertises it via `Link: rel="compression-dictionary"`
(`lib/security.js`). It also diffs the current page against the committed
`www/p-dict` snapshots from the previous release. The worker tries the
`src/dict/p-dict` snapshots from the previous release. The worker tries the
`Available-Dictionary` tag it receives. The family offer deliberately uses a
longer site-wide URLPattern than an exact page path, so RFC 9842 makes it the
preferred dictionary whenever both are cached; this prevents an uncaptured old
Expand Down Expand Up @@ -2417,12 +2417,12 @@ pnpm run deploy:direct
server-side proven (149-byte page delta decodes to the live page), svg OFF by
design (Chromium's image loader chokes). `pnpm run dcz:check` asserts both page
tiers against production, reading the family dictionary out of the live `Link`
header and the per-page candidate from `www/p-dict`. With `pnpm run dict:roll`
header and the per-page candidate from `src/dict/p-dict`. With `pnpm run dict:roll`
the source is production for both halves, so the old "roll only from the deployed
build, never from a feature branch" rule is satisfied by construction rather than
by remembering it. Plain `shell:roll` still reads `.build/` and still carries that
requirement. (Either writes into `www/a-dict/` the moment it runs, so if you run one
to read the code, `git checkout -- www/a-dict` after.)
requirement. (Either writes into `src/dict/a-dict/` the moment it runs, so if you run one
to read the code, `git checkout -- src/dict/a-dict` after.)

**`dcz:check`'s shell probe could only ever agree with itself, and the coverage
assertion beside it is the fix.** The probe picks an a-dict candidate that is NOT
Expand Down Expand Up @@ -2668,7 +2668,7 @@ pnpm run deploy:direct
claim that "the edge cannot rewrite HTML it did not compress" is true of the
zone-side Web Analytics injector and NOT true in general. Verify per feature.

What broke: `www/p-dict` snapshots were adopted from `.build/www/*.html`,
What broke: `src/dict/p-dict` snapshots were adopted from `.build/www/*.html`,
and a shared dictionary is matched by the SHA-256 a BROWSER computes over the
body it stored. The staged file has no injected tag, so every snapshot hashed to
bytes nobody held and the 93-97% per-page tier fell back to the family
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
8 changes: 4 additions & 4 deletions tools/build.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ async function checkInvariants() {
const VT_DIAGNOSTIC = /^garage\/vt-(check|b)\.html$/;
const vtSources = [
...(await readdir("www", { recursive: true }))
.filter((r) => /\.(html|css|js)$/.test(r) && !VT_DIAGNOSTIC.test(r) && !r.startsWith("a-dict/"))
.filter((r) => /\.(html|css|js)$/.test(r) && !VT_DIAGNOSTIC.test(r))
.map((r) => `www/${r}`),
"cal/src/templates.js", "serendipity/serendipity.js", "pipelines/lwe/generate.mjs",
];
Expand Down Expand Up @@ -1832,7 +1832,7 @@ for (const file of ["nav-run.css", "nav-tray.css", "infotip.css"]) {
// no `pnpm run shell:deltas` step to forget, and no staleness tripwire needed at all,
// because a delta is now a pure function of bytes this build just produced.
//
// Still committed, and unavoidably so: www/a-dict/, the DICTIONARY set. A dictionary
// Still committed, and unavoidably so: src/dict/a-dict/, the DICTIONARY set. A dictionary
// has to be bytes the BROWSER already holds, which no build can derive from source.
{
// HARD CHECK: is the `dictionary` option actually honored by this Node?
Expand All @@ -1859,7 +1859,7 @@ for (const file of ["nav-run.css", "nav-tray.css", "infotip.css"]) {
);
}

const dictDir = "www/a-dict";
const dictDir = "src/dict/a-dict";
const dicts = await readdir(dictDir).catch(() => []);
const parse = (n) => { const m = n.match(/^(.+)\.([0-9a-f]{8})\.(js|css|svg)$/); return m ? { base: m[1], hash8: m[2], ext: m[3], name: n } : null; };
const shell = files.map(parse).filter(Boolean);
Expand Down Expand Up @@ -2249,7 +2249,7 @@ for (const file of ["nav-run.css", "nav-tray.css", "infotip.css"]) {
// slug: the request path with separators folded, so it survives as one filename segment.
const slugOf = (assetPath) => assetPath.replace(/\.html$/, "").replace(/\//g, "__");

const dictDir = "www/p-dict";
const dictDir = "src/dict/p-dict";
const dicts = (await readdir(dictDir).catch(() => []));
const parseDict = (n) => {
const m = n.match(/^(.+)\.([0-9a-f]{16})\.html\.br$/);
Expand Down
18 changes: 9 additions & 9 deletions tools/check-dictionary-support.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,10 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
{
const home = await (await fetch("https://aadhar.sh/", { headers: { "accept-encoding": "identity" } })).text();
const live = home.match(/\/a\/nav\.([0-9a-f]{8})\.js/)?.[1];
const cand = (await readdir("www/a-dict")).find((n) => n.startsWith("nav.") && !n.includes(live));
const cand = (await readdir("src/dict/a-dict")).find((n) => n.startsWith("nav.") && !n.includes(live));
if (!cand) report("shell js (nav)", false, "no non-live a-dict candidate to probe with");
else {
const r = await get(`https://aadhar.sh/a/nav.${live}.js`, b64(await readFile(`www/a-dict/${cand}`)));
const r = await get(`https://aadhar.sh/a/nav.${live}.js`, b64(await readFile(`src/dict/a-dict/${cand}`)));
const ce = r.headers.get("content-encoding");
report("shell js (nav)", ce === "dcz", `ce=${ce} vs candidate ${cand}`);
}
Expand Down Expand Up @@ -63,7 +63,7 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
// until a deploy fixes it. Do NOT make dcz:check a required check or it deadlocks the very
// release that would clear it.
{
const committed = (await readdir("www/a-dict")).filter((n) => /\.[0-9a-f]{8}\.(js|css)$/.test(n));
const committed = (await readdir("src/dict/a-dict")).filter((n) => /\.[0-9a-f]{8}\.(js|css)$/.test(n));
const bases = new Set(committed.map((n) => n.replace(/\.[0-9a-f]{8}\.(js|css)$/, "")));

// Two pages, because no single document references every dictionary-carrying asset:
Expand All @@ -84,7 +84,7 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
// sha256, and the browser keys on the full digest of the bytes it stored, so the
// committed copy has to be those bytes exactly.
const served = Buffer.from(await (await fetch(`https://aadhar.sh/a/${name}`, { headers: { "accept-encoding": "identity" } })).arrayBuffer());
if (!served.equals(await readFile(`www/a-dict/${name}`))) missing.push(`${name} (filename matches, bytes do not)`);
if (!served.equals(await readFile(`src/dict/a-dict/${name}`))) missing.push(`${name} (filename matches, bytes do not)`);
}
const skipped = [...refs].filter(([, base]) => !bases.has(base)).map(([n]) => n);
report("live shell is covered by a-dict", missing.length === 0,
Expand Down Expand Up @@ -126,7 +126,7 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
ce === "dcz" ? `ce=dcz vs ${offered}` : `ce=${ce} — the deployed /pd/ deltas may predate this dictionary`);
}

const candidates = (await readdir("www/p-dict").catch(() => []))
const candidates = (await readdir("src/dict/p-dict").catch(() => []))
.filter((name) => name.startsWith("garage__pretext.") && name.endsWith(".html.br"));
if (!candidates.length) {
report("page html per-page tier", true, "no committed pretext snapshot — family tier is the only candidate");
Expand All @@ -139,7 +139,7 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
// usable against production right now" is the property worth asserting.
const tried = [];
for (const name of candidates) {
const raw = brotliDecompressSync(await readFile(`www/p-dict/${name}`));
const raw = brotliDecompressSync(await readFile(`src/dict/p-dict/${name}`));
const ce = (await get("https://aadhar.sh/garage/pretext", b64(raw))).headers.get("content-encoding");
tried.push(`${name}=${ce}`);
if (ce === "dcz") break;
Expand Down Expand Up @@ -170,7 +170,7 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
);
const liveDoc = await (await fetch("https://aadhar.sh/garage/pretext", { headers: { "accept-encoding": "identity" } })).text();
const held = await Promise.all(candidates.map(async (name) =>
srcs(brotliDecompressSync(await readFile(`www/p-dict/${name}`)).toString("utf8"))));
srcs(brotliDecompressSync(await readFile(`src/dict/p-dict/${name}`)).toString("utf8"))));
const unheld = [...srcs(liveDoc)].filter((src) => !held.some((set) => set.has(src)));
report("committed snapshots are WIRE bytes", unheld.length === 0,
unheld.length === 0
Expand Down Expand Up @@ -288,8 +288,8 @@ const report = (name, ok, detail) => { console.log(` ${ok ? "PASS" : "FAIL"} $
const url = `https://aadhar.sh/a/nav.${live}.js`;

const deltas = [];
for (const name of (await readdir("www/a-dict")).filter((n) => n.startsWith("nav.") && !n.includes(live))) {
const r = await get(url, b64(await readFile(`www/a-dict/${name}`)));
for (const name of (await readdir("src/dict/a-dict")).filter((n) => n.startsWith("nav.") && !n.includes(live))) {
const r = await get(url, b64(await readFile(`src/dict/a-dict/${name}`)));
const body = Buffer.from(await r.arrayBuffer());
// dcz is not an encoding undici knows, so these bytes arrive raw — which is the
// whole point here. Hash rather than compare lengths: two different deltas could
Expand Down
11 changes: 8 additions & 3 deletions tools/contract-tests.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3254,10 +3254,15 @@ test("LWE pages share one base stylesheet and the build derives one site-page di
const build = await readFile(new URL("tools/build.mjs", ROOT), "utf8");
assert.match(build, /site-page corpus/);
assert.match(build, /page-family\.\$\{hash8\(dictionary\)\}\.dict/);
assert.match(build, /www\/p-dict/);
assert.match(build, /src\/dict\/p-dict/);
assert.match(build, /site-page dictionary/);
const assetIgnore = await readFile(new URL("www/.assetsignore", ROOT), "utf8");
assert.match(assetIgnore, /^p-dict$/m, "page dictionary snapshots stay build input, not public assets");
// The snapshots live at src/dict/ now, OUTSIDE the served tree, which is a
// stronger statement than the .assetsignore entry this used to assert: an
// ignore line keeps a file from being uploaded, while being outside www/
// means the build never stages it in the first place.
const served = await readdir(new URL("www/", ROOT));
assert.ok(!served.includes("p-dict") && !served.includes("a-dict"),
"dictionary snapshots are build input and must not sit in the served tree");
const security = await readFile(new URL("src/worker/lib/security.js", ROOT), "utf8");
assert.match(security, /rel="compression-dictionary"/);
for (const name of ["index", "dac", "drivers", "encoding", "fhe", "knots", "mpc", "pcrypto", "tee", "utf8", "vigenere"]) {
Expand Down
12 changes: 6 additions & 6 deletions tools/roll-shell-dictionary.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ const only = process.argv.includes("--pages") ? "pages"
const live = process.argv.includes("--live");

const BUILT = ".build/www/a";
const DICTS = "www/a-dict";
const DICTS = "src/dict/a-dict";
// Each extra candidate costs one delta per asset per deploy and widens the cache-variant
// fan-out, while only serving visitors who skipped exactly that many deploys.
const KEEP = 3;
Expand Down Expand Up @@ -173,7 +173,7 @@ async function liveShell() {
return [...names].map(parse).filter(Boolean);
}

if (only === "pages") console.log("shell:roll — --pages given, leaving www/a-dict/ alone.");
if (only === "pages") console.log("shell:roll — --pages given, leaving src/dict/a-dict/ alone.");
else {
await mkdir(DICTS, { recursive: true });
const shell = live ? await liveShell() : (await readdir(BUILT)).map(parse).filter(Boolean);
Expand Down Expand Up @@ -213,10 +213,10 @@ for (const [, group] of byBase) {
}
}
console.log(`shell:roll — adopted ${adopted}, pruned ${pruned}, keeping ${KEEP} per asset.`);
console.log(" Commit www/a-dict/. build.mjs regenerates the deltas on its own from here.");
console.log(" Commit src/dict/a-dict/. build.mjs regenerates the deltas on its own from here.");
}

// ── PAGE DICTIONARIES (www/p-dict) ──────────────────────────────────────────
// ── PAGE DICTIONARIES (src/dict/p-dict) ──────────────────────────────────────────
// Static pages have two useful dictionary populations. The immutable family corpus
// reaches every visitor who has loaded any page; these committed snapshots recover
// the 93-97% per-page ratio for visitors returning to a page they already hold.
Expand Down Expand Up @@ -251,10 +251,10 @@ console.log(" Commit www/a-dict/. build.mjs regenerates the deltas on its own f
// required). And a staged page production does not serve yet is SKIPPED with a
// named line rather than adopted, because a browser cannot hold bytes that were
// never sent to it.
if (only === "shell") console.log("pages:roll — --shell given, leaving www/p-dict/ alone.");
if (only === "shell") console.log("pages:roll — --shell given, leaving src/dict/p-dict/ alone.");
else {
const BUILT_PAGES = ".build/www";
const PDICTS = "www/p-dict";
const PDICTS = "src/dict/p-dict";
const KEEP_PAGES = 2;
const FETCH_CONCURRENCY = 6;
const parse = (n) => {
Expand Down
11 changes: 5 additions & 6 deletions www/.assetsignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,11 @@ _worker.js
# at that path to ignore and a stale entry would silently exclude any future
# www/scripts from upload.

# Previous shipped versions of the content-hashed shell, kept ONLY as dictionary inputs
# for scripts/gen-shell-deltas.mjs. Uploading them would serve stale nav.js/luna.css at
# public URLs for no reason. www/ad/ (the .dcb deltas) is deliberately NOT ignored:
# the worker fetches those through ASSETS to answer an Available-Dictionary request.
a-dict
p-dict
# The previous shipped versions of the content-hashed shell, kept only as
# dictionary inputs, USED TO SIT HERE and be ignored. They live at src/dict/
# now, outside the served tree, so there is nothing at this path to exclude.
# www/ad/ (the deltas themselves) is deliberately NOT ignored: the worker
# fetches those through ASSETS to answer an Available-Dictionary request.

# Hand-authored Markdown twins for the Worker-rendered prose pages (/bot,
# /whoareyou, /security), whose text lives in template literals no build can read.
Expand Down
Loading