Repository navigation
fix(photos): re-bake the histograms against the thumbnails actually served - #535
Merged
Merged
Conversation
…erved #394 re-encoded 316 JPEG thumbnails on 2026-08-14, minting a new content hash for each, and re-baked no histograms. images/histograms.json has described thumbnails nobody is served ever since. All 158 photos move here: 31.2% of the 40,448 bins, mean 0.95 of 100, but 65 photos carry at least one bin past 15 and the worst is 59, so this is a visible bar rather than decoder noise. The decoder was the obvious suspect and is innocent. image 0.25.10 and zune-jpeg 0.5.15 are pinned identically at every commit involved, and Cargo.lock has not moved since the pipeline was relocated. What changed is the INPUT, which is the one thing a version pin cannot hold still: the j-tier hash for L1000069_3 went 0e092e5f -> 2b30aedc in #394, and the bars stayed. WHY NOTHING CAUGHT IT, which is the more useful half. photos:check compares histograms.json against images/meta/. Since #441 made meta/ build OUTPUT that build.mjs derives FROM histograms.json, those two agree no matter what the JPEGs contain. It is gotcha 24's shape: two values from one source, so the check proves that copying works. zenc histogram --check does read the pixels, and pointing it at a built tree is guaranteed to report drift whether or not any exists. The index packs each bin into ONE character over 64 levels while zenc computes 0..100, so a build-derived meta/ has been through a lossy 101 -> 64 -> 101 round trip. Measured: the derived tree reproduces roundTrip(committed) on 100.00% of 40,448 bins, so the packing is faithful and that drift was an artifact of where it was pointed. The real signal was underneath it and needed the pre-#441 committed meta to see. THE FIX AND THE GUARD. The data: materialise meta/, re-run zenc against the current j-tier, roll it back through build-histogram-index. The result satisfies zenc's own check ("histograms match on disk for 158 photos") and round-trips into the built meta/ at 256 of 256 bins for a spot-checked stem. The guard: hash-thumbnails.sh snapshots each stem's j hash and warns when one moves, naming the re-bake. That script is the choke point where new bytes become served bytes. add-photos.sh re-bakes the whole library every run and was never at risk; the gap is a standalone reencode-thumbnails.sh, whose own header tells you to re-run hash-thumbnails.sh and stops there. Controlled both ways: a no-op run stays silent, and a stem whose JPEG bytes change prints the warning naming it. The control mutated real files and the tree was restored to 632 tracked / 632 on disk afterwards. 353 tests, typecheck, lint, build, photos:check and the 142-route oracle green.
Contributor
Wire-size diff vs merge base
Brotli q11, the quality the edge serves. Every number here is deterministic: Client assetsNo change. 22 files, 77.74 KiB Brotli total. PagesTotal Brotli: 531.29 KiB → 531.30 KiB (+0.01 KiB, +0.0%) across 50 files, 1 changed. No single file moved by 0.13 KiB or more. Plus 1 file under the 0.13 KiB floor, net +0.01 KiB — usually content-hash churn in Worker bundleNo change. 237.86 KiB gzip. Dictionary deltas (
|
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
aadhar-sh | af571b1 | Aug 24 2026, 01:41 AM |
oddharsh
added a commit
that referenced
this pull request
Sep 10, 2026
…r the next one (#785) CLAUDE.md has carried TWO entries numbered 41 since 2026-08-24: #535's histogram re-encode entry landed on 2026-08-23 and #538 gave the cf-garage TypeScript-config entry the same number the next day. Twenty-four references pointed at `gotcha 41` across twelve files. Three meant cf-garage and twenty-one meant the histograms, and no reader could tell which without reading both entries. RENUMBERED THE HISTOGRAMS ENTRY TO 46 rather than shifting 42-45 up by one. The shift would have rewritten every cross-reference in a 6000-line file to fix a collision between two of them, over a file whose own layout note warns that a path rewrite cannot be run blind. So the number moved and the position did not, and the list now reads 40, 41, 42, 43, 44, 45, 46-where-41-was. It was already out of order: 22 has sat above 21 since 2026-08-06. Each reference was resolved from its own sentence, never by pattern. The reference surface was wider than a grep of CLAUDE.md suggests: eighteen of the twenty-four live in ci.yml, derivations.json, infra.json, a contract test, two derive modules and five photo-pipeline files, and two of those were buried mid-line where a fixed-width context grep missed them on the first pass. ONE JUDGMENT CALL, flagged rather than buried. infra.json's token-probe note ("Gotcha 41's lesson through another door") fits both entries: the discriminator reading suits the histograms entry's self-agreeing check, and the misread-refusal reading suits cf-garage's "which door it comes through". Git dates settle nothing, since both entries predate it. It reads as cf-garage, on the refusal shape and the echoed word, and the reference now NAMES cf-garage so the next reader inherits a decision instead of the ambiguity. contract-gotcha-numbers-resolve asserts every number names one entry and every reference repo-wide resolves to one. UNIQUENESS ONLY, never order: this entry and the 21/22 pair both break order, and asserting it would invite renumbering the file to satisfy a test. Both assertions carry a floor, on the a-dict precedent, since a scanner that has stopped matching reports a clean pass over an empty list. Three controls, each failing by name: reintroduce the duplicate, reference a gotcha that does not exist, move the section anchor. The heading scan is anchored at the section header deliberately. CLAUDE.md opens with several ordinary numbered lists at column 0, and a whole-document scan reads those as gotchas 1, 2 and 3 and reports duplicates that are not there. No served bytes move: a build here and a build from a detached worktree at HEAD produce all 1829 files byte-for-byte identical. Co-authored-by: Aadharsh Pannirselvam <19518661+oddharsh@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last of the three judgment calls from #525. I flagged
zenc histogram --checkreporting drift on all 158 photos whilephotos:checkpassed, guessed a decoder bump, and deliberately did not act. The guess was wrong and the real answer is worse.The bars describe thumbnails nobody is served
#394 re-encoded 316 JPEG thumbnails on 2026-08-14, minting a new content hash for each, and re-baked zero histograms.
images/histograms.jsonhas been stale ever since. The j-tier hash forL1000069_3went0e092e5f→2b30aedcin that commit and the bars stayed put.All 158 photos move here: 31.2% of 40,448 bins, mean 0.95 of 100, but 65 photos carry at least one bin past 15 and the worst is 59. That is a visible bar, not decoder noise.
The decoder was the obvious suspect and is innocent.
image0.25.10 andzune-jpeg0.5.15 are pinned identically at every commit involved, andCargo.lockhas not moved since the pipeline was relocated. What changed is the input, which is the one thing a version pin cannot hold still.Why nothing caught it, which is the more useful half
photos:checkcannot see this and never could. It compareshistograms.jsonagainstimages/meta/, and since #441 mademeta/build output thatbuild.mjsderives fromhistograms.json, the two agree no matter what the JPEGs contain. Gotcha 24's shape: two values from one source, so the check proves that copying works.zenc histogram --checkdoes read the pixels, and pointing it at a built tree is guaranteed to report drift whether or not any exists. The index packs each bin into one character over 64 levels (HIST_BASE63,HIST_LEVELS64) while zenc computes 0..100, so a build-derivedmeta/has been through a lossy 101 → 64 → 101 round trip.Measured: the derived tree reproduces
roundTrip(committed)on 100.00% of 40,448 bins, so the packing is faithful and the drift I reported in #525 was an artifact of where I pointed the tool. The real signal was underneath it and only showed up against the pre-#441 committedmeta/, which is unquantised:meta/==roundTrip(pre-#441 committed)meta/== pre-#441 rawThe fix
Materialise
meta/, re-run zenc against the current j-tier, roll it back throughbuild-histogram-index. The result satisfies zenc's own check (histograms match on disk for 158 photos) and round-trips into the builtmeta/at 256 of 256 bins for a spot-checked stem. Onlyhistograms.jsonchanges.The guard
hash-thumbnails.shnow snapshots each stem'sjhash and warns when one moves, naming the re-bake. That script is the choke point where new bytes become served bytes.add-photos.shre-bakes the whole library on every run and was never at risk; the gap is a standalonereencode-thumbnails.sh, whose own header tells you to re-runhash-thumbnails.shand stops there.Controlled both ways: a no-op run stays silent, and a stem whose JPEG bytes change prints the warning naming it. That control mutated real files; the tree was restored to 632 tracked / 632 on disk afterwards and re-verified.
Generalises past histograms
Anything derived from a content-addressed file has to be regenerated when the hash moves, and a hash moving is exactly the event that leaves no diff in the derived file to notice. Written up as gotcha 41.
Verification
354 tests, typecheck, lint, build (0 warnings),
photos:checkand the 142-route oracle green, re-run after rebasing onto #530 and #531.