feat(tools): derive:check, a derivation graph for committed artifacts by oddharsh · Pull Request #609 · oddharsh/site

oddharsh · 2026-08-26T14:55:58Z

Four incidents in this repo share one shape: a content-addressed input moved and
something derived from it did not. #394 re-encoded 316 thumbnails and re-baked no
histograms, so the bars described pixels nobody had been served for nine days. An
/a/ asset changing hash orphans every a-dict snapshot naming the old one. The
search index froze twice under two unrelated causes. Each bought one bespoke
tripwire, and each tripwire was written to catch the thing that had already
happened.

This is the general form. config/derivations.json records what each committed
derived artifact was made FROM as a content hash of its inputs, bun run derive:check re-hashes them, and adding an artifact is a data edit rather than a
new script. It is wired into validate.

What it found

Three dead generators, none of them found by looking for them.

generator	broken since	symptom
`gen-photo-semantics.ts`	2026-08-18	joined `ROOT + "www"` five times; 7 photos carried no retrieval terms
`tools/lib/lens-chips.ts`	2026-08-18	read `www/_worker.js/lens.js`, taking `lens-seed` and `lens-warm` with it
`gen-quadgram-table.ts`	never worked	writes to `tools/lwe/`, a directory that has never existed under any of this repo's three layouts

All three are gotcha 40, the tenth through twelfth instances, after nine had
already been swept for. The transferable part is in CLAUDE.md: a path
expression is not greppable and therefore never gets audited, so what finds one
is any check that has to resolve it for its own reasons. A derived artifact turns
out to be a witness for its generator.

semantics.json is regenerated here. Verified rather than assumed: all 158
pre-existing records are byte-identical and 7 stems are new, with no term dropped
and no vision tier to lose.

Design, and the parts that are load-bearing

It runs no generator and writes nothing without --lock. Both halves are
requirements. Several sessions work in this tree at once, so a check that rebuilt
artifacts in place would be writing over somebody else's uncommitted work, and
the families that actually go stale cannot be rebuilt from a checkout anyway
(images/meta/ is local, metadata.json reads the SOOC originals, the og cards
capture production). Hashing inputs works on all of them in about a second.

Two tiers. pinned is checkable here with no tool and no network.
unverifiable names which inputs are not in this repository and why, and never
counts as a pass. Same standing as the two tools in tools.json that report no
version.

Set mode, and why covers is the half that bites. An artifact keyed by stem
does not depend on pixel bytes, so pinning alt.json to them would report it
stale on every re-encode and teach somebody to re-record without looking. set
digests the key set instead. But a digest alone is too weak for an artifact built
one key at a time: a resumable generator that dies halfway leaves a file covering
most of the set, and --lock afterwards would call it fresh forever. covers
reads the output and asserts an entry per input, computed live rather than
recorded, which makes it the one verdict --lock cannot clear. It is what caught
semantics.json at 158 of 165.

The census answers the question the graph structurally could not: whether an
artifact has a declaration at all. That is the gap gotcha 41 actually fell
through, since histograms.json had no declaration and so there was nothing for
a check to be wrong about. 30 writers, 30 classified, each either named by a
regenerate command or exempt with a reason.

It uses two scanners, because shell has no exact answer to "does this write":
a script writes through redirects, cp, sed -i, and through the encoders it
drives. JS, TS and Python get an exact scan; shell gets a deliberately wider one
(does it mention a committed path at all), which over-approximates and therefore
fails closed. The shell tier was added after the exact scan shipped with shell
listed as a known hole, and the hole was real: add-car-photo.sh, both
encoding-grid scripts, and reencode-thumbnails.sh all write committed bytes and
none was classified. That last one is the exact script whose standalone use
caused gotcha 41, and it now names derive:check in its closing advice, with a
test that fails if the reference goes away.

36 of 36 classified (30 exact + 6 shell).

Evidence

The baseline is verified, not assumed. Recording a digest nobody had checked
would make this decoration on day one. So the library was re-baked from the
committed JPEG tiers with zenc histogram and packed through the exported
packHistogram: 165 of 165 stems byte-identical, 42,240 bins, zero drift.

Controls, because a check that has never gone red is decoration.

control	result
perturb one JPEG in place	`changed (1): public/i/L1000069_3.b60c4974.jpg`
rename to a new hash8 (the real re-encode shape)	`added (1)` + `removed (1)`, both named
lock a coverage gap deliberately	still stale, `NO ENTRY for (7)`
add a throwaway generator	`UNDECLARED`, exit 1; removing it goes green
add a throwaway shell generator	same, via the shell tier

The census also proved itself on a real change rather than a synthetic one:
relock-dependabot.ts landed on main mid-review and the rebase caught it
unclassified.

Incremental build is deliberately not here

It was the other planned follow-up. Measured first:

full build 2.0 to 2.15s over 1606 files
dev serves from the symlink farm, not the build
output byte-identical across back-to-back runs, with its one date derived
from the photo index rather than the wall clock

Skipping steps would save about two seconds while putting a now-verified
byte-identical property at risk against a content-addressed deploy, which is the
trade gotcha 28 already refused. The census was measured to be worth more and
cost less.

Extractability

tools/lib/derive.ts is node-only, zero dependencies, and knows nothing about
this site. The declaration is data. It lifts into any repository that commits
derived artifacts, which is most of them.

Checks

lint, typecheck (all 275 files held by a program, tools ratchet unmoved),
bun run test 436 pass, build, and derive:check all green.

cloudflare-workers-and-pages · 2026-08-26T14:57:27Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	aadhar-sh	`b658782`	Aug 26 2026, 05:09 PM

github-actions · 2026-08-26T14:57:59Z

Wire-size diff vs merge base

72df5735 → 5abf8c21

Brotli q11, the quality the edge serves. Every number here is deterministic:
an unchanged file produces no row. Advisory — this check fails on nothing.

Client assets

No change. 23 files, 83.33 KiB Brotli total.

Pages

No change. 54 files, 571.17 KiB Brotli total.

Worker bundle

No change. 241.33 KiB gzip.

Dictionary deltas (`pd/`)

No change. 158 deltas, 511.05 KiB total.

Four incidents here share one shape: a content-addressed input moved and something derived from it did not. #394 re-encoded 316 thumbnails and re-baked no histograms, so the bars described pixels nobody had been served for nine days. An /a/ asset changing hash orphans every a-dict snapshot naming the old one. The search index froze twice under two unrelated causes. Each bought one bespoke check, and each was written to catch the thing that had already happened. config/derivations.json records what each committed derived artifact was made FROM as a content hash of its inputs, and `bun run derive:check` re-hashes them. Adding an artifact is a data edit rather than a new script. It runs no generator and writes nothing without --lock. Both halves are requirements: several sessions work in this tree at once, so a check that rebuilt artifacts in place would be writing over somebody else's uncommitted work, and the three families that actually go stale cannot be rebuilt from a checkout anyway (images/meta/ is a local artifact, metadata.json reads the SOOC originals, the og cards capture production). Hashing inputs works on all of them for the cost of reading 15 MB, in about a second. Two tiers, because the alternative to an honest note is a summary line that covers more than the check does. `pinned` is checkable here with no tool and no network. `unverifiable` says which inputs are not in this repository and why, and never counts as a pass. Same standing as the two tools in tools.json that report no version. THE BASELINE IS VERIFIED, NOT ASSUMED. Recording a digest nobody had checked would make this decoration on day one, so the library was re-baked from the committed JPEG tiers with `zenc histogram` and packed through the exported packHistogram: 165 of 165 stems byte-identical, 42,240 bins, zero drift. Both failure shapes were then run as controls. Perturbing one JPEG names it under `changed`; renaming one to a new hash8, which is the real re-encode shape, names it under both `added` and `removed`. Deliberately not covered, so the claim stays honest: alt.json and semantics.json depend on the SET of stems rather than the pixel bytes, and pinning them to bytes would invent staleness on every re-encode; the MCP cards already have a contract test that deep-equals them against each server's live tools/list, which is stronger than a digest. Both are written down in the declaration. The kernel is node-only with zero dependencies and knows nothing about this site, so it lifts into any repository that commits derived artifacts.

…generator Two artifacts could not join the graph as it stood. alt.json and semantics.json are keyed by stem, and re-encoding a photo does not change what is in the picture, so pinning them to pixel bytes would report them stale on every encode and teach somebody to re-record without looking. `set` projects each input path to a key and digests the key SET instead, so they go stale for the one reason they can be: a photo arrived or left. The digest alone is still too weak for an artifact built one key at a time. A resumable generator that dies halfway leaves a file covering most of the set, and --lock afterwards would call it fresh forever. `covers` reads the output and asserts an entry per input. It is computed live rather than recorded, which makes it the one verdict --lock cannot clear, and there is a test for exactly that. IT FOUND A LIVE BUG ON THE DAY IT WAS WRITTEN. semantics.json covered 158 of 165 photos. The cause was not a missed run: gen-photo-semantics.ts joined ROOT + "www" five times, and www/ was deleted by the 2026-08-18 split, so the script had been unrunnable for a week and the seven photos added since carried no retrieval terms. tools/lib/lens-chips.ts had the same rot pointing at www/_worker.js/lens.js, taking lens-seed and lens-warm with it. Both repaired; lens-chips reads 7 chips again, over a floor of 5. Regenerating semantics is purely additive, verified rather than assumed: all 158 pre-existing records are byte-identical and 7 stems are new, with no term dropped and no vision tier to lose. The general lesson is in gotcha 40 alongside the nine sites it already lists. A derived artifact is a witness for its generator, so checking the OUTPUT found a broken producer that a sweep of path expressions had already walked past once: gen-photo-semantics.ts had been audited before, and that audit fixed ROOT while leaving www in every join beneath it.

…xist The graph answers "is this artifact stale". It could not answer the question underneath it, which is whether an artifact has a declaration at all, and that is the gap gotcha 41 actually fell through: histograms.json had no declaration, so there was nothing for any check to be wrong about. Every file under tools/ and pipelines/ that writes is now classified exactly once. It is either named by some derivation's regenerate command or exempt with a reason. 29 writers, 29 classified. A new generator is covered by EXISTING rather than by somebody remembering, which is the property worth copying from tools.json's declaration tier. Controls both ways: adding a throwaway generator fails the check by name and exits 1, and removing it goes green. It found two things immediately. og/cards was declared but invisible to the census, because its regenerate said `bun run og-cards` and named no script, so regenerate commands now name their file and a test asserts that file exists. And gen-quadgram-table.ts wrote to ../lwe/quadgrams.txt relative to tools/photos/, which is tools/lwe/, a directory that has never existed under any of this repo's three layouts; the artifact lives at public/lwe/quadgrams.txt. That is the twelfth instance of gotcha 40 and the third in one afternoon, none of them found by looking for them. The scanner reads JS, TS and Python and NOT shell, because a shell script writes with redirects, cp, mv, tee and sed -i, and a loose matcher would spend its life on false positives while still missing half of them. The limit is written down at the top of derive-writers.ts rather than left implied. INCREMENTAL BUILD WAS THE OTHER FOLLOW-UP AND IS DELIBERATELY NOT HERE. The full build is 2.0-2.15s over 1606 files, dev serves from the symlink farm rather than the build, and the output is byte-identical across back-to-back runs with its one date derived from the photo index rather than the wall clock. So skipping steps would save about two seconds while putting a verified byte-identical property at risk against a content-addressed deploy, which is the trade gotcha 28 already refused. The census was measured to be worth more and cost less.

It landed on main while this branch was open and writes docs/DEPENDENCIES.md, so the census caught it on the rebase. That is the mechanism working on a real change rather than on the synthetic control: a generator arriving from somewhere else has to be classified before it can merge. Exempt rather than declared. It rewrites version pins inside prose that a human edits, and contract-docs-dependencies-md-states-version-pins-in-prose.test.mjs already holds those pins to the lockfile, which is the check a digest here would duplicate.

The census shipped reading JS, TS and Python with shell written down as a known limit. Taking the limit seriously found four shell scripts writing committed bytes that nothing classified: add-car-photo.sh (public/cars), the two encoding-grid scripts (public/garage/enc), and reencode-thumbnails.sh, which is the exact script whose standalone use caused gotcha 41. Shell gets a DELIBERATELY WIDER question than the exact scan, because shell has no exact answer: a script writes through redirects, cp, mv, tee, sed -i and heredocs, and also through the tools it drives (avifenc -o, zenc square, exif-sooc --merge-into). A loose matcher for those spends its life on false positives and still misses some. So the shell tier asks whether a script mentions a committed path at all. Over-approximating fails CLOSED: a script that only reads still has to be classified, and its exemption says it only reads. 36 of 36 classified now, up from 30, with two new unverifiable derivations (cars/reference, garage/encoding-samples) and three exemptions. Control: an undeclared shell script fails by name and exits 1. Two things worth noting about building it. The first regex excluded `/` before the path segment, which is exactly the character that precedes every real case (`../../public/cars`, `$PROJECT_DIR/public/images`), so it found 3 of 6 and the floor caught it. And a script handed its target as an ARGUMENT stays invisible, which is deliberate and is the same reason perf-snapshot.ts is exempt: a tool that writes where it is told has no fixed artifact to pin. reencode-thumbnails.sh now names derive:check in its closing advice and says why, with a test that fails if the reference goes away. gotcha 41's residual gap was that script telling you to re-run hash-thumbnails.sh and stopping there, so the script is the one place the remedy has to be legible.

CI failed with ERR_INVALID_PACKAGE_CONFIG naming the repo root, on contract-the-deploy-time-page-renderers.test.mjs, which touches none of that. The cause is a race that has been latent for months. contract-the-bun-pin-is-declared-once.test.mjs called writePin(root, "9.9.9") against the REAL package.json and restored it in a finally, which is safe in one process and is not what happens: `node --test` runs test FILES in parallel processes, and writeFileSync truncates before it writes. For the width of those two writes any other file's module resolution could read a half-written package.json, and Node answers that by refusing the package scope. It is a race, so it passed locally and failed in CI under load, and it failed on a file unrelated to the cause while the test responsible reported green. Nothing about the diff on this branch caused it; more test files in one parallel run is the likeliest reason it surfaced now. writePin already took its root as an argument, so the fix is to hand it a copy in a temp directory. The final assertion now says the live file was never modified, which is the property the race was about. A narrow guard comes with it: no test may hand the repo root to a write-shaped call. It is deliberately narrow, catching that exact shape rather than every way a test could reach outside its sandbox, because a heuristic that also fired on temp-dir writes would be muted within a week. Control: reintroducing writePin(root, ...) fails it by name, and reverting goes green. It has to strip comment lines first, since the note explaining it names the call it forbids, and a test that fails on its own explanation can only be fixed by deleting the explanation. The JSDoc-tag contract learned that the same way. It also corrects an exemption reason this branch got wrong. tools/lib/bun-pin.ts was exempted as writing "a temp release zip", which is bump-bun-pin.ts. It writes the root package.json. The census caught the right file and the reason beside it was wrong, which is the failure mode an exemption list has, and is why every entry has to state something checkable rather than something plausible.

…n it skipped (#662) add-photos.sh runs five of the six stem-keyed generators and never ran the sixth. hashes and fingerprints go through hash-thumbnails.sh (the second at line 150), histograms through extract-photo-metadata.sh line 274 plus the zenc bake, alt text through the gen-alt-text.py block. images/semantics had no caller at all, which is how seven photos drifted out of it for a week. #609 fixed the producer and not the hole. Its `covers` verdict found semantics.json at 158 of 165, traced it to gen-photo-semantics.ts joining a www/ the 2026-08-18 split deleted (gotcha 40), repaired the joins and re-ran it. Nothing then made the ingest call it, so the next photo add reopens the gap and lands on derive:check as a red PR with a command to go find. It goes after the captioner because the derived tier folds alt[stem] into `terms` (gen-photo-semantics.ts:206). Captioning second leaves a new stem with camera vocabulary and no subject, and that failure is quiet by construction: queryPhotos still finds the photo through the alt field at weight 5, one tier above expansion at 3, so the search result is worse rather than absent. Cheap and mandatory rather than tolerated. The derived tier reads metadata.json, hashes.json and alt.json, so unlike gen-alt-text.py it cannot be rate-limited and needs no credential; `set -e` stops the run instead of degrading it. Measured here: 95ms over 165 stems, and re-running the whole library is byte-identical, so the step adds no diff on an add that changes nothing. --vision stays opt-in and is not passed. Co-authored-by: Aadharsh Pannirselvam <19518661+oddharsh@users.noreply.github.com>

oddharsh force-pushed the build/derivation-graph branch from 7227d7b to bc3c134 Compare August 26, 2026 14:56

github-actions Bot assigned oddharsh Aug 26, 2026

oddharsh enabled auto-merge (squash) August 26, 2026 15:29

oddharsh disabled auto-merge August 26, 2026 15:30

oddharsh closed this Aug 26, 2026

oddharsh reopened this Aug 26, 2026

oddharsh added 5 commits August 26, 2026 11:49

oddharsh force-pushed the build/derivation-graph branch from 610bf18 to c6e5d5e Compare August 26, 2026 15:49

oddharsh enabled auto-merge (squash) August 26, 2026 17:10

oddharsh merged commit b7526da into main Aug 26, 2026
6 checks passed

oddharsh deleted the build/derivation-graph branch August 26, 2026 17:13

oddharsh mentioned this pull request Aug 29, 2026

fix(photos): the ingest regenerates semantics.json, the one derivation it skipped #662

Merged

Navigation Menu

Repository navigation

feat(tools): derive:check, a derivation graph for committed artifacts - #609

feat(tools): derive:check, a derivation graph for committed artifacts#609
oddharsh merged 6 commits into
mainfrom
build/derivation-graph

oddharsh commented Aug 26, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

oddharsh commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What it found

Design, and the parts that are load-bearing

Evidence

Incremental build is deliberately not here

Extractability

Checks

Uh oh!

cloudflare-workers-and-pages Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

github-actions Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Wire-size diff vs merge base

Client assets

Pages

Worker bundle

Dictionary deltas (pd/)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

oddharsh commented Aug 26, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Aug 26, 2026 •

edited

Loading

github-actions Bot commented Aug 26, 2026 •

edited

Loading

Dictionary deltas (`pd/`)