Port the citymap builder to the T3 server

Part of #1

Question

Translate .repos/mindwalk/internal/citymap/builder.go (520 lines of Go) into Effect-flavored TS on apps/server, and expose it over HTTP.

What it does: git ls-files -co --exclude-standard -z for the file list (builder.go:91), stat + line-count every non-binary file, build a directory tree, then lay out a squarified treemap into a 120×120 world with weight sqrt(max(lines, bytes/4096, 16)) (builder.go:85). Output is a flat array of files with baked {x, z, w, d} rects plus directory plates. Trace-independent by design — the layout depends only on the repo, so the same file sits in the same place every session and spatial memory forms. Do not make it depend on the trace.

Faithful port. No exclusion globs, no float rounding, no aggregation, even though all three were measured as worthwhile during charting. They are deferred as post-port fog.

Delivery constraints:

HTTP, not the websocket — apps/server/src/httpCompression/HttpResponseCompression.ts already provides gzip. Measured payload for this repo is 744 KB gzipped, which is fine over HTTP and not fine over the wire budget AGENTS.md cares about.
Cache server-side keyed on git commit + dirty state, like mindwalk's fingerprint cache (internal/server/server.go:59, fileFingerprint at :85). The cost here is the walk — reading 15,685 files to count lines — not the bytes.
Root is the thread's worktree (thread.worktreePath) when it has one, falling back to the project root. Threads on branches should map the tree they actually ran against.

Note the existing FilesystemBrowseInput contract is one-level browse only, so this needs its own walk and its own contract.

Resolution

Ported and landed on branch feat/citymap-builder-server (commit 747a7810a).

Verified against the real Go binary. mindwalk build was run from ~/.local/bin/mindwalk and diffed against the TS output on the same tree:

Mixed fixture (62 files: nested dirs, PNG blobs, a NUL-byte .dat, a dotfile, a file with no trailing newline, an empty file) — identical.
This repo — 15,796 files / 1,714 dirs, zero diffs in files, dirs, and repo metadata. Payload 4.96 MB raw, 752 KB gzipped, matching the 744 KB charted. Cold build ≈1s; Go is 0.58s.

That differential run is the real proof the port is faithful, and it is repeatable — the harness is a ~35-line throwaway test, deleted after use.

Shape

Split in two, because the layout is the part that mistranslates invisibly and the filesystem walk is the part that has to become T3-shaped:

apps/server/src/citymap/CitymapLayout.ts — pure, IO-free port of the treemap: buildTree, computeWeight, squarify, layoutRow, capAspect, inset, fileWeight, langForPath. Near-verbatim, including Go quirks worth preserving (filepath.Ext treating .gitignore as all-extension; a root ghost's dir staying "." where a real file's flattens to "").
apps/server/src/citymap/CitymapBuilder.ts — the walk, git state, and cache.
apps/server/src/citymap/http.ts + a citymap group on EnvironmentHttpApi.
packages/contracts/src/citymap.ts — wire shape kept byte-identical to mindwalk's JSON (commit and lang as optionalKey), so the ported types.ts reads it unchanged.

Decisions made while porting

File listing goes through VcsDriver.listWorkspaceFiles, which is already exactly git ls-files --cached --others --exclude-standard -z with a 16 MB cap. mindwalk's non-git fallback walk is ported too (same skip list: .git, node_modules, .venv, dist, build), because a T3 project need not be a git repo. Effect's FileSystem has no lstat, so the fallback is depth-bounded at 64 to make symlink cycles impossible.
Line counting is one pass, not two. mindwalk opens each file twice — once to sniff 8 KB for a NUL, once to count newlines. Same result from a single chunked read; halves the syscalls on a 15k-file walk.
Cache key is commit:dirty, plus a 30s TTL when dirty. Worth flagging: commit + dirty alone is not sufficient, because "dirty" does not change when a dirty file's contents change. mindwalk's repo-map path has the same hole and papers over it with repoMapTTL = 30s; I kept that, bounded to 16 roots. A precise dirty key would need an mtime fingerprint over the tree — noted as fog, not built.
Ghosts kept as a ghostPaths: string[] parameter, not a Trace. The builder stays trace-independent as the ticket requires — it takes paths, never a trace — but mindwalk's ghost behavior is not deleted, so Project T3's activity stream onto mindwalk's Trace model can wire it without reopening the builder. Tested.
Root resolution lives in the HTTP handler, via ProjectionSnapshotQuery.getThreadCheckpointContext → worktreePath ?? workspaceRoot, the same expression CheckpointDiffQuery already uses. The builder itself only ever knows about a root.
gzip needed no new code. httpCompressionLayer is already global middleware over any application/json response ≥1024 bytes.

Tests

CitymapLayout.test.ts (10) — determinism, path/id ordering, directory plates, aspect ratios, world bounds, byte-vs-line weighting, ghosts, langForPath including the dotfile case.
CitymapBuilder.test.ts (6) — tracked + untracked, gitignore exclusion, dirty reporting, NUL-byte binaries, cache identity, non-repo fallback. Real git repos in temp dirs.
server.test.ts (3 new) — 200 with content-encoding: gzip, worktree-wins root resolution, 404 unknown thread, 401 unauthenticated.

138 server tests and 221 contracts tests pass; typecheck and lint clean at baseline.

Not done, deliberately

.repos/ exclusion globs, float rounding, aggregation — all measured as worthwhile, all post-port fog. Note the raw payload came out 32% smaller than Go's anyway (4.96 MB vs 6.94 MB) purely because Go's build command pretty-prints; compacted, the two are the same size.

Ported, differentially verified against the Go binary, and landed on feat/citymap-builder-server.

Review follow-ups

Reviewed after the fact; three changes on top of 747a7810a.

`ed39d7d29` — concurrent builds deduped

mindwalk holds repoMapMu across its entire build (internal/server/server.go:366, defer before Build()), so a second caller waits instead of walking again. The port had a bare Map with no such guard: two concurrent requests for one root each walked ~15.7k files and read every one, at 32 concurrent reads apiece — plausible with web and mobile both opening the surface.

Now deduped per root via an in-flight Deferred, claimed synchronously (makeUnsafe + set, no yield between lookup and claim) and completed on exit so a joiner cannot outlive its build. Per root rather than globally, because mindwalk serves one repo while this server hosts many, and one project's walk must not block another's.

Covered by collapses concurrent builds of one root into a single walk, verified to fail without the fix — three concurrent builds returned structurally equal but distinct objects, exactly the duplicate-walk signature.

`ed39d7d29` — output pinned to the Go binary

The byte-identical claim was a one-time manual check; nothing stopped a later refactor from breaking it. Now a 13-file fixture covering trailing and missing newlines, an empty file, binaries by extension and by NUL sniff, an extensionless file, a dotfile, three-level nesting, and a bytes-dominant file, with mindwalk v0.3.0's output captured via GET /api/repomap as a golden.

The tree is a manifest, not loose files — vp fmt would otherwise rewrite the fixtures on commit and silently invalidate the golden. Verified to fail on drift: changing the weight floor from 16 to 17 breaks it.

`4d80f7d9d` — layout tests pruned, 10 to 5

The golden subsumed three of them, and a fourth asserted that a pure function returns equal output for equal input (CitymapLayout has no Date, Clock, or Math.random, so it could not fail). World-bounds folded into the 80-file aspect test. Kept the narrow tests the golden does not reach — the bytes-over-lines weight branch, ghosts, missing HEAD — since a golden diff reports "13 files differ" rather than naming the rule.

Not changed

CACHE_MAX_ENTRIES = 16 looked like a constant copied from Go without accounting for JS object overhead. Measured instead of assumed: ~3.5 MB per citymap of this repo, so a full cache is ~57 MB worst case, against the largest tree we expect. Left at mindwalk's value, with the measurement recorded in the comment so it reads as justified rather than inherited.

All 140 citymap and server tests pass; typecheck clean apart from the pre-existing HostPowerMonitor.ts error on main.

fix(server): address Macroscope review on the citymap builder

fix(server): fail the citymap build on a truncated workspace listing

fix(server): never read a non-regular file during citymap inspection

fix(server): drop the manufactured cause on a truncated listing

feat(server): port mindwalk's citymap builder to the T3 server

fix(server): dedupe concurrent citymap builds and pin output to mindwalk

test(server): prune citymap layout tests the golden fixture subsumes

fix(server): address Macroscope review on the citymap builder

fix(server): fail the citymap build on a truncated workspace listing

fix(server): never read a non-regular file during citymap inspection

fix(server): drop the manufactured cause on a truncated listing

Navigation Menu

Repository navigation

Port the citymap builder to the T3 server #6

Question

Resolution

Shape

Decisions made while porting

Tests

Not done, deliberately

Review follow-ups

`ed39d7d29` — concurrent builds deduped

`ed39d7d29` — output pinned to the Go binary

`4d80f7d9d` — layout tests pruned, 10 to 5

Not changed

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Port the citymap builder to the T3 server #6

Description

Question

Activity

Dillpickleschmidt commented on Aug 1, 2026

Resolution

Shape

Decisions made while porting

Tests

Not done, deliberately

Dillpickleschmidt commented on Aug 1, 2026

Dillpickleschmidt commented on Aug 1, 2026

Review follow-ups

ed39d7d29 — concurrent builds deduped

ed39d7d29 — output pinned to the Go binary

4d80f7d9d — layout tests pruned, 10 to 5

Not changed

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`ed39d7d29` — concurrent builds deduped

`ed39d7d29` — output pinned to the Go binary

`4d80f7d9d` — layout tests pruned, 10 to 5