Reduce non-intersecting map archive downloads - #136
Open
FrogAi wants to merge 6 commits into
Open
Conversation
FrogAi
force-pushed
the
codex/optimize-map-downloads
branch
2 times, most recently
from
August 23, 2026 00:01
2c7e34c to
a35dbc1
Compare
FrogAi
force-pushed
the
codex/optimize-map-downloads
branch
from
August 23, 2026 00:29
a35dbc1 to
f538b5e
Compare
FrogAi
force-pushed
the
codex/optimize-map-downloads
branch
from
September 4, 2026 21:28
f538b5e to
8e3da52
Compare
FrogAi
added a commit
to FrogAi/mapd
that referenced
this pull request
Sep 4, 2026
Retain the original PR commits and the tested rewrite. The resulting file tree is identical to 8e3da52.
Replace the earlier implementation with the simplified version.
FrogAi
force-pushed
the
codex/optimize-map-downloads
branch
from
September 4, 2026 21:50
d8ffc33 to
46daaa2
Compare
This was referenced Sep 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Select map archives using conservative country/state outlines instead of requesting every cell in each rectangular menu box. Existing bounding-box menus and custom overrides still work; invalid optional rows fall back to the box.
Coverage stays within the existing menu extents and coarse source geometry. This does not promise complete administrative regions or worldwide road coverage. A 0.05-degree margin and explicit exceptions retain additional cells, but cannot bound every source error. Forks needing different regions should use the existing override mechanism.
The byte column uses server-reported Content-Length observations from September 4, 2026, 21:07:11–21:08:04 UTC. It is compressed archive size, not transfer speed, actual transferred payload, extracted size or protocol overhead. The dated observations and complete aggregation are embedded below; changing live archives cannot recreate that historical measurement.
Generation is an explicit command in the existing CLI using local GeoJSON and the existing geometry dependency. It runs separately from ordinary OSM tile generation and needs no planet dataset. Null menus, features and missing/null geometry return contextual input errors before output is written. Valid generation from the pinned inputs reproduces the committed menu exactly.
Runtime contract, generation/provenance, reproducible checks and dated size observations
download_rowsstores[latitude, first longitude, exclusive last longitude]intervals on the existing two-degree grid. For example[48,-124,-120]requests origins[48,-124]and[48,-122]. Sorted, disjoint, in-bounds rows select cells in row order. Missing/empty rows, nulls, malformed shape/type, null coordinates, off-grid/out-of-bounds/reversed/overlapping/unsorted rows fall back to the rounded box. The menu is resolved once per run for both totals and requests. Nested/repeated selections and last-occurrence detail counters retain their existing meaning.The generator combines matching country features and outer rings, including islands; holes conservatively retain extra archives. It matches
ISO_A2_EHfor countries andUS-*state codes, aliases the existing territory entries (includingus_state.GMto Guam), clones rings before clipping, pads each candidate by 0.05 degrees and preserves full boxes for Morocco/Somalia/Ukraine because of source territory assignments. Custom keys with no matching geometry retain box fallback.The small nullable-object preflight is necessary because orb0.1.3's feature decoder dereferences null geometry before returning. It checks object presence, then leaves geometry/type/property decoding with the existing library. There is no panic recovery, alternate GIS framework, dependency upgrade or runtime input-policy change. Empty feature collections leave entries using box fallback; unsupported inputs preserve the destination.
Tested source and validation scope
7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee.30e8ce9, tree0806b59aa4489483442628425034407adcc37932.Checks described here used Go 1.25.1 on Linux amd64 with isolated fixtures and networking disabled during behavior tests. Existing repository tests remain unchanged; the extra reproductions below are deliberately outside the committed source. These are local execution results, not physical-device validation. The PR's Checks tab provides the published workflow result.
Pinned valid-input regeneration
Natural Earth countries/states are pinned to
ca96624a56bd078437bca8184e78163e5039ad19. Downloadne_10m_admin_0_countries.geojsonandne_10m_admin_1_states_provinces.geojsonfrom that directory ascountries.geojsonandstates.geojson. Expected SHA256 values are239eec57ac17f100a11e2536cffc56752c318b50ae765b0918ff7aab4ce8f255and22d0e3ad85eb3e27f17cabf8ba2d50e554fbc27a87796ff891d958185da62fb5.Run from a scratch checkout at the PR head listed above with Go1.25.1 and cached dependencies. Use an isolated container with an empty temporary
/data:mainattempts settings loading before CLI dispatch. Do not invoke the daemon. The command is:CLI generation at the PR head matched
17d567562216db94dd662634b2b89ee2dd2acac6c9f1dabf6292427882feb964exactly. Independent intersection checks of this menu used the same pinned Natural Earth inputs. Separate Canada/Yukon checks provide bounded corroboration; agreement with those inputs does not establish worldwide road coverage.Complete input and runtime reproductions
The null-menu, null-feature and null-geometry rejection tests pass. The complete fixtures below also exercise malformed/wrong-shaped inputs, missing geometry, both source files, destination preservation, optional-row validation/fallback, exact URL order/counts, nested/repeated selections and menu replacement during a run. HTTP is intercepted with controlled responses; no production archive transfer occurs. Use Linux with no live
/dataor custom menu paths, as enforced by the fixtures.Use an exact checkout with Linux Go 1.25.1 and its pinned
go.moddependencies already cached. To populate a separate module cache, create a temporary directory, setGOMODCACHEto amodulessubdirectory there, and rungo mod downloadfrom that checkout. The commands below then run offline, archive source into another temporary directory, and write no tests into the source checkout. They invoke tests, not the daemon.Recoverable historical size calculation
The payload below is a zlib/base64-encoded CSV of all 2,250 unique observed
(latitude,longitude,Content-Length)rows, 29,547 decoded bytes. Every captured response was HTTP200 with positive size. The capture used eight workers, a 30-second timeout and no retries. Public URL template:https://map-data.pfeifer.dev/offline/LAT/LON.tar.gz. Per-response etags/timestamps remain outside this compact data projection; all rows were captured within the interval stated above. The CSV hash checks the embedded observation record; it does not certify present server contents. Save and run this complete Python3 snippet from a clone containing the upstream and PR-head commits pinned below. It reads Git objects and embedded data only, makes no network requests, and reproduces the table's counts/byte sums.#107 is now merged upstream. This branch incorporates upstream main at
f374b2b6020f4a4c85aeec01323a9387d37417e0and retains selected-row iteration withpublishProgress()inside the loop, preserving between-archive cancellation and extraction.Conflict-resolution checks on September 7, 2026 used Go 1.25.1 on Linux amd64 with isolated synthetic downloads: repository tests, selected-row and lifecycle regression fixtures, generator input cases, and the settings race check passed. The generator and committed region data are unchanged by this merge; new test fixtures remain outside the PR.
Combined validation: exact source tree
a4c306906627db3ac7a8ab768651c8628d55465acombines #1019d61f06a1288ec4ea6f74f7d56a3057444316a13, #10320e7c25b054b6399360676a7f539a39b4fbf855c, #105bfcfe77be066634e36054327b20cfa6541063b54, #1078e5e677d1196838069e9665d4e9d962bcc1e116b, #1166fd5bbd6cf617c24a7fefd5e302fd36688a1a63b, #13630e8ce98ea7a4c8401dbb5bfc62120c84fc689e4. The only overlapping file issettings/download.go; the resolution retains #136's selected-row loop and #107's progress publication inside it.Combined Linux amd64 tests (including the scratch regression fixtures), race checks, vet and build passed. Under ARM64 emulation, the existing Makefile build stage (
make GO_CAPNP_PATH=/usr/local/go-capnp/std), committed repository tests, vet and both CLI help commands passed with Go1.25.1;go.mod/go.sumstayed unchanged and the resulting executable is AArch64. The ARM64 run does not include the extra amd64 scratch tests. It used an isolated retained build image, not a new dependency-install/image rebuild or physical device. No archive payload or live params were accessed. #105 still requires runtime-first rollout before regenerated tiles are distributed.