Skip to content

Reduce non-intersecting map archive downloads - #136

Open
FrogAi wants to merge 6 commits into
pfeiferj:mainfrom
FrogAi:codex/optimize-map-downloads
Open

Reduce non-intersecting map archive downloads#136
FrogAi wants to merge 6 commits into
pfeiferj:mainfrom
FrogAi:codex/optimize-map-downloads

Conversation

@FrogAi

@FrogAi FrogAi commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Select map archives using conservative country/state outlines instead of requesting every cell in each rectangular menu box. Existing bounding-box menus and custom overrides still work; invalid optional rows fall back to the box.

Coverage stays within the existing menu extents and coarse source geometry. This does not promise complete administrative regions or worldwide road coverage. A 0.05-degree margin and explicit exceptions retain additional cells, but cannot bound every source error. Forks needing different regions should use the existing override mechanism.

Selection Box requests → selected Historical compressed bytes → selected
All 229 entries, each summed separately 17,238 → 8,841 Not measured
Canada 765 → 534 1,436,890,135 → 508,625,635
Alaska 1,980 → 145 2,416,185,887 → 11,670,601

The byte column uses server-reported Content-Length observations from September 4, 2026, 21:07:11–21:08:04 UTC. It is compressed archive size, not transfer speed, actual transferred payload, extracted size or protocol overhead. The dated observations and complete aggregation are embedded below; changing live archives cannot recreate that historical measurement.

Generation is an explicit command in the existing CLI using local GeoJSON and the existing geometry dependency. It runs separately from ordinary OSM tile generation and needs no planet dataset. Null menus, features and missing/null geometry return contextual input errors before output is written. Valid generation from the pinned inputs reproduces the committed menu exactly.

Runtime contract, generation/provenance, reproducible checks and dated size observations

download_rows stores [latitude, first longitude, exclusive last longitude] intervals on the existing two-degree grid. For example [48,-124,-120] requests origins [48,-124] and [48,-122]. Sorted, disjoint, in-bounds rows select cells in row order. Missing/empty rows, nulls, malformed shape/type, null coordinates, off-grid/out-of-bounds/reversed/overlapping/unsorted rows fall back to the rounded box. The menu is resolved once per run for both totals and requests. Nested/repeated selections and last-occurrence detail counters retain their existing meaning.

The generator combines matching country features and outer rings, including islands; holes conservatively retain extra archives. It matches ISO_A2_EH for countries and US-* state codes, aliases the existing territory entries (including us_state.GM to Guam), clones rings before clipping, pads each candidate by 0.05 degrees and preserves full boxes for Morocco/Somalia/Ukraine because of source territory assignments. Custom keys with no matching geometry retain box fallback.

The small nullable-object preflight is necessary because orb0.1.3's feature decoder dereferences null geometry before returning. It checks object presence, then leaves geometry/type/property decoding with the existing library. There is no panic recovery, alternate GIS framework, dependency upgrade or runtime input-policy change. Empty feature collections leave entries using box fallback; unsupported inputs preserve the destination.

Tested source and validation scope

  • Upstream comparison: 7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee.
  • PR head: 30e8ce9, tree 0806b59aa4489483442628425034407adcc37932.

Checks described here used Go 1.25.1 on Linux amd64 with isolated fixtures and networking disabled during behavior tests. Existing repository tests remain unchanged; the extra reproductions below are deliberately outside the committed source. These are local execution results, not physical-device validation. The PR's Checks tab provides the published workflow result.

Pinned valid-input regeneration

Natural Earth countries/states are pinned to ca96624a56bd078437bca8184e78163e5039ad19. Download ne_10m_admin_0_countries.geojson and ne_10m_admin_1_states_provinces.geojson from that directory as countries.geojson and states.geojson. Expected SHA256 values are 239eec57ac17f100a11e2536cffc56752c318b50ae765b0918ff7aab4ce8f255 and 22d0e3ad85eb3e27f17cabf8ba2d50e554fbc27a87796ff891d958185da62fb5.

Run from a scratch checkout at the PR head listed above with Go1.25.1 and cached dependencies. Use an isolated container with an empty temporary /data: main attempts settings loading before CLI dispatch. Do not invoke the daemon. The command is:

git show 7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee:settings/download_menu.json > base-menu.json
go build -buildvcs=false -o mapd .
./mapd generate-download-menu --menu base-menu.json --countries countries.geojson --states states.geojson --output generated-menu.json
cmp generated-menu.json settings/download_menu.json
sha256sum generated-menu.json

CLI generation at the PR head matched 17d567562216db94dd662634b2b89ee2dd2acac6c9f1dabf6292427882feb964 exactly. Independent intersection checks of this menu used the same pinned Natural Earth inputs. Separate Canada/Yukon checks provide bounded corroboration; agreement with those inputs does not establish worldwide road coverage.

Complete input and runtime reproductions

The null-menu, null-feature and null-geometry rejection tests pass. The complete fixtures below also exercise malformed/wrong-shaped inputs, missing geometry, both source files, destination preservation, optional-row validation/fallback, exact URL order/counts, nested/repeated selections and menu replacement during a run. HTTP is intercepted with controlled responses; no production archive transfer occurs. Use Linux with no live /data or custom menu paths, as enforced by the fixtures.

Use an exact checkout with Linux Go 1.25.1 and its pinned go.mod dependencies already cached. To populate a separate module cache, create a temporary directory, set GOMODCACHE to a modules subdirectory there, and run go mod download from that checkout. The commands below then run offline, archive source into another temporary directory, and write no tests into the source checkout. They invoke tests, not the daemon.

ref=30e8ce98ea7a4c8401dbb5bfc62120c84fc689e4 # Validated PR head.
repository=$(pwd)
scratch=$(mktemp -d)
module_cache=$(go env GOMODCACHE)
mkdir -p "$scratch/source" "$scratch/tmp" "$scratch/gocache" "$scratch/gopath"
git -c core.autocrlf=false archive "$ref" | tar -x -C "$scratch/source"
cat > "$scratch/source/settings/audit_download_selection_test.go" <<'GO'
package settings

import (
	"archive/tar"
	"bytes"
	"compress/gzip"
	"encoding/json"
	"fmt"
	"io"
	"log/slog"
	"net/http"
	"os"
	"reflect"
	"strings"
	"testing"
)

type selectionTestTransport func(*http.Request) (*http.Response, error)

func (transport selectionTestTransport) RoundTrip(request *http.Request) (*http.Response, error) {
	return transport(request)
}

type selectionTestResult struct {
	URLs          []string
	Progress      []DownloadProgress
	ResponseBytes int
}

func runDownloadSelection(t *testing.T, menu, paths, replacementMenu string) selectionTestResult {
	t.Helper()
	for _, path := range []string{"/data/media/0", "/data/openpilot/mapd_download_menu.json"} {
		if _, err := os.Stat(path); err == nil {
			t.Skip("test requires isolated download paths")
		}
	}
	t.Chdir(t.TempDir())
	previousMenu := boundingBoxesJson
	previousClient := http.DefaultClient
	previousLogger := slog.Default()
	boundingBoxesJson = []byte(menu)
	slog.SetDefault(slog.New(slog.NewTextHandler(io.Discard, nil)))
	t.Cleanup(func() {
		boundingBoxesJson = previousMenu
		http.DefaultClient = previousClient
		slog.SetDefault(previousLogger)
	})
	var archive bytes.Buffer
	compressed := gzip.NewWriter(&archive)
	writer := tar.NewWriter(compressed)
	if err := writer.Close(); err != nil {
		t.Fatal(err)
	}
	if err := compressed.Close(); err != nil {
		t.Fatal(err)
	}
	// All fixtures contain fewer than 400 archives, including the bounding-box baseline.
	// Keep progress buffered and copy it synchronously to avoid testing PR #107's separate lifecycle defects.
	progress := make(chan DownloadProgress, 512)
	result := selectionTestResult{}
	captureProgress := func() {
		for len(progress) > 0 {
			data, err := json.Marshal(<-progress)
			if err != nil {
				t.Fatal(err)
			}
			var snapshot DownloadProgress
			if err := json.Unmarshal(data, &snapshot); err != nil {
				t.Fatal(err)
			}
			result.Progress = append(result.Progress, snapshot)
		}
	}
	http.DefaultClient = &http.Client{Transport: selectionTestTransport(func(request *http.Request) (*http.Response, error) {
		captureProgress()
		result.URLs = append(result.URLs, request.URL.String())
		result.ResponseBytes += archive.Len()
		if replacementMenu != "" && len(result.URLs) == 1 {
			boundingBoxesJson = []byte(replacementMenu)
		}
		return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(bytes.NewReader(archive.Bytes())), Header: make(http.Header)}, nil
	})}
	Download(paths, progress, nil)
	captureProgress()
	return result
}

func assertDownloadSelection(t *testing.T, result selectionTestResult, archivePaths []string, perLocation map[string]int) {
	t.Helper()
	expectedURLs := make([]string, len(archivePaths))
	for index, path := range archivePaths {
		expectedURLs[index] = "https://map-data.pfeifer.dev/offline/" + path + ".tar.gz"
	}
	if !reflect.DeepEqual(result.URLs, expectedURLs) {
		t.Errorf("requested %d archives, want %d; URLs = %v, want %v", len(result.URLs), len(expectedURLs), result.URLs, expectedURLs)
	}
	if len(result.Progress) == 0 {
		t.Fatal("download produced no progress")
	}
	terminal := result.Progress[len(result.Progress)-1]
	if terminal.Active || terminal.Canceled || terminal.TotalFiles != len(archivePaths) || terminal.DownloadedFiles != len(archivePaths) {
		t.Errorf("completion totals do not match selection: %+v", terminal)
	}
	for _, snapshot := range result.Progress {
		if snapshot.TotalFiles != len(archivePaths) {
			t.Errorf("published total %d does not match selected request count %d", snapshot.TotalFiles, len(archivePaths))
			break
		}
	}
	for location, expected := range perLocation {
		detail := terminal.LocationDetails[location]
		if detail == nil || detail.TotalFiles != expected || detail.DownloadedFiles != expected {
			t.Errorf("location %q detail = %+v, want total/downloaded %d", location, detail, expected)
		}
	}
	t.Logf("HTTP requests=%d, response bytes=%d", len(result.URLs), result.ResponseBytes)
}

func TestDownloadSelectionSparseRows(t *testing.T) {
	tests := []struct {
		name   string
		bounds string
		rows   string
		paths  []string
	}{
		{
			name:   "negative coordinates and separated runs",
			bounds: `{"min_lat":-3.1,"min_lon":-5.2,"max_lat":2.1,"max_lon":4.1}`,
			rows:   `[[-4,-6,-4],[-2,0,4],[2,4,6]]`,
			paths:  []string{"-4/-6", "-2/0", "-2/2", "2/4"},
		},
		{
			name:   "exact grid boundaries",
			bounds: `{"min_lat":0,"min_lon":0,"max_lat":4,"max_lon":4}`,
			rows:   `[[0,0,2],[2,2,4]]`,
			paths:  []string{"0/0", "2/2"},
		},
		{
			name:   "both sides of dateline",
			bounds: `{"min_lat":0,"min_lon":-180,"max_lat":2,"max_lon":180}`,
			rows:   `[[0,-180,-178],[0,178,180]]`,
			paths:  []string{"0/-180", "0/178"},
		},
		{
			name:   "legacy fractional negative bounds",
			bounds: `{"min_lat":-1,"min_lon":-1,"max_lat":1,"max_lon":1}`,
			rows:   `[]`,
			paths:  []string{"-2/-2", "-2/0", "0/-2", "0/0"},
		},
	}
	for _, test := range tests {
		t.Run(test.name, func(t *testing.T) {
			menu := fmt.Sprintf(`{"test":{"place":{"bounding_box":%s,"download_rows":%s}}}`, test.bounds, test.rows)
			result := runDownloadSelection(t, menu, "test.place", "")
			assertDownloadSelection(t, result, test.paths, map[string]int{"test.place": len(test.paths)})
		})
	}
}

func TestDownloadSelectionBoundingBoxFallback(t *testing.T) {
	rows := map[string]string{
		"absent":               "",
		"null":                 `null`,
		"empty":                `[]`,
		"odd latitude":         `[[1,0,2]]`,
		"odd first longitude":  `[[0,1,2]]`,
		"odd last longitude":   `[[0,0,3]]`,
		"empty interval":       `[[0,2,2]]`,
		"reversed interval":    `[[0,4,2]]`,
		"latitude outside":     `[[4,0,2]]`,
		"first outside":        `[[0,-2,2]]`,
		"last outside":         `[[0,0,6]]`,
		"unsorted latitude":    `[[2,0,2],[0,0,2]]`,
		"unsorted longitude":   `[[0,2,4],[0,0,2]]`,
		"overlap":              `[[0,0,4],[0,2,4]]`,
		"partly invalid":       `[[0,0,2],[2,0,6]]`,
		"too few columns":      `[[0,0]]`,
		"too many columns":     `[[0,0,2,4]]`,
		"null row":             `[null]`,
		"null latitude":        `[[null,0,2]]`,
		"null first longitude": `[[0,null,2]]`,
		"null last longitude":  `[[0,0,null]]`,
		"null after valid row": `[[0,0,2],[2,null,2]]`,
		"nonnumeric column":    `[[0,"bad",2]]`,
	}
	for name, value := range rows {
		t.Run(name, func(t *testing.T) {
			metadata := ""
			if value != "" {
				metadata = `,"download_rows":` + value
			}
			menu := `{"test":{"place":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":4,"max_lon":4}` + metadata + `}}}`
			result := runDownloadSelection(t, menu, "test.place", "")
			assertDownloadSelection(t, result, []string{"0/0", "0/2", "2/0", "2/2"}, map[string]int{"test.place": 4})
		})
	}
}

func TestDownloadSelectionNestedAndRepeatedPaths(t *testing.T) {
	menu := `{
		"test":{"group":{"submenu":"regions"},"other":{"bounding_box":{"min_lat":8,"min_lon":6,"max_lat":10,"max_lon":8}}},
		"regions":{"islands":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":4,"max_lon":4},"download_rows":[[2,2,4]]}}
	}`
	paths := "test.group.islands,test.other,test.group.islands"
	result := runDownloadSelection(t, menu, paths, "")
	assertDownloadSelection(t, result, []string{"2/2", "8/6", "2/2"}, map[string]int{"test.group.islands": 1, "test.other": 1})
	terminal := result.Progress[len(result.Progress)-1]
	if !reflect.DeepEqual(terminal.LocationsToDownload, strings.Split(paths, ",")) {
		t.Errorf("selection paths changed: %v", terminal.LocationsToDownload)
	}
}

func TestDownloadSelectionRepeatedBoundingBoxesPreserveDetails(t *testing.T) {
	menu := `{"test":{"place":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":2,"max_lon":2}}}}`
	result := runDownloadSelection(t, menu, "test.place,test.place", "")
	assertDownloadSelection(t, result, []string{"0/0", "0/0"}, map[string]int{"test.place": 1})
}

func TestDownloadSelectionResolvesMenuOnce(t *testing.T) {
	menu := `{"test":{
		"first":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":2,"max_lon":2}},
		"second":{"bounding_box":{"min_lat":2,"min_lon":0,"max_lat":4,"max_lon":2}}
	}}`
	replacement := `{"test":{
		"first":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":2,"max_lon":2}},
		"second":{"bounding_box":{"min_lat":8,"min_lon":6,"max_lat":10,"max_lon":10}}
	}}`
	result := runDownloadSelection(t, menu, "test.first,test.second", replacement)
	assertDownloadSelection(t, result, []string{"0/0", "2/0"}, map[string]int{"test.first": 1, "test.second": 1})
}
GO
cat > "$scratch/source/settings/audit_generate_download_menu_test.go" <<'GO'
package settings

import (
	"encoding/json"
	"os"
	"path/filepath"
	"reflect"
	"testing"
)

func TestGeneratedDownloadMenuGeometry(t *testing.T) {
	directory := t.TempDir()
	menuPath := filepath.Join(directory, "menu.json")
	countryPath := filepath.Join(directory, "countries.json")
	statePath := filepath.Join(directory, "states.json")
	outputPath := filepath.Join(directory, "output.json")
	menu := DownloadMenu{"nation": {
		"AA": {FullName: "Two islands", BoundingBox: Bounds{MinLat: 0, MinLon: 0, MaxLat: 6, MaxLon: 6}},
		"AB": {FullName: "Split features", BoundingBox: Bounds{MinLat: 0, MinLon: 0, MaxLat: 6, MaxLon: 6}, Submenu: "custom"},
		"AC": {FullName: "Surrounding polygon with a hole", BoundingBox: Bounds{MinLat: 0, MinLon: 0, MaxLat: 6, MaxLon: 6}},
		"AD": {FullName: "Missing geometry", BoundingBox: Bounds{MinLat: 0, MinLon: 0, MaxLat: 6, MaxLon: 6}},
	}}
	encoded, _ := json.Marshal(menu)
	countries := `{"type":"FeatureCollection","features":[
		{"type":"Feature","properties":{"ISO_A2_EH":"AA"},"geometry":{"type":"MultiPolygon","coordinates":[[[[0.5,0.5],[1.5,0.5],[1.5,1.5],[0.5,1.5],[0.5,0.5]]],[[[4.5,4.5],[5.5,4.5],[5.5,5.5],[4.5,5.5],[4.5,4.5]]]]}},
		{"type":"Feature","properties":{"ISO_A2_EH":"AB"},"geometry":{"type":"Polygon","coordinates":[[[0.5,0.5],[1.5,0.5],[1.5,1.5],[0.5,1.5],[0.5,0.5]]]}},
		{"type":"Feature","properties":{"ISO_A2_EH":"AB"},"geometry":{"type":"Polygon","coordinates":[[[4.5,4.5],[5.5,4.5],[5.5,5.5],[4.5,5.5],[4.5,4.5]]]}},
		{"type":"Feature","properties":{"ISO_A2_EH":"AC"},"geometry":{"type":"Polygon","coordinates":[[[-1,-1],[7,-1],[7,7],[-1,7],[-1,-1]],[[1,1],[1,5],[5,5],[5,1],[1,1]]]}}
	]}`
	for filename, data := range map[string][]byte{menuPath: encoded, countryPath: []byte(countries), statePath: []byte(`{"type":"FeatureCollection","features":[]}`)} {
		if err := os.WriteFile(filename, data, 0o644); err != nil {
			t.Fatal(err)
		}
	}
	if err := GenerateDownloadMenu(menuPath, countryPath, statePath, outputPath); err != nil {
		t.Fatal(err)
	}
	output, err := os.ReadFile(outputPath)
	if err != nil {
		t.Fatal(err)
	}
	var generated DownloadMenu
	if err := json.Unmarshal(output, &generated); err != nil {
		t.Fatal(err)
	}
	for code, expected := range menu["nation"] {
		actual := generated["nation"][code]
		if actual.FullName != expected.FullName || actual.BoundingBox != expected.BoundingBox || actual.Submenu != expected.Submenu {
			t.Errorf("metadata changed for %s: %+v", code, actual)
		}
		if code == "AA" || code == "AB" {
			if !reflect.DeepEqual(actual.DownloadRows, DownloadRows{{0, 0, 2}, {4, 4, 6}}) {
				t.Errorf("island/duplicate-feature coverage for %s: %v", code, actual.DownloadRows)
			}
		} else if len(actual.DownloadRows) != 0 || actual.countFiles() != 9 {
			t.Errorf("full coverage should use bounding box for %s: %+v", code, actual)
		}
	}
	if err := GenerateDownloadMenu(outputPath, countryPath, statePath, outputPath); err != nil {
		t.Fatal(err)
	}
	repeated, _ := os.ReadFile(outputPath)
	if string(output) != string(repeated) {
		t.Fatal("regeneration changed its own output")
	}
	if err := os.WriteFile(countryPath, []byte(`not json`), 0o644); err != nil {
		t.Fatal(err)
	}
	if err := GenerateDownloadMenu(menuPath, countryPath, statePath, outputPath); err == nil {
		t.Fatal("malformed source accepted")
	}
	preserved, _ := os.ReadFile(outputPath)
	if string(preserved) != string(output) {
		t.Fatal("invalid source modified existing output")
	}
}
GO
cat > "$scratch/source/settings/audit_generator_inputs_test.go" <<'GO'
package settings

import (
  "os"
  "path/filepath"
  "testing"
)

func TestAuditGeneratorRejectsInvalidInputs(t *testing.T) {
  cases := []struct {name, menu, countries string}{
    {"null menu", `null`, `{"type":"FeatureCollection","features":[]}`},
    {"null feature", `{"nation":{"AA":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":4,"max_lon":4}}}}`, `{"type":"FeatureCollection","features":[null]}`},
    {"null geometry", `{"nation":{"AA":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":4,"max_lon":4}}}}`, `{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"ISO_A2_EH":"AA"},"geometry":null}]}`},
  }
  for _, test := range cases {
    t.Run(test.name, func(t *testing.T) {
      directory := t.TempDir()
      menu, countries, states, output := filepath.Join(directory,"menu.json"), filepath.Join(directory,"countries.json"), filepath.Join(directory,"states.json"), filepath.Join(directory,"output.json")
      for name, content := range map[string]string{menu:test.menu,countries:test.countries,states:`{"type":"FeatureCollection","features":[]}`,output:"unchanged"} {
        if err := os.WriteFile(name,[]byte(content),0600); err != nil {t.Fatal(err)}
      }
      defer func() {
        if value := recover(); value != nil {t.Errorf("panicked instead of returning input error: %v",value)}
        data, err := os.ReadFile(output)
        if err != nil || string(data)!="unchanged" {t.Errorf("invalid input changed output: %q error=%v",data,err)}
      }()
      if err := GenerateDownloadMenu(menu,countries,states,output); err == nil {t.Error("invalid input accepted")}
    })
  }
}
GO
cat > "$scratch/source/settings/generator_boundary_test.go" <<'GO'
package settings

import (
  "os"
  "path/filepath"
  "strings"
  "testing"
)

func TestGeneratorInputBoundary(t *testing.T) {
  empty := `{"type":"FeatureCollection","features":[]}`
  goodMenu := `{"nation":{"AA":{"bounding_box":{"min_lat":0,"min_lon":0,"max_lat":4,"max_lon":4}}}}`
  cases := []struct {name, menu, source, context string; states bool}{
    {"null menu", `null`, empty, "menu.json", false},
    {"wrong menu", `[]`, empty, "", false},
    {"malformed menu", `{`, empty, "", false},
    {"null collection", goodMenu, `null`, "source.json", false},
    {"wrong collection", goodMenu, `[]`, "source.json", false},
    {"wrong collection type", goodMenu, `{"type":"Feature","features":[]}`, "source.json", false},
    {"malformed collection", goodMenu, `{`, "source.json", false},
    {"null feature", goodMenu, `{"type":"FeatureCollection","features":[null]}`, "feature 0", false},
    {"missing geometry", goodMenu, `{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"ISO_A2_EH":"AA"}}]}`, "feature 0", false},
    {"null geometry", goodMenu, `{"type":"FeatureCollection","features":[{"type":"Feature","geometry":null}]}`, "feature 0", false},
    {"wrong feature", goodMenu, `{"type":"FeatureCollection","features":[1]}`, "source.json", false},
    {"wrong geometry", goodMenu, `{"type":"FeatureCollection","features":[{"type":"Feature","geometry":[]}]}`, "source.json", false},
    {"bad feature type", goodMenu, `{"type":"FeatureCollection","features":[{"type":"Bad","geometry":{"type":"Polygon","coordinates":[]}}]}`, "source.json", false},
    {"unsupported geometry", goodMenu, `{"type":"FeatureCollection","features":[{"type":"Feature","geometry":{"type":"Point","coordinates":[0,0]}}]}`, "source.json", false},
    {"null state feature", goodMenu, `{"type":"FeatureCollection","features":[null]}`, "feature 0", true},
    {"null state geometry", goodMenu, `{"type":"FeatureCollection","features":[{"type":"Feature","geometry":null}]}`, "feature 0", true},
  }
  for _, test := range cases {
    t.Run(test.name, func(t *testing.T) {
      directory := t.TempDir()
      menu, source, unused, output := filepath.Join(directory,"menu.json"), filepath.Join(directory,"source.json"), filepath.Join(directory,"empty.json"), filepath.Join(directory,"output.json")
      for filename, content := range map[string]string{menu:test.menu, source:test.source, unused:empty, output:"unchanged"} {
        if err := os.WriteFile(filename, []byte(content), 0600); err != nil { t.Fatal(err) }
      }
      countries, states := source, unused
      if test.states { countries, states = unused, source }
      err := GenerateDownloadMenu(menu, countries, states, output)
      if err == nil || !strings.Contains(err.Error(), test.context) { t.Fatalf("expected contextual input error, got %v", err) }
      data, readError := os.ReadFile(output)
      if readError != nil || string(data) != "unchanged" { t.Fatalf("destination changed: %q, %v", data, readError) }
    })
  }
}
GO
cd "$scratch/source"
GOTOOLCHAIN=local GOPROXY=off GOSUMDB=off GOMODCACHE="$module_cache" \
  GOCACHE="$scratch/gocache" GOPATH="$scratch/gopath" TMPDIR="$scratch/tmp" \
  go test -mod=readonly -buildvcs=false -count=1 -timeout=120s -v ./settings

Recoverable historical size calculation

The payload below is a zlib/base64-encoded CSV of all 2,250 unique observed (latitude,longitude,Content-Length) rows, 29,547 decoded bytes. Every captured response was HTTP200 with positive size. The capture used eight workers, a 30-second timeout and no retries. Public URL template: https://map-data.pfeifer.dev/offline/LAT/LON.tar.gz. Per-response etags/timestamps remain outside this compact data projection; all rows were captured within the interval stated above. The CSV hash checks the embedded observation record; it does not certify present server contents. Save and run this complete Python3 snippet from a clone containing the upstream and PR-head commits pinned below. It reads Git objects and embedded data only, makes no network requests, and reproduces the table's counts/byte sums.

import base64, hashlib, json, math, subprocess, zlib
data = """eNpNXdmu7agOfO9vOVciDIb8/4/dmsjaarW0vJMwGruwC85s//73zP7v2Xv/NyW0f8/7RBjn33PeJ0JRWBEmhFMRUMDZb4TG15qF
zgL2FVDAu+vtIzLKWHO+s1JO7//WXu8Y9wUUVbt2T3Oew4bWulU9LLDPeWbk+W/2+dRIIx+0a50zxm3O0/71Gv1d6VI7/8Z5O96J
jALR3mfdJrf5b0y8UGlB6//GrvbO1NDav1XVnmX5RQPHnLO6W/jWv3dP9MCfv+jwU7XO4/pftK+Pg0Fw/S+a117Unvac86/vgxFa
HpCD5u2F4Rnu8Jn/ej99rePmHrTunXPvk/fV3bFWRnyjvLf222bk+oe+rtXKzd3zH95dT0v3N9r3nrnP0y23fxMTFqnQ2bf5y9JM
5O+Tf3cPi5qR6SkqxvBUL+pFerHYq9cjtCZ/49sevTxtRKBejmZBeknt69HLd1xBlfcI0ssrSLHzGvVSetitl30845kz8vz3Towq
29StlgOacM59HyX1hoFNYVBLaO2skedQy2prtPdExrBiFs+T8qGWNTgL6c1DNX/H6WkptBIqWWhD5MK0nnl6xoVKiXUyqITdStlb
h1qlPVDK/r67TbcfSrnxPqqxWFDZTTWzONn4NrsbS5UcKHqkMVDJ0aD/bwqHSj5zo7I8p0oeLqEUd7hgGubkuHKoJNr7npORPyxv
76Ul0aWST/UJs+DOQSWfud6+y4MFlRxPPVhx/h4queY4NTLHm3OBBddcPXSyY6iXX6ZW9jrreS3SZr0ul4r5pg9UTFgV/ZZiNrdF
itn9vhTzoJx5DebbI7So0oxivq0iVHRxfoqZAqiYL9VhXsW838hg3nqgmHtCtVM49LJjGsfZ9w9QpGro+4qM9VmzwX5YhmJy+Y83
hUMxJ3p0ODcz9nLUHr1F7ng+e92uPZwsWqi0h4oJ8/HOfA/F7FXPojmbUcyzMTn3fShmR/922ifFPHtNlw/FhKo8qyKit5zn4da8
nPkxYMA9TtDM88J6NXceiokF27CO3FkoJvSioQ6L9e+MA8Pnt4/cFQyjWw6thGHdUoAppXxpxZuLgk4eDsrxt1TJg0E43Q2RSmK9
z+lhhUq+DXq18rj9gwbuJ3MNjcTKhXr5KVQSDvXtrxsCldytsLzyMtpVbxstLzf6w7Jgk+kGLzXJo07N1N/ramadCLJ4y4I08xkR
tGrvkxm/Xp9m3tdaFLiuZr55Qs2EjaEfLWvmWFx48/4BkwXF+GqHZtYDk/akafTks7f9rpY/wFP2CauQAqiacCJ7p0KoZn/Hq9mv
2MwFkzciQzWhl09PfXTkc8IEz8hcOrM0hRU/Dt0ZBDoVzcRKmo+rp2bCbc28LpMJTDDcemgmfCK6F5Fed/TFJVy2mNDbnac0mC/6
nZGRC99YMXlKCITOLw8y9RKWWOak7L/3PO/2INFWvuORipf0EuP9vt31Qi1hrJrgWkkroXRrPXna2Mh2pjtcWsnQco9XcSHPsWZE
e/Bd/pZaCZs67rdcxXscS1TLdp7UutSm1iNh1GF8Btf7udpJnTlXOzlm5wOaV6B20macD2iOCP2v0GLOj7UTzUdDIsJlYg4GkeIJ
1HwLXrXt/IEKsLCO7/dQUNpCOskT09l3W7cy6CeQz3lv66GfRJKz8j2hJj6f/b7f5Md2S9ehn6gDECDdEtJ8qxH5nSgoQUOlfTSd
AHLjlkcFXQCex2MEBV2YgT7d3JdaVVUp7eW4Y9RnXibMxOJ53FZbTqw8v3xY1Lu634V+FvBzc7VQT2DV8eSZfbnU4kg7YYRaT5eo
nWjiWC6WVrNt+4Yj7XxW60KaR9qJ4cf6dz304hsGJS8TWc6B3rsiunFgkJ6+QzsxLJWRNcLs7hpUE0vpyaxTNWk/h0uFbvaGBdQj
TvYUTv2/RSTP3tDBS0Bfar5onCXqIyyFBQKBuoJ2TiuCOpESSjgiT9gBgOgIKmBG6H+fcEBXimbjiTEsCCLnGxn8doW/LVhqQUqb
akGaM+tP7yYLeNI5g+zUwzUJw2ZBa3KmHq3Jlm+0JqEU69v8jYwi1iTWwExFdBkNSLPnu06njEWXCrgo4TYwlZEBP1qDlU6dWJM0
SfALlh8ZiU21Xtn+bQDnOe5zKAac556pjmuysJu5jeOaRG2rImtNwmucK1NB4JpXymtqH/xkRqdxHR1YlbQXaxLmeKV32vsBXL3u
PH0GtqLdbcWKhPdD2Zag/UVjYglLez4no8I9H1rwRkIXsUz8G6v6IZZb3uqhuO2WQnfnWRk2bvLQlelOCU+3J9Mo5T3cmCxv7waR
u7sD9QUynNEl6C82AXSvK9s7VH3yjOPSMdCWgMgByfMizWhLS6DCMKiMBaxs8xpMlF/8q8TS4crfpcJuHzX4LjUp8M5v9mJ5HBy7
cA3S3gz4OL+lMKT8nsShb92P0X910ZnsDHo/vzIZs9jT71Nf6TH1u//K6c2YyXbEyMoLmh4uykIHdBWTej7mVXMsGbiantXJvR8c
hXuh/cscT0/HMV/Vz5MvO40dfHjp5caNV0ef1SE86/Amx8+wocV+FXhNElZhQTFf6/Hh9BUcjxpNyA9LU7NrLNTUCT3zGuAKA26o
5SWDDnYGU7AIJVJ1OzY5Td9iWGCK4ci7vu0KxkAZPZoYTMBDGGRLWC2wTxUJ8AsFRd84NQ14e3kUMGv1LgSV1ATMJ0YEw6KeYaah
SMUNHyX58Hf7O2gHdhPYQHhEuvb8y3o7CQ+NuynRDmB3b7sCLYSbIlCFoBFCAMjzA8UFYsWwS5jcyWJsVCK1fcGW2bLSmMNqwnRo
/NlfxLHUSCyfd2T5cGHValYTrDlsxbg5osCtUn+sI3I06MvUMG9uAmEDLXAHhsWpseHax6x7fcMsYAFvt4fODhYiDop+fdBsqr+E
pJxxLwDiAejzUMW0TR07QlUFs4WZXzZitGjY91n3YOxo68pPUAB2JKqWJrLBclhbCGmAx6zMNK6A/9waSmI/JoMIkqRnsBdlETMy
GXSQRI2djN5YwsYD3tUulR7hoF9es4rLwL/aQihq01oGkb4GGrY9y/JEh7NniXsRhNmGJS5K6I29BH0cwx82BfR/BfvqMgcXzzot
EscV+DISFiGrz2fcvK73uL7JxdIfIw46aGwnu+25fPe2NY9Tzzfns6GGAV7qxge21I9jGB41giFsS92Y9edjB0M8Q8JWNmCGLTvv
CG762zqfYTYCMrQxNMrc7p/BFpiK4STM8rf9AjBW3A3AjIt68NdUJ/oFYCy4XwDG0roBWBXHoxuALeL3SBdA9Q+B3fd62tYvAmsR
PCppgxDYrdUDOSL0QKt+EdgVNCcc7/4hsCuogLRt9j9Fy4etvCYENtI7IbCVQZAyAT6nJmgazPC5b9LYQWcsdVploLVUoJgSbFuq
YExpcvHft4krEJnNGHRGILj1s6gAfIMbyeOH2yxErz8Z6w3WH94sMgNA2MffUdSu6LSzUh4DStgu1u0Wd0WICqw7Go0B6SOEkkAn
IJXAZuKce+1MOGwLQjI9gjRmpVru1xdgpwU2qXFLvRLepNX2XMK+IQzrurVRn/mNQTrVXACsJXYqwz0+LRFGSYRecGqZh23Ak7Hb
3G6smRdpX3p6vbU0Uy2XE1r0unn0AQhMRcCcY2zvIxptxDciyTRE26G42Cf2CJii3po7KNT1uA0rQUz9FnLy7+kki39f1N+Nus5j
hTPqyjtSeXfZqMvvC3VllQl15f3xR921hX/8d6Mu/12oK23rd8PVg7qi9EJdmW+hror6l+NUXk4ObVmXEYJChCiqya8ZppuZes4w
YERzRRg0pEXm7SKcO5bXjKpBAaDZABkqysFjuFZ9yfCZJ1meeDOQT6noCjpRGCWY+4db0SHpkSNEhMn6ykWDsM0+tjDsA9ZQMWRE
kcACSvZ6KthzQI4FxVD1xFxozfCK5HhhwTFwIJGYa6HLfigcWPKcnZhrMEbQNFqDzh3xh1ctGtwo4F1brMGoMXzjqw4PwhR47u7v
ALmw91hu66QhwoJ9Vd+UMVDujBLXFHCGlyUh18ZmxMsSKjgGnKNVmEgOrnLZaKjPiDF5ahRhopVW/2ikCwrihbUUix/DtRdxE4K4
qgCLB/5kemExntDesiIUt20tPqAUtMEy06MtXALgppHm5gkYxKtxM/EIq+L3MK7YSXkx0pFxzbpJR6gAyFbNZbSPW6dXb9LIYI00
Dw/Mz2AA6bGEYV3Yu0p4FfcaGceXzcQ2S+UzwYiwlFcUd6A0QPpNlH81qCmtmlFSpAkm8OlWviaHAI2yesEiF/On0xIUF0bW+iNr
/u4MqZEXrGe3xP3OXh45+QngrvvsMIk6PZ90McxpeuaV0kDuai9LRF4MG1lCh6Da3bXTXiAOZOevYCCwpedGMQmkHSoSSymvPLlR
qO+KxOUD7+iB4RYBMeDhrtPILSJwS9Mhd/2uxEz6BV5+R97fy0Kw4L7jrKZLXYyc7/SdO2REPFKsoFdeEywxYDFeWXmnPlAS6OWq
992/9n9/QZHRUsZ9/yBIoBe6Pb/Y12PB0OuNUNlGzw95XeHippnQlzo7E/p6aXTnF/rKa+gI9wg7EtVfcbVp6KWAnwTlOhhXmYFe
chLzBr9matIY9yuoDSlNs/Kme5quJ68Jet3WKfhF+3cTecxJr5vIU4zsJvIOXVkSeVCS+b3IxASA9BW503q1HbjpvF4wSmkKVBgm
Y96C6du4hFa+lr1GYnvez+UfYZAy6LT2Dy1hRoPOANbpvY3hquPrI98/UlzrZxJ6EwyAldllAGx7P3bTeTBAlcoY/mpdbvVL5iFH
lqbTiGAbtDys3NnBPQx38+Xm4snEMBgND+WJIfoad/4IvqpeV8c8CcLFrx8dp9D9kZJ3aeTp0eGZAHRzlYRefV2hAkOcrSvm8ixo
1TzNwyWtfVOYMsdvqpfWZpCL3p+B9uUE3VYUz9k5xLMz8d4s+KV1tzjJzimWNIO4dt7/s2K+ONe8cS5XIFXNbDjl7DL/Kqr0NHPi
GK17qhBtxm9c1DSDuMrvK6MXzTTicjv7/K2KriWWvwtxRR1E7IlW15+/0+5/6gyg1O/6ZPAE0DaispHYc95BTqT2teiUZrvTpjmH
CbB6Xdw3Bbb8K2wkxvSd18cvR+L261FWwvBg6TQ19HHgvRQZ5WYKtqiB96IRY54cEaBm5WagABvTaD7dVTHcoDe1YBl+sEp1r79z
Rdh6enHbzC7nCcenL7lJQ+ioW+NoRRgb84jKxCAYYn0aCksTRanQQXSPiMrQh4wVIML2ehon0Rn8pyucLJe8D/UYmoWYI7bEHjAA
AgTatVeZhFhoCFafhnApoemdC5UfxS6iOEm0EbBwtmCLsebHznsSYoE3ggyRJ4J4AVjYxsY+7HuTEKmOR4L+ginQqSJF6DgwJfqM
KAuj0rw6lERC55Y6S/bSGIo9TuIsWCpyXDjzCAQgl+QROhoTIDJLDKLDwNoiwsx0JulcPEwQoPK29sI4kS/gJr7CN48A3lR8CyY6
7zE29cRzEWYdQJTXGicmQFmJHeDq52ojNWkqBDv/OV09mDaVxPRXa926KMVd0TdadARfy4Vwu72sFYJYxx7PbuK1FVEWBUnlstBF
IfHi1uYfCVrrjgJbiBjb18hzgQhipRe82tPvyZzYhNMZAtumOO34dl6Clnb73EfbAJtjetyz03xHQTwC09Q2/z6fDQykcuONW/z+
0jy+HhTucedjvCDs0FrcCmHFEJqfwVS2y4QiwHZ5pxSK9u+TENa84SyX6nCW+7bn54MSznJbhZw493UxFfFRXUxF610XUxF8VzCV
hryCqaYMZhlUIYiaEur6iLqgaucjR7Pua9ymsJllSNWNTCqYCruoEUl5oZ3PNOoAxWnhuk62vqRiPrOrShc1Z+8VhKvua0KbKcAh
rTyBXoy6nafCkOmVntDswQKfK9KAGPldMkob/Xab1rSTqLrCRxmweytVct+ADd2bTzs3Zf25A0ua1GZcySJT/bA+LeNE4w9CTrtP
uSBhYFMU199kjjKiNi8tBTetu5E2Na3IkQ8bbaBCOeGfIJA+z32VLbra8irNeTxgMC8HCSAPMrd3zVk/807gJV3gK9Pb/Q3s2HBc
NYQTweKwTbSHqQCp118f2roM+yaR5vjv3jC4q/smzswt+cp31tstkpKm4dLRLAbi/jdDRg2t7QHThue41wJQGQEBqHJda/4U/0sU
VhKFr9vsQKvbMG/CrgKgtvviKKvLl0ZO/32cX1/GDahXQlblMg2gXO+4e7xKyCr19hsqq4Ss3vxdRie/lSiMCghA3QXNSEUUTtyb
rBYCnHZ1xPxXzRdTDr1boJbBnKR5tNxAqJVVw0jO+/qZCtCECDep2GkeIn4VowBuD7UYxJp29PKjBAy02itPbYJvGbYCj2wmGpan
tMTIELke9hG+vA9NPWNTwB+vi+VGHyCrWXW6nCDAkkecoSnggelVp9AUs6eSBkNcGC0XQ+DUwOTzd9zpA2VmHjCdGIe2bSSUDURG
qTRsUAHEEt7jJcMcC7YPr+tjaAoG4rHNYmgKtOXmMieJU/jPJnupLVM4tYybYH499kJ8yPdMtYwNgPEQ3i3hplYEFBr6xvjEq/Bt
CTchVlIetGJUCIFBfydCKxKh3c8Y8oYzVyH0FuDmDNUmSiIHQoIQAQjAKmITwLXhVSRGmXPmRdRE8oNVgGE3ZBz9gDCBg6xvyKxE
QsDqTVODLrsfMEKcCC9u7vRQQnkQyTImPUqFMyU4SQyVAMiELu3omtD1tB7SSsK6Ls+tkgFAcS3PpPfNTlWWF/rzqsPa5iLP1ywI
om9PrXhbj3fvJdCEzbIdlTmJu3e3hIQwcJv8XpdjTqu0umFWshzQmddzJcwEwrwF6ulXtgJSLMKSwFK6NYg+hkfQYQcwltxYhdZt
v3JIw5986aO6wMmjYDvo9z8uUAU4ZYRtT/0t19YSQC/jJmyBjCSIHYCzPUWK1JAea2H+KnE0yo2q86tQTqG7Ubv/OrF/RjXRKNe2
bxDtXOREC3uCnOS+zkVOVLxzo1H7ChexnEvEuoKIWD0FcN8x6ViOgROMcSpVMGqTuHSMnMC66ZGYT1mCl8fIaZOYGkmGq247aAKY
fk+p6+Y3jrHTem99HyHrfISsKzDO8l7BOfJ8NEQOuQMwaO+AKk/EydX3Xgk4eymFEqokYtWVNnedeqk8gwLPobRpiJKwbXvdh7AD
r2YsJMnOLqQ55Htg19oyMA/NgQgGIUiCbDHSDyZfYGjuhwyS9dth0bC+UWryIJVuiLfL4y8rpEiwL7TzPTcEhfyh5/M9YcKZD4nd
bHMhr7IeeaCAX363eG+TIJ+Txp6LAkyANHA/N/SU9zmxJxVIe1OBVDRDu+dPXZnxEzQy25HUHAt1ZOP8W2nLTKF3zPndE106AU3L
o+XTCn7H68uFOuDs97+o0wloSpkyFstdnjwYkFU2r4E4FzS5TBmj4XcEmpbLH/Xr/fhT77hG59w8n//+satOok5pW78Eg5OoU9rT
Q/O1bQhQOn+iTudPnu9c0HT1R7RfTxmUfvFohAaRhwUUJ2LPsFTgjVMdwoUK6p1/ISjTWYfghT0HQlqbfpwCFR6LstRJHf0Ccc/D
y5weNiCgqagOnURDKrCs9GovtviKDB3l9BB6BHTRsDDg9HCM1CGdNkKqSW9y/87Iw7LkszuZUaZH0NvynHKRF1hmXvTipaALNmkG
TXBvlhRVWeV5cxBx8bwCJRZESoSeTZFOEFlRn6aC1Yt8Wkr0afsRMDoETSQNdRUyZScQz1B/xL1BjzzezgDB7eoz7ep1kEBS6RxU
eVRohZF0tBZxETBn5/mn8cYmzeaFmw0lfyQQBWKOLDh6vv2e0u4wUCphK3nlKSaRfpDszt+M129PsI4qjVgGUi5xVs4OQ8dJ4En1
iUzGs6y9BEvz2EIyiXei1dyvtSMIz0AJxssInoiOHmRPPyHI6TFYL4kc87VuyviRp12W6HfAFbM6NaUVRhRRgIEnX0zV/8cDf8sS
g7KIsrtEhUTn8nL3vvc6L9Hf22vbLqAE1bILFEXkHSneQMm6o1MhiKvYNii8hMGzRaAPAsA8fgTtxLLx6Is1Vc1qK6SE4JW/oWqu
Ny2iCSLb3MtJQcPbPnpQ5Fx3JMZaeQRBws9I0h8D0tkgPCIoTm0ATgBTmiAcb8zwfBvTI7w0mKX3Ala+3KtX2rhtWwgyxntXeX0O
xGgpv3f78/sSws9FS67AfsUNv2ipfrz1CDub6Gpf7u4K8SbV/uTu6qOtA4lUaOtn3NdK2geFrfDWsX16rtR5vgqLscJcnzzlZ2mJ
MZlHQudEBXW56zT4Fe46TvZBNSP64AwJOHUJ7MCjI4+nMCqDOBUSO7HIlZgjP/dNajXi0pakKO25pRJuY+G9ad7g7MLirIg0R6Q1
RiTg6Ywf1uWzM9wfSTTRjEfXSYnbMwaceHojEvme/X6mTTE45CmTB58WvUTdE8yDh5zqEtjP+iSlbvJZEzk/HW5l2n6FtQ7jmg4Q
MZ26ryXsUWGr0yaUqeqo5rUKvHHDlfPJTLyVWepyydWClh635ISqU+3Gl1zBSf67zE8nbnMzbEYf17B1DOZ107eSMClr09O96eEW
6XD5SSWYWiGmc6FWaOlM2lXOHV8Nr+wnKueOdzRq1U8XpZcr74gy6SaJHfmm21BIoLPMAD1cu4Mw529FzgR86jLSo0zCTMPvXEZ6
XUZ65Z3+K2f8WZc30FSXkZ6ZMzcq79h8+3cO8dfHSI9ylANQdRnpny4bP1lNiJn8+yhi011BglTqF1xe1JL5CJLSKYQHz7gzsZn6
MWknJuM6PBsg5hCiPhTkjYrH1f+TO6cVRZ7VRTHOig1YWRseAh3EN7dawiVDrDRVlTAhT4HoTW3b4dJsVJK9bFmY3NwgY80QjUSF
GBCjsXJ0unjEKafeHUxOPwqwlkjnWOKYSktK54CSpQaQdD510E7JBl4IMNxQTj9syLFBFV5aPJUoicmtk4mXbcMy6KqaeobuuEDR
rs/2R+JNte0FLcf0do+XdqU49KV6Cf/g/rqax0bhcw9WibuGjb0E5Zrq8UTR2iMY4MnmGgNBbbtwbliOSKElvjmM+o7qML0OS219
9JkvssgkEXaeqPnxlk+vMbg0Yid0ADJLRKFq23fFlXCORH+H6cEpHSsNTdJ840+Ild7pNhIqkTrvt2io2VFrFkd7Wl8Vedc51DLN
/K4xGlF4S2uRL4YABd6PSDHv8Z7aysKk+z2R81oskOgYmI/XNXF7DP1JVWLmrSwPEcwZObTABCzp5WV6Ob6xd1CSY/MUgKShLC2v
XyizywmUd57ZbOe7wbgagzFlcjm9rJs424cDxCx/uh3Cc7d9ZWL5QiTRrZviyaeHAe+PHy0dybbjcA6p23M8io021y+0FlAhjtN8
Pa9OyKVOKONUQLxMMKcXdlk6QQ2sK2Hr++7Xdv8QTpJy+Xt9ZjGQCHX8OObDwk6asS7HXNPY/5zxq1DMO/Byj0Q1HSnuHvKrSzHv
PBVQ4ZhjN1RXEvXxlui55PIOzRyHiloEXhgy8p5TtD3NNV+s3UJE+ieLziJmifmvFCOYixzQfaj9QPX7lEEdnu+oMM4H+YTp4PDK
49xd1vnQfSYV3vlLhx2J3dryy/eGDYRu8qXYJSOdtFan6czO8VRGJJ1OvUV6t5/xpWkHbrjtfnSeZZ0U82gv3lMdliU4ozsD7qT5
Sv9bLlWoyzEne6ZCMH+Y469+wdK5TwSWXIAyco8rElh63CIdBMc9GxYEHVOL0ZJ/nwu1Lqu8XOrJqYjqFy25284c+1vRR5muLxPK
yZ/1W9ylylf25ONez8JWcvNxFZV0bZlLPphhr1DJdyZKizDq7fMP/vs9/lDfTRj+1hnw/JYZcyN0NGv776bfudWzft2fObtRl0N+
fytF7HaOCy7DIT/pixhNGSKdOc3MK7Y03X4zmjw5SshleEV0mNppmEW+GMJ1udy/Lt0RYhMBY8PYRPWbl8uDGUjTb4jp6hNBj5tC
otC641GmJKl7Orfq2kxZKvHGX3e5+5Kl6gJLM4PFNPSr4x/cazFi+A4Py8PYPK81sqRNAmIoKoEn9HgizV19lFs4x4ubDgkZmeif
TtduMtgkmZextiUdVyUhSxLzQUimv2o3YRIistMDRpgEWo/2GCKKA+x0d4u2grZSrRw6hnBsOZmK294DiCS+2Q49mdr8b6+7yWze
O2xepij5j4eRBHHE8pqecOsOCk+pHgbgQezoKkDEKl6tI4Et4Nk4CYy62loKwfFWIwkcl5cb8BIxHGGO1zpEkghdmj5n0Fa3IpVp
4chmucnetZTHlNcpIcBgQ7e5mUaAQQpPaIRNoFcj2UrkQ+s3Z3CVl6O2U/76iCDkj4mNeKa9RAXv291jEClz8v6+YyB8NDs5xo/E
AS9zwJH/tGmQ4YNDmhbY69n8m3H0d/m3Pp/WOe1Fm1/S1QctTlE72LN7HnGLjJhoJAYzkVOyJP2DOd+WOPA1vJCUhVgKqJV53zzg
6X7Sr+BQLQ+V+ggM04eV9vKgwuBVTpaUDGdYv0z85tp0IdRA8X/KtG8UYf1xJGEclw4NXHco6VTBbEx751GG2q/pnGSz0iiqgeXg
QVN6CEY+EveMPQt1Od3oxS6uztgpXGu1DCyUb8O2353ggaB1DDIcQ7KP+2JIdfnfO38nRdBughgHU+Nu0rEsDX4I4JuHmCoE8M18
ZF0C+CMXFwb46JrOUMB5eH9FIqohy6NCAkd0s6d83eBwtMbnBUfSsfDAgShpWsMD5+bOwvK5npSydH8c3dBlgk/BllDBeewlXVAo
iYRKi1NByyLNo0IIR1t4gKVCCYfVTbFTG4znpFyiI0RnWgrWQU2t1csLb4Jg94YnxSXuDU+KG4UUrqNfFUo48NRtGrNv2LSkv10Y
+qRd/YZQPir4yQA+PKh2C3zsJPOeUgs7rdDxbV6OUWF/Y8OQqoSJ3jwRJppXkMJdof+ptrVfp16aYO2wZu65e/yWA0juwttN0qlc
2MTcVoXsrWDSvDfiePSPeFvHNZwkkCpUb6bVylRvbjT8YIc3WDP5tuiYQ5x55+555g0euQIFjzJuDh65pZXob80Ej7bHzBQl12WO
t8vUIaSZj3VQ6A7fCjmqLst7ujaHofNbNbsGJ/ldkHZqr2tjGqb3NMMJN780LpC7NO/yzAsUZd4dPPLfuy76yRLiPXfYsjxuB2/E
6Yzx56H2d/1bJNQ4nnvMKqUHaUwnVUjfRfZJVNVXIFhlTDAqU755GV/NXygpVO8ofPdFaiWKN0+/lSnej/o9P0XU9UAqU6f2bDmY
cOs8VKTPHhGfQBZUfWwfmcJDBT2OWC7PEQ8jzUSSxe5mYpApvRK7u/MEhMvUNv1dUc8u6jB8g79jxAWGYaiBTLeTV6IXmc9ACNkz
LIrjLM+YzAcAigaCU4zvXxWHuT889qQnU9cv8aKaMqcbsfXu18TnGG6e9ukxo3RIS5scsbl5g5tNMRHRo8vISlxuUBKP55QJpsGD
JUI9vNwo603ejkcxyixuBGI8NSRTPF5uJd6zB43Bonfbfm75626zvBVuWl64oiFtniQrkbdhhLsNBcmIz3TXyd3GgT5Vd3JjSIm2
PRwjEWl78IBeibLdva8VY3u+nn0bJJWnc8GC+uJqwxHZAtKyDSc3zNTGyd/l3/p/R8XqW1W2n9ZWH4RLk80QjWmgNcaNdZ5uUbSf
TLcOwYlGUuZoU6U9gLL7yBnZPggL1b6P6LliFMXjeFYKp869sSP0PrW9quiWSufdyhRtJrlcMjna3M17iTBI9th8KIOGqFbGY4pH
HYvo01SreVHTtZJ46ke6ZWN6Th+dx0y12jNaHUXWRt7MrWZ2WBfclLna5OlHICGnPf7+noArs7WHsJbZ2mTR67eOZ+40i3SL65GI
Wngxmn9T55t1RCCoeFqlLmMb404TFc42I8cnEoHXvILuOVwROu94rQhELxy2cLZ5RVgE8sTYhXC2eaFaSjCwW/crIsGVmpzuu1Up
l/bcz8zZZ+/D2WbA837no4Spm2CVlMyqC36Yt6zQtkua+9G23ys0cSIzGkNBvK8DBNPkM0ai4eQxsrq8baWZLmsboG1Y6mIspupu
ExAhJ9OrPuBzhZb8xb03UHvNMLXRm1ucgM9JG57cSFwhaePQaF5r509xrUy2rfpiQVe4m/LQs0V0r7o0I7da164t99RWJu/0QJq6
sMfqwQMmPU00y8jvnAuZLi/7uEzndt1wxzH9zhesvLzsdG5f+HB52cv1VqhRlav+HjLWy8TsKatjYvbZmcO6yb76nWyrS8zOKuC5
oPFkvBgIcrzxUrPzxfzztc+2ueY5//y9J0p1qdnNNShjNl3mqN/wCvSMvJMzHRVqtjall5qdMs0yyt9l966eMAzPQ4cWaEB0gKXC
z+YdkNEhpYpuIYoFfWpsMnV9HO3Mb8I/FbTjt3ULlF8WstFgNB9ZKxOzbT1mEFIJ7byqV9c/arxdkX/6cr/lZUt662SoSEogNjYA
ksp+qHwrqshsROPVX2oscQ7STy2N0O2BSIPJenDwEPkpWzgdusb1UF5Auu0NsSD1behW46f0HnHO4DlYCSxzueKhc1pquQDuY4si
h9Pzm8CrW/uYiXgeW0PyNZYSYSJeo5tHxeueA947K4FOZQnJm3QNe261Mf1DyUxRromJSw0pJ3xtLeWbn+XVXmJtT4+06EOjbH7l
bdZj7dS5ncNzGSW6te5K0G+OkYghZbI1tiSvOnYEjF+P42FE63hUBHOsA0fj5knRvcuiRohnfU3Hm1McVYE5fiHnlUv0agcwTK4e
x2or+7Yrf59JkJtWfUvx8WCbH9rQN/OQe/UUvgujerweYwV8Rqbat8XYlMmKxyTqctjdvbJk+PtOq3pOkFXI1EMRz5CpSxdklNnU
sGFPlJyk/0bXb0lU5GdGEqVmZG0oZeT2mGXbbSBFJgFKO3mPJjLO6Lk0yTKn2nezlEnVk4RI94l7QN3JbUkNeY/HVCDneJU8znXZ
2DvQ894nxNrd60wZMNDq0yld/xVXT4wBeOpvdv+sv1DJe39zeT22NEIyNtuXXr0UEjrfybQZ4W6IP341B/zjV3MUPn41TWH41aI/
1/kOpl1hanszIt2gVPjVTtydi3FWnsjDjPskl11VqNVQrNSqO9u0wTn3VBp9Y5jVeC0lfMSM7/rZNz3yZYGp1aeo03FiZAeN7/2z
xZuGI4ZoXt8NtE/q7TdJFFJ18WhvhVPNq6gj9D/1iqcxUrSIDE86qKDOuU9k1VPPkztw614ze25zyO1TtiQ86jmVtjwX27z3kcKL
mczWfl1/r9MMhVrXyJQp1K9Q+qVQN7dM1yc9mQYFdaa/PjmhWOFQ32Yd3Zj3+Ot75WpdDnV3M+7x/boc6oydld2F7vZrap3f+z52
5pYK4qcuE4I8uE4Ru5x108Mfg9rfLt2xn1brmhvealiXQ11+yzEdd3/+WTyO6fjj+69r1OVQH4+pTNP9Xb/eOIzoFo1cwFaXQ91c
170rqT4OtevtuZiswqFWRDgc6jfl+BbVaFb5gFldDnXaI1DzaVLgyUkIJ4P4K2UGKh+DmqF39ZVqt43jr/lpbiUjJtq0EPgxqLG5
cPjmOSrU0ZuXt/qWuNJwN7rnXel3RU2Wx5fkn6M7xRV+5uHoaCpTzgjcWjt5LH9t7SeP75qkK6fAQ/kkOeobAhrEuJZaNPRvOJQ6
5LBvaXhMrmgq17fn6hVdLOIvnUPwC9cvixPNS0T4U5dALg38unQ9UaFhsDxGCnxPz+XS+Ux3kVyh/VgPmFbD+Vf/nYmzNJzAhVFz
/T7OlKmvvBFcl7ZL4L0Wyo+K/wzbPVTSlg/15O6ckSixn2HE/JNf2q7rnNhwgYe4rHmMeK3afD1G7w0hi/aMQlSlMlbWgjfnVUuE
5+ubvAU7y7+ZW3RDjF0qv+szZT/sYpazM9AhOePy67ylG4PfqNnzc16GLqm6t8/k69rwaTvqc2CC/mY397igp+fywjK1GakFe06F
ZfB5BG/cUsnQ/feuhEkG8yhNbS5dhFlmNvNuLa8RhQK3tcXsWDDg8554y48nWrCFl99bUL5q9TwiTlf0wuxm5KDSPHLIxm5Zc2Le
VD4q0d3TCgY/n9jGx7dnPRG4DFZW7RYVTmGBsJwR3bTjEHBZisqF6NzLq1boZJKMuEN13uSx7VCddQf4/qjOWNz7ozqPK+QEy/6o
zpUnlxuxvxu6PyF70x2ic/G05A7PmReTWeAILYaS9t8rundozmSTR6BR5L8gtMNyLjEHdzjOvPI6QjlKtsNvxqGUK3Sfm9v3km66
r33ZzYz67XtJ976j5ZuO881NR+37LzTt2+6rrPte0M2czr7XcxOK7TCafwPUc7ZwXz5zI/tg33+S6Sv7+VuCb1zJKDztT60tXnRf
LjMR2743cH9T2ZKW2e0LybieN6z33e5N+B6CNzmdHSozA9q73YNfLvWcX1culXmHynzL4fGOsdIpXZ3LIPAOl/n2dufY2L73bG83
9ZLUdnjM7Y06Vv6pmX15zFEY37TqZlT+xYR9ecwZmZXru/blMUdDlyITwx/oXr2Tl9pv9O+9pPveqx1l8r3absRfjfMxVH87/oyX
YzIeozF/+uGYjMsf7bf++p8V1+s3H4pwZxGIOb88ij2hsx0W87eO69d+qdhdnv2P7rXfcjy/V+pPrwyPVJER1xaDeXvE+mcD5jc1
9elT8Nc2bzmzmDuF9HP6yjER141Ot6nKIJ96+Ng7hrRUMilJ2DBYoZ1oem0jSH7gaPoJIcJgjnP7XmychvHCGSGtb7OTsxw0Q7YO
mjgr9Gxfi2b/3tXsd7XGF3uosJnN2fbd10x/bvGQASS6VGIlGrd96XW0cp1Pl6S3Q0XUz4rU/JqmSJJXSDnGNSWoxJ7atygohOra
/3L37jJ3EP8W43gxs759w3XG6YhfMPXymd/SP7koYottfM3M27551y399jCvWBTq+vsz0q+uzNzWAo67R4+G6nQPtS2Yu2PTZh22
zfPCtjG0pX4cznu8AkSmqTfK89S39PIv4blq3avg5aDzuvD7/q2s7HE7fD1Fi8Cc2Yx6ksHFW/otiEXkcdOJ4r7SYLLcydOyQIBA
NtzOddXF2KGZUN9sKLcw3pEFYUawR04ZifmkAF8Rnd/9U9Ns/92w9bMXuZjZ75v66yocx3MFTgK5H6U7LN0NomCer9nhEusM2r5c
4pnf8zfSf8x6uMRo0O/a5hGh/Xkyrjr1vz65fz453/gepRbhutT++eQU0MMV2P365JPS+vVplzirlfzd00yNurRZmcL+ueR881xU
cCmzX9va+dOcdsHRZczK8V7GLANLu38u2d/IJVMNwpi9bbZLzt+vmwhf9jb4XONqviwSpo8/PqF47xBmtX7uvyk33I2df4HRQvIk
u8cl97x057rfQIJ/1wUi/QYSPBwfIPzosh6Myj9WsUOXHbdrpL+/qWDFrv4fZIdkjg=="""
csv = zlib.decompress(base64.b64decode(data))
assert hashlib.sha256(csv).hexdigest() == "7956a24e739c23d8fb946d99f45686e948f25e48bc1771eb82a6d19cf2676077"
sizes = {(int(lat), int(lon)): int(size) for lat, lon, size in (line.split(",") for line in csv.decode().splitlines())}
assert len(sizes) == 2250
base = json.loads(subprocess.check_output(["git", "show", "7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee:settings/download_menu.json"]))
current = json.loads(subprocess.check_output(["git", "show", "30e8ce98ea7a4c8401dbb5bfc62120c84fc689e4:settings/download_menu.json"]))
def cells(location):
  rows = location.get("download_rows")
  if rows:
    return {(lat, lon) for lat, first, end in rows for lon in range(first, end, 2)}
  box = location["bounding_box"]
  return {(lat, lon) for lat in range(2*math.floor(box["min_lat"]/2), 2*math.ceil(box["max_lat"]/2), 2)
    for lon in range(2*math.floor(box["min_lon"]/2), 2*math.ceil(box["max_lon"]/2), 2)}
for group, code in [("nation", "CA"), ("us_state", "AK")]:
  before, after = cells(base[group][code]), cells(current[group][code])
  print(group, code, "requests", len(before), len(after), "compressed bytes", sum(sizes[p] for p in before), sum(sizes[p] for p in after))
for label, menu in [("upstream", base), ("selected", current)]:
  print(label, "separately summed requests", sum(len(cells(entry)) for group in menu.values() for entry in group.values()))

#107 is now merged upstream. This branch incorporates upstream main at f374b2b6020f4a4c85aeec01323a9387d37417e0 and retains selected-row iteration with publishProgress() inside the loop, preserving between-archive cancellation and extraction.

Conflict-resolution checks on September 7, 2026 used Go 1.25.1 on Linux amd64 with isolated synthetic downloads: repository tests, selected-row and lifecycle regression fixtures, generator input cases, and the settings race check passed. The generator and committed region data are unchanged by this merge; new test fixtures remain outside the PR.

Combined validation: exact source tree a4c306906627db3ac7a8ab768651c8628d55465a combines #101 9d61f06a1288ec4ea6f74f7d56a3057444316a13, #103 20e7c25b054b6399360676a7f539a39b4fbf855c, #105 bfcfe77be066634e36054327b20cfa6541063b54, #107 8e5e677d1196838069e9665d4e9d962bcc1e116b, #116 6fd5bbd6cf617c24a7fefd5e302fd36688a1a63b, #136 30e8ce98ea7a4c8401dbb5bfc62120c84fc689e4. The only overlapping file is settings/download.go; the resolution retains #136's selected-row loop and #107's progress publication inside it.

Combined Linux amd64 tests (including the scratch regression fixtures), race checks, vet and build passed. Under ARM64 emulation, the existing Makefile build stage (make GO_CAPNP_PATH=/usr/local/go-capnp/std), committed repository tests, vet and both CLI help commands passed with Go1.25.1; go.mod/go.sum stayed unchanged and the resulting executable is AArch64. The ARM64 run does not include the extra amd64 scratch tests. It used an isolated retained build image, not a new dependency-install/image rebuild or physical device. No archive payload or live params were accessed. #105 still requires runtime-first rollout before regenerated tiles are distributed.

@FrogAi
FrogAi force-pushed the codex/optimize-map-downloads branch 2 times, most recently from 2c7e34c to a35dbc1 Compare August 23, 2026 00:01
@FrogAi
FrogAi force-pushed the codex/optimize-map-downloads branch from a35dbc1 to f538b5e Compare August 23, 2026 00:29
@FrogAi
FrogAi force-pushed the codex/optimize-map-downloads branch from f538b5e to 8e3da52 Compare September 4, 2026 21:28
FrogAi added a commit to FrogAi/mapd that referenced this pull request Sep 4, 2026
Retain the original PR commits and the tested rewrite. The resulting
file tree is identical to 8e3da52.
Replace the earlier implementation with the simplified version.
@FrogAi
FrogAi force-pushed the codex/optimize-map-downloads branch from d8ffc33 to 46daaa2 Compare September 4, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant