feat(telemetry): report safe app setting values

CHANGELOG.md

@@ -52,6 +52,11 @@ between releases do not reach them. Remote-PC runtime bundles are versioned sepa
   ([#606](https://github.com/devswha/herdr-web-ui/pull/606) by @sobir-git)
 
 ### Changed
+- Install and update counts include app setting values such as theme, text size and notification
+  choices, rather than only the names of changed appearance settings. Custom text and account
+  identifiers stay out; custom choices use flags or counts. The expanded notice is shown again
+  before sending, and existing opt-outs and install/update-only delivery stay unchanged.
+  ([#866](https://github.com/devswha/herdr-web-ui/pull/866))
 - Markdown file previews are parsed in a worker with a 2 s budget and a 60 000-element limit,
   keeping the existing HTML blocks, images and reader-page navigation. A preview that cannot
   be drawn offers its source. Partial text and preview, highlighting or line-number limits are

docs/guide.md

@@ -470,9 +470,11 @@ This is for Linux and macOS. On Windows, and for a remote PC, update herdr on th
 
 ## Anonymous usage counts
 
-The app tells its maintainer how many installs there are, which versions they run, which countries they are in and which appearance settings people change from their defaults, and nothing else. It sends one small message when it is first installed and one each time it runs a new version, never in between. The first time you open the app, a line under the header says so, with **What is sent** and **Turn off**. Nothing is sent before that line has been shown, and the first message waits ten minutes after it, so **Turn off** on that line stops it. An install that saw the line before it mentioned the settings sees it once more, and sends nothing until then.
+The app tells its maintainer how many installs there are, which versions they run, which countries the messages come from and which app settings people choose. It sends one message when it is first installed and one each time it runs a new version, never in between. The first time you open the app, a line under the header says so, with **What is sent** and **Turn off**. Nothing is sent before that line has been shown, and the first message waits ten minutes after it, so **Turn off** on that line stops it. An install that saw an older notice sees the expanded-settings notice once more, and sends nothing until then. An existing choice to turn counts off stays off.
 
-A message holds the event (`install` or `update`), a random ID made on this PC, the app's version and the version an update replaced, the OS and CPU architecture, how the app was installed (herdr plugin, `bun run start`, or `bun run server`), and the names of the appearance settings changed from their defaults in the browser that last opened the app: theme, density, colors, sidebar rows, Agents order, workspace cards, pin agents, plan flow and the plan count in the header. It holds the names only, never what a setting is set to, and never a setting you type (a font, a shortcut, the key bar). Nothing about your terminals, agents, files, workspaces or accounts is in it. The receiver ([`telemetry/`](../telemetry/README.md)) keeps those fields, the day and the country the message came from (two letters, such as `KR`, worked out by its host from the connection), and never stores your IP address or anything finer than the country.
+A message holds the event (`install` or `update`), a random ID made on this PC, the app's version and the version an update replaced, the OS and CPU architecture, how the app was installed (herdr plugin, `bun run start`, or `bun run server`), and an allowlisted `settings` snapshot. The snapshot contains actual choices for appearance, terminal and chat behavior, notifications, language, voice controls, usage-meter display and the file viewer, including defaults. Custom fonts, shortcuts, key-bar layouts and quick replies are represented by customization flags, not their text or key combinations; quick replies and hidden usage accounts also have counts. Account identifiers and ordering, paths, addresses, credentials, commands, terminal content, conversations and files are not collected.
+
+The snapshot is the last one a browser reported to this server, not a list of devices or a complete picture of a person. Changing a setting updates that local snapshot but does not send another telemetry event. Before a browser reports, `settings` is `null`. The receiver ([`telemetry/`](../telemetry/README.md)) keeps the event fields, the day and the country the message came from (two letters, such as `KR`, worked out by its host from the connection), and never stores your IP address or anything finer than the country. The random install ID and settings can link events from an installation; they do not guarantee anonymity.
 
 **Settings → About → Anonymous usage counts** shows the next message exactly as it would be sent, and its switch turns the counts off for this PC. `HERDR_WEB_TELEMETRY=0` or `DO_NOT_TRACK=1` in the server's environment turns them off whatever the switch says, and they are always off in CI. A remote PC's bridge sends nothing.
 

scripts/telemetry-regression.ts

@@ -0,0 +1,198 @@
+/** The real telemetry UI and local API, with a fixture-only transport. Run after `bun run build`. */
+import assert from "node:assert/strict";
+import { randomUUID } from "node:crypto";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import type { Browser, BrowserContext } from "playwright-core";
+import { NOTICE_VERSION, type TelemetryChange, type TelemetryEvent, type TelemetryStatus } from "../shared/telemetry.ts";
+import { createServer } from "../server/index.ts";
+import { Telemetry } from "../server/telemetry.ts";
+import { UsageService } from "../server/usage.ts";
+import { openSettingsPage } from "./settings-page.ts";
+
+const SEND_GRACE_MS = 24 * 60 * 60 * 1_000;
+
+async function until(check: () => boolean | Promise<boolean>, label: string): Promise<void> {
+  const deadline = Date.now() + 15_000;
+  while (!(await check())) {
+    if (Date.now() >= deadline) throw new Error(`Timed out: ${label}`);
+    await Bun.sleep(50);
+  }
+}
+
+async function statusAt(origin: string): Promise<TelemetryStatus> {
+  const response = await fetch(`${origin}/api/telemetry`);
+  assert.equal(response.status, 200, "the owned local server serves the telemetry API");
+  return await response.json() as TelemetryStatus;
+}
+
+function fixtureSettings() {
+  return {
+    language: "en",
+    palette: "forest",
+    terminalInputMode: "line",
+    terminalFontSize: 17,
+    chatFontSize: 19,
+    voiceLanguage: "en-GB",
+    terminalFontFamily: "Fixture Terminal Font Canary",
+    chatFontFamily: "Fixture Chat Font Canary",
+    quickReplies: ["Fixture quick reply canary"],
+    usageOrder: ["fictional-account-order-canary"],
+    usageHidden: ["fictional-account-id-canary"],
+  };
+}
+
+/** A real app page and local API; the injected transport cannot reach the production receiver. */
+export async function checkTelemetrySettings(browser: Browser): Promise<void> {
+  const root = mkdtempSync(join(tmpdir(), "herdr-web-ui-telemetry-regression-"));
+  const serverState = join(root, "server-state");
+  const telemetryState = join(root, "telemetry-state");
+  const sent: TelemetryEvent[] = [];
+  const changes: TelemetryChange[] = [];
+  const fixture = fixtureSettings();
+  let fixtureNow = Date.now();
+  const privateMarkers = [
+    fixture.quickReplies[0]!,
+    fixture.terminalFontFamily,
+    fixture.chatFontFamily,
+    fixture.usageOrder[0]!,
+    fixture.usageHidden[0]!,
+  ];
+  let telemetry: Telemetry | undefined;
+  let server: ReturnType<typeof createServer> | undefined;
+  let context: BrowserContext | undefined;
+  try {
+    mkdirSync(telemetryState, { recursive: true });
+    writeFileSync(join(telemetryState, "telemetry.json"), JSON.stringify({
+      install_id: randomUUID(),
+      enabled: true,
+      notice_seen_at: new Date(0).toISOString(),
+      notice_version: NOTICE_VERSION - 1,
+      reported_version: null,
+      settings: null,
+    }), { mode: 0o600 });
+
+    telemetry = new Telemetry({
+      stateDir: telemetryState,
+      version: "0.0.0-telemetry-fixture",
+      env: {},
+      fetch: (async (_input: string | URL | Request, init?: RequestInit) => {
+        if (init?.body) sent.push(JSON.parse(String(init.body)) as TelemetryEvent);
+        return new Response(null, { status: 204 });
+      }) as typeof fetch,
+      now: () => fixtureNow,
+      noticeGraceMs: SEND_GRACE_MS,
+    });
+    server = createServer({
+      port: 0,
+      hostname: "127.0.0.1",
+      token: "",
+      stateDir: serverState,
+      telemetry,
+      usage: new UsageService(undefined, []),
+      registerBridge: false,
+    });
+    await server.statusReady;
+    const origin = `http://127.0.0.1:${server.port}`;
+    const initial = await statusAt(origin);
+    assert.equal(initial.enabled, true);
+    assert.equal(initial.notice_seen, false, "a persisted older notice version needs reacknowledgement");
+
+    context = await browser.newContext({ viewport: { width: 1280, height: 800 }, locale: "en-US" });
+    await context.addInitScript((settings) => {
+      localStorage.setItem("herdr-web-ui:settings", JSON.stringify(settings));
+    }, fixture);
+    const page = await context.newPage();
+    page.setDefaultTimeout(10_000);
+    const errors: string[] = [];
+    page.on("pageerror", (error) => errors.push(error.message));
+    page.on("request", (request) => {
+      if (request.method() !== "POST" || new URL(request.url()).pathname !== "/api/telemetry") return;
+      changes.push(request.postDataJSON() as TelemetryChange);
+    });
+
+    await page.goto(origin);
+    const notice = page.locator(".update-notice").filter({ hasText: "herdr web ui sends install and update counts" }).first();
+    await notice.waitFor({ state: "visible" });
+    await until(async () => (await statusAt(origin)).notice_seen, "the visible notice is reacknowledged");
+    assert.ok(changes.some((change) => change.notice_seen === true && change.notice_version === NOTICE_VERSION), "the browser acknowledges exactly the current notice version");
+    assert.equal(changes.findIndex((change) => change.settings !== undefined), changes.findIndex((change) => change.notice_seen === true),
+      "no settings snapshot leaves the browser before the notice is acknowledged; the first rides with the acknowledgement");
+    await until(async () => {
+      const settings = (await statusAt(origin)).next?.settings;
+      return settings?.palette === "forest" && settings.terminalFontSize === 17;
+    }, "the server receives the full initial settings snapshot");
+
+    await page.keyboard.press("ControlOrMeta+Shift+Comma");
+    const dialog = page.locator(".settings-dialog");
+    await dialog.waitFor({ state: "visible" });
+    await openSettingsPage(page, "About");
+    const telemetrySection = dialog.locator(".settings-telemetry");
+    await telemetrySection.locator("summary").click();
+    const preview = telemetrySection.locator("pre.update-output");
+    await preview.waitFor({ state: "visible" });
+    const previewText = await preview.innerText();
+    const event = JSON.parse(previewText) as TelemetryEvent;
+    assert.equal(event.settings?.palette, "forest", "the preview shows the selected palette enum");
+    assert.equal(event.settings?.terminalInputMode, "line", "the preview shows the selected input-mode enum");
+    assert.equal(event.settings?.terminalFontSize, 17, "the preview shows the selected numeric font size");
+    assert.equal(event.settings?.chatFontSize, 19);
+    assert.equal(event.settings?.voiceLanguage, "en-GB");
+    assert.equal(event.settings?.terminalFontCustom, true);
+    assert.equal(event.settings?.chatFontCustom, true);
+    assert.equal(event.settings?.quickRepliesCustom, true);
+    assert.equal(event.settings?.quickReplyCount, 1);
+    assert.equal(event.settings?.usageHiddenCount, 1);
+    for (const marker of privateMarkers) {
+      assert.equal(previewText.includes(marker), false, `the preview omits private fixture value ${marker}`);
+    }
+
+    await openSettingsPage(page, "Appearance");
+    const palette = page.locator("#settings-palette");
+    assert.equal(await palette.inputValue(), "forest");
+    await palette.selectOption("lilac");
+    await until(async () => {
+      const settings = (await statusAt(origin)).next?.settings;
+      return settings?.palette === "lilac";
+    }, "a non-default palette change reaches the server snapshot");
+    for (const marker of privateMarkers) {
+      assert.equal(JSON.stringify(changes).includes(marker), false, `the browser submission omits private fixture value ${marker}`);
+    }
+
+    await openSettingsPage(page, "About");
+    await telemetrySection.locator("summary").click();
+    const updatedPreview = JSON.parse(await preview.innerText()) as TelemetryEvent;
+    assert.equal(updatedPreview.settings?.palette, "lilac", "the preview refreshes with the server's updated snapshot");
+    await dialog.getByRole("button", { name: "Close settings", exact: true }).click();
+    await notice.getByRole("button", { name: "Turn off", exact: true }).click();
+    await until(async () => !(await statusAt(origin)).enabled, "Turn off cancels the pending event");
+    await notice.waitFor({ state: "hidden" });
+    fixtureNow += SEND_GRACE_MS + 1;
+    await telemetry.report();
+    assert.deepEqual(sent, [], "the grace period and fixture-only transport prevent outgoing telemetry");
+    // turned off, a page keeps its settings to itself: the status is read, nothing is told
+    const beforeReload = changes.length;
+    const statusRead = page.waitForResponse((response) => new URL(response.url()).pathname === "/api/telemetry" && response.request().method() === "GET");
+    await page.reload();
+    await statusRead;
+    await page.locator(".conn-live").waitFor();
+    await page.waitForTimeout(500);
+    assert.equal(changes.slice(beforeReload).some((change) => change.settings !== undefined), false, "an opted-out install sends no settings snapshot to its server");
+    assert.deepEqual(errors, []);
+    console.log("PASS telemetry settings: current notice reacknowledged, snapshot first with the acknowledgement, safe preview and snapshot update, opt-out stops pending delivery and later snapshots");
+  } finally {
+    await context?.close();
+    telemetry?.stop();
+    server?.stop();
+    rmSync(root, { recursive: true, force: true });
+  }
+}
+
+// Focused QA entry point, e.g. `bun run check run --build bun scripts/telemetry-regression.ts`.
+if (import.meta.main) {
+  const { chromium } = await import("playwright-core");
+  const browser = await chromium.launch({ executablePath: process.env.CHROME_PATH ?? chromium.executablePath(), headless: true, args: ["--no-sandbox", "--accept-lang=en-US"] });
+  try { await checkTelemetrySettings(browser); }
+  finally { await browser.close(); }
+}

scripts/ui-regression.ts

@@ -42,6 +42,7 @@ import { checkCommandArrows } from "./terminal-command-arrows-regression.ts";
 import { checkFolderFilter } from "./folder-filter-regression.ts";
 import { checkPaletteKeys } from "./palette-keys-regression.ts";
 import { checkUpdateNotice } from "./update-notice-regression.ts";
+import { checkTelemetrySettings } from "./telemetry-regression.ts";
 import { UsageService } from "../server/usage.ts";
 import { openSettingsPage } from "./settings-page.ts";
 import { assertCspClean, watchCsp } from "./csp-violations.ts";
@@ -872,6 +873,7 @@ try {
   await checkBackgroundTabKeepsTerminalSize(browser, origin);
   await checkFocusedWindowReclaimsTerminalSize(browser, origin);
   await checkUpdateNotice(browser, origin);
+  await checkTelemetrySettings(browser);
 
   // The pending protocol has a real foreground agent and an owned byte recorder: merely
   // reporting an idle shell as working would not prove that server-side queueing happened.

server/AGENTS.md

@@ -78,7 +78,7 @@ Bun backend: HTTP API + WebSocket fan-out over herdr's unix-socket API, plus the
 | Subscription plan limits | usage.ts + GET /api/usage in index.ts | read-only, from each CLI's own sign-in files; a provider is asked only when a client asks, at most every FRESH_MS unless forced, and a forced ask no more than once per MIN_REFRESH_MS. Where each sign-in lives follows OpenUsage |
 | Voice input | voice.ts + /api/voice, /api/voice/config, /api/voice/transcribe in index.ts | BYOK OpenAI key in `stateDir/voice.json` (0600) or `HERDR_WEB_OPENAI_API_KEY`; the route raises Bun's request timeout to 120 s; the transcript streams back as NDJSON and the provider picks its decoder from the file name, so the browser's container has to survive the multipart framing |
 | Mirrored terminals | mirror.ts + mirror-input.ts | for a herdr with no `terminal attach`: Windows (herdrdev/herdr#4821) and any bridge runtime that ships no PTY sidecar |
-| Anonymous install/update counts | telemetry.ts + /api/telemetry in index.ts; shared/telemetry.ts; receiver in telemetry/ | one event per install and per new version, never before the app's current notice (`NOTICE_VERSION`; a file that saw an older one counts as unseen) was shown (`notice_seen`) and NOTICE_GRACE_MS has passed since, so the notice's Turn off still stops it; turning the switch off aborts a send in flight, and the state file is re-read before each decision (two servers can share it); never while the switch is off or `telemetryBlocked` (HERDR_WEB_TELEMETRY=0, DO_NOT_TRACK, CI, test env); only the real entrypoint passes a `Telemetry`, so tests and remote bridges answer 404 and send nothing. A failed send is retried at the next start, never in a loop. An event carries `changed_settings`: the last list a browser POSTed, names from `REPORTED_SETTINGS` only (anything else is a 400); telling it sends nothing by itself, and a new reported setting goes into shared/telemetry.ts and the Worker's `SETTINGS` together |
+| Anonymous install/update counts | telemetry.ts + /api/telemetry in index.ts; shared/telemetry.ts; receiver in telemetry/ | one event per install and per new version, never before the app's current notice (`NOTICE_VERSION`; a file that saw an older one counts as unseen) was shown (`notice_seen`) and NOTICE_GRACE_MS has passed since, so the notice's Turn off still stops it; turning the switch off aborts a send in flight, and the state file is re-read before each decision (two servers can share it); never while the switch is off or `telemetryBlocked` (HERDR_WEB_TELEMETRY=0, DO_NOT_TRACK, CI, test env); only the real entrypoint passes a `Telemetry`, so tests and remote bridges answer 404 and send nothing. A failed send is retried at the next start, never in a loop. An event carries `settings`: the last browser's allowlisted scalar snapshot, validated by shared/telemetry.ts on both sender and receiver. Raw custom text and account identifiers never enter it; custom choices use flags or counts. Reporting settings to the local server sends no event by itself. Expand the notice version when the disclosed collection scope changes |
 | What an update brings | release-notes.ts | CHANGELOG.md sections plus `release-summaries.json`, newest first and bounded; `## [Unreleased]` is no release, and another level-two heading belongs to the notes it sits in |
 | Agent stores beyond claude/codex/omo | gjc-runtime.ts, pi.ts, claude-store.ts, omp.ts (transcript-records.ts and skill-activity.ts parse what they find) | each store lookup answers null rather than guessing which transcript a pane writes. pi's store moves with `PI_CODING_AGENT_SESSION_DIR` or `PI_CODING_AGENT_DIR`, and a session file is not written until the first answer, so an absent path can still be the pane's. gjc on Windows is found through the PC's process table, shared between polls for PROCESS_TABLE_MS; an empty table is a failed read, not an answer |
 | Paired devices | devices.ts + /api/devices* in index.ts | six-digit code valid ten minutes, one per pairing; the device's token is stored only as a hash in devices.json (0600, written whole), and revoking also closes that device's live streams |