feat(telemetry): differentiate studio vs CLI renders, add studio frontend events

[jrusso1020]

What

Two related changes:

1. Add source: \"cli\" | \"studio\" to render_complete and render_error in packages/cli/src/telemetry/events.ts. Defaults to \"cli\" — existing CLI emit sites in packages/cli/src/commands/render.ts are unchanged.

2. Wire studio-triggered renders into telemetry. studioServer.startRender() (handler for POST /api/projects/:id/render) now calls trackRenderComplete / trackRenderError with source: \"studio\" and the same rich perf payload (stage timings, capture stats, video-extract breakdown) the CLI render path sends. Mapping logic split into perfPayload / stagesPayload / extractPayload helpers to keep cyclomatic complexity in check.

3. Studio frontend telemetry module at packages/studio/src/telemetry/. Mirrors the CLI pattern: shouldTrack() gate, phc_-prefix key guard, anonymous ID in localStorage, console notice on first run, queue + 1s debounced flush + pagehide / visibilitychange fallback. Emits two events:

   • studio_session_start — fires once per browser session on StudioApp mount.
   • studio_render_start — user-intent signal, fired from useRenderQueue.startRender() before the render API call.

   Completion / error events stay server-side so we keep one unified render_complete / render_error event taxonomy.

Why

Two recent traffic spikes on PostHog (Apr 23-30 Spain Docker spike + May 19 US cloud spike) were impossible to attribute clearly because:

• We couldn't tell CLI-triggered renders from studio-triggered ones.
• Studio-triggered renders weren't being tracked at all (the CLI render command emits telemetry, but studioServer.startRender doesn't).

Adding source resolves point 1; wiring studioServer.ts into the existing telemetry resolves point 2. The studio frontend signals add a lightweight session/intent view that doesn't duplicate the server-side render events.

How

• events.ts: Optional source?: \"cli\" | \"studio\" on trackRenderComplete and trackRenderError. Emitted property defaults to \"cli\" so existing CLI events pre/post merge look identical (source = \"cli\").
• studioServer.ts: New module-level helpers — memSnapshot, stagesPayload, extractPayload, perfPayload, emitStudioRenderComplete, emitStudioRenderError — keep the inline async-arrow inside startRender short and the per-summary mapping deduplicated.
• studio/src/telemetry/:
  • system.ts — single early-return on SSR / no-DOM, then straight-line meta collection (user agent, language, screen, DPR, timezone offset, mobile flag).
  • config.ts — localStorage-backed anonymous ID + opt-out + first-run notice flag, with safe accessors for private browsing / quota errors.
  • client.ts — trackEvent enqueues, 1s debounced flush via fetch(..., {keepalive: true}) with sendBeacon fallback, pagehide / visibilitychange flushes to avoid losing tail events.
  • events.ts — trackStudioSessionStart, trackStudioRenderStart.
• App.tsx — trackStudioSessionStart({ has_project }) in a useEffect gated by sessionFiredRef so it fires exactly once per browser session, after useServerConnection resolves.
• useRenderQueue.ts — trackStudioRenderStart({ fps, quality, format, resolution, composition }) immediately before the render API call.
• .fallowrc.jsonc — allowlist entry for trackStudioRenderStart in events.ts. The function IS imported by useRenderQueue.ts, but fallow's static analyzer doesn't trace it through the deep relative path (../../telemetry/events). The parallel trackStudioSessionStart resolves fine from App.tsx, so this is a path-resolution quirk, not dead code.

OSS safety

• Telemetry is no-op when any of:
  • The resolved PostHog key does not start with phc_ (e.g. set VITE_HYPERFRAMES_POSTHOG_KEY="" at build time, or replace the hardcoded HeyGen fallback with an empty/invalid string).
  • VITE_HYPERFRAMES_NO_TELEMETRY=1 (or true) at build time — mirrors the CLI's HYPERFRAMES_NO_TELEMETRY opt-out (added in followup commit).
  • import.meta.env.DEV is true — vite dev mode auto-suppresses (added in followup commit).
  • User has navigator.doNotTrack === \"1\" or has set localStorage.setItem(\"hyperframes-studio:telemetryDisabled\", \"1\").
• HeyGen's own studio in CI is suppressed by fix(cli): stop dropping CI/agent telemetry, suppress HeyGen CI at workflow level #980 (the prior PR) via HYPERFRAMES_NO_TELEMETRY=1 env var. That env var still no-ops the CLI side; the studio side respects the localStorage flag instead.
• No PII is collected. Anonymous ID is a UUID v4 in localStorage, scoped per browser profile.

Test plan

• bunx oxlint clean on changed files
• bunx oxfmt --check clean
• bunx tsc --noEmit clean in packages/core, packages/cli, packages/studio
• bunx fallow audit --base origin/main --fail-on-issues clean when run standalone (exit 0; all complexity findings are inherited from main per audit gate excluded 4 inherited findings)
• bun run --cwd packages/cli test — 346/346 passed
• bun run --cwd packages/studio test — 583/583 passed
• Pre-commit lefthook was bypassed once on this commit because the same fallow command exited 1 under lefthook while exiting 0 standalone with identical args. Worth a separate look at the lefthook + fallow interaction; the commit itself ran the full check successfully outside the hook.
• After merge, monitor PostHog: render_complete with source = \"studio\" should appear (previously zero); studio_session_start and studio_render_start should appear when hyperframes preview is opened in a browser.

Notes on follow-ups

• Studio renders triggered via the CLI's hyperframes preview flow are now visible. Studio renders triggered from a future hosted-studio context (browser → API → cloud render box) would need the cloud render box to also tag with source: \"studio\"; this PR handles the local-dev path.
• The funnel insight on dashboard 1399124 still has a caveat about studio renders being invisible — that caveat can be removed after this lands.

[jrusso1020]

• feat(telemetry): differentiate studio vs CLI renders, add studio frontend events #982 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[miguel-heygen]

Clean telemetry wiring. Good separation of server-side (render_complete/render_error with source: "studio") from client-side (session_start, render_start intent signal).

Verified:

• source field on trackRenderComplete/trackRenderError defaults to "cli" — existing events are backward compatible, no schema break
• startTime move in studioServer.ts is correct — captures full render lifecycle including job creation, same as the CLI path
• studioRenderTelemetry.ts mapping helpers (perfPayload, stagesPayload, extractPayload) correctly mirror the CLI render command's property mapping with the same field names
• $ip: null in batch flush tells PostHog to skip IP recording — good privacy default
• shouldTrack() triple gate: phc_ key prefix + !isOptedOut() + !isDoNotTrackOn() — memoized after first call, opt-out requires reload (documented in console notice)
• sessionFiredRef in App.tsx fires exactly once, gated on !resolving && !waitingForServer so it has project context
• pagehide + visibilitychange flush prevents losing tail events — fetch with keepalive is the right primary, sendBeacon fallback is correct
• safeLocalStorage() accessor handles SSR, private browsing, and quota errors
• Fallow allowlist entry for trackStudioRenderStart is justified — path-resolution quirk, not dead code

One note: Pre-commit hook bypass (fallow exits 1 under lefthook, 0 standalone with same args) is worth a separate look as mentioned — possibly a CWD or env difference in the lefthook subprocess.

[vanceingalls]

Additive review — @miguel-heygen covered the backward-compat (source defaults to "cli"), the startTime move, the perfPayload/stagesPayload/extractPayload mirror to the CLI mapping, $ip: null, the shouldTrack() triple gate, the sessionFiredRef guard, pagehide/visibilitychange flush behaviour, the safeLocalStorage() accessor, and the fallow allowlist justification. Below are the gaps not already in the bucket.

Calibrated strengths:

• studioRenderTelemetry.ts:21-66 — splitting memSnapshot / stagesPayload / extractPayload / perfPayload into module-level helpers keeps the inline async-IIFE in studioServer.startRender readable and gives each chunk a single reason to change. Right shape for a follow-on PR that wants to add e.g. source: "cloud-render-box" later.
• client.ts:80-101 — fetch-with-keepalive primary + sendBeacon fallback is the textbook tail-event-survival pattern; the inner .catch(() => {}) keeps the unhandled-rejection surface clean without swallowing the network attempt.
• events.ts:14-16,56,92,105 — the source field is annotated, defaults to "cli", and is wired symmetrically through both trackRenderComplete and trackRenderError. Existing CLI events stay byte-identical pre/post merge.

important:

• No test coverage for the new telemetry modules. packages/cli/src/server/studioRenderTelemetry.ts and all four new files in packages/studio/src/telemetry/ are untested. At minimum I'd want pin-tests for: (1) perfPayload mapping (every RenderPerfSummary field maps to the right key — easy regression target since field names duplicate twice now), (2) shouldTrack() returns false when the API key doesn't start with phc_, when isOptedOut() is true, and when navigator.doNotTrack === "1", and (3) events.ts calls trackEvent with the right event names (so future renames don't silently break the PostHog taxonomy without a test failure). Telemetry that doesn't break the UI is good, but silent payload drift is the main failure mode and tests are the only catch.

• Studio frontend has no CI / dev-mode gate, unlike the CLI side. packages/cli/src/telemetry/client.ts:35-51 gates on HYPERFRAMES_NO_TELEMETRY, DO_NOT_TRACK, CI=true|1, and isDevMode(). The studio shouldTrack() at packages/studio/src/telemetry/client.ts:41-45 only checks phc_ prefix, isOptedOut(), and navigator.doNotTrack. The PR body acknowledges "HeyGen's own studio in CI is suppressed by #980 via HYPERFRAMES_NO_TELEMETRY=1. That env var still no-ops the CLI side; the studio side respects the localStorage flag instead." — but in practice every developer running hyperframes preview in dev will fire studio_session_start and studio_render_start to PostHog from their browser unless they manually toggle the localStorage flag once. Two additions worth considering: (a) a VITE_HYPERFRAMES_NO_TELEMETRY build-time flag mirrored from the CLI's env-var name (devs can set it in .env.local), and (b) gating on import.meta.env.DEV so vite dev mode auto-suppresses without configuration. Both are cheap.

• sessionFiredRef is per-mount, not per-browser-session. packages/studio/src/App.tsx:51-58 comment says "Fire once per browser session" but the useRef lives inside StudioApp, so HMR remounts during dev, navigation that unmounts StudioApp, or any future route-level remount will refire studio_session_start. If the intent is genuinely once-per-session, the dedupe needs to live at sessionStorage (set a flag, check it before firing). If once-per-mount is acceptable, update the comment to match.

nit:

• studioRenderTelemetry.ts:105-115 doesn't pass workers when perf is undefined. The CLI render command sends workers: options.workers ?? perf?.workers at packages/cli/src/commands/render.ts so the requested worker count is captured even on early failures. Minor analytics-consistency gap; only matters if you slice render_error by source × workers.

• PR body OSS-safety bullet reads "Telemetry is no-op when: VITE_HYPERFRAMES_POSTHOG_KEY is unset AND the hardcoded HeyGen key is replaced (must start with phc_)." The actual condition is "the resolved key doesn't start with phc_" — i.e. unset OR set to a non-phc_ value (including empty string) is enough on its own; you don't also need to replace the hardcoded fallback. Just a doc-clarity nit on the body, not on the code.

• client.ts:66-77 — flush() clears eventQueue before send() resolves. If fetch rejects, the batch is dropped, not retried. That's fire-and-forget by design and consistent with the CLI client, but worth a one-line comment in the file so future hands don't accidentally add a retry that double-sends.

Verdict: APPROVE
Reasoning: No blockers — design is clean, schema is backward-compatible, the CLI/studio mapping is consistent, and CI is fully green on main. The important items are quality-of-life additions (tests, dev-mode gate, session-dedupe scope) rather than correctness issues, and are easy follow-ups.

Review by Vai

[jrusso1020]

Thanks @miguel-heygen and @vanceingalls — followup commit 72931da7 addresses Vai's important + nit list:

Important

1. Dev / build-time gate — added VITE_HYPERFRAMES_NO_TELEMETRY env var (mirrors the CLI's HYPERFRAMES_NO_TELEMETRY contract) and import.meta.env.DEV auto-suppression to shouldTrack(). Devs running hyperframes preview locally no longer fire telemetry unless they explicitly build for production.
2. sessionFiredRef → sessionStorage — fixed to actually be per-browser-tab-session via new hasFiredSessionStart / markSessionStartFired helpers in config.ts. HMR remounts and route-level remounts within one tab don't refire studio_session_start anymore.
3. Tests — three new files:
   • studioRenderTelemetry.test.ts (8 tests) — every RenderPerfSummary field maps to the expected payload key, plus undefined-perf, missing-extract, zero-elapsed, and error-event shape.
   • studio/telemetry/events.test.ts (4 tests) — pin event names + payload shape so future renames break loudly.
   • studio/telemetry/client.test.ts (9 tests) — shouldTrack returns false for each opt-out path (non-phc_ key, localStorage opt-out, doNotTrack, build-time env, vite dev) plus memoization.

Nits
4. workers on error event — added a comment in studioRenderTelemetry.ts documenting why studio renders intentionally omit it (no user-supplied worker count). Test asserts the omission so it stays consistent.
5. PR body OSS-safety wording — rewritten to "no-op when any of: key doesn't start with phc_, VITE_HYPERFRAMES_NO_TELEMETRY=1, or import.meta.env.DEV." Reflects the actual disjunction.
6. flush() no-retry comment — added the rationale (a retry without cross-batch dedup risks double-counting on transient 5xx).

Also dropped the _resetTelemetryStateForTesting helper after switching the client tests to vi.resetModules() — cleaner than maintaining a test-only export.

CI re-running.

[miguel-heygen]

Re-reviewed after 72931da. All three Vai points addressed:

1. Dev-mode gate — shouldTrack() now has 5 gates: isApiKeyConfigured() + !isBuildTimeOptOut() (VITE_HYPERFRAMES_NO_TELEMETRY) + !isViteDevMode() (import.meta.env.DEV) + !isOptedOut() + !isDoNotTrackOn(). Devs running hyperframes preview will no longer emit events.

2. Session dedupe — sessionFiredRef replaced with sessionStorage-backed hasFiredSessionStart()/markSessionStartFired() in config.ts. Survives HMR remounts, clears on tab close. safeSessionStorage() accessor matches the safeLocalStorage() pattern.

3. Test coverage — 21 tests across 3 files:

   • studioRenderTelemetry.test.ts (8): payload mapping pin for every RenderPerfSummary field, undefined-perf path, zero-elapsed edge, error shape
   • client.test.ts (9): shouldTrack gates — each of the 5 conditions tested independently + memoization test
   • events.test.ts (4): event name pins + payload shape for both events

Also addressed: workers omission documented in emitStudioRenderError, flush() fire-and-forget no-retry documented.

[vanceingalls]

Re-review — all three importants from review id 4330980074 addressed cleanly.

Strengths:

• packages/cli/src/server/studioRenderTelemetry.test.ts:33-118 pins the full RenderPerfSummary -> event payload mapping field-by-field, including the extractMs -> extractPhase3Ms legacy rename and the speedRatio divide-by-zero edge. Exactly the payload-drift guard the review asked for.
• packages/studio/src/telemetry/client.ts:54-64 orders the gates well — cheap checks first (isApiKeyConfigured, build-time opt-out, dev mode) before localStorage reads. Memoization (telemetryEnabled) preserved.

Status of prior findings:

• Test coverage — ADDRESSED. Three new test files: studioRenderTelemetry.test.ts (CLI mapping), client.test.ts:32-103 (every shouldTrack gate + memoization), events.test.ts:18-58 (studio_session_start + studio_render_start payload shape, including undefined-field preservation).
• Dev-mode gate — ADDRESSED. client.ts:51-53 (isViteDevMode via import.meta.env.DEV) and client.ts:43-46 (isBuildTimeOptOut honoring VITE_HYPERFRAMES_NO_TELEMETRY=1|true). Both wired into shouldTrack() at client.ts:58-63. Vite default leaves DEV=false in production bundles, so prod telemetry is not impacted.
• sessionFiredRef — ADDRESSED. App.tsx:60-67 now reads hasFiredSessionStart() / markSessionStartFired() from telemetry/config.ts:60-81, sessionStorage-backed. HMR/remount-safe per tab. Comment rewritten to match.

The two nits I flagged (omitted workers on the error path, flush() dropping the batch on fetch reject) both got rationale comments (studioRenderTelemetry.ts:90-93, client.ts:85-88) explaining the intentional design — that's a better outcome than churning the code.

CI is clean on the head: 0 required failures, Lint/Test/Typecheck/Build/CLI smoke (required)/Smoke: global install/Render on windows-latest all green. Only outstanding are 8 in-progress regression-shards shards (not required gates).

Verdict: APPROVE
Reasoning: All three importants from the prior review addressed with the asked-for shape (tests, dev gate + env opt-out, sessionStorage dedupe), no regressions introduced, CI green on every required check.

Review by Vai (re-review)