Repository navigation
feat(console): add PostHog monitoring and workflow analytics - #3700
Open
AbigailDeng wants to merge 12 commits into
Open
AbigailDeng wants to merge 12 commits into
AbigailDeng wants to merge 12 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. @@ Coverage Diff @@
## dev #3700 +/- ##
==========================================
- Coverage 83.43% 83.41% -0.02%
==========================================
Files 1685 1685
Lines 193217 193217
Branches 23239 23239
==========================================
- Hits 161201 161171 -30
- Misses 21204 21236 +32
+ Partials 10812 10810 -2
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
AbigailDeng
marked this pull request as ready for review
October 11, 2026 05:32
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and resulting behavior
Add optional PostHog monitoring and product analytics to Aevatar Console. Collection is disabled when build-time key/host are absent, and analytics failures cannot change application or request behavior. All source changes are inside
apps/aevatar-console-web/.page_ttfbwith numericttfbMs, navigation type, metric ID and sanitized original document URL. SPA navigation preserves document attribution; bfcache restores are excluded from network-TTFB analysis. This measures document response timing, not editor readiness, API latency or model first-token time.api_requestfrom the shared fetch transport with sanitized URL, method, status, duration, success and a bounded error category. Duration ends at response headers or rejection. The wrapper preserves response/rejection identity and never consumes bodies or waits for an SSE stream to finish. No new dependency is needed for API instrumentation.long_taskwith task duration, blocking time above 50 ms, start time, and matching sanitized URL/path. A single native observer usesbuffered: trueto include startup work completed before Umi initializes analytics. Early entries use the original document URL; later entries follow the current SPA route at delivery. No raw performance entries or iframe attribution are sent. Summed blocking time is not Lighthouse TBT.RUN_STARTED; the first realSTEP_STARTEDnow confirms execution in that case.start_sourcedistinguishesrun_started, exact-runactivity, andfirst_step. First-step duration excludes the wait before the first observed step; earlier authoritative start timestamps can restore that interval before completion. HTTP acceptance, clicks andaevatar.run.contextalone never count as execution. Starts/terminals are deduplicated, and normal EOF/navigation are excluded from disconnects. Each invocation now generates one client-onlyinvocationIdreused by start/finish, alongside the editor's explicitscopeId. A crypto failure falls back to a local time/random/sequence key; these analytics fields do not enter execution requests or replace backend identities.workflow_timeoutandsse_reconnectremain reserved types without producers: current APIs expose neither authoritative timeoutMs nor a safe POST-SSE resume contract. No guessed events or backend changes are included.Replay masks inputs, text and content attributes, blocks the code editor, and excludes console logs and request/response bodies/headers. URLs are scrubbed of query/hash and recognized credentials. Identify uses the existing opaque NyxID user ID and resets on logout/account change. Arbitrary exception text cannot be guaranteed free of every secret.
PostHog configuration and documentation
docs/posthog.mddocuments environment variables, privacy, event properties, metric semantics, direct PostHog UI configuration and verification. It defines the start-cohort execution count, failure rate and terminal coverage, plus the missing product/data prerequisites for cross-scope Workflow reuse and an honest ROI estimate. There are no dashboard-management scripts.The US Cloud project Aevatar Console (657961) and Workflow health dashboard are configured. Production workflow/error insights filter
app=aevatar-console,environment=production. Enabled hourly alerts cover success <90%, SSE disconnect >5%, and affected-session exception rate >3x the prior-day baseline with at least 10 exceptions. Workflow alerts guard idle windows. All three email schedules were read back; actual email receipt and production ingestion remain unverified. Native crash-free measures sessions, not users.The SQL page tables use a fixed last-30-days window and do not follow dashboard date overrides. Duration analysis should use
workflow_finished.start_sourceto distinguish complete-run from first-step timing. Existing Workflow health alerts retain the documented event-window start/finish ratios; the product metrics use the correlated cohort below.Product metrics identify a Workflow by
(scopeId, workflowId)and an invocation by(scopeId, workflowId, invocationId). They select starts inside the reporting window and match their observed terminals, deduplicating both sides:Zero-denominator percentages are
NULL/ unavailable. Events missing correlation fields are shown as legacy counts and excluded from exact cohort calculations.start_sourcecan change between start and finish after a timing-baseline upgrade, so it is not the correlation key. Time savings / ROI remains unavailable until per-Workflow manual-effort baselines and remaining human effort are supplied; execution latency is not substituted for labor time. These metrics cover browser-observed Console invocations, not schedules, channels or all backend runs.Real browser and Cloud verification
Reused existing Chrome with normal NyxID SSO and the local frontend at
127.0.0.1:5201. Independently loaded Workflows, Activity, Channels, Settings and two real workflow editors, waiting for visible content. Exercised search, Canvas/YAML, zoom/Fit View and existing run inputs. No artificial blocking loop or manual event injection was used for these samples.The TTFB verification snapshot from PostHog Cloud on 2026-10-12 confirmed 33 real development TTFB samples, up from 15. At that snapshot the page table showed:
For the sampling window beginning 2026-10-12 02:22:49 UTC, Cloud also returned 85 api_request, 38 long_task, and 18 page_ttfb, all development. The saved TTFB table and its dashboard membership were read back, and the table was visually checked.
Seven genuine workflow invocations were observed across the browser verification: six successful executions of an existing four-node report workflow with no external tools, and one failed document-reading run with
NYXID_PROXY_SERVICE_SCOPE_FORBIDDEN. The failed document workflow was not retried and no permission configuration was changed. Two pre-correlation report executions proved the missing-RUN_STARTEDfix with exactly one start and finish each, nodeCount=4, durations 11208 ms and 11751 ms, andstart_source=first_step.After adding
scopeId/invocationId, two further safe report runs were verified in Cloud as two distinct correlated starts, two successes, zero failures and 100% known-terminal coverage, allenvironment=development. The two older starts are reported separately as legacy events rather than retroactively paired. No events were manually injected or backfilled. There are no correlated production samples yet, so production failure rate is unavailable, not 0%. This small development cohort proves delivery and query association, not a production reliability estimate.The product dashboard and saved reuse insight were read back with the corrected cross-scope definition: reuse is unavailable and its percentage is NULL. The two development executions verify start/finish correlation only. They do not prove another user copied or used the Workflow.
Earlier real-route verification also returned native LCP/INP/CLS for editor, Activity, Channels and Settings; Workflows returned LCP/CLS. Masked Cloud replay was visually inspected. Synthetic contract/alert tests are separate from the real-route evidence. Development document timing is not production/CDN performance, and this small sample does not establish representative production percentiles. An intermittent local proxy read failure recovered with the page's Retry action.
Latest dev merge verification
Merged
origin/devat481eb042d3f6fd1c7d66e564bd2c74e0e7204649. Resolved the single conflict insrc/app.tsxby retaining PostHog initialization/user synchronization and the newConsoleLocaleGate, while preserving dev's lazy LiveOps imports. Reviewed the auto-merged PostHog configuration, Umi stable-polyfill patch/dependencies and workflow-editor publication snapshot integration; normalized editor import ordering. No new feature behavior was introduced for the conflict resolution.pnpm --dir apps/aevatar-console-web install --frozen-lockfile— passed; merged lockfile accepted and dev's Umi patch installed.pnpm --dir apps/aevatar-console-web exec jest --runInBand --runTestsByPath src/app.layout.test.ts src/shared/i18n/ConsoleLocaleGate.test.tsx src/shared/i18n/createLocaleCatalogLoader.test.ts src/shared/analytics/posthog.test.ts src/shared/analytics/longTasks.test.ts src/pages/workflow-activity-vnext/hooks/useWorkflowEditor.telemetry.test.tsx src/pages/workflow-activity-vnext/workflows/WorkflowEditorDeferredPanels.test.tsx— 7 suites / 44 tests passed. The app entry has broad dependency fan-out, so direct owning tests were selected instead of expanding through the entry module.pnpm --dir apps/aevatar-console-web exec jest --runInBand --runTestsByPath src/pages/workflow-activity-vnext/index.test.tsx --testNamePattern 'keeps a live transport failure visible ahead of running Activity'— 1 focused regression passed; 132 unrelated cases skipped.pnpm --dir apps/aevatar-console-web exec biome check src/app.tsx config/config.ts package.json src/pages/workflow-activity-vnext/workflows/WorkflowEditorPage.tsx src/pages/workflow-activity-vnext/index.test.tsx— 5 files passed. Initial check identified import ordering in the auto-merged editor; corrected withpnpm --dir apps/aevatar-console-web exec biome check --write src/pages/workflow-activity-vnext/workflows/WorkflowEditorPage.tsx.PATH=/opt/homebrew/opt/python@3.12/libexec/bin:$PATH bash tools/ci/test_stability_guards.sh— passed.git diff --cached --check— passed; no unmerged index entries remain. Independent merge review found no blocker.Full frontend tests, typecheck and production build remain delegated to GitHub CI; no full local validation was run.
Focused local verification
Documentation-only cross-scope reuse clarification:
git diff --check -- apps/aevatar-console-web/docs/posthog.mdpassed. Inspected the current Console copy-reference, creation/template and run-fork paths and their contracts; no cross-scope user-Workflow copy/provenance feature exists. PostHog title, description, NULL result and dashboard text were verified through the existing Chrome UI. No runtime changes or new dependencies; prior focused tests below remain applicable and were not rerun for this documentation/configuration correction.Workflow correlation and stream verification:
python3 /Users/abigaildeng/.codex/skills/frontend-incremental-pr/scripts/frontend_change_scope.py --repo . --base origin/dev— inspected full PR scope; repeated with--base 32dbb6e04f1e2b9355f39ac4131eebb576bd07beto isolate the current correlation increment.pnpm --dir apps/aevatar-console-web exec jest --listTests --findRelatedTests src/shared/analytics/workflowTelemetry.ts src/pages/workflow-activity-vnext/hooks/useWorkflowEditor.ts— selected 6 files including 4 unrelated Channel suites; rejected under the fan-out guard and used the explicit owning suites below. The shared event type was not used to expand test scope.pnpm --dir apps/aevatar-console-web exec jest --runInBand --runTestsByPath src/pages/workflow-activity-vnext/hooks/useWorkflowEditor.telemetry.test.tsx src/shared/analytics/posthog.test.ts src/shared/analytics/longTasks.test.ts— 3 suites / 25 tests passed. Protects first-step starts, real success/failed completion, accepted-only rejection, deduplication, timing-baseline recovery, observer isolation and matching invocation IDs. Two new integration cases verify distinct IDs across repeated runs and the same start/finish ID with native crypto available or throwing; both calls still complete.pnpm --dir apps/aevatar-console-web exec biome check src/shared/analytics/workflowTelemetry.ts src/shared/analytics/types.ts src/shared/analytics/posthog.test.ts src/shared/analytics/longTasks.test.ts src/pages/workflow-activity-vnext/hooks/useWorkflowEditor.ts src/pages/workflow-activity-vnext/hooks/useWorkflowEditor.telemetry.test.tsx— 6 files passed. Initial check found formatting only in the new test;pnpm --dir apps/aevatar-console-web exec biome check --write src/pages/workflow-activity-vnext/hooks/useWorkflowEditor.telemetry.test.tsxcorrected it before the successful check.PATH="/opt/homebrew/opt/python@3.12/libexec/bin:$PATH" bash tools/ci/test_stability_guards.sh— passed, including 9 Python tests. Initial system Python 3.9 could not parse the repository's existingPath | Noneannotation; rerunning with the installed Python 3.12 passed without changing the guard.bash tools/docs/lint.sh— 94 files passed.python3 apps/aevatar-console-web/docs/design-baselines/workflow-activity-vnext/verify-baseline.py— passed declared hashes, deterministic output and frame inventories.git diff --check— passed. Independent reviews of the stream-start and invocation-correlation changes found no blocker; actual Cloud ingestion and the new correlated development cohort were also verified in the browser.Existing focused transport and startup verification retained:
pnpm --dir apps/aevatar-console-web exec jest --runInBand --runTestsByPath src/shared/analytics/longTasks.test.ts src/shared/analytics/posthog.test.ts src/shared/api/http/fetch.test.ts— 3 suites / 23 tests passed for Long Task startup and API isolation.pnpm --dir apps/aevatar-console-web exec jest --runInBand --runTestsByPath src/shared/api/http/fetch.test.ts— 8 tests passed for timing, HTTP/network/abort/timeout classification, unconsumed SSE bodies and unchanged response/rejection identity.pnpm --dir apps/aevatar-console-web exec biome check src/shared/api/http/fetch.ts src/shared/api/http/fetch.test.ts src/shared/analytics/types.ts src/shared/auth/fetch.ts— 4 files passed.pnpm --dir apps/aevatar-console-web exec biome lint src/shared/auth/backend.ts src/pages/chat/nyxIdServiceApi.ts— 2 files passed.pnpm --dir apps/aevatar-console-web exec biome check config/config.ts src/shared/analytics/posthog.ts src/shared/analytics/types.ts src/shared/analytics/posthog.test.ts package.json— 5 files passed for TTFB/config wiring; served-bundle inspection confirmed configuration serialization and observer presence without exposing values.The full frontend suite, typecheck and production build are delegated to GitHub CI under the personal incremental-validation policy. No reliable repository-native affected typecheck exists. No full local E2E suite was run.
Design baseline
apps/aevatar-console-web/docs/design-baselines/workflow-activity-vnext/aevatar-workflow-activity-vnext.excalidraw30e74d7b410ae72c4c91432355436679033679c54c10b1702908435b001577deapps/aevatar-console-web/docs/superpowers/specs/2026-08-04-workflow-activity-vnext-design.mdapps/aevatar-console-web/docs/superpowers/specs/2026-08-04-workflow-activity-vnext-user-paths.md