[wizard-ci-bot]

Automated wizard CI run

Source: wizard-pr
Trigger ID: 69eeefc
App: vercel-ai/nextjs-support-chat
App directory: apps/vercel-ai/nextjs-support-chat
Workbench branch: wizard-ci-69eeefc-vercel-ai-nextjs-support-chat
Wizard branch: posthog/functional-a1-min
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-09-22T16:02:22.966Z
Duration: 356.7s

YARA Scanner

✓ 76 tool calls scanned, 0 violations detected

No violations: ✓ 76 clean scans

[wizard-ci-bot]

Now I have everything I need to produce the evaluation.

---

PR Evaluation Report

Summary

This PR instruments a Next.js + Vercel AI SDK support chat app for PostHog AI observability. It uses the @posthog/ai withTracing model wrapper approach instead of the OTel + PostHogSpanProcessor + experimental_telemetry approach that the README specifies. While generations and identity are correctly wired, the chosen instrumentation path cannot capture tool call spans, resulting in an incomplete trace tree.

Files changed	Lines added	Lines removed
7	+131	-19

Confidence score: 4/5 👍

• Wrong instrumentation path: The README explicitly requires OTel SDK with PostHogSpanProcessor in instrumentation.ts and experimental_telemetry: { isEnabled: true } on the generateText call. The PR uses withTracing model wrapper instead, which is a valid PostHog path but does not match the app's expected structure. [CRITICAL]
• Missing tool call spans: The withTracing wrapper only captures model-level calls (generations). Tool execution (the lookupOrder span) is captured automatically by Vercel AI SDK's telemetry system (experimental_telemetry), not by model wrappers. The expected generation → span(lookupOrder) → generation tree is incomplete. [CRITICAL]
• No .env.example: The PR introduces POSTHOG_API_KEY and POSTHOG_HOST environment variables used in instrumentation.ts, but no .env.example file is created to document them. The .env file is gitignored (correctly) but undocumented for other developers. [MEDIUM]

---

File changes

Filename	Score	Description
app/api/chat/route.ts	3/5	Wraps model with withTracing, adds per-request identity. Wrong approach per README (should use experimental_telemetry).
instrumentation.ts	3/5	Creates PostHog Node client at module scope. Should be OTel SDK + PostHogSpanProcessor per README.
package.json	4/5	Adds @posthog/ai and posthog-node correctly.
tsconfig.json	2/5	Pure formatting changes with no functional impact. Unnecessary diff noise.
.gitignore	5/5	Correctly excludes .env from version control.
.posthog-wizard-cache/.posthog-ai.json	3/5	Cache file correctly notes provider and package but says otel_init_file: instrumentation.ts even though no OTel is set up.
posthog-ai-observability-report.md	3/5	Accurately describes what was done but doesn't acknowledge the deviation from the expected OTel approach.

---

App sanity check ⚠️

Criteria	Result	Description
App builds and runs	Yes	Report confirms npm run typecheck and npm run build pass.
Preserves existing env vars & configs	Yes	Existing app behavior, response shape, and tool implementation are preserved.
No syntax or type errors	Yes	TypeScript compiles successfully.
Correct imports/exports	Yes	@posthog/ai/vercel withTracing and posthog-node PostHog are real exports. @/instrumentation resolves via path alias.
Minimal, focused changes	No	tsconfig.json reformatting is unnecessary scope creep.
Pre-existing issues	None	Base app is clean.

Issues

• No .env.example: POSTHOG_API_KEY and POSTHOG_HOST are required but not documented in an .env.example file. The .env exists locally but is gitignored. New developers would not know which variables are needed. [MEDIUM]
• Unnecessary tsconfig formatting: tsconfig.json changes are purely cosmetic reformatting with zero functional impact. [LOW]

Other completed criteria

• No credentials are hardcoded in source code; .env is properly gitignored
• Dependency versions (@posthog/ai ^8.13.1, posthog-node ^5.52.5) are plausible
• PostHog client is flushed before handler returns via finally block
• Tool definition and lib/orders.ts are untouched

---

PostHog implementation ❌

Criteria	Result	Description
PostHog SDKs installed	Yes	@posthog/ai and posthog-node added to dependencies.
PostHog client initialized	Yes	posthog-node client created at module scope in instrumentation.ts with flushAt: 1, flushInterval: 0.
capture()	Yes	withTracing implicitly captures `` events on each model call.
identify()	N/A	Server-side; identity via posthogDistinctId per call.
Error tracking	Yes	enableExceptionAutocapture: true on the PostHog Node client.
Reverse proxy	N/A	Server-only app.

Issues

• Wrong instrumentation path: The README requires instrumentation.ts to start the OTel SDK with PostHogSpanProcessor and experimental_telemetry: { isEnabled: true } on the generateText call. The PR instead uses withTracing model wrapper, which is a fundamentally different approach that cannot capture Vercel AI SDK's built-in telemetry spans (tool calls, step metadata). [CRITICAL]
• Missing tool call spans: withTracing wraps only the model's doGenerate/doStream calls. The lookupOrder tool execution is a Vercel AI SDK-level step that only emits OTel spans when experimental_telemetry is enabled. Without it, the tool call span is silently absent from the trace tree. [CRITICAL]

Other completed criteria

• PostHog client initialized once at module scope, not per-request
• posthogDistinctId uses the request's userId (per-request, not process-global)
• `` uses the correct property key
• Session ID derived from threadId — same thread yields same session across requests
• posthogTraceId shared UUID per request groups all generations into one trace
• No hand-authored spans around helper functions
• No pre-existing PostHog code was modified

---

PostHog insights and events ⚠️

Filename	PostHog events	Description
app/api/chat/route.ts	``	Each model call (initial tool_use + final answer) produces a generation event via withTracing. Tool span (lookupOrder) is NOT captured.
instrumentation.ts	Client initialization	PostHog Node client created; no OTel TracerProvider or SpanProcessor configured.

Issues

• Incomplete trace tree: The README expects generation → span(lookupOrder) → generation per request. The PR only produces generation → generation — the tool call span is missing because withTracing does not hook into Vercel AI SDK's step-level telemetry. [CRITICAL]
• Session and trace grouping is correct: Despite the wrong path, the identity wiring (distinct ID, trace ID, session ID per request) is correct and would produce proper grouping for the events that ARE emitted. [LOW]

Other completed criteria

• Person attribution uses the request's userId, not a hardcoded value
• Session ID derives from the app's natural threadId concept
• No fabricated structure — session and trace map to real app concepts
• No credentials or secrets placed in event properties

---

Reviewed by wizard workbench PR evaluator