wizard-ci-bot · Sep 22, 2026
@@ -0,0 +1,83 @@
+---
+name: ai-observability-vercel-ai
+description: PostHog AI Observability integration for Vercel AI SDK
+metadata:
+  author: PostHog
+  version: dev
+---
+
+# PostHog AI Observability for Vercel AI SDK
+
+Wire up PostHog's AI Observability so calls made through Vercel AI SDK land in LLM Analytics as a full **session → trace → span → generation** tree — not just isolated `$ai_generation` events.
+
+## Prerequisite — vendor LLM SDK
+
+This skill instruments the LLM calls the project *already makes*. It does **not** install the vendor SDK for you.
+
+Check the project's manifest for an LLM package. The catalog is far wider than the obvious providers — 69 variants covering agent frameworks (`openai-agents`, `claude-agent-sdk`, LangGraph, CrewAI, Mastra, …) and OpenAI-compatible gateways (Groq, OpenRouter, Together, Ollama, …), which an app reaches through the `openai` package plus a `baseURL` override. `1-begin.md` carries the ordered decision rules; follow them rather than matching on the first familiar package name. If no LLM SDK is present, switch to the `manual-capture` variant — it posts `$ai_generation` events directly and works standalone.
+
+Everything else this skill needs — PostHog credentials, instrumentation packages, env vars — the skill installs and configures itself. It does **not** require a pre-existing `posthog.init(...)`. If one is already there, reuse its env-var names in `3-instrument.md`; if not, that step sets fresh values via `set_env_values`.
+
+## Steps
+
+Read every referenced file **before editing**. Then work through them in order:
+
+1. **Begin** — see `references/1-begin.md`. Pick the variant with the ordered rules (framework before provider, gateway base URL before the SDK it borrows), then read four facts from the code: the conversation, the user, the turn, and whether the app registers tools.
+2. **Install** — see `references/2-install.md`. Declare the variant's packages in the manifest — and only those. For providers and gateways that's the PostHog SDK alongside the vendor SDK, with no OpenTelemetry packages.
+3. **Instrument** — see `references/3-instrument.md`. Swap the vendor client for PostHog's wrapper, attach `$ai_session_id`, a per-turn `posthog_trace_id`, and the distinct id to every call, and capture tool runs as `$ai_span` events. This step is what turns isolated generations into a session tree.
+4. **Verify** — see `references/4-verify.md`. Describe a request the user can trigger, and grade what lands in PostHog — one session, grouped traces, right attribution — rather than what the diff contains.
+
+## Reference files
+
+- `references/1-begin.md` - Pick the variant that matches this project, then read the four facts the instrumentation needs
+- `references/2-install.md` - Declare the packages the variant needs, and no others
+- `references/3-instrument.md` - Swap in the wrapper client, then attach identity and tool spans so the calls form a session tree
+- `references/4-verify.md` - Give the user a way to trigger one turn, and grade the tree that reaches PostHog
+- `references/vercel-ai.md` - Vercel ai SDK observability installation
+- `references/node.md` - Node.js
+- `references/basics.md` - Ai observability basics
+- `references/generations.md` - Generations
+- `references/traces.md` - Traces
+- `references/sessions.md` - Sessions
+- `references/spans.md` - SPAns
+- `references/COMMANDMENTS.md` - Framework-specific rules the integration must follow
+
+The linked install page carries the exact code blocks for this variant's language. Prefer copying from there over reconstructing from memory — package names and initialization shapes change between AIO releases.
+
+## Key principles
+
+- **Environment variables.** Read `<ph_project_token>` and `<ph_client_api_host>` from env, using the framework's env-var convention. Never hardcode either value.
+- **The SDK wrapper is the default, not OpenTelemetry.** OTel makes the session tree awkward to build and maintain, so provider and gateway variants use PostHog's drop-in wrapper client. Reserve OTel for the `opentelemetry-*` variants and LlamaIndex, and never swap a framework's own tracing hook for an instrumentor. Go is the exception: it has no wrapper SDK, so model calls made from Go use the `opentelemetry-go` bridge.
+- **Minimal changes.** The wrapper swaps a client constructor and adds parameters to existing calls. Don't restructure the app, and don't wrap the setup in an init function or module globals.
+- **Match the docs.** Package names and wrapper imports change between AIO releases. The install page for this variant is the source of truth.
+- **Cardinality is what gets graded.** One `$ai_session_id` per conversation, one `posthog_trace_id` per turn, shared by every call in it. An id minted per call is worse than none — it looks instrumented and groups nothing.
+- **Tools become spans.** When the app registers tools, capture each execution as an `$ai_span` event sharing the turn's trace id — the wrapper never sees your dispatch loop. Framework variants emit these themselves; an app with no tools correctly has none.
+- **Don't touch what isn't yours.** This skill instruments LLM observability only — generations, traces, sessions, spans. Identify calls, event tracking, error tracking, and dashboards belong to the base `integration` skill — do not add or edit them here.
+
+## Emit a run record
+
+When you finish, write `.posthog-wizard-cache/.posthog-ai.json` at the project root:
+
+```json
+{ "provider": "openai", "package": "@posthog/ai", "otel_init_file": "src/instrumentation.ts" }
+```
+
+`otel_init_file` keeps its name for the report's sake, but on the wrapper path there is no OTel init — set it to the file where the wrapper client was constructed (or, on the manual path, where the capture helper lives).
+
+The `report/` step reads this file to render an AI Observability section in the setup report. If the cache directory does not exist, create it.
+
+## Framework guidelines
+
+- A missing PostHog configuration must never break the app — read keys optionally (never a required setting), guard init and capture behind their presence, and keep build and boot working with no PostHog environment set — but never silently: in development or debug builds fail loudly, using the language's idiomatic error, with the message "<VAR> variable required by PostHog is missing or un-configured, this causes events to be silently missed. This error stops appearing once <VAR> is configured" (substituting the actual variable name); production stays a no-op
+- AI Observability carve-out: this skill instruments LLM calls and is not product-analytics coverage. Do NOT add posthog.capture() events for user actions, captureException() error handlers, or a reverse proxy unless the user explicitly asks for them
+- AI Observability carve-out: only the wrapper-client and manual-capture install paths construct a PostHog client. The OTel and framework-hook paths have no client at all, so any rule in this file about the Posthog()/PostHog() constructor, exception autocapture, atexit/shutdown registration or flushing simply does not apply — never invent a client just to satisfy one
+- AI Observability carve-out: the $ai_* payload properties ($ai_input, $ai_output_choices, and the rest) intentionally carry user-generated prompt and completion text, so this file's PII rules do NOT apply to them. Those rules still govern every other property you set
+- AI Observability carve-out: the Go OpenTelemetry path uses the nested github.com/posthog/posthog-go/otel module, not the core github.com/posthog/posthog-go SDK. This file's Go install and import line does not apply, and neither do its posthog.NewWithConfig client rules: the bridge registers a span processor and builds no PostHog client
+- AI Observability carve-out: read the PostHog key and host exactly as the variant's install doc reads them. A direct os.environ["POSTHOG_API_KEY"] / process.env lookup already fails loudly and idiomatically when unset, which satisfies this file's missing-configuration rule — do NOT add a separate presence check, guard branch, or custom raise around a bootstrap that is only a few lines long
+- posthog-node is the Node.js server-side SDK package name; posthog-js is browser-only, so use posthog-node on the server instead
+- Include enableExceptionAutocapture: true in the PostHog constructor options
+- Add posthog.capture() calls in route handlers for meaningful user actions – every route that creates, updates, or deletes data should track an event with contextual properties
+- Add posthog.captureException(err, distinctId) in the application's error handler (e.g., Express error middleware, Fastify setErrorHandler, Koa app.on('error'))
+- The SDK batches events and flushes asynchronously. await flush() or await shutdown() before letting that process exit. If unsure, set flushAt 1 and flushInterval 0.
+- `posthog.capture()` enqueues synchronously and returns; the batched HTTP send happens afterwards. Treat every per-request handler as short-lived even when the framework feels like a server: Next.js / Nuxt / SvelteKit / Remix route handlers, serverless and edge functions, and Lambda are torn down per invocation before the send runs. Create the client with flushAt 1 and flushInterval 0, then await the send before returning. Always use `await posthog.flush()` for a shared/singleton client, `await posthog.shutdown()` for a per-request client. Never skip the awaited flush or risk the enqueued event being silently dropped.
+- Reverse proxy is NOT needed for server-side Node.js – only client-side JavaScript needs a proxy to avoid ad blockers
@@ -0,0 +1,84 @@
+---
+title: AI Observability Setup - Begin
+description: Pick the variant that matches this project, then read the four facts the instrumentation needs
+---
+
+Pick the variant, then read the code. Do not edit anything in this step.
+
+## Pick the variant
+
+This skill ships 69 variants. Call `load_skill_menu` with `category: "ai-observability"`. That list is the source of truth.
+
+Apply these rules in order. The first match wins. Frameworks wrap providers, and gateways look like OpenAI, so the order matters.
+
+### 1. Model calls made from Go take OpenTelemetry
+
+Go has no PostHog wrapper SDK, so a project whose model calls are in Go takes `opentelemetry-go`, whatever SDK or framework makes them. The bridge is `github.com/posthog/posthog-go/otel`, and the install doc carries the code.
+
+A `go.mod` on its own is not the signal. If the repo also has a `package.json`, `pyproject.toml`, or `requirements.txt`, find the call sites before you route. A Go service beside a Node or Python LLM app takes the rules below. If it stays unclear, use `wizard_ask`.
+
+The bridge needs Go 1.25 or newer. An older toolchain fails with `module requires go >= 1.25.0`. Say so in the report rather than raising the project's Go version yourself.
+
+### 2. A framework wins over the provider under it
+
+| Package in the manifest | Variant |
+|---|---|
+| `openai-agents` | `openai-agents` |
+| `claude-agent-sdk` | `claude-agent-sdk` |
+| `langchain`, `@langchain/core` | `langchain-{python,node}` |
+| `langgraph`, `@langchain/langgraph` | `langgraph-{python,node}` |
+| `ai` (Vercel AI SDK) | `vercel-ai` |
+| `llama-index`, `llamaindex` | `llamaindex` |
+| `crewai` | `crewai` |
+| `pyautogen`, `autogen-agentchat` | `autogen` |
+| `dspy`, `dspy-ai` | `dspy` |
+| `pydantic-ai` | `pydantic-ai` |
+| `semantic-kernel` | `semantic-kernel` |
+| `smolagents` | `smolagents` |
+| `mirascope` | `mirascope` |
+| `instructor` | `instructor-{python,node}` |
+| `litellm` | `litellm` |
+| `mastra`, `@mastra/core` | `mastra` |
+| `@google/adk` | `google-adk` |
+| `convex` | `convex` |
+
+Instrument the framework, not the provider below it. A provider instrumentor keeps the model calls and loses the agent, tool, and handoff structure.
+
+The framework wins only when it carries the app's generations. A package in the manifest is not enough. Some apps pull in `llama-index`, `langchain`, and the like for a side path only, such as a vector store or datastore provider, a retrieval helper, or an eval harness, while the real generations run through a direct provider SDK (`openai`, `anthropic`) at other call sites. Open the generation call sites before routing. If the framework wraps the generations, instrument it. If a direct provider SDK makes the primary generations and the framework is only a side path, route by that provider under rule 4 and instrument its call sites; add the framework only if it also generates. When both carry real generations, use `wizard_ask` with each as an option.
+
+### 3. An `openai` client with a base URL override means a gateway
+
+Most OpenAI-compatible providers ship no SDK. Apps call them with the `openai` package aimed at another host. Check the client constructor and `OPENAI_BASE_URL`. Common hosts are `api.groq.com`, `openrouter.ai`, `api.together.xyz`, and `localhost:11434`.
+
+Pick the variant that names the provider. The install shape matches plain OpenAI, but the provider name does not. Step 3 explains why that matters.
+
+### 4. A plain provider SDK maps to that provider
+
+`openai`, `anthropic`, `@anthropic-ai/sdk`, `google-genai`, `@google/genai`, `mistralai`, and `cohere` each have a variant. `boto3` with `bedrock-runtime` maps to `aws-bedrock`.
+
+### 5. Anything else
+
+- Several candidates and no framework: prefer the higher-level one. If it stays unclear, use `wizard_ask` with the candidates as options.
+- The app already emits its own OTel spans: `opentelemetry-{python,node}`.
+- No LLM SDK at all: `manual-capture`.
+
+Language follows the manifest. A `package.json` means Node. A `pyproject.toml` or `requirements.txt` means Python. Go call sites went to `opentelemetry-go` under rule 1. Framework variants have no language suffix.
+
+Report the variant and the reason in a `[STATUS]` line, then call `install_skill` with the full id.
+
+## Read four facts from the code
+
+The install doc holds the code. It cannot know this app. Step 3 uses these answers and nothing else.
+
+1. **Conversation.** The field that groups turns, such as `thread_id` or `conversation_id`. If the app has none, the process run is the conversation.
+2. **User.** The user id in scope at the call sites. If the app has none, the events stay anonymous. Do not invent one.
+3. **Turn.** The function that takes one question and returns one answer. It may call the model several times.
+4. **Tools.** Does the app register tools with its calls? Look for a `tools=` argument or a tool decorator. Find the loop that runs them.
+
+Note the module that builds the vendor client. Step 3 replaces that constructor. If the project already calls `posthog.init(...)` or `PostHog(...)`, reuse its env-var names and its client.
+
+Go to `2-install.md`.
+
+---
+
+**Upon completion, continue with:** [2-install.md](2-install.md)
@@ -0,0 +1,50 @@
+---
+title: AI Observability Setup - Install
+description: Declare the packages the variant needs, and no others
+---
+
+Declare the packages in the project manifest. Do not run the package manager. The build step installs them later.
+
+Read the manifest first. If a package is already there, keep its version and say so in the report. Match the style of the entries around it.
+
+## Providers and gateways
+
+Every provider and every OpenAI-compatible gateway needs the PostHog SDK next to the vendor SDK.
+
+| Runtime | Packages |
+|---|---|
+| Python | `posthog` |
+| Node | `@posthog/ai`, `posthog-node` |
+
+**Do not add OpenTelemetry packages.** The wrapper path needs none. If you reach for `opentelemetry-sdk`, `posthog[otel]`, or an `opentelemetry-instrumentation-*` package, you picked the wrong mechanism. Go back to `3-instrument.md`.
+
+The vendor SDK is already in the manifest. Do not add or upgrade it.
+
+**Watch the pinned major version.** The openai wrapper and every `opentelemetry-instrumentation-*` bridge target the modern client API (`openai>=1.x`, `client.chat.completions.create`). A project pinned to a pre-1.0 `openai` (the flat `openai.ChatCompletion` / `openai.Embedding` module API) gives them nothing to patch, and you must not upgrade it — so the instrumentation silently emits no events. Use `manual-capture` and wrap the real calls with `posthog.capture()` instead. The same holds for any provider pinned below its current client interface.
+
+Portkey also needs `portkey-ai`.
+
+## Other variants
+
+| Variant | Packages |
+|---|---|
+| Agent frameworks | The list the install doc names |
+| `opentelemetry-*`, LlamaIndex, AWS Bedrock | The OTel packages the install doc names |
+| `manual-capture` | `posthog` or `posthog-node` |
+
+AWS Bedrock has no wrapper client. It instruments the AWS SDK through OpenTelemetry, so its package list differs from every other provider.
+
+`opentelemetry-go` declares `github.com/posthog/posthog-go/otel` and the OpenTelemetry SDK modules in `go.mod`. There are no Go instrumentation libraries for provider SDKs; step 3 hand-authors the `gen_ai.*` spans instead.
+
+Go is the exception to the no-package-manager and no-lockfile rules on this page. A `go.mod` require line with no matching `go.sum` entry fails the build with `missing go.sum entry for module`, so let the tool write both files: `go get github.com/posthog/posthog-go/otel go.opentelemetry.io/otel go.opentelemetry.io/otel/sdk`, then `go mod tidy`. Do not hand-edit `go.mod` or `go.sum`.
+
+## Do not
+
+- Do not run `npm install` or `pip install`.
+- Do not edit the lockfile. On Go, `go get` and `go mod tidy` write `go.mod` and `go.sum` for you.
+- Do not upgrade the vendor SDK.
+- Do not add OpenTelemetry to a wrapper variant.
+
+---
+
+**Upon completion, continue with:** [3-instrument.md](3-instrument.md)