[wizard-ci-bot]

Automated wizard CI run

Source: wizard-pr
Trigger ID: 3f39e0e
App: openai-agents/python-travel-triage
App directory: apps/openai-agents/python-travel-triage
Workbench branch: wizard-ci-3f39e0e-openai-agents-python-travel-triage
Wizard branch: release-please--branches--main--components--wizard
Context Mill branch: main
PostHog (MCP) branch: master
Timestamp: 2026-09-23T21:36:56.023Z
Duration: 157.6s

YARA Scanner

✓ 31 tool calls scanned, 0 violations detected

No violations: ✓ 31 clean scans

[wizard-ci-bot]

Now I have all the information needed. Let me compile the evaluation.

Key findings from the research:

• instrument() accepts client, distinct_id (a callable/lambda), privacy_mode, and groups — it does not accept a properties parameter
• group_id from RunConfig does map to `` — this is correct
• distinct_id should be set via a lambda on instrument() that reads from trace.metadata, and the metadata is passed via RunConfig(trace_metadata={"posthog_distinct_id": "..."})
• The README says "on a real distinct id" — the PR provides none, and doesn't set distinct_id on instrument() or pass posthog_distinct_id in trace_metadata

---

PR Evaluation Report

Summary

This PR instruments an OpenAI Agents SDK travel triage app with PostHog AI observability via posthog.ai.openai_agents.instrument(). It correctly initializes PostHog, registers the tracing processor at module scope, sets up shutdown flushing, and uses RunConfig(group_id=...) for session grouping. However, it omits any distinct ID — neither the distinct_id callable on instrument() nor posthog_distinct_id in trace_metadata — leaving all events anonymous despite the README requiring "a real distinct id."

Files changed	Lines added	Lines removed
5	+66	-3

Confidence score: 4/5 👍

• No distinct ID configured anywhere: The instrument() call omits the distinct_id parameter (a lambda to extract user identity from trace metadata), and RunConfig has no trace_metadata with posthog_distinct_id. All events are anonymous, directly contradicting the README's requirement of "a real distinct id." [CRITICAL]
• No .env.example committed: POSTHOG_API_KEY and POSTHOG_HOST are required but no .env.example documents them. The .env itself is gitignored. [MEDIUM]

---

File changes

Filename	Score	Description
main.py	3/5	Core instrumentation added correctly — PostHog client init, instrument(), atexit shutdown, RunConfig(group_id=...) for session ID. Missing distinct_id lambda and trace_metadata.
pyproject.toml	4/5	Adds posthog dependency correctly, though unpinned.
.gitignore	5/5	Correctly ignores .env.
.posthog-wizard-cache/.posthog-ai.json	4/5	Wizard cache metadata.
posthog-ai-observability-report.md	3/5	Acknowledges missing distinct_id but doesn't remediate.

---

App sanity check ⚠️

Criteria	Result	Description
App builds and runs	Yes	No syntax errors; dependencies declared correctly
Preserves existing env vars & configs	Yes	No pre-existing PostHog; app logic untouched
No syntax or type errors	Yes	Clean Python, valid imports
Correct imports/exports	Yes	posthog.ai.openai_agents.instrument is a real module/function
Minimal, focused changes	Yes	Only AI observability instrumentation added
Pre-existing issues	None	Base app is clean

Issues

• No .env.example: POSTHOG_API_KEY and POSTHOG_HOST are read from environment but no .env.example file documents them for developers. [MEDIUM]

Other completed criteria

• Dependency version resolves — posthog latest includes posthog.ai.openai_agents
• Flushes before exit via atexit.register(posthog_client.shutdown)
• enable_exception_autocapture=True correctly set
• .gitignore prevents .env from being committed

---

PostHog implementation ⚠️

Criteria	Result	Description
PostHog SDKs installed	Yes	posthog added to pyproject.toml dependencies
PostHog client initialized	Yes	Posthog() constructed at module scope from env vars
capture()	Yes	instrument() registers a tracing processor that auto-captures , events
identify()	No	No distinct_id lambda on instrument(), no trace_metadata with posthog_distinct_id — all events are anonymous
Error tracking	Yes	enable_exception_autocapture=True
Reverse proxy	N/A	Server-only CLI

Issues

• No distinct ID — events are anonymous: The instrument() function accepts a distinct_id parameter (a callable like lambda trace: (trace.metadata or {}).get("posthog_distinct_id")) and RunConfig supports trace_metadata={"posthog_distinct_id": "user_123"}. Neither is used. The README explicitly requires "a real distinct id." Fix: add distinct_id=lambda trace: (trace.metadata or {}).get("posthog_distinct_id") to instrument() and add trace_metadata={"posthog_distinct_id": "travel-desk-user"} to RunConfig. [CRITICAL]

Other completed criteria

• Instrumentation initialized once at module scope
• Uses posthog.ai.openai_agents.instrument() — the correct, non-OTel path per README
• Session ID set via RunConfig(group_id=f"travel-triage-{uuid4()}") which maps to ``
• Session cardinality correct — one session per run, shared across all events
• No hand-authored spans — all spans come from the SDK's tracing processor
• No pre-existing PostHog code modified (additive only)
• Trace grouping correct — single Runner.run_sync call produces one trace

---

PostHog insights and events ⚠️

Filename	PostHog events	Description
main.py	, (via processor)	The Agents SDK tracing processor auto-captures: TriageAgent span, handoff span, BookingAgent span, generation events, and get_flight_price tool span. All grouped under one trace and one session.

Issues

• Anonymous events reduce observability value: Without person attribution, traces cannot be linked to users in PostHog's AI Observability dashboard. The tree structure renders correctly, but "who ran this?" is unanswerable. [CRITICAL]

Other completed criteria

• Events would render as the expected tree: session → trace → (TriageAgent span, handoff, BookingAgent span with generation + tool)
• No fabricated structure — all spans come naturally from the Agents SDK's tracing
• No credentials or secrets in event properties
• privacy_mode=False correctly captures prompts and completions

---

Reviewed by wizard workbench PR evaluator