docs: lead with OTel-traced agent as the recommended target path - #27
Merged
Conversation
Customer-preview docs previously described OpenTelemetry trace capture as `optional`, `opt-in`, an `upgrade`, and reassured users with `new to OTel? skip it on the first run`. That framing shipped the weakest version of the eval first: with final text only, the judge cannot score tool calls, routing, sub-agent decisions, or "right-answer-for-the-wrong-reason" failures. Restructure the target docs around two clearly-labelled paths: 1. **Recommended path: callable target with OTel traces.** For 33+ frameworks Phoenix/OpenInference auto-instruments in two lines; for anything else, emit your own OTel SDK spans. This is what every flagship example uses and what the judge needs to score real behavior. 2. **Customization path: plain callable without traces.** Explicitly marked as a fallback for black-box APIs you cannot instrument or for pipeline smoke tests, and labelled `not recommended for real agents`. Files updated for consistent voice: - README.md (intro bullet, quickstart blurb, "Choose your target" table now has 3 rows and drops the "new to OTel?" callout, status section) - docs/targets/overview.md (decision tree + at-a-glance table) - docs/targets/callable.md (full restructure: OTel path first, plain-callable framed as fallback) - docs/targets/model-and-tools.md (cross-link wording) - docs/concepts.md (Execute stage YAML now includes target.trace) - docs/quickstart.md (intro + Phoenix UI block reframed: span capture happens regardless of whether `phoenix serve` runs) - docs/status-and-roadmap.md - examples/README.md, examples/pipes/README.md - AGENTS.md (so AI assistants also recommend the OTel-traced path by default) No code changes. `uv run pytest -q` -> 528 passed, 14 skipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ad model+tools with TDD framing callable.md: - restructure into Recommended (OTel-traced, 33 frameworks) and Customization (with OTel spans for unsupported frameworks; without traces for plain callable / HTTP endpoint) - explicitly document history shape (OpenAI/LiteLLM dicts with current user turn split off into 'message') with a litellm round-trip example - add a return-type table that calls out litellm/OpenAI-style returns as a middle ground for tool-call visibility without OTel - document target.endpoint (HTTPEndpointSession) alongside plain callable model-and-tools.md: - lead with test-driven prompt + toolset design — eval the prompt and toolset YAML against simulated tools before any agent code exists - present real-Python-tools as a one-line swap from the simulator - demote the bare hosted-model snippet to 'smoke' position README.md: update deep link anchor to #customization-without-traces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The callable doc claimed history contains only **prior** turns and showed a litellm round-trip that re-appended message. Tracing p2m/stages/rollout.py:682,701 and p2m/core/session.py:498-501 confirms history is the full OpenAI/LiteLLM messages list (user/assistant only) including the current user turn at history[-1]. The published example would have duplicated the latest user turn into every LiteLLM call. Fix the framing, the inline example, and the round-trip snippet to use history directly. While here, align the neurosan example and the t5_ns_safety regression fixture on trace.backend: phoenix and clarify in comments that the backend value is informational — the in-process exporter captures spans from any TracerProvider regardless of which auto-instrument or manual path produced them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lain-callable section - Add `What the judge sees, by integration path` table near top: 8 observability dimensions across plain str / LiteLLM-style / OTel paths (1/8, 4/8, 8/8). Replaces verbose prose with a structured comparison. - Reframe `Customization without traces`: introduce LiteLLM with hyperlink and `100+ model providers` one-liner; show the recover-tool-calls-without-OTel pattern as a single concise example. - Tighten `Plain Python callable` section to roughly mirror the OTel-traced section's brevity. Drop the inline value-comment example (text already conveys `current turn at history[-1]`). - Fix return-types table: collapse the duplicate-looking `LiteLLM.ModelResponse` and `p2m.core.model_client.ModelResponse` rows. They are different classes; the LiteLLM/OpenAI shape is normalized into the p2m dataclass internally (per session.py:478-482). Single row now covers both with hyperlinks to LiteLLM, OpenAI ChatCompletion, and the p2m source file. - Add hyperlinks to OpenAI messages-format spec, OpenAI ChatCompletion object spec, aiohttp. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Customer-facing docs previously used five inconsistent names for the same shape ( arget.model + arget.tools): model and tools target, hosted model with system prompt, prompt agent, prompt-shaped agents, single prompt agent. Settle on **Prompt Agent** as the canonical name across docs, READMEs, and examples READMEs. Spine of the rewrite: the distinction that matters is **who owns the tool-call loop**: - Callable target: your code runs the loop; p2m reads OTel spans. - Prompt Agent target: declared in YAML; p2m runtime owns the loop (up to 10 rounds, real Python tools or LLM-simulated tool responses). Concrete changes: - `docs/targets/overview.md`: lead with a Who owns the tool-call loop? framing section; add Loop ownership column to the at-a-glance table; rename the alternate-path section. - `docs/targets/model-and-tools.md`: H1 -> Prompt Agent Target (model + tools); reframe intro and the When to switch to the callable target section around loop ownership. - `docs/targets/callable.md`: cross-reference fix (system_prompt is consumed only by the Prompt Agent target). - `README.md`, `AGENTS.md`, `examples/README.md`, `examples/pipes/README.md`, `docs/status-and-roadmap.md`: consistency sweep. - `AGENTS.md`: add a Terminology divergence note documenting that the customer-facing name Prompt Agent target corresponds to the runtime class HostedSession (`p2m/core/session.py`); divergence is intentional and not worth renaming. Filename `docs/targets/model-and-tools.md` is intentionally preserved for URL stability. No code changes; `uv run pytest -q` -> 528 passed, 14 skipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y default Browsing to `docs/targets/` previously showed only a directory listing; GitHub auto-renders `README.md` files. Rename the entry-point doc and update the three internal references in `README.md` and `AGENTS.md`. The top-level `README.md` link target switches to the directory itself (`docs/targets/`) rather than the explicit filename, so the link will keep working if the file is reorganized again later. No content changes; `uv run pytest -q` -> 528 passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Follow-up to ec396e9: the rename commit accidentally lost the staged edits to `AGENTS.md` and `README.md` because the deleted-pathspec `git add` failed silently. Three references to the old filename are now updated to the new one (or to the directory itself). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…er story p2m calls models through LiteLLM, which supports 100+ model providers (Azure OpenAI, OpenAI, Anthropic, Bedrock, Vertex, Ollama, …). The shipped example configs happen to use `azure/gpt-5.4-mini`, so by default a customer running them as-is sets `AZURE_API_KEY` and `AZURE_API_BASE`, but that is a property of the example, not of p2m. `.env.example` and `docs/targets/callable.md` already reflect this; three other places framed Azure as a prerequisite or implied it was the only option. Three docs corrected: - `docs/quickstart.md` (Prerequisites bullet + Run-it code-block comment): rewrite to `Credentials for whichever model provider the example calls` with a hyperlink to the LiteLLM providers list and a one-liner on switching providers via `model.name`. - `README.md` (Quickstart code-block comment): same framing. - `examples/README.md` (Quickstart code-block comment): same framing. `AGENTS.md` mentions of `AZURE_API_KEY` / `AZURE_API_BASE` are placeholder-name guidance for credential safety and are intentionally left alone. No code changes; `uv run pytest -q` -> 528 passed, 14 skipped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tangym
approved these changes
May 8, 2026
Last straggler from the prompt-agent rename sweep (ec396e9). The `target.model` description in CONFIG_REFERENCE.md still said `hosted model or simple model+tools targets`; update to the canonical name and link to the customer-facing target doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tangym
approved these changes
May 8, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Restructure the customer-preview target docs around two clearly-labelled paths:
from phoenix.otel import register; register(auto_instrument=True)). For anything else, emit your own OTel SDK spans. Either way,target.tracefeeds tool calls, routing, model calls, and latency to the judge.not recommended for real agentsand reserved for black-box APIs you cannot instrument or for pipeline smoke tests.Why
Previous wording shipped the weakest version of the eval first:
Optional trace-grounded judgment/opt-in OpenTelemetry captureYou do not need to understand OpenTelemetry to startNew to OTel? Skip it on the first run; add it later when you need itOpenTelemetry trace capture is an optional upgradeWith final text only the judge cannot score tool calls, routing, sub-agent decisions, or right-answer-for-the-wrong-reason failures — exactly the failure modes adaptive eval is supposed to catch. Framing OTel as an obstacle ("new to OTel?") nudges customers away from the integration that makes the product work.
Files updated
README.mddocs/targets/overview.mddocs/targets/callable.mddocs/targets/model-and-tools.mddocs/concepts.mdtarget.tracedocs/quickstart.mdphoenix serveblock now correctly distinguishes the optional UI from the always-on span capturedocs/status-and-roadmap.mdexamples/README.md,examples/pipes/README.mdAGENTS.mdNo code changes.
Verification
uv run pytest -q-> 528 passed, 14 skipped, 0 failed.rg -i "optional.*OTel|opt-?in|new to.*[Tt]elemetry|skip.{0,15}OTel|do not need to understand"overREADME.md docs/ examples/ AGENTS.md-> no matches.Note on companion PR
This sits cleanly on top of #26 (which pins the Phoenix auto-trace instrumentors so the recommended path actually works for customers without manual
pip install). Either order of merge is fine; together they make the OTel-traced happy path land smoothly.