feat(langfuse): evaluate existing traces and return ASSERT scores

[Chang Liu (changliu2)]

Customer need

Teams already using Langfuse should be able to apply ASSERT behavior specs to existing conversations without moving to a second inspection workflow or re-running the agent. This PR completes that round trip:

Langfuse traces -> ASSERT conversations -> custom judgments -> Langfuse traces and scores

What this PR adds

• import Langfuse trace-detail JSON into ordered ASSERT conversations with message, tool, retrieval, and session evidence
• inspect reconstructed conversations before spending judge tokens
• run a judge-only ASSERT evaluation against explicit permissible and impermissible behavior categories
• export completed ASSERT traces and verdict dimensions to Langfuse through its OTLP trace and public Scores APIs
• include deterministic synthetic examples for both directions, with no customer data or model call required for the bridge checks

Customer flow

1. Export a narrow Langfuse cohort or fetch it through the public API.
2. Convert and inspect the reconstructed conversations.
3. Replace the sample behavior spec and taxonomy with the application's requirements.
4. Run only the ASSERT judge stage; the agent is not run again.
5. Inspect evidence-cited verdicts locally.
6. Export the completed ASSERT-owned traces and scores to the existing Langfuse project.

Data and integrity contract

• HTTPS is required outside local loopback development, and Basic Auth is never forwarded through redirects.
• API import fails closed if any selected trace-detail request fails; it does not write a partial conversation set.
• Imported session IDs are collision-safe.
• Every score is bound to the exact inference row it judged, so stale scores cannot be attached to changed transcripts.
• Skipped and failed judgments export the trace without an invalid score and are reported separately.
• Export creates deterministic ASSERT-owned trace objects; it does not mutate the original Langfuse traces.
• Credentials, real exports, generated artifacts, and customer conversations remain uncommitted.

Validation

• 105 tests and 6 subtests passed across the Langfuse mapping, clients, importer, exporter, judge provenance, schema, and stage contracts
• deterministic fixture check: 20 traces reconstruct into 8 ordered conversations, with each user, assistant, and retrieval turn represented exactly once
• independent current-head review found no blocker

Scope boundary

Direct API import uses Langfuse's public v1 trace list/detail shape; deployments that no longer expose that shape can export through the Langfuse SDK and use file import. This PR does not add automatic online judging, Langfuse evaluation-rule registration, dashboard creation, retries, or a new dependency.

[Jake Present (jakepresent)]

The deterministic fixture passes, but the converter drops real conversation turns in a supported/default input shape.

trace_to_spans() tracks seen (role, content) counts across the whole Langfuse session, while each trace recomputes occurrence counts from zero (langfuse_to_assert.py:581-592, shared at :659-677). If a Langfuse session stores one user message per trace and the user sends the same text twice, such as two separate “yes” messages, the second trace’s user turn is treated as already seen and omitted. I reproduced this with two traces in one session, each containing only its current [{role: user, content: yes}] input: the emitted transcript roles are user, assistant, assistant instead of user, assistant, user, assistant. The README explicitly says each user message can be one Langfuse trace, so this corrupts the judge input on the advertised path.

The checked-in verifier only covers fixture traces that replay growing history, and it is not run by any current required check; this PR’s check rollup is CodeQL-only. Please fix the cross-trace deduplication and add the repeated-identical-turn case to an automated test surface that CI executes. The existing fixture verification itself is green: 20 traces become 8 conversations with the expected taxonomy.

[Chang Liu (changliu2)]

Fixed in 4e81b50. I replaced session-wide (role, content) counts with sequence-aware reconciliation: repeated system prefixes and prior-history suffix/incoming-prefix overlap are removed, while assistant outputs anchor later identical user turns as new. Added CI-gated pytest coverage for the two-yes regression, replayed full/suffix histories, reversed trace input, and deterministic ordering. verify.py remains green; the targeted regression plus 182 relevant trace/import tests pass (1 skipped). Ready for re-review.