feat(langchain): capture document relevance score on retrieval spans

[dlowzzxx]

Description

Fixes #584

Per the OpenTelemetry Semantic Conventions for Generative AI systems (gen_ai.retrieval.documents), each retrieved document item should capture its relevance score under the score key when available from the retriever or vector search engine.

This PR adds document relevance score extraction to OpenTelemetryCallbackHandler._get_retrieval_documents in opentelemetry-instrumentation-genai-langchain.

Key Changes:

• Polymorphic Score Extraction: Extracts score by checking document attributes first (getattr(doc, "score", None) or dictionary doc.get("score")), then falling back to document metadata (metadata.get("score") / metadata.get("relevance_score")).
• Validation & Guard Rails:
  • Validates that the score is a numeric int or float.
  • Excludes bool instances (since isinstance(True, int) is True in Python).
  • Preserves valid 0 and 0.0 scores (using is not None and type checks instead of truthiness).
  • Rejects non-finite floats (NaN, +Inf, -Inf) via math.isfinite.
  • Rejects non-numeric types (e.g. strings, lists, dicts).
• Duck-Typed Document Support: Symmetrically supports both standard langchain_core.documents.Document objects and arbitrary duck-typed document objects/dictionaries with page_content or content and metadata.
• SemConv Conformance: Populates the score key alongside content and id in gen_ai.retrieval.documents JSON serialization.

Type of change

• New feature (non-breaking change which adds functionality)

How has this been tested?

• Unit tests in test_callback_handler.py covering score extraction, attribute precedence, metadata fallbacks, 0/0.0 preservation, boolean exclusion, NaN/Inf exclusion, invalid types, and duck-typed objects.
• Integration tests in test_retriever.py covering synchronous and asynchronous retrievers (invoke, ainvoke, get_relevant_documents, aget_relevant_documents) with scores.
• Full package test suite: 430 passed across all tests in instrumentation/opentelemetry-instrumentation-genai-langchain.
• Static type checking: pyright --level error (0 errors) and mypy strict type check clean.
• Lint & formatting: ruff check (0 errors) and ruff format --check clean across all files.

Checklist

• Followed the style guidelines of this project
• Changelog updated if the change requires an entry
• Unit tests added
• Documentation updated

[Copilot]

🟡 Changes recommended

The implementation and tests currently don’t cover the documented metadata["relevance_score"] fallback, so behavior diverges from the PR description and can drop valid scores.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR enhances the LangChain GenAI instrumentation to include per-document relevance scores in retrieval span content, aligning emitted gen_ai.retrieval.documents data more closely with the GenAI semantic conventions.

Changes:

• Add polymorphic score extraction and include score in retrieval document serialization for spans when content capture is enabled.
• Add/extend unit and integration test coverage to validate score extraction behavior and JSON safety (non-finite floats).
• Add a changelog fragment documenting the new behavior.

File summaries

File	Description
instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py	Adds _extract_document_score / _document_to_dict and uses them to populate retrieval documents (including score) on retrieval spans.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_callback_handler.py	Adds unit tests around score extraction and document-to-dict conversion.
instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_retriever.py	Adds end-to-end/integration assertions that retrieval spans include score where applicable and exclude invalid/non-finite scores.
instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/670.added	Changelog entry for capturing document relevance scores on retrieval spans.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Merged · refreshed 2026-09-17 05:30 UTC

Status above doesn't look right?

• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[dlowzzxx]

Added explicit
elevance_score fallback for document attributes and metadata dictionaries along with dedicated unit tests in est_callback_handler.py. Ready for review!

/dashboard route:reviewers