fix(otel): record full error message on standard exception event in otel v2#30380
Conversation
…tel v2 The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead.
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR fixes truncation of OTel v2 error messages by adding a standard
Confidence Score: 5/5Additive-only change to the OTel v2 error path; no existing attributes or status fields are removed, and the new event is skipped entirely on the success path. The change is narrowly scoped to finish_span's error branch: it adds one span.add_event() call using standard OTel semconv constants and leaves every existing attribute and status field intact. The two new tests exercise both the error and success code paths with an in-memory exporter and make no real network calls. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/integrations/otel/emitter.py | Adds a standard OTel exception span event carrying exception.type and exception.message on the error path; refactors two repeated expressions into named variables. No existing behaviour changed. |
| litellm/integrations/otel/model/semconv.py | Adds ExceptionEvent semconv constant class (NAME, TYPE, MESSAGE), matching standard OTel exception.* attribute keys. Clean additive change. |
| tests/test_litellm/integrations/otel/test_otel_v2_components.py | Adds two new regression tests and two helper functions. Tests use the in-memory exporter only (no real network calls), covering both the long-message survival path and the no-event-on-success path. |
Reviews (2): Last reviewed commit: "fix(otel): record full error message on ..." | Re-trigger Greptile
3b84150
into
litellm_internal_staging
…tel v2 (BerriAI#30380) The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com>
…tel v2 (BerriAI#30380) The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com>
…tel v2 (BerriAI#30380) The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com>
…tel v2 (#30380) The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit 3b84150)
…tel v2 (BerriAI#30380) The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com>
….2) (#327) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.89.1` → `v1.89.2` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.89.2`](https://github.com/BerriAI/litellm/releases/tag/v1.89.2) [Compare Source](BerriAI/litellm@v1.89.2...v1.89.2) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.2/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30703](BerriAI/litellm#30703) - chore(release): backport [#​30380](BerriAI/litellm#30380), [#​30503](BerriAI/litellm#30503), [#​30558](BerriAI/litellm#30558), [#​30130](BerriAI/litellm#30130), [#​30588](BerriAI/litellm#30588), [#​30495](BerriAI/litellm#30495), [#​30690](BerriAI/litellm#30690) to stable/1.89.x and cut 1.89.2 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30681](BerriAI/litellm#30681) **Full Changelog**: <BerriAI/litellm@v1.89.1...v1.89.2> ### [`v1.89.2`](https://github.com/BerriAI/litellm/releases/tag/v1.89.2) [Compare Source](BerriAI/litellm@v1.89.1...v1.89.2) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.2/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30703](BerriAI/litellm#30703) - chore(release): backport [#​30380](BerriAI/litellm#30380), [#​30503](BerriAI/litellm#30503), [#​30558](BerriAI/litellm#30558), [#​30130](BerriAI/litellm#30130), [#​30588](BerriAI/litellm#30588), [#​30495](BerriAI/litellm#30495), [#​30690](BerriAI/litellm#30690) to stable/1.89.x and cut 1.89.2 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30681](BerriAI/litellm#30681) **Full Changelog**: <BerriAI/litellm@v1.89.1...v1.89.2> </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMjQuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIyNC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19--> Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/327
…to v1.89.2 (#206) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [https://github.com/BerriAI/litellm.git](https://github.com/BerriAI/litellm) | patch | `v1.89.1` → `v1.89.2` | --- ### Release Notes <details> <summary>BerriAI/litellm (https://github.com/BerriAI/litellm.git)</summary> ### [`v1.89.2`](https://github.com/BerriAI/litellm/releases/tag/v1.89.2) [Compare Source](BerriAI/litellm@v1.89.1...v1.89.2) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.2/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30703](BerriAI/litellm#30703) - chore(release): backport [#​30380](BerriAI/litellm#30380), [#​30503](BerriAI/litellm#30503), [#​30558](BerriAI/litellm#30558), [#​30130](BerriAI/litellm#30130), [#​30588](BerriAI/litellm#30588), [#​30495](BerriAI/litellm#30495), [#​30690](BerriAI/litellm#30690) to stable/1.89.x and cut 1.89.2 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30681](BerriAI/litellm#30681) **Full Changelog**: <BerriAI/litellm@v1.89.1...v1.89.2> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMjAuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIyMC4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=--> Co-authored-by: Renovate Bot <renovate@bhamm-lab.com> Reviewed-on: https://codeberg.org/blake-hamm/bhamm-lab/pulls/206
….2) (#143) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.89.1` → `v1.89.2` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.89.2`](https://github.com/BerriAI/litellm/releases/tag/v1.89.2) [Compare Source](BerriAI/litellm@v1.89.2...v1.89.2) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.2/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30703](BerriAI/litellm#30703) - chore(release): backport [#​30380](BerriAI/litellm#30380), [#​30503](BerriAI/litellm#30503), [#​30558](BerriAI/litellm#30558), [#​30130](BerriAI/litellm#30130), [#​30588](BerriAI/litellm#30588), [#​30495](BerriAI/litellm#30495), [#​30690](BerriAI/litellm#30690) to stable/1.89.x and cut 1.89.2 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30681](BerriAI/litellm#30681) **Full Changelog**: <BerriAI/litellm@v1.89.1...v1.89.2> ### [`v1.89.2`](https://github.com/BerriAI/litellm/releases/tag/v1.89.2) [Compare Source](BerriAI/litellm@v1.89.1...v1.89.2) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.2/cosign.pub \ ghcr.io/berriai/litellm:v1.89.2 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(ui): rebuild ui by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30703](BerriAI/litellm#30703) - chore(release): backport [#​30380](BerriAI/litellm#30380), [#​30503](BerriAI/litellm#30503), [#​30558](BerriAI/litellm#30558), [#​30130](BerriAI/litellm#30130), [#​30588](BerriAI/litellm#30588), [#​30495](BerriAI/litellm#30495), [#​30690](BerriAI/litellm#30690) to stable/1.89.x and cut 1.89.2 by [@​yuneng-berri](https://github.com/yuneng-berri) in [#​30681](BerriAI/litellm#30681) **Full Changelog**: <BerriAI/litellm@v1.89.1...v1.89.2> </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yMzIuMCIsInVwZGF0ZWRJblZlciI6IjQzLjIzMi4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL3BhdGNoIl19--> Reviewed-on: https://forgejo.hayden.moe/hayden/phoebe/pulls/143
…tel v2 (BerriAI#30380) The v2 span engine only stamped error.type and stuffed the message into the span status description; it never recorded the standard OTel exception event. Backends that dynamic-map unknown string fields (e.g. Elasticsearch) index the message as a keyword capped at ignore_above:1024, truncating it. Emit the full message under the recognized exception.message semconv field via a span event so it is mapped as full text instead. Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit 3b84150)
Relevant issues
Error messages emitted by the OpenTelemetry v2 integration are truncated downstream. Reported via EAIP-2334; the root cause analysis there is that the backend (Elasticsearch) applies a dynamic index field-type mapping that converts the error message field from
stringtokeyword, andkeyworddefaults toignore_above: 1024, so anything past 1024 chars is droppedLinear ticket
Pre-Submission checklist
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewScreenshots / Proof of Fix
Ran the proxy with otel v2 + the console exporter (
LITELLM_OTEL_V2=true OTEL_EXPORTER=console), pointing a model at a dead endpoint so the call fails and the v2 failure path emits the LLM-call span.Config used:
Request:
Before the fix, the emitted
chat broken-modelspan carried the message only in the status description; its event list was empty, so the only place the message lived was a field a backend dynamic-maps to a truncating keyword:After the fix, the same span carries the full message on the standard
exceptionevent underexception.message(recognized semconv field, mapped as full text), whileerror.typestays a low-cardinality attribute:Type
🐛 Bug Fix
Changes
The v2 span engine (
emitter.py) previously stamped onlyerror.typeand put the message into the span status description; it never recorded the standard OTel exception event that v1 emits viarecord_exception. A non-semconv field name (e.g.error_message) falls into the backend's default dynamic template, which maps strings to akeywordcapped atignore_above: 1024and truncates them.finish_spannow also records the standardexceptionspan event carryingexception.typeand the fullexception.message, so the message rides a recognized semantic-convention field that backends map as full text rather than a truncated keyword. AddedExceptionEventsemconv constants and a regression test asserting a 5000-char message survives intact on the event (and that success spans record no exception event)Generated by Claude Code