Skip to content

Implement error counters categorized by semantic error type for better operational visibility #465

Closed

Description

@psschwei

Description:
Implement error counters categorized by semantic error type for better operational visibility.

Detailed Requirements:

  1. Create error type enum/constants in mellea/telemetry/metrics.py:
    • rate_limit - Rate limiting / 429 errors
    • timeout - Request timeouts
    • content_policy - Content policy violations
    • auth - Authentication/authorization errors
    • invalid_request - Malformed request errors
    • transport_error - Network/connection errors
    • server_error - Provider server errors (5xx)
    • unknown - Uncategorized errors
  2. Create counter:
    • mellea.llm.errors - Error count by type
  3. Create error classifier function:
    • classify_error(exception) -> str - Map exceptions to semantic types
    • Handle provider-specific exception types (OpenAI, etc.)
  4. Create helper:
    • record_error(error_type, model, backend, exception_class)
  5. Integrate with backend error handling

Files to Modify:

  • mellea/telemetry/metrics.py - Add error counter and classifier
  • mellea/telemetry/backend_instrumentation.py - Add error recording
  • mellea/backends/openai.py - Record errors on exceptions
  • Other backends as applicable

Metric Definition:

Metric Type Unit Attributes
mellea.llm.errors Counter {error} error_type, model, backend, system

Error Type Mapping (examples):

Exception Pattern Error Type
RateLimitError, status 429 rate_limit
Timeout, TimeoutError timeout
ContentPolicyViolation content_policy
AuthenticationError auth
BadRequestError, status 400 invalid_request
ConnectionError, NetworkError transport_error
status 5xx server_error

Acceptance Criteria:

  • All errors classified into semantic types
  • Counter increments with correct labels
  • Provider-specific exceptions handled
  • Unknown errors categorized as "unknown"

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions