Skip to content

refactor: subscribe metrics plugins to generation_batch_* hooks #1182

Closed

Description

@ajbozarth

Background

PR #1181 (Phase 1 of #444) introduces three new hooks for raw-path generations:

  • GENERATION_BATCH_PRE_CALL
  • GENERATION_BATCH_POST_CALL
  • GENERATION_BATCH_ERROR

The tracing plugin subscribes to all three so that generate_from_raw emits one span per call (per OTel GenAI semconv). The metrics plugins do not — mellea/telemetry/metrics_plugins.py only subscribes to per-MOT hooks:

Plugin Subscribed hooks
TokenMetricsPlugin generation_post_call
LatencyMetricsPlugin generation_post_call
CostMetricsPlugin generation_post_call
ErrorMetricsPlugin generation_error

As a result, generate_from_raw calls produce zero metrics today. Tracing covers it; metrics don't.

Scope

Pure additive. For each metrics plugin, add a parallel hook handler that subscribes to the corresponding generation_batch_* hook and records the same metrics from the batch payload. Estimated ~30–40 LOC.

Acceptance criteria

  • All four metrics plugins subscribe to the matching generation_batch_* hook.
  • Token, latency, cost, and error metrics are recorded for generate_from_raw calls in addition to generate_from_chat_context.
  • Test added that calls generate_from_raw with metrics enabled and asserts each counter/histogram is incremented.

Reference

PR #1181 (the tracing plugin in mellea/telemetry/tracing_plugins.py shows the batch-hook subscription shape to mirror).

Activity

  1. self-assigned this
    on Jun 1, 2026
  2. added a commit that references this issue on Jun 10, 2026
    1479a5b
  3. added a commit that references this issue on Jun 11, 2026
    7e23948
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions