[BUG] TokenCalcHandler drops token counts when usage is a dictionary #8170

Description

TokenCalcHandler.log_success_event accepts response_obj as a dictionary, but counts token fields only when its nested usage value exposes attributes. With dictionary usage it increments successful_requests while silently leaving prompt, completion, total, and cached-prompt token counters unchanged.

This is specific to the callback accumulator. BaseLLM's separate internal tracker already accepts provider dictionaries through UsageMetrics.from_provider_dict; no claim is made that every Crew/Flow total is affected. LiteLLM response handling also explicitly passes usage_info to TokenCalcHandler in a response dictionary.

Steps to reproduce

Source main aa5f2acd9b48782b6b6a756bcef24cc54ac0530b:

from crewai.agents.agent_builder.utilities.base_token_process import TokenProcess
from crewai.utilities.token_counter_callback import TokenCalcHandler

process = TokenProcess()
callback = TokenCalcHandler(process)
callback.log_success_event({}, {"usage": {
    "prompt_tokens": 12,
    "completion_tokens": 3,
    "total_tokens": 15,
    "prompt_tokens_details": {"cached_tokens": 4},
}}, 0, 0)
print(process.get_summary())

Expected behavior

The summary records prompt=12, completion=3, total=15, cached_prompt=4, successful_requests=1, matching equivalent SDK usage objects.

Actual behavior

The token counters remain zero while successful_requests is 1.

Proposed fix and verification

For dictionary usage, reuse UsageMetrics.from_provider_dict and feed the existing TokenProcess accumulation methods. Keep the existing SDK-object path, empty-usage handling, and absent-process behavior unchanged.

New offline regression/control cases use actual TokenCalcHandler and TokenProcess plus real OpenAI CompletionUsage objects. Before the production fix: 2 failed / 5 passed. After the fix: 32 passed across the callback tests, Agent/Crew usage-shape tests and Flow usage-aggregation tests (23 existing deprecation warnings). Targeted Ruff lint/format and mypy pass.

Python 3.13.12 / macOS arm64; repository-lockfile dependencies, telemetry disabled. No provider/model/registry calls. Full monorepo tests and live LiteLLM provider integration were not run.

AI assistance: Codex investigated, implemented regression coverage and the proposed fix, and ran the checks above.