feat(ai-monitoring): Fetch model context size and rename task to fetch_ai_model_info

[constantinius]

Closes https://linear.app/getsentry/issue/TET-2219/sentry-map-llm-context-size-to-relay-cost-calculation-config

Introduces a new AIModelMetadata schema that nests costs under a costs field and adds an optional contextSize. Context size is fetched from OpenRouter's context_length field and models.dev's limit.context field, following the same precedence logic as costs (OpenRouter takes priority).

Architecture

Both tasks run independently on the same cron schedule (every 30 min):

Task	Cache key	Format
fetch_ai_model_costs (legacy, TODO remove)	ai-model-costs:v2	Flat AIModelCostV2
fetch_ai_model_metadata (new)	ai-model-metadata:v1	Nested AIModelMetadata

They share helper functions (_normalize_model_id, _create_prefix_glob_model_name, safe_float_conversion) but fetch and cache independently. The old task + cache key will be removed once all consumers have migrated.

GlobalConfig serves both fields side by side:

• aiModelCosts — legacy flat format (read by Relay today)
• aiModelMetadata — new nested format with contextSize (not yet consumed by Relay)

New schema (ai-model-metadata:v1)

{
  "version": 1,
  "models": {
    "gpt-4": {
      "costs": {
        "inputPerToken": 0.0000003,
        "outputPerToken": 0.00000165,
        "outputReasoningPerToken": 0.0,
        "inputCachedPerToken": 0.0000015,
        "inputCacheWritePerToken": 0.00001875
      },
      "contextSize": 1000000
    },
    "claude-3-5-sonnet": {
      "costs": {
        "inputPerToken": 0.000003,
        "outputPerToken": 0.000015,
        "outputReasoningPerToken": 0.0,
        "inputCachedPerToken": 0.0000015,
        "inputCacheWritePerToken": 0.00000375
      }
    }
  }
}

contextSize is optional — only present when the source API provides it.

New types

Type	Purpose
AIModelCost	Token cost fields (nested under costs)
AIModelMetadata	Per-model entry: costs + optional contextSize
AIModelMetadataConfig	Top-level config: version + models dict

Legacy types (AIModelCostV2, AIModelCosts) are kept unchanged for the old task.

Co-Authored-By: Claude Sonnet 4 noreply@anthropic.com

[linear-code]

TET-2219 Sentry: map LLM context size to relay cost calculation config

[constantinius]

Maybe we should update the config version, or use another structure.

[vgrozdanic]

Maybe adding a new field in the config would be the best approach, because I can see us wanting to expand this even further for non cost related things. I am aware that it complicates this all a bit now, but it will allow us to do a quick metadata changes in future.

That new field could be called LLMModelMetadata and for each model would contain cost and context information for now, with a chance to expand it in the future.

[constantinius]

Added type defs for a new schema.

[vgrozdanic]

Maybe adding a new field in the config would be the best approach, because I can see us wanting to expand this even further for non cost related things. I am aware that it complicates this all a bit now, but it will allow us to do a quick metadata changes in future.

That new field could be called LLMModelMetadata and for each model would contain cost and context information for now, with a chance to expand it in the future.

[github-actions]

Backend Test Failures

Failures on 80438bc in this run:

tests/sentry/api/endpoints/test_relay_globalconfig_v3.py::test_global_config — log

[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/api/endpoints/test_relay_globalconfig_v3.py:72: in test_global_config
    assert normalized == config
E   AssertionError: assert {'aiModelCost....0, ...}, ...} == {'aiModelCost.....]}]}}}, ...}
E     
E     Omitting 5 identical items, use -vv to show
E     Right contains 1 more item:
E     {'llmModelMetadata': None}
E     
E     Full diff:
E       {
E           'aiModelCosts': None,
E     -     'llmModelMetadata': None,
E           'measurements': {
E               'builtinMeasurements': [
E                   {
E                       'name': 'app_start_cold',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'app_start_warm',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'cls',
E                       'unit': 'none',
E                   },
E                   {
E                       'name': 'connection.rtt',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'fcp',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'fid',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'fp',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'frames_frozen_rate',
E                       'unit': 'ratio',
E                   },
E                   {
E                       'name': 'frames_frozen',
E                       'unit': 'none',
... (1493 more lines)

tests/sentry/api/endpoints/test_relay_globalconfig_v3.py::test_global_config_valid_with_generic_filters — log

[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/api/endpoints/test_relay_globalconfig_v3.py:127: in test_global_config_valid_with_generic_filters
    assert config == normalize_global_config(config)
E   AssertionError: assert {'aiModelCost...ts': 10}, ...} == {'aiModelCost.....]}]}}}, ...}
E     
E     Omitting 5 identical items, use -vv to show
E     Left contains 1 more item:
E     {'llmModelMetadata': None}
E     
E     Full diff:
E       {
E           'aiModelCosts': None,
E           'filters': {
E               'filters': [
E                   {
E                       'condition': {
E                           'inner': {
E                               'name': 'event.contexts.browser.name',
E                               'op': 'eq',
E                               'value': 'Firefox',
E                           },
E                           'op': 'not',
E                       },
E                       'id': 'test-id',
E                       'isEnabled': True,
E                   },
E               ],
E               'version': 1,
E           },
E     +     'llmModelMetadata': None,
E           'measurements': {
E               'builtinMeasurements': [
E                   {
E                       'name': 'app_start_cold',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'app_start_warm',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'cls',
E                       'unit': 'none',
E                   },
E                   {
E                       'name': 'connection.rtt',
E                       'unit': 'millisecond',
E                   },
E                   {
E                       'name': 'fcp',
... (1483 more lines)

[jjbayer]

Was this schema already added to Relay?

[constantinius]

There is a PR in Relay for that: getsentry/relay#5814

[jjbayer]

A general problem with config migrations like this is that customer Relays also receive the global config, and they might still expect the old format. So the safer way forward is to leave the old format in place and add an additional key.

[constantinius]

A general problem with config migrations like this is that customer Relays also receive the global config, and they might still expect the old format. So the safer way forward is to leave the old format in place and add an additional key.

@jjbayer the previous task and config remains for a time

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f81a89a. Configure here.}

[cursor]

Identical TypedDict duplicated under a new name

Low Severity

AIModelCost is field-for-field identical to AIModelCostV2. A simple type alias (AIModelCost = AIModelCostV2) or reusing the existing type in AIModelMetadata would avoid the duplication while still allowing the legacy type to be independently removed later.

Additional Locations (1)

• src/sentry/relay/config/ai_model_costs.py#L24-L32

 

^{Reviewed by Cursor Bugbot for commit f81a89a. Configure here.}

[jjbayer]

A general problem with config migrations like this is that customer Relays also receive the global config, and they might still expect the old format. So the safer way forward is to leave the old format in place and add an additional key.

@jjbayer the previous task and config remains for a time

OK as long as we double-write in sentry before we deploy the relay change this should be fine. I discussed with @Dav1dde that customer relays should not be a problem because they will just skip this part of normalization once the old config key is gone, which means our own relays will populate the attributes. Since metrics extraction is essentially dead, we no longer require doing this on the edge.