fix(grafana): route distinct-metric PromQL arithmetic off native PROMQL

[giorgi-imerlishvili-elastic]

Fixes #376.

Is the issue valid?

Yes. Reproduced on the unmodified base at 0a5340ad64b43764d74d8ae3a64fe22a8bb23a56, and confirmed against a live Elasticsearch 9.5.0-SNAPSHOT rather than taken on faith.

Root cause

Elasticsearch's native PROMQL command keeps __name__ in the implicit vector-matching key; Prometheus excludes it when matching binary-operator operands. Two different metric names can therefore never match, so A / B returns zero rows while the source panel shows real values — and the panel was still reported migrated at confidence 0.9 with no warning.

Bisected against the same index:

Expression	Result
A / B alone	16 rows each
A / B (distinct names)	0 rows
B / B (same name)	16 rows
sum(A) / sum(B)	8 rows
B * 2 (vector × scalar)	16 rows

Probing further showed the divergence is wider than the report: Elasticsearch also propagates __name__ through rate(), abs(), *_over_time() and scalar arithmetic, where Prometheus drops it. So rate(A[5m]) / rate(B[5m]) is broken too, not just the bare-selector form. Aggregations do drop the name (and topk/bottomk preserve it), which is why sum(A)/sum(B) was unaffected and must stay native.

Code paths checked

• can_use_native_promql and its existing decline gates (_PROMQL_UNSUPPORTED_RE, unsupported comparisons, range-on-nonselector, nested aggregation, known server bugs).
• _translate_panel_native_promql and build_native_promql_query.
• The PromQL AST layer in promql.py (_parse_fragment, _ast_from_node, _ast_binary_fragment, _ast_matchers), since the PromQLFragment IR loses the detail the decision needs.
• The alert path: _has_source_faithful_query, _generate_esql_for_alert, classify_automation_tier, build_es_query_rule_params.
• Prior work this interacts with: Grafana → Kibana: migrate label-aligned vector-matching joins via a per-key ES|QL aggregation instead of marking them not-feasible #156 (explicit on()/ignoring() joins) and grafana-migrate: PromQL ratio/difference panels migrate as ES|QL approximations instead of native PromQL #138/grafana-migrate: distinct-metric PromQL arithmetic on single-value (gauge/stat) panels still degrades to ES|QL approximation #146 (which deliberately routed distinct-metric arithmetic onto the native path).

The fix

Option 1 from the issue — detect and re-route.

promql_has_unmatchable_distinct_metric_binop walks the AST and resolves the set of metric names each operand's result carries (_ast_result_metric_names). When both sides are determinate, non-scalar and disjoint, the operation can never match, so can_use_native_promql declines it and the panel degrades to the existing per-key ES|QL STATS/EVAL translator, carrying an operator-visible note.

Excluded by design: set operators (and/or/unless), where distinct names are normal and correct; explicit on()/ignoring() matchers and __name__ regex selectors, which Elasticsearch rejects loudly with a 400 and so never fail silently; and anything indeterminate, which suppresses the flag rather than triggering it.

Alerts get the same reroute for a stronger reason — a rule that never fires is quieter than an empty panel. _has_source_faithful_query keeps them automated via ES|QL, but only promises a query when the translator can actually emit one, since it has no ES|QL form for changes(), absent() or predict_linear(). It and _generate_esql_for_alert now share a single feasibility predicate so they cannot disagree.

Side effects considered

• Panel status changes shape, not just label. Statefulset replicas moves migrated/0.9 → migrated_with_warnings/0.6. That reads as a downgrade but is an upgrade in correctness: the old status described a query returning nothing.
• Artifact diff is contained. Across 51 panels of dashboard 13332, the normalized native JSON changes by 10 lines in exactly one panel (query, x, y, breakdown_by). Two HPA panels were already on the ES|QL path and only picked up the explanatory note; their queries are byte-identical.
• Aggregated forms stay native — verified, so no fidelity loss for sum(A)/sum(B).
• Alert regression caught and fixed. Tightening can_use_native_promql initially made these alerts fall to manual, breaking test_group_interval_reaches_kibana_schedule with KeyError: 'schedule'. That was a real regression, not a stale test.
• Predicate cost measured, since _has_source_faithful_query is called ~9 times per alert: 0.588 ms vs 0.184 ms baseline for affected alerts, and zero probes for unaffected ones.
• Probe/real-call parity proven across 4 expressions × 4 resolver configurations (including a match-everything custom rule pack and label rewrites): 0 divergences, now pinned by a test.

Tests

make test 6260 passed, 53 skipped, 497 subtests. make lint (ruff + 502 source headers + skill mirror/structure) and make typecheck (mypy, 10 source files) both clean.

• New tests/test_grafana_issue_376_vector_matching.py: 38 tests / 24 subtests covering unmatchable shapes, matchable shapes that must not regress, macro resolution, native eligibility, alert routing, and probe/real-call parity.
• Five tests in test_migrate.py that encoded the grafana-migrate: PromQL ratio/difference panels migrate as ES|QL approximations instead of native PromQL #138/grafana-migrate: distinct-metric PromQL arithmetic on single-value (gauge/stat) panels still degrades to ES|QL approximation #146 behavior were renamed and inverted; three were added, including an end-to-end guard for the exact reported panel. Three previously asserted only "not PROMQL" and now also require STATS, computed_value, and exact status/confidence.

No pre-existing failures on the base. One environment caveat: test_bump_version_script.py fails under a restrictive sandbox with PermissionError creating a .cursor temp directory — an environment artifact, passing normally.

Kibana visual verification

Chrome DevTools MCP against a local 9.5.0-SNAPSHOT stack.

Dashboard obs-migrate-kube-state-metrics-v2 opened in view mode with a cache-bypassing hard reload. The previously-empty Statefulset replicas panel now renders a line at 100% with a prometheus series, matching Grafana.

Because the panel is driven by dashboard variables, a green default-state render alone would not prove filters work, so the namespace control was exercised from monitoring → production: the series switched prometheus → postgres and still rendered at 100%, correlating control selection with the affected query. Console showed only a CSP violation from the MCP's own injected script (an artifact, not an application error), and none after the interaction. All 89 xhr/fetch requests returned 200, including every esql_async call. Neighbouring Cluster tiles held at 1.82% / 12.50% / 3.13%, and the two not_feasible changes() panels kept their expected "Migration Required" state.

Re-verified after the follow-up review round by re-running the migration and diffing the normalized native artifact against the uploaded one: identical, so the saved object was unchanged.