Grafana: element-wise distinct-metric PromQL arithmetic via native PROMQL silently returns zero rows (panel reported clean, renders empty)

[giorgi-imerlishvili-elastic]

Summary

Element-wise PromQL arithmetic between two different metric names with no explicit
matcher (A / B, the implicit label-aligned case) is routed to the native PROMQL ES|QL
command, where it silently returns zero rows. The panel is reported as migrated
(green, confidence 0.9, no warning) and renders empty in Kibana while the source panel shows
real values.

Elasticsearch's PROMQL command appears to include __name__ in the implicit vector-matching
key. Real PromQL excludes the metric name when matching binary-operator operands, so two
different metrics can never match and the result set is empty.

Reproduction

Dashboard: grafana.com 13332 "kube-state-metrics-v2",
panel Statefulset replicas.

kube_statefulset_status_replicas_ready{namespace=~"$namespace",cluster=~"$cluster"}
  / kube_statefulset_status_replicas{namespace=~"$namespace",cluster=~"$cluster"} * 100

Grafana renders two series (prometheus, postgres) at 100%. The migrated panel is
empty; the emitted query returns 0 rows:

PROMQL index=metrics-k8s.prometheus-parity start=?_tstart end=?_tend buckets=50
  value=(kube_statefulset_status_replicas_ready{namespace=~?namespace, cluster=~?cluster}
        /kube_statefulset_status_replicas{namespace=~?namespace, cluster=~?cluster}*100)
| GROK _timeseries """(?:\A\{(?:"labels":\{)?|,)"statefulset":"%{DATA:statefulset}\""""
| KEEP step, value, statefulset

Evidence — bisected against the same index

Both operands have identical label sets (cluster, namespace, statefulset, job,
instance, origin_prometheus) and both are populated.

#	Expression	Result
A	kube_statefulset_status_replicas_ready{…}	16 rows
B	kube_statefulset_status_replicas{…}	16 rows
C	A / B	0 rows
D	A / B * 100 (as migrated)	0 rows
E	A / B, no variable filters	0 rows
F	sum(A) / sum(B)	8 rows ✅
G	B * 2 (vector × scalar)	16 rows ✅
H	B / B (same metric name)	16 rows ✅
I	A / on(statefulset) B	HTTP 400 — VectorBinaryArithmetic queries with group modifiers are not supported at this time
J	A / ignoring(__name__) B	HTTP 400 — same
K	kube_hpa_status_current_replicas / kube_hpa_spec_max_replicas	0 rows

H succeeding while C/K return nothing is the key datum: the operands differ only in
__name__, so the implicit matching key evidently includes it.

Note that I/J fail loudly (400), which the engine can detect, whereas the default
matcher-less form fails silently — which is what makes this dangerous.

Relationship to prior work

• Grafana → Kibana: migrate label-aligned vector-matching joins via a per-key ES|QL aggregation instead of marking them not-feasible #156 handled joins with explicit on()/ignoring()/group_left() matchers and
  documented the same 400 error for those.
• grafana-migrate: PromQL ratio/difference panels migrate as ES|QL approximations instead of native PromQL #138 / grafana-migrate: distinct-metric PromQL arithmetic on single-value (gauge/stat) panels still degrades to ES|QL approximation #146 deliberately routed distinct-metric arithmetic onto the native PROMQL path
  (first for line/timeseries, then for single-value tiles).

This report is the gap between the two: distinct-metric, element-wise, matcher-less
arithmetic now takes the native path that #138/#146 opened, and that path yields nothing.
Aggregated forms (sum(A)/sum(B), case F) are unaffected — the same dashboard's
"Cluster CPU Requested" tile uses that shape and matches Grafana exactly.

Expected

Either:

1. Detect and re-route. When an expression is an element-wise binary op between distinct
   metric names with no aggregation wrapper, use the per-key ES|QL aggregation shape from
   Grafana → Kibana: migrate label-aligned vector-matching joins via a per-key ES|QL aggregation instead of marking them not-feasible #156 (STATS a = …, b = … BY <shared labels>, TBUCKET | EVAL result = a / b) instead of
   native PROMQL; or
2. Degrade gracefully. If the native path is kept, the panel must not be reported as clean
   migrated. Silently emitting a query that always returns zero rows hides a semantic gap,
   which the repo's "degrade gracefully" rule explicitly forbids.

A validation check would also catch this class generally: a panel whose query returns zero
rows against a target that does contain both referenced metrics is a translation failure,
not a data gap.