feat(grafana): emit native PROMQL for vector matching on capable targets

[giorgi-imerlishvili-elastic]

Fixes #440.

Was the issue valid?

Yes — with one correction to its own acceptance table, explained under Deviation below.

Root cause

_PROMQL_UNSUPPORTED_RE in observability_migration/adapters/source/grafana/panels.py
listed on(, ignoring(, group_left, and group_right among the PromQL constructs
blocked from native emission unconditionally. That list predates
elastic/elasticsearch#155634, so every vector-matched panel fell through to the ES|QL
join/ratio approximation regardless of what the target could evaluate — dropping labels
and emitting semantic-loss warnings on clusters that could have run the operator's PromQL
verbatim.

The fix

Native emission now requires two independent facts, each established rather than assumed,
because emitting a query the target cannot plan replaces a rendering panel with a hard
Kibana error — strictly worse than an approximation.

1. The target evaluates it. New PROMQL_VECTOR_MATCHING runtime feature, detected by
a probe rather than by reading version.number, so Serverless (which has the capability
without a 9.6 version string) is recognized. The probe uses vector(1) operands so it
needs no index or data:

PROMQL index=metrics-* step=1m start=… end=…
  value=(vector(1) * on(job) group_left(pod) vector(1) / ignoring(pod) vector(1))

vector(1) is deliberate: a probe over real selectors cannot answer the question, because
a capable target still rejects a bare selector and returns HTTP 500 for a matched binary
op over metrics the index lacks — either shape would report "unsupported" on a capable
cluster. The probe fails closed (only HTTP 200 enables native), the opposite default from
PROMQL_COMMAND_V0, and its verdict is printed under Target PromQL profile.

2. The operand shapes are plannable. Support is necessary but not sufficient:
Elasticsearch only vector-matches operands whose label set it can determine statically, and
rejects the rest with vector matching requires operands with concrete label sets. A new
AST predicate (promql_vector_matching_has_indeterminate_operand) models that rule —
aggregations with by (...), bare sum(...), and vector(n) are concrete and propagate
through functions, unary ops, parens, and arithmetic; raw selectors, rate()/*_over_time()
over them, and without (...) are not.

Panels that stay on ES|QL carry a note naming which half declined, distinguishing an
unverifiable probe (auth error, transport failure) from a verified absence, so the operator
knows whether to upgrade, fix access, or reshape the query. Panels that go native record
minimum_kibana_version: 9.6.0, gated on the emitted query so a fallback never raises the
floor. or/and/unless with a matcher stay blocked (elastic/elasticsearch#158181).

Deviation from the issue's acceptance table

The issue lists panel 4 — node_hwmon_temp_celsius * on(chip) group_left(chip_name) node_hwmon_chip_names
— as "expected: PROMQL index=…". Against a live 9.6.0-SNAPSHOT that query returns:

HTTP 400  vector matching requires operands with concrete label sets

Both operands are raw selectors. Emitting it natively would convert a rendering panel into
a hard error, so it stays on ES|QL and explains why in its notes. Panels 1–3 behave as the
issue specifies.

Related code paths checked

• can_use_native_promql and every other gate it runs (_PROMQL_UNSUPPORTED_RE,
  _PROMQL_HISTOGRAM_QUANTILE_RE, _PROMQL_EMPTY_METRICLESS_SELECTOR_RE,
  _promql_has_unsupported_comparison, _promql_has_known_server_bug).
• Every consumer of _PROMQL_UNSUPPORTED_RE — only can_use_native_promql, so removing the
  four tokens has no hidden blast radius.
• Both native-emission note sites (_translate_panel_native_promql,
  _translate_multi_target_native_promql).
• The alert path: _generate_esql_for_alert calls can_use_native_promql without a
  runtime-feature profile, so matched-expression alerts keep ES|QL. Pinned by a test, since
  the shared gate now has a path that can return True.
• _dashboard_minimum_kibana_version and its existing 9.5 floors.
• _promql_has_unmatchable_distinct_metric_binop / _promql_has_unmatchable_vector_match
  from Grafana: element-wise distinct-metric PromQL arithmetic via native PROMQL silently returns zero rows (panel reported clean, renders empty) #376, which handle implicit matching and are unaffected.
• parity-rig/verifier/classifier.py, so a shape that ever slips past the gate is
  classified as a translator bug rather than going unlabeled.

Side effects considered

• Emitting native on an incapable target — the probe fails closed; inconclusive keeps ES|QL.
• Emitting a shape the target rejects — the AST predicate is conservative: unparseable
  input, unmodelled node types, and analysis errors all keep the fallback.
• Portability of the artifact — native matcher panels raise the version floor to 9.6.0.
• Alerts — unchanged, and pinned by test.
• Comments hiding structure — PromQL allows a comment between a token and its
  parenthesis, so on # note\n(x) parses as on(x). Comments are stripped from the raw
  expression, where their extent is exact. The gates that scan _clean_promql_for_native
  output deliberately keep scanning it with comments intact: flattening newlines destroys a
  comment's extent, and stripping there would swallow the operand after it and hide or /
  and / unless / comparisons. Quote tracking covers ", ', and backtick raw strings so
  a # inside a label value is data, not a comment.
• Existing probe-count test — test_detect_target_runtime_features_uses_capability_names
  now counts label-matcher probes by query instead of total POSTs, so an unrelated feature
  probe cannot look like a regression.

Empirical verification

Control run on unmodified base d80e8ec vs this branch, same input, same target, same
seeded data:

Panel	Base	Branch
1 Control (no matcher)	native PROMQL	native PROMQL (byte-identical)
2 on() aligned ratio	ES|QL TS … SUM(CASE(...))	native PROMQL … / on(instance) …
3 ignoring()	ES|QL; warned dropped device, labels not retained	native PROMQL … / ignoring(device) …; warnings gone, status migrated
4 on() + group_left	ES|QL	ES|QL + explanatory note
minimum_kibana_version	9.5.0	9.6.0

Direct _query against 9.6.0-SNAPSHOT: panel 2 → HTTP 200, 183 rows, columns
value, step, instance; panel 3 → HTTP 200, 61 rows; panel 4 → HTTP 400 as quoted above.

Negative case against a real pre-9.6 cluster (Elasticsearch 9.5.0 container): the probe
returns HTTP 400 VectorBinaryArithmetic queries with group modifiers are not supported at this time → state supported: False, confidence: verified. The profile prints
unsupported, all three matcher panels stay ES|QL, and the emitted queries are
byte-identical to what base code emits against that same target. On a pre-9.6 target
this change is a no-op.

Tests

• New tests/test_grafana_issue_440_vector_matching.py: 60 tests / 55 subtests covering the
  AST shape model (concrete and indeterminate operands), the eligibility gate, the capability
  probe's five status paths, comment and quote handling, end-to-end panel emission on capable
  and incapable targets, note wording, the version floor, and alert routing.
• make test: 6432 passed, 61 skipped, 586 subtests passed.
• make lint: clean. make typecheck (mypy): clean.
• No pre-existing failures on the base.

Kibana visual verification

Performed by driving the already signed-in Chrome over CDP. The Chrome DevTools MCP could
not attach — the profile's DevToolsActivePort held a stale browser GUID — so the same real
browser and session were used through Playwright's connect_over_cdp, which yields the same
evidence (screenshots, panel text, console, network).

Dashboard obs-migrate-promql-on-ignoring-eval in view mode, cache-bypassing reload, at
Last 1 hour and re-checked at Last 3 hours:

• All four panels render charts; none show No results found, Migration Required, or any
  error text.
• Zero failed requests (xhr/fetch). The only two console errors are CSP violations from
  the driver's own injected reload script, not the application.
• Panel 2 renders three instance_* series; panel 3 renders a single series ≈1.34; panel 4's
  ES|QL fallback still renders its nine chip_N / chip_name_M series unchanged.
• Each panel's Lens x / y / breakdown_by accessors match the columns its query actually
  returns — the mismatch that produces "invalid column" render failures.
• The base build was uploaded and captured for contrast, then the branch build restored. Base
  and branch produce the same values, which is the expected result: native evaluation did not
  shift numbers, it removed the approximation and its semantic-loss warnings.

Re-run after every review fix; the emitted native/ and ir/ artifacts are byte-identical
to the verified build.

Pre-existing issues found, not fixed here

1. _clean_promql_for_native flattens newlines, so a PromQL comment swallows the rest of the
   expression in the emitted native query. Base already emits native PROMQL for
   sum(a) # note\n+ sum(b), producing a silently truncated query. Out of scope; filed
   separately.
2. grafana-validate-uploaded has no --insecure or --ca-cert flag and ignores
   OBS_MIGRATE_INSECURE, so it cannot validate a self-signed local stack — it fails with
   CERTIFICATE_VERIFY_FAILED. Worked around by replaying the uploaded saved object's
   queries directly.