fix: correct Datadog migration defects exposed by a live cluster

[shmsr]

Fixes #459

Why

A user demo hit this on a migrated Datadog dashboard:

verification_exception - first argument of [http.response.status_code == "500"]
is [numeric] so second argument must also be [numeric] but was [keyword]

Running that to ground — migrate → seed-sample-data → verifier.live_validate, once
per field profile, against a real cluster — turned up a family of defects sharing one
cause: these paths had never been executed against a cluster holding the shape they
assumed. Four committed assertions had pinned the buggy behaviour, so the suite was
green throughout.

What's fixed

Each bullet is one commit; the rationale and evidence are in the commit messages.

Translation

• Tag-filter literals now typed from target field caps — the reported error, plus !=, a|b, IN (…), monitor scopes, metric_map filters
• LIKE used SQL %/_; ES|QL wants */?. {host:web-*} matched nothing and {!host:canary*} excluded nothing — valid ES|QL, wrong rows, no error
• Identifier quoting is per-segment and idempotent — nested backticks and reserved words (system.network.in.bytes) were hard parse errors on 4 profiles
• status:(error OR warn) tokenized into status == "(error", destroying the surrounding boolean structure
• log_stream columns were hardcoded to ECS, so every log panel failed under passthrough
• A composited breakdown column read itself (EVAL series_group = CONCAT(…, TO_STRING(series_group)) → Unknown column); the splice is idempotent now
• Datadog 1mo/1y live spans were unparsed, silently collapsing "compared to last month" panels to a 1-hour window

Seeding / mapping

• subobjects: false on generated templates: ES rejects redis.keys beside redis.keys.evicted only under the default mapping. passthrough goes 255/261 → 261/261
• Backfill clamped to the 7d TSDS ceiling — 53,256 of 125,532 docs were being generated outside the writable window and rejected
• Field limit derived from the properties being declared; at realistic corpus size (1,043 props vs the 1000 default) template creation failed and every panel went unseeded, which read downstream as "telemetry not ready"
• Control default values are seeded, so a dashboard's own control no longer blanks every panel on first open (14 empty → 22/22)

Alerts

• The canonical Datadog monitor format ({"definition": {…}}) was unreadable — 60 of 60 real monitors skipped as "No monitors found". Identification keys on type now, since a third carry no id
• 30 monitors were tiered manual_required with no reason; every MANUAL_ONLY_KIND now names the construct to build, with a drift guard

Reporting

• assess_field_usage's capability is None branch was unreachable, so a run whose caps proved a metric absent still scored the panel OK — while its own readiness contract said missing
• DATA READINESS was Grafana-only; Datadog has a separate reporter. Now shared
• One PreflightResult printed Preflight: issues in-run and Preflight: pass in the report. Only a blocking issue fails a preflight — which the manifest already published — so both derive from that
• An empty source dashboard was reported UPLOAD FAILED; 3 of 6 dashboards on a real account have zero widgets. Now skipped, without relaxing the check that catches silently dropped panels

Gates

• New verifier.filter_semantics_gate — asserts a WHERE selects exactly the intended rows. Nothing existing caught the wildcard bug: live_validate calls a wrongly-filtering query ok, and the render audit calls the empty panel a warn
• The render audit no longer excuses a panel's own synthetic column as a target field gap — that downgrade let a panel that couldn't render pass as a warning
• run_cross_profile_corpus.py takes --source and fails closed on a profile with no leakage rules, instead of passing vacuously

Verified

Elasticsearch + Kibana 9.6.0-SNAPSHOT, clean cluster before each run.

Corpus	Result
All 7 Datadog + 5 Grafana profiles, migrate→seed→live_validate	ok=261/261 and 239/240, data_gap=0 REAL_BUGS=0
47 pinned integrations-core dashboards (1,291 widgets)	92.1% migrated; live_validate 901/901; render audit 1,280/1,282, 0 render_error
5 authored dashboards (62 widgets) — IN (…), boolean scopes, p50–p99, globs, powerpack	100% on all 7 profiles; 0 render_error
Live Datadog API (6 dashboards, 2 monitors) — where the last 4 defects surfaced	ok=27/27; render audit 84 → 88 rendered, 0 render_error
60 real monitor assets	20 automated / 10 draft / 30 manual, each with a stated reason

~600 new test cases across 24 files. The filter matrix, the new gate and the
render-audit reclassification were each mutation-tested — reverting the fix makes them
fail — so they aren't vacuous.

Elasticsearch behaviour was established by probe, not docs: subobjects: false is
accepted and composes with index.mode: time_series ("auto" is rejected on 9.6),
index.look_back_time caps at 7d, and 19 of 77 probed ES|QL keywords are rejected as
a bare dotted segment.

Not verified here: alert-rule creation. All 30 fail HTTP 500 — but a hand-written
minimal .es-query rule fails identically on this stack, so it's the environment
(Kibana alerting needs security enabled; the local stack runs with it off), not the
payload.

Follow-ups

• compare replays migrate-time verdicts for non-PromQL panels instead of re-executing, so its ERROR count can never be cleared #456 — compare replays migrate-time verdicts instead of re-executing, so a stale ERROR can't be cleared
• Grafana: a dashboard with no panels is still reported as a failed upload (Datadog half fixed in #454) #457 — Grafana still reports an empty dashboard as a failed upload (the Datadog half landed here)
• live_validate files "Cannot mix time-series aggregate" as other, so --fail-on-bug misses it #460 — live_validate files Cannot mix time-series aggregate … as other, so --fail-on-bug misses it
• Grafana: non-le numeric label matchers are dropped instead of translated #461 — Grafana drops non-le numeric label matchers it could now translate — corpus-wide blast radius
• [Datadog] Default --field-profile otel is a poor fit for Datadog-Agent-origin dashboards #306 — no built-in profile fits an OTLP-native target (dotted metrics and ECS attributes); a new profile is a product decision
• Investigated and dropped: wrapper functions (per_second(…)) are already handled via the formula path (parse_legacy_query). The residue is capability — top(), timeshift, autosmooth — not parsing, so there's no parser work worth doing

Splitting

Natural cut if you'd prefer smaller PRs: (a) filter typing + wildcards + quoting,
(b) telemetry contract / seeding / subobjects, (c) readiness reporting, (d) gates,
(e) the live-API defects. (b) must precede (c)'s tests — subobjects changes what
counts as unmappable.

[stefans-elastic]

Review findings

The overall direction is strong: identifier quoting is centralized and idempotent, typed filter handling is substantially improved, readiness reporting is shared, and the focused regression coverage is extensive. I found the following issues that should be addressed before merge.

Critical

1. The new filter-semantics verifier can delete unrelated Elasticsearch indices.

   parity-rig/verifier/filter_semantics_gate.py:142 deletes predictable names such as filter-semantics-otel-long on any endpoint supplied through --es-url, then line 216 deletes them again. Concurrent runs can also erase one another's indices. The delete/create/bulk responses are ignored and cleanup is not in finally, so failures can additionally leave stale state or query stale data.

   Please use collision-resistant run-specific names, verify API responses and resource ownership, and clean up only resources owned by this invocation in finally.

Important

2. The “exact selected rows” gate only verifies cardinality.

   filter_semantics_gate.py:201-214 ends every query with STATS n = COUNT(*) and compares the result with len(expected). A broken predicate that selects a different row with the same cardinality passes, despite the module and docs promising exact document-set validation. Seed a stable identity/value column and compare the normalized selected set with expected; add a same-count/wrong-row regression test.

3. Grouped Datadog log values bypass safe value parsing and typed rendering.

   At log_parser.py:597-600, grouped members are split as raw strings and passed directly to _render_attr_predicate. Reproductions against this PR head include:

   • status:("error" OR "warn") emitting comparisons against values that literally contain quote characters.
   • Numeric @http.status_code:5* emitting http.status_code LIKE "5*", which ES|QL rejects for a numeric field.
   • @http.status_code:(500 OR error) emitting a bare http.status_code == error, treating error as a column.
   • Quoted numeric group members remaining quoted against numeric fields.

   Please parse grouped members as proper quoted/unquoted values and route equality and wildcard members through a shared typed operand renderer. Add numeric, string, unknown, quoted, wildcard, and mixed-member cases.

4. Explicit --data-hours still bypasses the seven-day TSDS ceiling.

   telemetry_data.py:846-850 clamps only contract-derived lookback. data_hours=240 still generates timestamps 240 hours old even though the generated TSDS template accepts at most 168 hours, recreating the rejected-document failure this PR intends to fix. Clamp the effective dense window, include data_hours in truncation reporting, and test data_hours > 168.

5. Wildcard replacement can generate a stream that does not match its source pattern.

   telemetry_data.py:47 replaces a run containing ? with the multi-character string generic. For example, metrics-?-* becomes metrics-generic-default, which does not match the original pattern because ? matches exactly one character. Replace each ? with one safe character and each * with a safe run, and test the invariant that the generated concrete name matches the input pattern.

6. Datadog cross-profile coverage omits elastic_agent.

   scripts/run_cross_profile_corpus.py:48-54 includes both aliases of the same Prometheus layout but excludes the unique built-in elastic_agent profile. The new fail-closed rule check therefore never sees that elastic_agent lacks leakage rules. Add those rules and profile coverage; preferably derive unique canonical profiles from the adapter registry so new profiles cannot silently escape the gate.

7. Mixed literal/template scope values escape readiness assessment while still emitting a field reference.

   translate.py:1900 drops every value containing a template from _scope_filter_tag_keys. A value such as service:prod-$svc still emits service LIKE "prod-*", but readiness records no dependency on service; if live caps prove it absent, the panel receives no DATA READINESS reason and later fails with an unknown-column error. Skip readiness only when no clause is emitted or when the field key itself is dynamic, and add mixed literal/template coverage.

The current Ruff and unit-test failures appear inherited from main and occur in untouched Grafana code, so I have not counted them as PR findings. Packaging, E2E, type checking, security, and dependency checks are green.

[shmsr]

Leaving this. FailingCreate subclasses FakeES and defines __call__, and the test passes an instance as request. The call is valid. A callable(request) guard would not change this finding.

[shmsr]

Same as the note above: FailingBulk implements __call__, so the instance passed as request is callable. Not changing the gate for this.

[shmsr]

Review findings addressed — all 7, one commit each

@stefans-elastic thanks, these were good catches. Every one reproduced before
fixing, and none turned out to be wrong. Two were slightly broader than reported.

#	Finding	Fix	Commit
1	Critical — the gate DELETEs predictable index names on any --es-url, ignores seed responses, cleans up outside finally	Per-invocation token in every index name; nothing deleted that the run did not create; cleanup in finally; seed responses checked, including a 200 bulk that rejected every document	4bc3d6b0
2	The "exact rows" gate compared only COUNT(*)	Compares the selected value set, sorted and stringified so row order and long-vs-keyword do not matter. Same-count/wrong-row is now a test	4bc3d6b0
3	Grouped log values bypass safe parsing and typed rendering	Quote-aware split; the tag path's typed renderer moved to emit/esql_utils and shared by both paths; one comparison family per group	0fa3f2f7
4	--data-hours escapes the 7d TSDS ceiling	Clamped like the contract lookback; the truncation warning takes the wider of the two	627c54b2
5	Wildcard replacement can generate a stream its pattern cannot match	Each ? → one character, each * run → one token; the match invariant is now a test	deba7880
6	Datadog cross-profile coverage omits elastic_agent	Both profile lists derived from the adapter registries, rules added, drift-guard test	22a534dc
7	Mixed literal/template scope values escape readiness	Assessment keys on whether a clause is emitted, asked of the emitter itself	dd028f76

Broader than reported. #6 also omitted default, not just elastic_agent —
deriving the list from BUILTIN_PROFILES covers both and stops the next one
escaping. #7 additionally recorded a bogus dependency on the literal $k when the
key was dynamic; that now records nothing, per your "skip when the field key itself
is dynamic".

#3 was the most consequential. @http.status_code:("500" OR "503") against a
numeric field emitted == "500" — the exact error this PR opens with, reached
through the log path instead of the tag path. _tag_comparison_mode's docstring
already claimed to mirror log_parser._render_attr_predicate, but nothing shared the
implementation, so they had drifted. All four of your reproductions were confirmed as
hard rejections on 9.6.0 before the fix and execute cleanly after it. The 307-case
tag matrix is unchanged by the move.

Verification

filter_semantics_gate   140/140 shapes select the expected rows; 0 indices left behind
unit suite              3591 passed (2 inherited failures, see below)
ruff / mypy             clean, except the inherited F811

CI

The three red checks are inherited from main, not from this branch —
reproducible on a clean origin/main worktree with nothing here applied. Bisected
to #444, tracked in #455.

One note for whoever picks #455 up: the F811 and the two test failures are
unrelated. The duplicate _strip_promql_comments shadows the promql.py import
and the shadowed symbol is the one the failing test exercises, which makes it look
causal — but fuzzing both implementations over 20,000 inputs gives 0 differing
outputs. Removing the duplicate is a clean lint-only fix and will not move the test.

Ready for another look.

🤖 Generated with Claude Code

[stefans-elastic]

[P2] Scope the seven-day clamp to TSDS metric streams — observability_migration/core/telemetry_data.py:889

_document_timestamps is shared by every stream and timestamps are calculated before stream types are known. Consequently, --data-hours 240 now truncates logs and traces to 168 hours even though their templates do not use index.mode: time_series. Compute timestamps per stream or apply the ceiling only to metric TSDS streams.

[P2] Do not classify readable flattened roots as missing — observability_migration/core/verification/field_capabilities.py:219

Both readiness builders now treat flattened capabilities as absent. On Elasticsearch 9.6, ES|QL can read a flattened root directly; only accessing its subfields requires FIELD_EXTRACT. This produces false status: missing results for valid root-field queries. Separate non-queryable object path nodes from readable container field types. Reference: https://www.elastic.co/docs/reference/query-languages/esql/esql-flattened-fields

[P2] Establish the Datadog baseline before comparing profiles — scripts/run_cross_profile_corpus.py:63

The new Datadog profile list is sorted, placing prometheus_native last. The loop only compares profiles after baseline_count is assigned, so none of the six preceding Datadog profiles undergoes the advertised feasibility-parity check. Run the native profile first or collect all counts before comparing.

Overall assessment: the PR-targeted tests pass, as do Ruff and git diff --check, but these three issues should be addressed. The full non-e2e suite stops on two test_grafana_issue_440_vector_matching failures; both reproduce unchanged on the PR base, so they are pre-existing rather than PR regressions. Live Elasticsearch verification was not rerun during this review.

[shmsr]

Second review, addressed in b12eda4.

TSDS ceiling. _document_timestamps now clamps to 7 days only when apply_tsds_ceiling is set, and generate_documents sets that flag from the stream type after it is known. A mixed contract with --data-hours 240 keeps logs and traces past 200h and holds metrics to 168h. lookback_truncation_warning is silent for a logs/traces-only contract and only counts metric-stream lookbacks, so a 14-day log span next to a short metric span does not claim the metrics were truncated. docs/command-contract.md and the --data-hours note in docs/contributing/dev-commands.md say the cap is metric TSDS only.

Flattened roots. OBJECT_CONTAINER_TYPES is object / nested / passthrough. A flattened root stays readable; ES|QL 9.6 only needs FIELD_EXTRACT for its subfields. Readiness now reports that root as confirmed, not missing.

Feasibility parity. Counts are collected for every profile, then feasibility_shortfalls compares them to prometheus_native. Order no longer matters, so the Datadog profiles that sort before the baseline are checked. A run that never produced the baseline reports no shortfalls.

Code-quality: removed unused _QUOTED_AGAINST_BARE_FIELD and _NUMERIC_TAG_VALUE_RE, and the cleanup test now uses pytest.raises(RuntimeError). The two “non-callable” notes on FailingCreate / FailingBulk are false positives: those test doubles subclass FakeES and define __call__, so the instances passed as request are callable. A callable() guard would not change the finding.