feat(providers): Elasticsearch and OpenSearch, two type-ids over one SQL transport (#424)

[cevheri]

Phase 1 of #424 — the largest gap on the coverage map. One implementation in src/lib/db/providers/sql/search/ serves both type-ids; everything the two products disagree about is data in a dialect table, never a branch on the type-id.

The query surface was decided by measurement

	Elasticsearch 9.1.4 (basic)	OpenSearch 3.8.0
SQL	_sql?format=json, {columns, rows}	_plugins/_sql, {schema, datarows}
ES|QL	works	does not exist (405)
Licence	basic, no gate	Apache-2.0

Both speak SQL with no licence gate, so SQL is the shared language and ES|QL cannot be. That puts the provider on SQLBaseProvider, which is what makes the explorer, ER diagrams, schema diff, the query limiter and agent plan mode work with no per-provider cost. The schema still comes from _mapping, never from SQL — SELECT * describes the statement, and an index mapping flattened/nested answers it with zero columns.

Three defects the live probes found

A truncated result presented as complete. An Elasticsearch aggregation is paged by the engine even with no fetch_size: a GROUP BY over 1500 distinct values answers 1000 rows plus a cursor. Dropping it returned two thirds of the buckets and labelled the result complete — worse than an error, because a user reading a GROUP BY has no way to notice 500 missing groups. query() now follows the cursor (page two carries rows and no column declaration, so page one's names are carried forward) and refuses loudly at a bound rather than truncating quietly. Verified live: 1500 rows where it previously returned 1000.

A wrong column label. SELECT customer AS who declares name: "who" on Elasticsearch and name: "customer", alias: "who" on OpenSearch. The transport read name only, so the same statement's column was labelled who on one product and customer on the other.

Silently wrong rows from a generated query. query-generators.ts derives its dialect from defaultPort — an assumption that held only while every engine had a distinct one. Both products are 9200 and they disagree about the quote character: on OpenSearch a double-quoted identifier is a string literal, so WHERE "customer" = 'acme' answers HTTP 200 with total: 0 while the backtick form returns the row. No error, just no rows for data that exists. Fixed with a declared capability (ProviderCapabilities.identifierQuoting) rather than a type-id check; absent means "keep deriving it from the port", so no existing provider changed.

Two places the honest answer was to offer less

• Multi-fields are not listed as columns, on either product. OpenSearch refuses SELECT note.keyword in every spelling (SemanticCheckException) while Elasticsearch selects it fine, and dynamic mapping creates such a child for every text field — so keeping them would break the generated starter query on nearly any dynamically-mapped OpenSearch index. Keeping them only on Elasticsearch would make the dialect decide a behaviour, which introspect.ts's own recorded rule forbids. The Elasticsearch-side loss is written down, not papered over.
• supportsExplain: false. Neither SQL endpoint returns a plan tree, so an Explain tab would render for a plan nothing can parse.

Registration surfaces

Recon measured which surfaces the compiler enforces by adding the union member and running tsc: eight fail the build, and roughly as many fail silently — the factory switch and the literal type list in its error message, the seed zod enum, selectableTypes, the icon exports, the grammar maps. All are registered, and three that had no test at all now do: the icon list, both factory cases (each asserted to produce its own class), and the error-message list, which now iterates SHIPPED_DATABASE_TYPES instead of naming three engines.

SHIPPED_DATABASE_TYPES itself was the one mandatory surface with no guard — a hand-written array whose count test read the same constant on both sides. It is now derived from an exhaustive Record, so an omission is a compile error.

Claim surfaces this work falsified

docs/BRAND_MESSAGING.md carried a standing rule: "There is no Elasticsearch or OpenSearch provider. Neither may appear as an example, in any list of engines, or in any story about polyglot teams." Shipping the provider makes that false while the rule keeps forbidding us to mention it. Nothing in the repo would have flagged this — it is not a provider file, a test, or a doc anyone greps when adding an engine. Updated, along with its engine count and the candidate row in docs/ADDING_A_PROVIDER.md.

Verification

Seven gates of #424's Definition of Done:

Gate	Evidence
1 Tri-sync	provider + docs/providers/{elasticsearch,opensearch}.md + one integration test per type-id
2 Registration	8 type-enforced + ~8 silent surfaces, listed above
3 Unit + integration	check-coverage: OK — 37635/37635 lines (100.00%)
4 Live probe	both engines, real provider, every introspection surface: zero failures, QueryError for a missing index, versions Elasticsearch 9.1.4 / OpenSearch 3.8.0
5 E2E	e2e/search-providers.spec.ts — both drivers selectable, port prefilled, no connection-string toggle, and no unearned compatibility hint
6 Functional + gates	format · lint (0 errors) · typecheck · knip · test · build all clean
7 Agent plan mode	both type-ids ground via provider-inventory (#414's non-catalog path), asserted in context-snapshot.test.ts

Container fixtures added to database-compose.yml with the exact configuration the probed containers ran, so gate 4 is reproducible.

Recorded limitations

No EXPLAIN. Aliases and data streams are not in the schema tree (_cat/indices does not list them). No maintenance operations, no active sessions, no slow queries. totalHits is unavailable on Elasticsearch. A trailing semicolon is a syntax error there, and OFFSET is too. Nested-mapping document counts exceed what a SELECT returns. No result-export.ts dialect entry: both engines reject CREATE TABLE, so no type spelling makes an exported file replayable into its own engine — the #422 criterion cannot be met here, and inventing a row would bury a promise the product cannot keep.

[cevheri]

Live-cluster verification (Chrome, both engines)

Driven in a real browser against Elasticsearch 9.1.4 and OpenSearch 3.8.0 (the pinned database-compose.yml services), seeded with a microservice-shaped dataset: orders with an object mapping and a text+keyword multi-field, customers, a 1500-document app_logs, Turkish names and city names throughout, plus .internal_bookkeeping and OpenSearch's own .plugins-ml-config / top_queries-* to test index hiding.

What already worked, measured rather than assumed

Surface	Result
Connection dialog	Elasticsearch and OpenSearch both offered, default port 9200, no connection-string field, TLS/SSH panels present; Test Connection → Connected successfully (19ms) with no database name
Schema explorer	"Search indices or fields…", 3 indices; .internal_bookkeeping, .plugins-ml-config and the dotless top_queries-2026.08.19-* all hidden; object leaves flattened to shipping.city / shipping.country; multi-field customer_name.raw absent — and SELECT customer_name.raw is refused live on OpenSearch (SemanticCheckException), which is the measurement that rule was written from
Engine-initiated paging	GROUP BY trace_id over 1500 groups returned 1500 rows through the cursor, trace-0001 … trace-1500
ES has no OFFSET	Load More refused with the provider's own sentence as a toast, row count stayed at 500 — no fabricated page
OpenSearch has OFFSET	Load More 500 → 1000, no duplicates; LIMIT 5 OFFSET 5 returns o-006
Writes	INSERT / UPDATE refused by both grammars with the engine's own message; 1/0 → HTTP 500 categorised as a user error
Gating	No EDIT, no Explain button, no Explain tab, no Create Table
ER diagram	3 boxes, 0 edges, per-field types — exactly what declaresForeignKeys: false promises
Monitoring	OpenSearch 3.8.0, DB size 381.68 KB, 3 tables, "no limit published" for connections, "Not measured" for cache hit; Sessions empty and honest
Turkish data	Şeyma Çağlar, İbrahim Öztürk, Gökhan Şahin, İzmir, Şanlıurfa intact through query, grid and ER labels

Four defects the browser found, all fixed in ca98d72

1. The first click on an index failed on Elasticsearch. "Select Top 50 Documents" generated SELECT * FROM orders LIMIT 50; — parsing_exception, "extraneous input ';' expecting <EOF>". "Generate Query" failed identically. elasticsearch.md §5.4 had already recorded this as a fact about the engine rather than fixing it. Now ProviderCapabilities.statementTerminator: "none" on both products (the absence runs on OpenSearch too, so no dialect branch) and the generators ask the capability.
2. The grid rendered NULL over values the API returned. TanStack reads a dot in accessorKey as a deep path, so shipping.city was looked up as row.shipping.city while the row carries that flat key. The API answered {"order_id":"o-001","shipping.city":"İzmir"}; the cell said NULL; the CSV export wrote İzmir. Every object field in a mapping flattens to a dotted leaf, so this was most columns on a search cluster — and the screen was the only surface that lied. Fixed with id + accessorFn.
3. Plan mode wrote the wrong language. A plan run on OpenSearch, asked for one runnable statement, answered with a native aggregation body ({"size":0,"aggs":{…}}) — correct for the product, unrunnable through the SQL endpoint. The guard declined to classify it (NO_STATEMENT), so nothing ran, but the deliverable was unusable. queryLanguage: "sql" was already true and settled nothing: the engine's name carries the stronger prior. ProviderLabels.statementLanguage now names the endpoint and rules out each product's own alternatives (ES|QL upstream, PPL on the fork). After the fix the same objective drafted SELECT service, COUNT(*) AS error_count, AVG(latency_ms) … GROUP BY service ORDER BY error_count DESC, the guard read it as a bounded read, and the statement runs on the cluster.
4. Two maintenance entries led nowhere. An index is addressable, so Redis schema-explorer actions generate MongoDB syntax: nothing the sidebar produces can be executed #427's tablesAreDerivedGroupings gate did not catch it: "Index Statistics" and "Merge Segments" rendered on an engine declaring supportsMaintenance: false, and /admin/operations gates its Global Operations card on that same capability — so the click landed on a page with no controls, no error, and none of the wording written for exactly that moment. The gate now requires an addressable row and a declared capability.

Plus a tri-sync fix: both docs said "multi-fields kept" in their §11 coverage summaries while introspect.ts drops them by a measured rule.

Re-verified in the browser after the fixes: the generated query is SELECT * FROM orders LIMIT 50 (runs, 40 rows), shipping.city renders İzmir / Ankara, the two maintenance entries are gone while Profile Table / Generate Code / Generate Test Data remain, and "Generate Query" emits "shipping.city" double-quoted (ES's form, from identifierQuoting) and runs.

Deliberately not fixed here — recorded in docs/BACKLOG.md, each pre-dating this PR and spanning every engine: the monitoring Queries tab tells all eleven engines to enable pg_stat_statements (U12); BEGIN/SANDBOX render where no provider declares transactions and /api/db/transaction refuses with 400 (U13); monitoring reports Buffer Pool 0% / Deadlocks 0 as measurements on engines that keep no such counter (U14).

Gates after the fixes: format · lint · typecheck · knip · test (7827 pass, 0 fail) · build, plus test:coverage + coverage:check at 38144/38144 lines (100.00%), build:lib and attw-relevant surfaces rebuilt.

[sonarqubecloud]

Quality Gate passed

Issues
13 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.5% Duplication on New Code

See analysis details on SonarQube Cloud