ES|QL Data Federation: Query files in Amazon S3 (Experimental) #307

Value Proposition

Not all enterprise data lives in Elasticsearch. Historical logs, compliance archives, security reference data, and partner feeds accumulate in object storage because moving them is expensive or impractical. Reaching that data from Elasticsearch has required a second query engine, a separate interface, and manual reconciliation of results.

ES|QL Data Federation extends Elasticsearch to query data in external sources including object storage, with no second query engine and no pipelines to build. Register a data source, name a dataset, and query it with the same ES|QL syntax and Kibana interface you use for indexed data.

Expected Outcome

Shipped as Experimental in 9.5 with Amazon S3 as the first supported object store.

• Datasets as virtual indices — queried via FROM dataset_name with ES|QL (WHERE, STATS, EVAL, SORT, LIMIT, LOOKUP JOIN, and more)
• Parquet, NDJSON, and CSV/TSV formats, with schema inferred at query time
• gzip and zstd compression support for text formats
• Filter pushdown, column projection, and LIMIT pushdown for query performance
• Queries can combine Elasticsearch indices and external datasets
• Kibana UI for managing data sources and datasets
• RBAC via standard Elasticsearch index privileges on dataset names
• Encrypted credential management (stored at rest in cluster state, decrypted only at query time)