ES|QL Data Federation: Query files in Amazon S3 (Tech Preview) #685

Value Proposition

Not all enterprise data lives in Elasticsearch. Historical logs, compliance archives, security reference data, and partner feeds accumulate in object storage because moving them is expensive or impractical. Reaching that data from Elasticsearch has required a second query engine, a separate interface, and manual reconciliation of results.

ES|QL Data Federation extends Elasticsearch to query data in external sources including object storage, with no second query engine and no pipelines to build. Register a data source, name a dataset, and query it with the same ES|QL syntax and Kibana interface you use for indexed data. Explore data where it lives, and when a dataset becomes part of your regular workload, bring it into Elasticsearch with ease.

Expected Outcome

Ships as Tech Preview on stateful deployments (Elastic Cloud Hosted and self-managed) with Amazon S3 as the first supported object store. Enterprise license. Builds on the Experimental foundation.

Core capability (unchanged scope)

• Datasets as virtual indices — queried via FROM dataset_name with the same ES|QL pipeline used for indexed data (WHERE, STATS, EVAL, SORT, LIMIT, LOOKUP JOIN, and more)
• Parquet, NDJSON, and CSV/TSV in Amazon S3, with schema inferred at query time
• gzip and zstd compression support
• Filter pushdown, column projection, and LIMIT pushdown for query performance
• Queries can combine Elasticsearch indices and external datasets in one session
• RBAC via standard Elasticsearch index privileges on dataset names
• Encrypted credential management (stored at rest in cluster state, decrypted only at query time)

Improvements in this Tech Preview

• On by default for Enterprise stateful deployments (admins can disable)
• Keyless federated identity for S3 (workload identity / no long-lived access keys required)
• Dataset creation wizard and stronger Kibana management UX
• Discover and dashboard time-range and filter support for federated datasets
• ES|QL alerting against federated datasets that have a mapped timestamp field
• Schema discovery and request-filter handling hardened for production preview use

Google Cloud Storage, Azure Blob Storage, and additional source types remain on the roadmap.