Repository navigation
[datafusion-contrib] AWS Glue Integration #2206
Description
Activity
I created issue on the official aws-sdk-rust repo to add support for Glue. I believe they prioritize based on getting 👍 so anyone interested in this please up vote there if you get the chance.
I suppose this would mean using rusoto in the meantime. im unsure what this would mean if we tried to use rusoto for glue and aws-sdk-rust for s3 (via
datafusion-objectstore-s3). If needed / any type of incompatibility perhaps we could add a rusoto feature todatafusion-objectstore-s3that could be used until official support lands.Reacted by Raphael Taylor-Davies, Ayush Bindlish and naseerscorpioI have some sample code which uses the aws-sdk-glue client, iterates over all databases and tables, and registers them in a memorycatalogprovider
https://gist.github.com/timvw/84246389d9c79fc0bf07570c625fdaf4
Reacted by Raphael Taylor-Davies, Matthew Turner, Zach Paden, Hoang Pham, DJ Spatoulas and feniljain
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This has been discussed in various places, #907 and datafusion-contrib/datafusion-objectstore-s3#53 to name a few, so creating an issue for visibility.
Describe the solution you'd like
I would propose creating a new datafusion-contrib crate, perhaps
datafusion-catalog-glue, which communicates with an AWS Glue Data Catalog.I'll leave the exact design for whoever picks this up, but I might expect something along the following lines.
GlueCatalogwith an optional catalog IDasync fn GlueCatalog::list_databases(&self) -> Vec<String>to list the databasesasync fn GlueCatalog::get_database(&self, name: &str) -> Result<GlueDatabase>to get a databaseSchemaProviderforGlueDatabaseI think it should be possible to reuse the
FileScanConfigstructure used byListingTableto simplify implementation of theTableProvider.Describe alternatives you've considered
We could not support AWS Glue
Additional context
This will help with datafusion-contrib/datafusion-objectstore-s3#53 by alleviating the need to infer the schema from the files on every query, and only listing files in non-pruned partitions.
This may need to depend on https://github.com/datafusion-contrib/datafusion-objectstore-s3 as I think it will still need to list S3 in order to get the files within a given partition.
The Glue API is not the snappiest of things, so a future extension might be to cache the metadata returned, as is done by the Java client.