Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Whilst AWS Glue can act as a Hive compatible metastore, and #2206 tracks adding support for it, this is of course not an option for people running outside of AWS. It would be good to provide some story for these users.
Describe the solution you'd like
I'm not very familiar with the Hive ecosystem, but it would appear that when people refer to the Hive metastore, they are referring to a client-side implementation, as opposed to a some service like Zookeeper, or Kafka.
I found this exceptionally confusing, but there appear to be three options:
- An implementation using an embedded Derby database
- An implementation using a remote JDBC database
- An implementation using a remote Hive server
I don't think 1. is useful for DataFusion's purposes, and 2. is likely tricky to implement without running into subtle incompatibilities with the Java version, I therefore think 3. is likely the best option. From my understanding this effectively takes the implementation of 2. and runs it as a network service with a thrift API. This should be relatively straightforward to support and can likely share a lot of code with any AWS Glue support.
Describe alternatives you've considered
We could not support this
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Whilst AWS Glue can act as a Hive compatible metastore, and #2206 tracks adding support for it, this is of course not an option for people running outside of AWS. It would be good to provide some story for these users.
Describe the solution you'd like
I'm not very familiar with the Hive ecosystem, but it would appear that when people refer to the Hive metastore, they are referring to a client-side implementation, as opposed to a some service like Zookeeper, or Kafka.
I found this exceptionally confusing, but there appear to be three options:
I don't think 1. is useful for DataFusion's purposes, and 2. is likely tricky to implement without running into subtle incompatibilities with the Java version, I therefore think 3. is likely the best option. From my understanding this effectively takes the implementation of 2. and runs it as a network service with a thrift API. This should be relatively straightforward to support and can likely share a lot of code with any AWS Glue support.
Describe alternatives you've considered
We could not support this