Add hive metastore catalog support

[Xuanwo]

Tasks

• feat: Add hms catalog layout #112
• Implement catalog trait for hms

[marvinlanhenke]

@Xuanwo I've started looking into this (just started yesterday taking a look at the codebase in general) - how would you go about unit testing the catalog trait for hms? The integration test could probably be done using the docker-compose utils etc. but I am not sure how to best unit test this? Do you have any suggestions or pointers for me to look at? Thank you very much and best regards.

EDIT:
I've done a simple prototype with my hive-metastore running locally via docker-compose, which works just fine. So testing via integration tests seems to be no problem.

Still unsure about how to unit-test this.

[Xuanwo]

I've done a simple prototype with my hive-metastore running locally via docker-compose, which works just fine. So testing via integration tests seems to be no problem.

It's fine to have integration tests. It doesn't make sense to me to mock HMS.

[marvinlanhenke]

@Xuanwo
Yes, I thought the same - mocking the HMS would not be beneficial.
I've setup the basic test-infra for the hms-catalog. In the next couple of days I will try to work on / finish the implementation of the catalog.

[Xuanwo]

I setup the basis test-infra for the hms-catalog. In the next couple of days I will try to work on / finish the implementation of the catalog.

That's great!

[marvinlanhenke]

@Xuanwo @liurenjie1024
...just a quick update and a request for feedback.

I 'finished' the work on all the namespace operations.
Since this would be my first contribution to this repo, I would appreciate it if someone could take a quick look at the current status.

I just want to make sure it aligns with the rest of the codebase, before I put in the effort to implement the rest of the catalog.
https://github.com/marvinlanhenke/iceberg-rust/blob/hms/crates/catalog/hms/src/catalog.rs#L260-L428

sidenote:
For the hive-metastore docker-compose, I used a file I had from a previous project; however this needs some optimization since the connection to the mysql backend slows down the integration tests. My approach would be to use derby instead of mysql to speed things up - but I havent gotten around to take a look at it.

Anyway best regards and thanks in advance for the feedback.

[Xuanwo]

I 'finished' the work on all the namespace operations.

Awesome!

I just want to make sure it aligns with the rest of the codebase, before I put in the effort to implement the rest of the catalog.

Let's merge this PR first and then start another one. There's no need to combine them into a single large PR.

In short, submit a PR and then we can polish it together.

[marvinlanhenke]

In short, submit a PR and then we can polish it together.

Sounds good to me - looking forward to it.

Best regards.

[marvinlanhenke]

@Xuanwo @liurenjie1024
I'm currently looking at the table operations in particular create_table and I'm having some trouble understanding the handling of the metadata. Looking at the REST-Catalog implementation, I don't see any metadata.json actually being written (physically)? Is this correct or am I missing something?

Anyway, is it correct to assume to use FileIO::from_path(metadata_location) and then create a new_output(...) -> Result<OutputFile> this would align with the python impl:
https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/hive.py#L309

[liurenjie1024]

Hi, @marvinlanhenke I think currently we can leave create_table unimplemented since it has some dependencies, such as transaction api and table metadata builder, otherwise it's currently not practical.

[marvinlanhenke]

[...] since it has some dependencies, such as transaction api and table metadata builder, otherwise it's currently not practical.
Makes sense to me.

So I'd leave create_table, load_table, update_table and drop_table as unimplemented, since they all depend on those dependencies you mentioned - or they need create_table; so those other methods can be tested.

@liurenjie1024
I think it'd be best to create another issue that sketches out the next steps (dependencies) for the upcomming week or two? (Andrew Lamb does this in Datafusion and it helps a lot coordinating the effort).

[liurenjie1024]

Hi, @marvinlanhenke Thanks for the suggestion, I think creating tracking issue and clarifying their dependency would be quite helpful, let me create one for 0.3 release. I'm not sure if an issue for upcoming week or two helps.