ci: Maintaining docker image for integration tests.

[liurenjie1024]

Is your feature request related to a problem or challenge?

Currently in the integrations, we create docker images on the fly, see

iceberg-rust/crates/integration_tests/testdata/docker-compose.yaml

Line 73 in 83ca98f

build: spark/

iceberg-rust/crates/catalog/hms/testdata/hms_catalog/docker-compose.yaml

Line 42 in 0942e3c

image: iceberg-hive-metastore

While this is easy to maintain, there are drawbacks about this approach:

1. It slows down tests
2. It increases device space usage on github action runner, which make tests unstable.

Describe the solution you'd like

We should consider maintaining the docker images used in integration tests to speed up integration tests.

Willingness to contribute

I can contribute to this feature at this time

[xxchan]

These images are mainly just thin wrappers of curl the jars (seems to be a convention of the java ecosystem). Therefore parhaps switching to pre-built image won't give too large improvement.

[xxchan]

Some improvement ideas:

• The slowest part is actually apt update. Therefore, perhaps we can utilize GHC docker cache instead.
• For spark image, there's actually a pre-built one: https://hub.docker.com/r/tabulario/spark-iceberg (I remember there's discussion in the community to change it from tabulario to an official one).
• Sunsetting the old integration tests in favor of the new one: test: Introduce integration test framework. #581 (review)

[liurenjie1024]

The slowest part is actually apt update. Therefore, perhaps we can utilize GHC docker cache instead.

This seems making things more complicated, why not just maintaining a docker image?

[liurenjie1024]

cc @Xuanwo @Fokko Do you have ideas what's the best practice of publishing docker images in apache projects?

[kevinjqliu]

Currently in the integrations, we create docker images on the fly

i think this is copied over from pyiceberg. I have this issue opened from the pyiceberg side for the same purpose apache/iceberg-python#1527

Do you have ideas what's the best practice of publishing docker images in apache projects?

we built the iceberg-rest-fixture https://hub.docker.com/r/apache/iceberg-rest-fixture via https://github.com/apache/iceberg/blob/main/.github/workflows/publish-iceberg-rest-fixture-docker.yml

i think theres some amount of set up required from apache infra.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'