Skip to content

HDDS-9031. Upgrade acceptance tests to Docker Compose v2#6667

Merged
adoroszlai merged 4 commits intoapache:masterfrom
adoroszlai:HDDS-9031
May 16, 2024
Merged

HDDS-9031. Upgrade acceptance tests to Docker Compose v2#6667
adoroszlai merged 4 commits intoapache:masterfrom
adoroszlai:HDDS-9031

Conversation

@adoroszlai
Copy link
Copy Markdown
Contributor

@adoroszlai adoroszlai commented May 11, 2024 •
edited
Loading

What changes were proposed in this pull request?

Change acceptance tests to use Docker Compose v2, if available.

ozonescripts environment has some non-test scripts (start.sh, etc.), these require Docker Compose v2 after the change.

env-compose.robot is unused, can be removed.

https://issues.apache.org/jira/browse/HDDS-9031

How was this patch tested?

Acceptance tests with Docker Compose v1 uninstalled:
https://github.com/adoroszlai/ozone/actions/runs/9041151300

Regular CI:
https://github.com/adoroszlai/ozone/actions/runs/9041151733

@adoroszlai adoroszlai self-assigned this May 11, 2024
@adoroszlai adoroszlai requested review from errose28 and smengcl May 13, 2024 07:27
@kerneltime
Copy link
Copy Markdown
Contributor

kerneltime commented May 14, 2024 •
edited by smengcl
Loading

ozonescripts environment has some non-test scripts (start.sh, etc.), these require Docker Compose v2 after the change.

Can you elaborate on why they require docker compose v2 vs an optional use like the hadoop-ozone/dist/src/main/compose/testlib.sh script?

@smengcl
Copy link
Copy Markdown
Contributor

smengcl commented May 15, 2024 •
edited
Loading

ozonescripts environment has some non-test scripts (start.sh, etc.), these require Docker Compose v2 after the change.

Can you elaborate on why they require docker compose v2 vs an optional use like the hadoop-ozone/dist/src/main/compose/testlib.sh script?

Actually testlib.sh is already using v2 if it is available. It is a clever trick to override docker-compose to use v2 if that is available:

https://github.com/apache/ozone/pull/6667/files#diff-f26853740acf7f82a794bddb61e32ae145d12fdb0e08cd4a4c18a991e93742d0R34-R36

Back to your point I think it is possible to use the same trick as well.

smengcl
smengcl approved these changes May 15, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@smengcl smengcl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @adoroszlai !

@adoroszlai
Copy link
Copy Markdown
Contributor Author

adoroszlai commented May 15, 2024 •
edited
Loading

Can you elaborate on why they require docker compose v2

For these scripts I wanted to avoid using testlib.sh or duplicating the code.

I think it is possible to use the same trick as well.

Done by extracting the docker-compose function definition to its own script. Updated ozonescripts to use it. Also in ozone/run.sh, which I completely missed earlier.

v2-only run with updated patch:
https://github.com/adoroszlai/ozone/actions/runs/9090438317

@adoroszlai adoroszlai merged commit d80c45b into apache:master May 16, 2024
@adoroszlai adoroszlai deleted the HDDS-9031 branch May 16, 2024 07:27
@adoroszlai
Copy link
Copy Markdown
Contributor Author

adoroszlai commented May 16, 2024

Thanks @kerneltime, @smengcl for the review.

jojochuang pushed a commit to jojochuang/ozone that referenced this pull request May 23, 2024
@adoroszlai @jojochuang
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
de22899
errose28 added a commit to errose28/ozone that referenced this pull request May 28, 2024
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10372-re…
97a76d7 
…concile-cli

* HDDS-10239-container-reconciliation: (296 commits)
  HDDS-10897. Refactor OzoneQuota (apache#6714)
  HDDS-10422. Fix some warnings about exposing internal representation in hdds-common (apache#6351)
  HDDS-10899. Refactor Lease callbacks (apache#6715)
  HDDS-10890. Increase default value for hdds.container.ratis.log.appender.queue.num-elements (apache#6711)
  HDDS-10832. Client should switch to streaming based on OpenKeySession replication (apache#6683)
  HDDS-10435. Support S3 object tags for existing requests (apache#6607)
  HDDS-10883. Improve logging in Recon for finalising DN logic. (apache#6704)
  HDDS-8752. Enable TestOzoneRpcClientAbstract#testOverWriteKeyWithAndWithOutVersioning (apache#6702)
  HDDS-10875. XceiverRatisServer#getRaftPeersInPipeline should be called before XceiverRatisServer#removeGroup (apache#6696)
  HDDS-10514. Recon - Provide DN decommissioning detailed status and info inline with current CLI command output. (apache#6376)
  HDDS-10878. Bump zstd-jni to 1.5.6-3 (apache#6701)
  HDDS-10877. Bump Dropwizard metrics to 3.2.6 (apache#6699)
  HDDS-10876. Bump jackson to 2.16.2 (apache#6697)
  HDDS-6116. Remove flaky tag from TestSCMInstallSnapshot (apache#6695)
  HDDS-2643. TestOzoneDelegationTokenSecretManager#testRenewTokenFailureRenewalTime fails intermittently.
  HDDS-10699. Refactor ContainerBalancerTask and TestContainerBalancerTask (apache#6537)
  HDDS-10861. Ozone cli supports default ozone.om.service.id (apache#6680)
  HDDS-10859. Improve error messages when decommission and maintenance fail-early (apache#6678)
  HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
  HDDS-10559. Add a warning or a check to run repair tool as System user (apache#6574)
  ...

Conflicts:
    hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
adoroszlai added a commit to adoroszlai/ozone that referenced this pull request Jul 12, 2024
@adoroszlai
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
6f7c25e 
(cherry picked from commit d80c45b)
adoroszlai added a commit to adoroszlai/ozone that referenced this pull request Jul 12, 2024
@adoroszlai
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
6d803c4 
(cherry picked from commit d80c45b)
spacemonkd pushed a commit to spacemonkd/ozone that referenced this pull request Jul 12, 2024
@adoroszlai @spacemonkd
HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667)
4fe44f4
@spacemonkd spacemonkd mentioned this pull request Jul 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smengcl smengcl smengcl approved these changes

@errose28 errose28 Awaiting requested review from errose28

Assignees

@adoroszlai adoroszlai

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adoroszlai @kerneltime @smengcl