HDDS-8773. [S3G] Improve list performance in FSO bucket

[whbing]

What changes were proposed in this pull request?

Option --delimiter '/' in ListObjectsV2 Is a commonly used option. Common scenarios are as follows:

• simple aws s3 cmd like aws s3 --endpoint http://<ip>:9878 ls buk/dir/
• aws s3api cmd like aws s3api --endpoint http://<ip>:9878 list-objects-v2 --bucket buk1-ln --prefix '' --delimiter '/'
• fuse mount s3 bucket and ls cmd

Only listing immediate children node of prefix is needed in the above scenario.

In the current implementation of FSO bucket, the object is listed by Depth-First-Search algorithm, and then filtered by delimiter, which greatly reduces the performance.

It was reduced from tens of seconds to 3 seconds in my test environment after optimization.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-8773

How was this patch tested?

• Unit test
• CLI commands in ozone cluster environment as follow
  info : s3v/buk1-ln linked fso bucket, and simulate multiple calls to the iterator by reducing the parameter ozone.client.list.cache

$ hadoop fs -count -v ofs://om/s3v/buk1-ln/*
   DIR_COUNT   FILE_COUNT       CONTENT_SIZE PATHNAME
          14         1703               2177 ofs://om/s3v/buk1-ln/test
           7            1                238 ofs://om/s3v/buk1-ln/test0
          93       373113             373113 ofs://om/s3v/buk1-ln/test1
          31       227379             227379 ofs://om/s3v/buk1-ln/test2

before optimization:

time aws s3api --endpoint http://`hostname -i`:9878 list-objects-v2 --bucket buk1-ln --prefix '' --delimiter '/'
{
    "CommonPrefixes": [
        {
            "Prefix": "test/"
        },
        {
            "Prefix": "test0/"
        },
        {
            "Prefix": "test1/"
        },
        {
            "Prefix": "test2/"
        }
    ]
}

real	0m35.005s
user	0m0.961s
sys	0m0.181s

after optimization:

time aws s3api --endpoint http://`hostname -i`:9878 list-objects-v2 --bucket buk1-ln --prefix '' --delimiter '/'
{
    "CommonPrefixes": [
        {
            "Prefix": "test/"
        },
        {
            "Prefix": "test0/"
        },
        {
            "Prefix": "test1/"
        },
        {
            "Prefix": "test2/"
        }
    ]
}

real	0m3.106s
user	0m0.915s
sys	0m0.131s

Detail data are shown in the following table:

time with prefix	test0/	test/	test1/	test2/	''
before optimization	3.171s	3.268s	21.861s	15.120s	35.005s
after optimization	3.127s	3.143s	3.182s	3.170s	3.106s

[adoroszlai]

@tanvipenumudy please take a look

[whbing]

@adoroszlai Added test, please help trigger ci, Thanks ! ( ci passed in my branch https://github.com/whbing/ozone/actions/runs/5233785220)

[ChenSammi]

Hi @kerneltime , could you please help to review this patch?

[adoroszlai]

@duongkame please review

[zhtttylz]

Great catch and patch here,just leave some nit comments inline,JFYI.

[ChenSammi]

Hi @whbing , looks like the build has some issue, could you take a look?

[whbing]

Thanks @captainzmc for merging #5003 upon which this PR depends. Now this PR is ready and I have successfully tested some scenarios in my environment. @captainzmc @ChenSammi @adoroszlai If you have time, thanks for helping to review this pr.

[adoroszlai]

@captainzmc @duongkame @kerneltime @tanvipenumudy please review

[whbing]

Thanks, and look forward to the review. The PR is working well in our cluster. Just fine-tuned and added comments in above new commit.

[kerneltime]

Thank you @whbing for this very useful contribution. I will get this review done this week. cc @tanvipenumudy @duongkame

[tanvipenumudy]

Thank you @whbing for this important change, please find a comment.

[ChenSammi]

Thanks @whbing , the patch overall looks good.