HDDS-9976. Ozone StateContext Memory leak for DeleteBlocksCommand when queue is full

[xichen01]

What changes were proposed in this pull request?

Solving StateContext memory leaks.

Root cause

The cmdStatusMap in the StateContext will only be cleaned up if the command is not in Pending status.

ozone/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/report/CommandStatusReportPublisher.java

Lines 76 to 85 in 077e09b

	iterator.forEachRemaining(key -> {
	CommandStatus cmdStatus = map.get(key);
	// If status is still pending then don't remove it from map as
	// CommandHandler will change its status when it works on this command.
	if (!cmdStatus.getStatus().equals(Status.PENDING)) {
	map.remove(key);
	}
	builder.addCmdStatus(cmdStatus.getProtoBufMessage());
	});
	return builder.getCmdStatusCount() > 0 ? builder.build() : null;

When the queue of DeleteBlocksCommand is full, the command in commandQueue cannot be added to the queue of DeleteBlocksCommand and will be discarded from commandQueue, so the commands are never updated, resulting in a memory leak.

Solution

Here we set the commands that can't be added to DeleteBlocksCommand queue to FAILED state, so that they can be cleaned up.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9976

How was this patch tested?

unit test

[adoroszlai]

Thanks @xichen01 for the patch.