HDDS-12495. Add metadata flag to check block existence in ozone debug verify replicas

[sarvekshayr]

What changes were proposed in this pull request?

Introduced a --metadata flag under ozone debug replicas verify command to check for block existence using GetBlock calls to the datanodes. For each key, it iterates through all replicas and verifies block presence.

What is the link to the Apache JIRA

HDDS-12495

How was this patch tested?

Tested the patch on a docker cluster.

1. If block data is present on all nodes, it will be indicated with "status": "BLOCK_EXISTS".

bash-5.1$ ozone debug replicas verify --metadata / --output-dir /tmp | jq  
{
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/1679091",
  "blockID": "conID: 1 locID: 115816896921600007 bcsId: 25 replicaIndex: null",
  "status": "BLOCK_EXISTS",
  "pass": true
}
{
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/45c48cc",
  "blockID": "conID: 1 locID: 115816896921600010 bcsId: 38 replicaIndex: null",
  "status": "BLOCK_EXISTS",
  "pass": true
}
{
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/8f14e45",
  "blockID": "conID: 1 locID: 115816896921600008 bcsId: 29 replicaIndex: null",
  "status": "BLOCK_EXISTS",
  "pass": true
}

2. If block data is missing in some or all nodes, it will be indicated with "status": "MISSING_REPLICAS".

bash-5.1$ ozone debug replicas verify --metadata / --output-dir /tmp | jq
{
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/1679091",
  "blockID": "conID: 1 locID: 115816896921600007 bcsId: 25 replicaIndex: null",
  "status": "MISSING_REPLICAS",
  "pass": false
}
{   
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/45c48cc",
  "blockID": "conID: 1 locID: 115816896921600010 bcsId: 38 replicaIndex: null",
  "status": "MISSING_REPLICAS",
  "pass": false
}
{
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/8f14e45",
  "blockID": "conID: 1 locID: 115816896921600008 bcsId: 29 replicaIndex: null",
  "status": "MISSING_REPLICAS",
  "pass": false
}

3. If some error is encountered while fetching details, it throws "status": "ERROR" along with the error message.

bash-5.1$ ozone debug replicas verify --metadata / --output-dir /tmp | jq
{
  "key": "ockrwvolume/ockrwbucket/vnmnn1ltsx/45c48cc",
  "status": "ERROR",
  "message": "No Route to Host from  10a39f1ddd74/172.31.0.7 to scm:9860 failed on socket timeout exception: java.net.NoRouteToHostException: No route to host; For more details see:  http://wiki.apache.org/hadoop/NoRouteToHost",
  "pass": false
}

[sadanand48]

This is exactly what ozone debug chunkinfo does i.e gets block info from the OM and performs getBlock from all its replica in the pipeline and prints it. Why add another tool to do the same thing? If the block info is missing from any node in the debug chunkinfo output it means MISSING else BLOCK_EXISTS

[sarvekshayr]

This is exactly what ozone debug chunkinfo does i.e gets block info from the OM and performs getBlock from all its replica in the pipeline and prints it. Why add another tool to do the same thing? If the block info is missing from any node in the debug chunkinfo output it means MISSING else BLOCK_EXISTS

Both the commands do have similar functionality.
@errose28 please take a look.

[errose28]

Thanks for looking at this @sadanand48

Although the two tools use the same backend API, they don't do the same thing. ozone debug replicas verify is designed to be run for large amounts of data and produce minimal pass/fail output for each key and replica. In the final version output will be able to be further filtered to only show failures and omit failure reasons. See HDDS-12207 for a better idea of what this will look like.

ozone debug replicas chunk-info should give far more verbose output for a single key. The information does not make sense to aggregate at scale the same way a list of pass/fail checks does. After looking at this command it is going to need some work to improve usability which is described under HDDS-12645.

[errose28]

Thanks for adding this @sarvekshayr. Can we add an integration test for the failure cases, and an acceptance test for the CLI in the regular cases?

[errose28]

Thanks for adding this! Just some minor comments.

[errose28]

I think this has been rolled into #8248. @sarvekshayr should we close this and the corresponding Jira and continue work there?

[sarvekshayr]

I think this has been rolled into #8248. @sarvekshayr should we close this and the corresponding Jira and continue work there?

Yes, we can close this PR.