Skip to content

HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN#5793

Merged
sumitagrawl merged 4 commits intoapache:masterfrom
devmadhuu:HDDS-9883
Jan 4, 2024
Merged

HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN#5793
sumitagrawl merged 4 commits intoapache:masterfrom
devmadhuu:HDDS-9883

Conversation

@devmadhuu
Copy link
Copy Markdown
Contributor

@devmadhuu devmadhuu commented Dec 14, 2023

What changes were proposed in this pull request?

There is a difference between SCM's IncrementalContainerReportHandler and Recon's IncrementalContainerReportHandler , Recon always connects to SCM and verify each container before adding new container to its own containerStateManager cache. This could be a bottle neck if SCM may respond slow and frequent ICR requests may pile up in queue. So as of now, this PR will improve below multiple things:

  • Recon to verify containers in batches from SCM on receive of ICR request from DNs.

  • Reduce the scmClient configs for Recon before connecting to SCM:
   - `hdds.scmclient.rpc.timeout` - 1 min (Default value is 15 mins)
   - `hdds.scmclient.failover.max.retry` - 3 (Default value is dynamic and computed, but based on default values, computed value is 15)
    Above 2 SCM client configs will be updated to respective new values as mentioned for recon to connect to SCM. These 2 SCM client configs will be exposed and mapped with new recon configs to be able to adjust independently in recon.

  **New configs in Recon:**
       -  `ozone.recon.scmclient.rpc.timeout`
       -  `ozone.recon.scmclient.failover.max.retry`

  • Merge the Incremental container report (ICR) to existing list of ICR reports.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9883

How was this patch tested?

Tested using existing Junit tests after updating existing test - TestReconIncrementalContainerReportHandler

@devmadhuu
Copy link
Copy Markdown
Contributor Author

devmadhuu commented Dec 14, 2023

@sumitagrawl Pls review.

@devmadhuu devmadhuu marked this pull request as ready for review December 15, 2023 06:07
@adoroszlai adoroszlai changed the title HDDS-9883. Recon - Improve the performance of processing of IncrementalContainerReport requests from DN. HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN Dec 15, 2023
@kerneltime
Copy link
Copy Markdown
Contributor

kerneltime commented Dec 18, 2023

cc @tanvipenumudy

sumitagrawl
sumitagrawl reviewed Jan 3, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@sumitagrawl sumitagrawl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devmadhuu Thanks for working over this, have few comments

Comment thread ...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconContainerReportQueue.java Outdated
Comment thread ...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java Outdated
sumitagrawl
sumitagrawl approved these changes Jan 4, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@sumitagrawl sumitagrawl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@devmadhuu LGTM

@sumitagrawl sumitagrawl merged commit 6021582 into apache:master Jan 4, 2024
@jojochuang
Copy link
Copy Markdown
Contributor

jojochuang commented Feb 23, 2024

I'm getting this exception which loosk related to this PR

2024-02-22 22:33:39,568 WARN [IPC Server handler 22 on 9891]-org.apache.hadoop.ipc.Server: IPC Server handler 22 on 9891, call Call#36 Retry#0 org.apache.hadoop.ozone.protocol.ReconDatanodeProtocol.submitRequest from 10.140.112.130:34368
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.addAll(Collections.java:1067)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher$IncrementalContainerReportFromDatanode.mergeReport(SCMDatanodeHeartbeatDispatcher.java:384)
at org.apache.hadoop.ozone.recon.scm.ReconContainerReportQueue.mergeIcr(ReconContainerReportQueue.java:41)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addIncrementalReport(ContainerReportQueue.java:115)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addValue(ContainerReportQueue.java:173)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:187)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:42)
at org.apache.hadoop.hdds.server.events.FixedThreadPoolWithAffinityExecutor.onMessage(FixedThreadPoolWithAffinityExecutor.java:178)
at org.apache.hadoop.hdds.server.events.EventQueue.fireEvent(EventQueue.java:220)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.dispatch(SCMDatanodeHeartbeatDispatcher.java:159)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.sendHeartbeat(SCMDatanodeProtocolServer.java:282)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.processMessage(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:113)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.submitRequest(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:92)
at org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$StorageContainerDatanodeProtocolService$2.callBlockingMethod(StorageContainerDatanodeProtocolProtos.java:45283)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)

@devmadhuu
Copy link
Copy Markdown
Contributor Author

devmadhuu commented Feb 23, 2024

I'm getting this exception which loosk related to this PR

2024-02-22 22:33:39,568 WARN [IPC Server handler 22 on 9891]-org.apache.hadoop.ipc.Server: IPC Server handler 22 on 9891, call Call#36 Retry#0 org.apache.hadoop.ozone.protocol.ReconDatanodeProtocol.submitRequest from 10.140.112.130:34368
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.addAll(Collections.java:1067)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher$IncrementalContainerReportFromDatanode.mergeReport(SCMDatanodeHeartbeatDispatcher.java:384)
at org.apache.hadoop.ozone.recon.scm.ReconContainerReportQueue.mergeIcr(ReconContainerReportQueue.java:41)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addIncrementalReport(ContainerReportQueue.java:115)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addValue(ContainerReportQueue.java:173)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:187)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:42)
at org.apache.hadoop.hdds.server.events.FixedThreadPoolWithAffinityExecutor.onMessage(FixedThreadPoolWithAffinityExecutor.java:178)
at org.apache.hadoop.hdds.server.events.EventQueue.fireEvent(EventQueue.java:220)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.dispatch(SCMDatanodeHeartbeatDispatcher.java:159)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.sendHeartbeat(SCMDatanodeProtocolServer.java:282)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.processMessage(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:113)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.submitRequest(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:92)
at org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$StorageContainerDatanodeProtocolService$2.callBlockingMethod(StorageContainerDatanodeProtocolProtos.java:45283)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)

This issue is being tracked at HDDS-10413

swamirishi pushed a commit to swamirishi/ozone that referenced this pull request Jun 10, 2024
@devmadhuu @singhdev76
CDPD-63086. HDDS-9883. Recon - Improve the performance of processing …
a5c3f41 
…IncrementalContainerReport from DN (apache#5793)

(cherry picked from commit 6021582)
Change-Id: Ic9fa1c235a2c347cf9375f6a1da9a47f5dcba213
xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024
@devmadhuu @xichen01
HDDS-9883. Recon - Improve the performance of processing IncrementalC…
c3cedd0 
…ontainerReport from DN (apache#5793)

(cherry picked from commit 6021582)
xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024
@devmadhuu @xichen01
HDDS-9883. Recon - Improve the performance of processing IncrementalC…
4554b36 
…ontainerReport from DN (apache#5793)

(cherry picked from commit 6021582)
xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024
@devmadhuu @xichen01
HDDS-9883. Recon - Improve the performance of processing IncrementalC…
82c6118 
…ontainerReport from DN (apache#5793)

(cherry picked from commit 6021582)
@xichen01 xichen01 mentioned this pull request Jul 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sumitagrawl sumitagrawl sumitagrawl approved these changes

@kerneltime kerneltime Awaiting requested review from kerneltime

@duongkame duongkame Awaiting requested review from duongkame

Assignees

No one assigned

Labels

recon

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@devmadhuu @kerneltime @jojochuang @sumitagrawl @adoroszlai