[close #749] Fix health checking issue (#748)

[ti-chi-bot]

This is an automated cherry-pick of #748

What problem does this PR solve?

Issue Number: close #749

pingcap/tispark#2707

Problem Description: StoreHealthyChecker fails to check the status of TiFlash

What is changed and how does it work?

Use isMppalive RPC to probe the status of TiFlash.

Check List for Tests

Manual test：

I have tested this locally with TiSpark. TiSpark will request TiFlash using client-java

1. TiSpark will with the client-java before this PR will throw errors even if TiFlash is alive

23/05/20 15:31:12 WARN StoreHealthyChecker: fail to check TiFlash health, regrade as unhealthy
org.tikv.shade.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 1.999882667s. [closed=[], open=[[buffered_nanos=2005712916, waiting_for_connection]]]
	at org.tikv.shade.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:287)
	at org.tikv.shade.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:268)
	at org.tikv.shade.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:175)
	at org.tikv.common.region.StoreHealthyChecker.checkTiFlashHealth(StoreHealthyChecker.java:99)
	at org.tikv.common.region.StoreHealthyChecker.checkStoreHealth(StoreHealthyChecker.java:86)
	at org.tikv.common.region.StoreHealthyChecker.run(StoreHealthyChecker.java:154)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
23/05/20 15:31:13 WARN RangeSplitter: Cannot find valid store on TiFlash(engine=tiflash)

2. TiSpark success with the client-java build by this pr

Side effects

• Possible performance regression, WHY: TBD
• Increased code complexity, WHY: TBD
• Breaking backward compatibility, WHY: TBD
• NO side effects

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation
• Need to be included in the release note
• NO related changes

[shiyuhang0]

LGTM

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (release-3.3@e67b1b6). Click here to learn what that means.
Patch has no changes to coverable lines.

❗ Current head aa1b417 differs from pull request most recent head ff338f4. Consider uploading reports for the commit ff338f4 to get more accurate results

Additional details and impacted files

@@              Coverage Diff               @@
##             release-3.3     #753   +/-   ##
==============================================
  Coverage               ?   37.90%           
  Complexity             ?     1610           
==============================================
  Files                  ?      278           
  Lines                  ?    17511           
  Branches               ?     1991           
==============================================
  Hits                   ?     6637           
  Misses                 ?    10211           
  Partials               ?      663           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.