Scheduler problems don't have valuable metrics on health endpoint

[poy]

We encountered a situation where we had mismatched certs between the scheduler and the log cache nodes. I would have expected to look at the health endpoints and have them better guide me.

Proposed new metrics (all counters):

Scheduler

• SuccessfulRoutingTableSets
• FailedRoutingTableSets

Log Cache

• SuccessfulRoutedEnvelopes
• FailedRoutedEnvelopes
• RoutingTableSets

In the given failure, if I had seen LogCache.RoutingTableSets set to 0 for all the nodes, I would have quickly been guided to the Scheduler. Instead I first looked at log cache nodes (assuming there was routing problems between nodes). The lack of metrics about the intra communication (SuccessfulRoutedEnvelopes and FailedRoutedEnvelopes would better inform) leaves operators in the dark a bit here.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/157951242

The labels on this github issue will be updated when the story is started.