Statsd emitter can stall checks when statsd is down; opens a new connection per metric

[dolph]

Summary

The statsd subsystem has reliability and efficiency problems that matter most in monitor mode but are visible everywhere:

1. Buffer of 100 with no overflow handling. queue = make(chan string, 100). The Increment/Count/Timer/Gauge helpers all do a blocking queue <- .... If StatsdSender can't drain (statsd down, slow TCP, etc.), the channel fills up and every metric call in the check path blocks. The blocking happens inside Check / Dial / HTTPS / Ping, so checks themselves stall.
2. New connection per metric. StatsdSender does Dial/Write/Close for every packet. That's expensive even for UDP (kernel work + ephemeral port churn) and catastrophic for TCP statsd (full 3-way handshake + close per metric, plus the connect blocks the queue drain — feeding back into problem check takes 10x longer to run than wait for the same set of URLs #1).
3. All errors are silently swallowed. if conn, err := net.Dial(...); err == nil — if dial fails there is no log, no fallback, no metric (it's the metric pipeline itself), and the queue grows.
4. Channel is never closed. StatsdSender runs forever and there's no graceful shutdown / drain on os.Exit. The last batch of metrics around a check failure may be lost.

Code

statsd.go:11-50:

var queue = make(chan string, 100)

func Increment(metric string, tags []string) {
	Count(metric, 1, tags)
}

func Count(metric string, value int, tags []string) {
	queue <- fmt.Sprintf("%s:%d|c|#%s", metric, value, formatTags(tags))   // blocking
}

...

func StatsdSender(config *Config) {
	for s := range queue {
		statsdHostPort := fmt.Sprintf("%s:%d", config.StatsdHost, config.StatsdPort)
		if conn, err := net.Dial(config.StatsdProtocol, statsdHostPort); err == nil {
			io.WriteString(conn, s)            // ignored error
			conn.Close()
		}
		// err != nil: silently dropped, no backoff, no log
	}
}

Impact

• With TCP statsd misconfigured or unreachable, connectivity check can hang in Increment after ~100 events.
• Even with UDP, the connection-per-packet pattern is a needless syscall storm; over time in monitor mode this is the single largest source of overhead in the process.
• validate-config does not start StatsdSender (see connectivity.go:25-39), but it also doesn't currently call any metric helpers, so it happens to work — a fragile invariant.

Suggested fix

• Hold a single persistent statsd connection (or a *net.UDPConn with WriteToUDP), reconnecting on error with backoff.
• Make the metric helpers non-blocking via select { case queue <- s: default: drop_and_count_drops } so a failing statsd path can never block the check path.
• Log dial/write errors at most once per N seconds.
• Drain the queue on shutdown (signal handler + wg.Wait()).