templates: set sysctl net.ipv4.tcp_keepalive_time to 30sec

[squeed]

This sets the interval for which the kernel will send TCP keepalives, in
seconds. The default, 7200, is too high and causes idle but alive
connections to be dropped by middleboxes and load balancers.

- How to verify it

$ sysctl net.ipv4.tcp_keepalive_time
net.ipv4.tcp_keepalive_time = 30

- Description for the changelog
Update tcp_keepalive_time to 30 seconds.

[squeed]

@mfojtik this should help with api issues.

[sttts]

/lgtm

[sttts]

/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: squeed, sttts
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: jlebon

If they are not already assigned, you can assign the PR to them by writing /assign @jlebon in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kikisdeliveryservice]

@squeed are there any open issues as reference for us on this?

[wking]

e2e-aws:

fail [github.com/openshift/origin/test/extended/util/cli.go:730]: Apr 16 16:56:22.332: Get https://api.ci-op-8xffl195-c4a31.origin-ci-int-aws.dev.rhcloud.com:6443/apis/user.openshift.io/v1/users/~: dial tcp 3.210.98.127:6443: connect: connection refused
...
fail [github.com/openshift/origin/test/extended/operators/operators.go:164]: Apr 16 16:56:13.822: Some cluster operators never became available <nil>/authentication
...
Flaky tests:

[Conformance][templates] templateinstance impersonation tests should pass impersonation update tests [Suite:openshift/conformance/parallel/minimal]
[Feature:Platform][Smoke] Managed cluster should start all core operators [Suite:openshift/conformance/parallel]
[sig-apps] DisruptionController evictions: enough pods, replicaSet, percentage => should allow an eviction [Suite:openshift/conformance/parallel] [Suite:k8s]

Failing tests:

[Feature:Builds][Conformance] oc new-app  should succeed with a --name of 58 characters [Suite:openshift/conformance/parallel/minimal]

That connection refused might be related, or it might just be a flake.

/retest

[kikisdeliveryservice]

thanks @wking

[cgwalters]

portal article on TCP keepalive which also implies we may want to be tuning the other values too?

Would like to have at least one or two real world bugs that are motivating this change.

[kikisdeliveryservice]

Would like to have at least one or two real world bugs that are motivating this change.

+1

[smarterclayton]

I'm confused why we need to set this at the machine level.

We already set this in many places in the server - for instance, the servers set it with:

	dialer := &net.Dialer{
		Timeout:   30 * time.Second,
		KeepAlive: 30 * time.Second,
	}

which updates the descriptor:

	if tc, ok := c.(*TCPConn); ok && d.KeepAlive > 0 {
		setKeepAlive(tc.fd, true)
		setKeepAlivePeriod(tc.fd, d.KeepAlive)
		testHookSetKeepAlive()
	}

[smarterclayton]

/hold

[smarterclayton]

Kube clients already set:

		dial = (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
		}).DialContext

in k8s.io/client-go/transport/cache.go - we have no need to set an explicit OS value.

[sttts]

@smarterclayton that KeepAlive value is interval only to my understanding. The idle value is still high.

Not sure what I looked at yesterday after an 8h+ working day, but I see this now in Golang's library code:

func setKeepAlivePeriod(fd *netFD, d time.Duration) error {
	// The kernel expects seconds so round to next highest second.
	d += (time.Second - time.Nanosecond)
	secs := int(d.Seconds())
	if err := fd.pfd.SetsockoptInt(syscall.IPPROTO_TCP, syscall.TCP_KEEPINTVL, secs); err != nil {
		return wrapSyscallError("setsockopt", err)
	}
	err := fd.pfd.SetsockoptInt(syscall.IPPROTO_TCP, syscall.TCP_KEEPIDLE, secs)
	runtime.KeepAlive(fd)
	return wrapSyscallError("setsockopt", err)
}

And we call this from k8s.io/apiserver with 120s, plus the mentioned client-go code.

[squeed]

Yup, kube clients have a 30sec keepalive timeout:

[root@ip-10-0-139-201 core]# netstat -town
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer
tcp        0      0 10.0.139.201:35362      10.0.130.31:6443        ESTABLISHED keepalive (24.41/0/0)

This probably isn't necessary, except for the (very minimal) potential non-client-go clients.

[cgwalters]

This PR has a global effect on all TCP connections, including outbound etc. So if we're just trying to affect kube clients then it's a pretty heavy hammer.

Bigger picture it feels like this is a kind of "race to the bottom" - NAT gateways want to clean up idle connections, apps want their TCP connections to stay open and not have spurious closed errors...

[openshift-ci-robot]

@squeed: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-upgrade	1310dda	link	/test e2e-aws-upgrade

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.