Fix pod deletion to be compatible with Kubernetes 1.9#164
Conversation
| switch controllerRef.Kind { | ||
| case "DaemonSet": | ||
| return kc.ExtensionsV1beta1().DaemonSets(sr.Reference.Namespace).Get(sr.Reference.Name, v1meta.GetOptions{}) | ||
| return kc.Extensions().DaemonSets(namespace).Get(controllerRef.Name, v1meta.GetOptions{}) |
There was a problem hiding this comment.
Even if drain uses Extensions, I think the versioned client is probably correct.
There was a problem hiding this comment.
Also, DaemonSet is now also in the apps API group. I don't know if Extensions() will list everything.
* Check each pod's OwnerReferences to determine whether it is owned by an existing Daemonset. If so, exclude it from deletion as the pod would just be recreated by the Daemonset controller * Drop usage of CreatedByAnnotation to deserialize the owner object of a pod when listing pods to delete during agent drains
|
Retested on another v1.9.1 cluster with |
| switch controllerRef.Kind { | ||
| case "DaemonSet": | ||
| return kc.ExtensionsV1beta1().DaemonSets(sr.Reference.Namespace).Get(sr.Reference.Name, v1meta.GetOptions{}) | ||
| return kc.ExtensionsV1beta1().DaemonSets(namespace).Get(controllerRef.Name, v1meta.GetOptions{}) |
There was a problem hiding this comment.
still concerned this won't identify daemonsets installed through the apps API group.
There was a problem hiding this comment.
For what its worth, kubernetes/kubernetes drain (master) only seems to use client.ExtensionsV1beta1 in detecting DaemonSets.
https://github.com/kubernetes/kubernetes/blob/48f69ac964b9a96b55351a3541f285b4cb5611bb/pkg/kubectl/cmd/drain.go#L341
https://github.com/kubernetes/kubernetes/blob/48f69ac964b9a96b55351a3541f285b4cb5611bb/pkg/kubectl/cmd/drain.go#L362
Referring to a specific hash (post 1.9) so the links don't break in future.
There was a problem hiding this comment.
Alright. A typical cluster has a few daemonsets to experiment with:
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system calico-node 4 4 4 4 4 <none> 19h
kube-system kube-apiserver 1 1 1 1 1 node-role.kubernetes.io/master= 19h
kube-system kube-proxy 4 4 4 4 4 <none> 19h
kube-system pod-checkpointer 1 1 1 1 1 node-role.kubernetes.io/master= 19h
monitoring node-exporter 4 4 4 4 4 <none> 13h
reboot-coordinator container-linux-update-agent 4 4 4 4 4 <none> 13h
At present, these daemonsets are all defined as "extensions/v1beta1 DaemonSet" objects. We can
- Switch a few to "apps/v1 DaemonSet"
- Intentionally orphan the DaemonSet pods with
kubectl delete daemonset node-exporter --cascade=false - Watch
update-agentlogs to see the orphaned pod is deleted (log line) - Watch
update-agentlogs to see the non-orphaned pod is not deleted
There was a problem hiding this comment.
Changed node-exporter to apps/v1 DaemonSet, it was correctly drained. Change update-agent itself to apps/v1 DaemonSet, it was correctly skipped because it is owned by an existing DaemonSet.
I0117 19:48:14.635108 1 agent.go:130] Setting annotations map[string]string{"container-linux-update.v1.coreos.com/reboot-in-progress":"true"}
I0117 19:48:14.655798 1 agent.go:142] Marking node as unschedulable
I0117 19:48:14.673086 1 agent.go:147] Getting pod list for deletion
I0117 19:48:14.690520 1 agent.go:156] Deleting 1 pods
I0117 19:48:14.690619 1 agent.go:159] Terminating pod "node-exporter-rsgbn"...
I0117 19:48:14.715594 1 agent.go:171] Waiting for pod "node-exporter-rsgbn" to terminate
I0117 19:48:24.731064 1 agent.go:374] Deleted pod "node-exporter-rsgbn"
I0117 19:48:24.731089 1 agent.go:180] Node drained, rebooting
ericchiang
left a comment
ericchiang
left a comment
There was a problem hiding this comment.
The API server is magic
$ curl -s --cert cluster/tls/admin.crt --key cluster/tls/admin.key -k https://172.17.4.100:6443/apis/extensions/v1beta1/namespaces/kube-system/daemonsets | jq .items[].metadata.name
"kube-apiserver"
"kube-flannel"
"kube-proxy"
"pod-checkpointer"
$ curl -s --cert cluster/tls/admin.crt --key cluster/tls/admin.key -k https://172.17.4.100:6443/apis/apps/v1/namespaces/kube-system/daemonsets | jq .items[].metadata.name
"kube-apiserver"
"kube-flannel"
"kube-proxy"
"pod-checkpointer"
Sorry for not testing that earlier.
lgtm
|
merge whenever |
2 participants
Testing
Running
locksmithctl send-need-rebootis not enough to reproduce the original issue. I deployed a Typhoon 1.9.1 cluster with an intentionally old version of Container Linux. CLUO v0.4.1 reproduced the issue in which one node became unscheduable at a time, with no nodes able to actually upgrade.Edited to test image and Container Linux upgrades were able to be applied.
quay.io/dghubble/container-linux-update-operator:77ac24a86c4e74d30e1a04bc6aa808371a1deb8eA proper release will be forthcoming.
See #163