Before and after reboot documentation and examples#124
Conversation
dghubble
left a comment
dghubble
left a comment
There was a problem hiding this comment.
Seems a tad verbose. We might be able to do more showing and less telling.
| @@ -0,0 +1,48 @@ | |||
| # Before and After Reboot Checks | |||
|
|
|||
| CLUO provides optional functionality for running arbitrary actions before and | |||
There was a problem hiding this comment.
CLUO can require custom node annotations before a node is allowed to reboot or after a node has rebooted to allow it to become scheduable.
| it's respective label, the check performs whatever function it needs to, and | ||
| then should set one of the annotations provided to the `update-operator` to | ||
| true. | ||
|
|
There was a problem hiding this comment.
Link to the examples directly
|
|
||
| ## Making a check | ||
|
|
||
| A check that needs to run before reboot should look for the |
There was a problem hiding this comment.
Seems like we could be more concise:
Pods that should run before a reboot should node select on the label blah and set the required before-reboot annotation to allow the reboot to occur. Pods that should run after a reboot should node select on the label blah and annotate the node with the required after-reboot annotation to allow the node to become schedulable again. A DaemonSet with a node selector is the recommended way to achieve this.
- before-reboot-daemonset.yaml
- after-reboot-daemonset.yaml
| update-operator --before-reboot-annotations anno1,anno2 --after-reboot-annotations anno3,anno4 | ||
| ``` | ||
|
|
||
| ## Making a check |
There was a problem hiding this comment.
I was trying to avoid referring to them as pods here, since they really don't have to be. It could be a cronjob on someones machine checking the labels every minute and then running kubectl annotate. Not that that's the recommended way to do things (hence the examples being pods) but I think there is value in keeping it flexible.
There was a problem hiding this comment.
My concern was that the vagueness doesn't help the user trying to figure it out. For most purposes, I'd expect people to be writing pods and the description of how its "just a label" and "just an annotation" makes it clear other things could be used?
There was a problem hiding this comment.
Yeah, I guess we can be opinionated on how it should be used in the docs even if the implementation itself is flexible.
There was a problem hiding this comment.
I made some suggestions about the contents of this section. Maybe that will inform our title. I won't be hurt if we have the title stay as is
| `before-reboot` and `after-reboot` label, and then annotate the node with a | ||
| configured annotation. | ||
|
|
||
| ## `update-operator` behavior |
There was a problem hiding this comment.
Let's move this section above custom check pods and call it "Before and After Reboot Labels"
|
|
||
| ## `update-operator` behavior | ||
|
|
||
| When a node requests a reboot, the `update-operator` will label the node with |
There was a problem hiding this comment.
Before and After Reboot Labels
update-operator labels nodes that are about to reboot container-linux-update.v1.coreos.com/before-reboot=true and labels nodes that have just completed rebooting (but not yet scheduable) with container-linux-update.v1.coreos.com/after-reboot=true. If you've required before or after reboot annotations, update-operator will wait until all annotations are applied before proceeding.
| effect: NoSchedule | ||
| containers: | ||
| - name: example-after-reboot-check | ||
| image: quay.io/stephen_demos/kube-annotate:latest |
There was a problem hiding this comment.
How much effort is it to use the hyperkube image (which contains kubectl) and service account or host kubeconfig and just run ``kubectl annotation node blah`?
There was a problem hiding this comment.
Probably not too much effort. I can check it out. These were just the same pods I used to test the feature.
sdemos
force-pushed
the
before-after-reboot-docs
branch
3 times, most recently
from
b187403 to
be8b772
Compare
| the binary - | ||
|
|
||
| ```bash | ||
| update-operator --before-reboot-annotations anno1,anno2 --after-reboot-annotations anno3,anno4 |
There was a problem hiding this comment.
I think my comment here was lost.
Users will be running the update-operator image on Kubernetes, likely with our example deployment, let's show the flags in a manifest context:
...
command:
- "/bin/update-operator"
- "--before-reboot-annotations=custom-before-reboot"
- "--after-reboot-annotations=custom-after-reboot,"
env:
...
| CLUO can require custom node annotations before a node is allowed to reboot or | ||
| before a node is allowed to become schedulable after a reboot. | ||
|
|
||
| ## Configuring the `update-operator` |
|
|
||
| ## Configuring the `update-operator` | ||
|
|
||
| When depolying the `update-operator`, you can provide a comma-separated list of |
There was a problem hiding this comment.
Perhaps:
Configure update-operator with comma-separated lists of --before-reboot-annotations and --after-reboot-annotations that should be required.
|
|
||
| ## Making a Custom Check | ||
|
|
||
| A check that needs to run before reboot should look for the |
There was a problem hiding this comment.
Hm, starting with a verb here might hint the user needs to go write some code or pick something. what about:
Write your logic to perform custom before-reboot behavior and after-reboot behavior. When successful, ensure your code sets the annotations you've passed to update-operator. If your code finds a serious issue, leaving the annotations unset will ensure cluster upgrades halt at the problematic node, for an admin to intervene.
It is recommended that custom checks be implemented by a container image and deployed using a Daemonset with a node selector on the before-reboot or after-reboot labels.
spec:
nodeSelector:
container-linux-update.v1.coreos.com/before-reboot: "true"
Be sure your custom check can handle being rescheduled to a node on which it has previously run as the update-operator does not remove the before-reboot and after-reboot labels instantaneously.
sdemos
force-pushed
the
before-after-reboot-docs
branch
from
be8b772 to
08a66a2
Compare
sdemos
force-pushed
the
before-after-reboot-docs
branch
from
08a66a2 to
3533e41
Compare
2 participants
Added documentation and examples for before- and after-reboot checks.