Repository navigation
[occm] dnsPolicy default value recent change blocks occm / cluster from coming up. #2611
Description
[occm] Recent change #2594 changed the default behaviour of dnspolicy breaking the start up process of occm when coredns has not yet been started.
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
Default value change of dnspolicy broke the bootstrapping of occm on new clusters.
The default value of dnspolicy is Default the change #2594 sets it to ClusterFirstWithHostNet.
If coredns is pending waiting for occm to set nodes to initialized this creates issue where occm fails and won't start blocking creation of new clusters.
F0531 14:46:10.225237 10 main.go:71] Cloud provider could not be initialized: could not init cloud provider "openstack": Get "https://<cloud provider dns>:5000/": dial tcp: lookup <cloud provider dns> on 10.96.0.10:53: write udp 10.6.0.240:32899->10.96.0.10:53: write: operation not permittedWhat you expected to happen:
Default behaviour not to change.
How to reproduce it:
Create a new cluster and use the default values of occm chart or the manifest.
Anything else we need to know?:
Environment:
- openstack-cloud-controller-manager(or other related binary) version:
- OpenStack version:
- Others:
Activity
mdbooth commentedon May 31, 2024
The documentation of this field is quite spectacularly bad: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
To my eyes that clearly says that pods running with host networking should set
ClusterFirstWithHostNet. CCM runs with host networking, so we'd seem to be covered. It doesn't actually say what it does, though.Some RTFS:
https://github.com/kubernetes/kubernetes/blob/790dfdbe386e4a115f41d38058c127d2dd0e6f44/pkg/kubelet/network/dns/dns.go#L303-L323
For reasons I don't understand, most likely legacy API compatibility,
ClusterFirstsets the DNS policy toDefaultif the pod uses host networking. Internally that is mapped to apodDNSTypeofpodDNSHost, which is a much better name. i.e.ClusterFirstmeans:Use cluster networking unless the pod uses host networking, in which case use the host's DNS.
ClusterFirstWithHostNetremoves the fallthrough and usesClusterFirstin all cases.As an early bootstrap service, I don't think CCM can rely on cluster DNS being up. I suspect it is correct to revert this.
@xinity, what was the issue you were hitting which caused you to change it? It's not clear to me from reading #2594 or #2592? I appreciate that CCM is not able to resolve service names internal to the cluster, but why did that matter?
xinity commentedon May 31, 2024
@mdbooth occm wasn't able to query the internal coredns instance without this new value
It matters because of specific internal dns zone with squid proxy that should be resolved from occm
mdbooth commentedon May 31, 2024
Right, but why? What was the internal DNS zone, and why was it important that CCM could resolve it?
jichenjc commentedon Jun 3, 2024
our CI passed so it should be a smaller portion of error case
and I am also curious why the internal DNS zone is needed here
mdbooth commentedon Jun 3, 2024
I also wondered about that. Does that mean CNI comes up an an uninitialized node, and coredns tolerates uninitialised?
yankcrime commentedon Jun 21, 2024
I've just tested the new release of OCCM on a cluster with 1.30 and have hit this issue as well. For anyone else who is struggling to understand the root cause (being this change), the nondescript error from the CCM Pod is:
I only found the underlying error when I SSH'd onto the node where the CCM had been scheduled and looked at the container logs in
/var/log/containers.zetaab commentedon Jul 2, 2024
@jichenjc @mdbooth the occm test was not really executed in #2594 that is why it did not fail. Only helm chart tests were executed. Like now we can see, our CI is basically broken. So yes, this change should be reverted
testing CI in #2620 and also PR #2621 which will set the default value back what it was.
zetaab commentedon Jul 2, 2024
blah at least our CI do work with this new value.. should we still use old default value just in case?
jichenjc commentedon Jul 3, 2024
yes, I think we may still use the old default value ..