Repository navigation
[occm] Helm chart DNS default can block cluster bootstrap while CoreDNS waits for node initialization #3258
Description
/kind bug
What happened:
The current OCCM(2.36.5) Helm chart defaults to hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet
and this configuration makes OCCM depend on cluster DNS to resolve OpenStack api hostnames.
On clusters, CoreDNS cannot start until occm initialize nodes, so this creates a circular dependency like this:
- OCCM needs to resolve an OpenStack api hostname before initialize nodes.
- doing lookup behavior requires cluster DNS.
- CoreDNS remains
Pendingbecouse its target nodes have thenode.cloudprovider.kubernetes.io/uninitialized taint. - and OCCM cannot complete initialization and remove the taint.
relevant log from occm when create cluster
Error from server: Get "https://matta-dev-occm-control-plane-5r4pj:10250/containerLogs/openstack-system/openstack-cloud-controller-manager-tcgjt/openstack-cloud-controller-manager": dial tcp: lookup matta-dev-occm-control-plane-5r4pj on 192.168.3.3:53: no such host
This chart default values changed from ClusterFirst to ClusterFirstWithHostNet in #3097. and with hostNetwork: true, the previous ClusterFirst policy selected nodeDNS.
also found the same bootstrap dependency reported in #2611 and fixed by #2621.
What you expected to happen:
With the default setting of bootstrap configuration, OCCM should be able to resolve OpenStackAPI endpoints before cluster DNS becomes available, and provided the node resolver can get those endpoints.
How to reproduce it:
The current chart configuration can be inspected without an openstack deployment,
helm template occm charts/openstack-cloud-controller-manager \
--namespace kube-system \
--show-only templates/daemonset.yamlthe expected failure condition is that occm cannot resolve its API endpoint while CoreDNS waiting for node initialization.
Anything else we need to know?:
I found kubespray encountered the same problem in kubernetes-sigs/kubespray#10914 in 2024, and restored node DNS behavior while retaining configuration support in kubernetes-sigs/kubespray#11168.
Environment:
- Helm chart version: 2.37.0.
- Chart appVersion: v1.37.0.
Activity
Lucasgranet commentedon Oct 15, 2026
Hello
It looks like an error on your K8s Cluster and not from the OCCM.
The APIServer is not able to dial any Kubelet because it cannot resolve the node's name.
Two options:
--kubelet-preferred-address-typestoInternalIP(instead ofHostname). We will be able to get logs through the APIServer. (ensure the InternalIP is reported to the Node resource).winiciusallan commentedon Oct 16, 2026
Hi @jihyun-huh, as @Lucasgranet commented, I think this is more a configuration issue on the cluster/node rather than in OCCM itself. AFAIK, OCCM doesn't need to resolve your hostname machine (your kubelet does), but it actually fetches the hostname from the cloud.
However, after searching and testing to get more context, I think doesn't make sense to have
ClusterFirstWithHostNetas the default.When this value is set, the pod always uses the cluster DNS to resolve name. Considering that OCCM is a bootstrap service, it can't rely on ClusterDNS to resolve names, in which is not running at this point, so if someone is using a cloud auth endpoint that needs to be resolved using an external DNS, it will not work.
Another thing is that, even though the recommended value by the docs is
ClusterFirstWithHostNet, OCCM doesn't make any request to an cluster-internal service, so technically speaking this is not strictly necessary.We likely haven't gotten this in the CI because it uses a fixed IP for the
auth-url, thus there is no need to make a request to CoreDNS 😅.jihyun-huh commentedon Oct 16, 2026
hi @Lucasgranet @winiciusallan thanks for looking at this
and you are right that the error in my original one came from the API server trying to retrieve container logs,
so I checked a newer one from here failed Azimuth CI run.
This run uses OCCM chart
2.37.1and imagev1.37.1, with no DNS override.The released chart still defaults to
hostNetwork: trueanddnsPolicy: ClusterFirstWithHostNet.the OCCM event was:
and the test eventually failed while waiting for the Cluster to become Available:
also from workload events:
from the logs, OCCM is crash looping, all three Nodes still have the cloud-provider uninitialized taint, and CoreDNS cannot be scheduled.
so.. these logs do not yet show whether DNS resolution caused directly OCCM to failed.
but, I still think(as @winiciusallan mentioned) we should set
dnsPolicy: defaultin the Helm chart, cuz OCCM needs to work during bootstrap, when coreDNS may not be running yet, so it should be able to use the nodes DNS to resolve OpenStack API endpoints.jihyun-huh commentedon Oct 16, 2026
@winiciusallan and could i add a bootstrap CI test PR that uses DNS hostname for
auth-urlwhile CoreDNS is not running?it should install OCCM through the Helm chart without overriding dnsPolicy, then just check that OCCM can initialize the node and remove the uninitialized taint. I hope it could help another future dns issues.
winiciusallan commentedon Oct 16, 2026
You can try. How are you planning to do that? Asking because, during my tests, even with
ClusterFirstWithHostNet, the OCCM installed successfully when I put a hostname for the OpenStack API in the/etc/hosts. I think it's because the pod is sharing the same network namespace with the host. That set up doesn't represent neither test a scenario that's currently failing.We can stick with what our CI already uses, the manifests have the same value as the chart's values afaik.
Fyi, jobs are registered here https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes/cloud-provider-openstack, but we would need maintainers opinions to add that job =).
jihyun-huh commentedon Oct 16, 2026
Thanks, @winiciusallan and I agree that
/etc/hostswould bypass the DNS lookup, so it would not test current dns failure.my plan is using a small DNS server outside the Kubernetes cluster, possibly on the existing DevStack host. It would resolve a test hostname for keystone. the hostname would not be added to the node or pod
/etc/hosts, or to hostAliases.if that so, the node can use that DNS server, with node DNS working and CoreDNS stopped. and the node gets a providerID and IP address, and the uninitialized taint is removed.
Fail condtion might be: these steps do not complete within the timeout.
as a check,
ClusterFirstWithHostNetshould fail with a DNS error under the same conditions.also, using the existing manifests sounds good! We can keep the Helm default covered by a unit test.
I’ll first try to reproduce this and share the results. then we can discuss whether it should extend an existing job or run as a separate job.
fix(k3s/OCCM): Helm chart DNS default can block cluster bootstrap whi…
fix(k3s/OCCM): Helm chart DNS default can block cluster bootstrap whi…