perm denied error when using rhcos with selinux enabled

[derekwaynecarr]

bootkube fails with perm denied error when reading /assets/kco-config.yaml

fix
disabled selinux
restarted bootkube, got further.

[derekwaynecarr]

this was using libvirt with rhcos qcow

[derekwaynecarr]

I see this will be fixed tomorrow

[derekwaynecarr]

tried with new image, still have issues with bootkube output:

[core@test1-master-0 ~]$ journalctl --no-pager -u bootkube
-- Logs begin at Wed 2018-08-15 13:56:27 UTC, end at Wed 2018-08-15 14:00:01 UTC. --
Aug 15 13:56:48 test1-master-0 systemd[1]: Starting Bootstrap a Kubernetes cluster...
Aug 15 13:56:48 test1-master-0 bash[1404]: Rendering Kubernetes core manifests...
Aug 15 13:56:48 test1-master-0 bash[1404]: Unable to find image 'quay.io/coreos/kube-core-renderer-dev:df42b97af403702013f4739fc82cd005cfd0c766' locally
Aug 15 13:56:48 test1-master-0 bash[1404]: Trying to pull repository quay.io/coreos/kube-core-renderer-dev ...
Aug 15 13:56:49 test1-master-0 bash[1404]: df42b97af403702013f4739fc82cd005cfd0c766: Pulling from quay.io/coreos/kube-core-renderer-dev
Aug 15 13:56:49 test1-master-0 bash[1404]: 4d472840d001: Pulling fs layer
Aug 15 13:56:49 test1-master-0 bash[1404]: dd5cfda89e72: Pulling fs layer
Aug 15 13:56:50 test1-master-0 bash[1404]: 4d472840d001: Verifying Checksum
Aug 15 13:56:50 test1-master-0 bash[1404]: 4d472840d001: Download complete
Aug 15 13:56:50 test1-master-0 bash[1404]: dd5cfda89e72: Verifying Checksum
Aug 15 13:56:50 test1-master-0 bash[1404]: dd5cfda89e72: Download complete
Aug 15 13:57:03 test1-master-0 bash[1404]: 4d472840d001: Pull complete
Aug 15 13:57:05 test1-master-0 bash[1404]: dd5cfda89e72: Pull complete
Aug 15 13:57:05 test1-master-0 bash[1404]: Digest: sha256:b7441413d170e803ca71a020863ac66f435d9b713664ba8074e994288268e712
Aug 15 13:57:05 test1-master-0 bash[1404]: Status: Downloaded newer image for quay.io/coreos/kube-core-renderer-dev:df42b97af403702013f4739fc82cd005cfd0c766
Aug 15 13:57:07 test1-master-0 bash[1404]: F0815 13:57:07.218416       1 main.go:24] Failure reading config from "/assets/kco-config.yaml": read config from "/assets/kco-config.yaml": open /assets/kco-config.yaml: permission denied
Aug 15 13:57:07 test1-master-0 systemd[1]: bootkube.service: main process exited, code=exited, status=255/n/a
Aug 15 13:57:07 test1-master-0 systemd[1]: Failed to start Bootstrap a Kubernetes cluster.
Aug 15 13:57:07 test1-master-0 systemd[1]: Unit bootkube.service entered failed state.
Aug 15 13:57:07 test1-master-0 systemd[1]: bootkube.service failed.

the perm denied still appears related to selinux.

[derekwaynecarr]

journalctl | grep -i avc
Aug 15 13:56:33 test1-master-0.tt.testing ignition[687]: "source": "data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBbk5YSWtsM2VtRFdyeU03aFNPQlN1WVlqYkFYd1VaN0hwZE9YVEpTck8xbzQ1YURHCmN6QUF1ZFl5ZTVnem9zeWUzWDZXL3hNNGVJR3JCR3ZGTS9Ucno1TUdHd3dpUnRRdk5ydHJMKzBoTlFSRXB4T00KMXRRWW5yeFc4N3pzV2w5bmFBWEY5cG45RFBYU0dmOCs2aXpab3NrYk5aWmt5aVY0SzRjM2dzYVVJR2s3bXRGbwora3JqQVFzQldvQWN0Ykt5SlJITS9wSkozTUI1K0lQQkZzK0Q4OXlTQ3ZjcGVyWUZZcXRPYlF3NGpWUnI0bmhEClZpR3VrbVFYdDBDOUhETDIwWm9GT1lKbjQ0YWJKemM2TEZuaDZXYjlaa1pSNFQyalF1bThOSTdPOUlDemhCOFYKclFOdm1rRnZkQzd0K1FpQVc2L0Z3c2NUVVdYa2lNamdoUTJDK1FJREFRQUJBb0lCQUFqMlNuVGF1bHFXVG8rMgpDcmVnWWZuS0NZSWx3THJaU08xWDd3Qm9TblNrempXSS8yNGJveDc3ZDMwS2tJRFFFby96cU90QWpPeU45RmpYClU3aUpXV1JPTVg0Z0xtRS94TWJxNU5BalM4OTh3L09NTVhNaFFacm9oa3Q5VTBCQ3pXVHJWNG1rK1FuaGpqVUEKR2ZkRnd0WURpZk9BK1pkM2xxdGVHYlQyWmdhSUJmNXVMNDBMSEZqdlA3TlhlQ3dTbkFQZFBwZ1FSSENDOVEyMgprT242cFhDazM0bVVzZVI4UlVpUWhYTFpTSWFrcUxVWE1ndndHL0pGNDNQZDZFQVdiYXkveTlraWtYZDRyUk1vCkRManB2YkhyeWpyVDM0d0N0UXpCNXJZRUd1MytEVGpvNGRKQ1pHeVg2QWw0T1ZFdFMxRUxTajdmT1FBelhMYysKTEZ5SVpKRUNnWUVBd1IyNlVpU21pWlhwdGdEL1YvMUF2cVNHdEJ2clhROEliOTF3OTNCc0haNlZLNXA2WEkwUQpyOEQwUTJjczB6ckJ3cXhKVThwU0FNYituVHY1dnFjaTdyK24wV2RmbGpwOE8rYVl3V1dXYldnMFNTZVNVUCtZClBsVzdmR3NtUi9KZUxNSmxKWlRFYjBNOGVHcFlTZUZUcWhmSVUzQzVoNVFvcjY2SXVnbFhyalVDZ1lFQXorZWwKMGxNSWlQNE8vOWVjamhESFFYZE1HN3crazV5Y1ltRFRDUXg1LzEzTlhucVdyUDR0WWVFSEZ0TFQ0dFVmcU1BUgpVOUZqaTNQbzRZTjBwaU9yb1ZEd2x1QkI4dkttOFVheHdjajgxWWNMTHRZdFVCMjNHRVAvbU56Vm9WSEY0d3JjCnJwa25nZ1FDcW9VMDJrWGYzbUJ2cWZQa1p6VUxweHIrTmt0QzZqVUNnWUVBZ09ZMjExMWZTN2FrcUxkQnVKbHgKL2M0VG0yU0hWVFlUaTVkakw4WDZaRXJWaHFVMXgxRGhNbTY0bThUaVJwdVJlVDlHTW9kNDlNdmVaMVVBL2lEUgpVRXJjMlFrRzVGOWxUUlkrSDlpTzc3ZitMbFliYzdVbkNYUndFRHYwOFZEMVN5cjJHSCtVSGkvaXpQMHVzU0dWCmxwTUpRNmlhTGNUVzQyeThGbkRsOVlFQ2dZQXJNcFY0c3ZuMkJOdTIrdVN6ZS9iNnVqL2REMnJ0SHNBN2pLU3MKbjZRRmxFYmtsNUlSRmFyMlNGeEJ1TUovd2dxVzlIbGxNZjk5N1RKNUVPZyswUENMVHhiK01sQmhtMXRtakdySQp1ZXNXcnIxN0dOTkhielVvM0pBU0FlaDlZVkU5a0hjejYreVNqaVREcTNQRTJubmVhYWtwNWR3U09hcFhLVHVpCnFsYVg5UUtCZ1FDV3J5dWUxYk9pdmE2TzdHb01FQzFtOGlhd3NkLzFzZ2VTZnBJZnRWUGh6bnM2RU1PMVVHa2UKaUVhOUFaMmdCY1RXd3Y2MWpzaFgwWXhmaC9Lc1ZOVkk1cUhBbG9VOGdDUmlCblkxL2xRQy9SRGJMNzJWcjd0Ngp3cjk2bzd0WFN3dzlaY0ZKc3c1TG85K3AwYkxWUWFNMUFpYVlBUk44Zm54V3FRaVcyQ3hDTFE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=",
Aug 15 13:56:34 test1-master-0.tt.testing ignition[687]: "source": "data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBd2MrWHdWZEFxa0hlSERuQjcwMkxOcXJaYXJ0b1ZRSTM5L2dkN2h3WnpqdnVCU1hvCmprTjFpK004dTRNRi9IWUIzdzdBZHd1N1dOeXIwN2tSazNBemZpTitBNDd4cjE1ajZ0djEraklIVDlrRDdveSsKUzBlRC8wQUM4bmFYeHdlNFdJUGV2TGU1Qm1LOTMvTkR6cjRBdXR2ZTVsUDliQnFQK25INElMck82WlF0Sm1BNgpFTmhsOHBnRGxZVFZDanBYbUxhZFBBbHEwSm1TZytSVSt5NEpyTkpybGk4NWFWRS91UW04b2gzK1BrSWhKNncyCkxaTTEva2tzNEVCQml2cUVwZUVHYXZ4TnUxUFo0YVl0S1lweVc0SHJ2R25tQ2tua3Z0VzJ1ZjFaTFgzaUl2aG4KdWdBT1AyeEFqSktxbWdSVVBYcXRIdnhaQ2RLZ3g5L0ZLaFArcFFJREFRQUJBb0lCQVFDM0VyTVV6S2ltcXdWMQp3QkV6VFJwZGoxRkVnclp3NW1HYitHRzlWQW9FUjVQMGhQU0J2Yk5CYW1zcDdRQXdrLy84aGVERUV1N3JaN2RmCmpZZk9yOFBVT1E2RnFmY2VZcGtiZHArSnNNdzdYcEZhT3RSZUk3WEozTnRyMFI1WndTOGZYYUYrdmtVbWhRczYKaW4zWXdwM0o1SHRQTXJORjlHbGdkMXVjL3hrSWhXd0Q2Q2lhQTUyRmgvUkRGYUc1bmM3cTNDNHVnNEsxeCtwUApldG5Obzd3U01aczI5bW5MS0pmR1h5NDU1RnZHQmlOQmJHLzNXL1pNT0lCKzI3aURib2l0QzIvYkZrWGJMQzJECnUrUXpOb25SNzJvZXVmTWh0aFNWUUw1WnNIU3NVRENoSkI5cEIzYWhvdlo0ZFdqWEpoYjY4bzVxdjFrQ3RuZmsKa25QUXlIeFpBb0dCQU93QUhvNm13U3QyeGVXNVh4azErWi9BR2JObXljdy9iMk9XV2dubHBWWVVYaEhGMVp2Zwpmcm1nUEZ6VWl6TGZTckVlbXNNWTV3Qk9pcTJ5cUh4TGlPMjU0MDZ3VlovTDYrLzBGTkliUE5yc1RxSDk5NDRoCnM3aGdqMmtZNklYSGdqc3ozTThkdHVmODFzYXZkc1kzZlNCMGpzYWFzS0w5citqaDROQlRpL2hiQW9HQkFOSTgKTXAxdnRyN1psamFvd1JvWG9JZm9McFJKaXZGb2lYZnptUWExZ1lQOWYyK1dpQ1NGVUdQcEZjNUN3ZGJJWm94RAp5bXlTbDhwL3JBemx4U0VMK0s1ZTZTRGhJQnJERjRWS3FnYjByNXhIYkFIMFdxcE5QQjhyejJnY0l1YWR0dzFDCnZLNHZDU1VJRUJJY2d3dkdoQkV6ZjVkY21RclNJdExJSjNiV3FaVC9Bb0dCQUpIaWdRRXRrN3VLY0VyUmpEZkoKWmNXYXVraHNBZEtBWkJycmxqMEgrR3g5cXFqUjRubTVESjB5c0IyeVJWbnRMZjdQTEZ2dHlONG5yeEl3bm5ZMwpPeTI0K3dwcGRvU1JTZ2ZLbWhSSFFoY1NmSWttdFNEbk5IR0ZQeUY0aEVRdVVCTEl2SFpMcUFWQUJvUkxjdUNVCjdJUmppTjY4UVBTQVhYMVlJK0NqeEtLQkFvR0JBTFFacThhaFlDMUkyMTFCM2dNTFFKT00vUEk5dWxDcW5ERnQKTnFlL3NBOHhpQTFCS0tvWXB0Q2dhZlREemFqQkR0Q1Vkb0hpWnpTcmdPbWZvT3Q1aFBWa0MxVUdadWxtUGUwTApGSE5YQkdYZDdaSVRFZVNZdTZ0OGJYYWp1K1pTTC9HbFBWditvVmZlKzExNG5XN21CbGR5QlpqV1U2a29jWHFlCnl1Z01aMFJqQW9HQVQ5WnY5Uk44cVBwMGRHaml1TmxGYWs3ZC9DRXBiMDI2bEovSmtkN0Z6bXVGekNrL0ZHc1QKL0w4U0x1dXZHY3dLSnFUNWZLdm9TT0FwUmdBUlhYMHpwdVZ3cGhJNWlQby9EM2VwaXJJMVlkdVZPM016WUQrWApiU3V1SFNpZU1aT0Z0aWx3NlJCQjNubnJUV0ErWHRWSCtpQVFlbUJEUnJCSnlSNWlNVjJROEFFPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=",
Aug 15 13:57:07 test1-master-0 kernel: type=1400 audit(1534341427.214:4): avc:  denied  { read } for  pid=1475 comm="renderer" name="kco-config.yaml" dev="dm-0" ino=5728740 scontext=system_u:system_r:container_t:s0:c115,c381 tcontext=system_u:object_r:var_t:s0 tclass=file

[derekwaynecarr]

looks like this should be privileged
https://github.com/openshift/installer/blob/master/modules/bootkube/resources/bootkube.sh#L7

[eparis]

or all of the volume mounts should be with :z

[rhatdan]

Correct the /var/.../kco-config.yml file is mislabeled. Just needs to be loaded in with a volume mount telling the container engine to relabel the content.

[wking]

It looks like #137 is related to this to, although I don't understand the situation clearly enough to know how #134 and #137 interact. Do we need both of them?

[eparis]

I don't know which problem alex was trying to fix in #137, but #134 is definitely right....

[jlebon]

Since we can't really customize the ignition configs, one way to work around this for now is:

$ cp /path/to/rhcos{,.permissive}.qcow2
$ virt-edit -a /path/to/rhcos.permissive.qcow2 -m /dev/sda1 /grub2/grub.cfg
$ # add enforcing=0 to the cmdline