I am using Kubernetes v1.2.4 (on top of CoreOS stable 1010.5.0) and would like to mount rbd/ceph volumes. Basically I’ve followed https://github.com/kubernetes/kubernetes/tree/master/examples/rbd except that I prefer YAML over JSON.
Noticed there has to be both:
secretRef:
name: ceph-secret
and
keyring: /etc/ceph/keyring
else kubectl complained. Is this expected behavior?
Seems kubelet tries to invoke rbd binary directly on the host (which is a problem for a "bare system" like CoreOS). Since copying over the binary and dependencies would be a bit cumbersome I did this trick:
$ cat /opt/bin/rbd
#!/bin/sh
docker run -v /etc/ceph:/etc/ceph ceph/rbd $@
Took care of /etc/ceph configuration, made the shell script executable and so on - if I do "rbd list" on CoreOS everything works fine. /opt/bin (beside being on PATH on CoreOS by default) is also in the PATH for the kubelet process (which I can confirm through /proc/kubelet pid/environ).
However if I start the (test) pod I get this error (in kubectl pod describe):
Events:
FirstSeen LastSeen Count From SubobjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
5s 5s 1 {default-scheduler } Normal Scheduled Successfully assigned busybox4 to some-host
4s 4s 1 {kubelet some-host} Warning FailedMount Unable to mount volumes for pod "busybox4_default(5386c7f3-3959-11e6-a768-aa00009a7832)": rbd: map failed fork/exec /opt/bin/rbd: invalid argument
4s 4s 1 {kubelet some-host} Warning FailedSync Error syncing pod, skipping: rbd: map failed fork/exec /opt/bin/rbd: invalid argument
so either fork() or execve() returns EINVAL? By reading through a few man pages I found only exec might actually fail with EINVAL due to
An ELF executable had more than one PT_INTERP segment (i.e., tried to name more than one interpreter)
but that seems quite obscure.
Any idea what the matter is or how I could fix / workaround the problem?
Edit: I tried strace -fp pid and there are a lot of stat() calls which I presume come from golang os/exec LookPath. However I don't see any execve() on "rbd" nor is there any system call failing with EINVAL. To make sure it is not related to fleet (systemd) I also tried running kubelet directly on the console as root. Results are the same.