2

I ran into the below error when trying to deploy an application in a kubernetes cluster. It looks like kubernetes doesn't allow to mount a file to containers, do you know the possible reason?

deployment config file

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: model-loader-service
  namespace: "{{ .Values.nsPrefix }}-aai"
spec:
  selector:
    matchLabels:
      app: model-loader-service
  template:
    metadata:
      labels:
        app: model-loader-service
      name: model-loader-service
    spec:
      containers:
      - name: model-loader-service
        image: "{{ .Values.image.modelLoaderImage }}:{{ .Values.image.modelLoaderVersion }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        env:
        - name: CONFIG_HOME
          value: /opt/app/model-loader/config/
        volumeMounts:
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
        - mountPath: /opt/app/model-loader/config/
          name: aai-model-loader-config
        - mountPath: /var/log/onap
          name: aai-model-loader-logs
        - mountPath: /opt/app/model-loader/bundleconfig/etc/logback.xml
          name: aai-model-loader-log-conf
          subPath: logback.xml
        ports:
        - containerPort: 8080
        - containerPort: 8443
      - name: filebeat-onap-aai-model-loader
        image: {{ .Values.image.filebeat }}
        imagePullPolicy: {{ .Values.pullPolicy }}
        volumeMounts:
        - mountPath: /usr/share/filebeat/filebeat.yml
          name: filebeat-conf
        - mountPath: /var/log/onap
          name: aai-model-loader-logs
        - mountPath: /usr/share/filebeat/data
          name: aai-model-loader-filebeat
      volumes:
      - name: localtime
        hostPath:
          path: /etc/localtime
      - name: aai-model-loader-config
        hostPath:
          path: "/dockerdata-nfs/{{ .Values.nsPrefix }}/aai/model-loader/appconfig/"
      - name: filebeat-conf
        hostPath:
          path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml

Details information of this issue:

message: 'invalid header field value "oci runtime error: container_linux.go:247:

        starting container process caused \"process_linux.go:359: container init

        caused \\\"rootfs_linux.go:53: mounting \\\\\\\"/dockerdata-nfs/onap/log/filebeat/logback/filebeat.yml\\\\\\\"

        to rootfs \\\\\\\"/var/lib/docker/aufs/mnt/7cd32a29938e9f70a727723f550474cb5b41c0966f45ad0c323360779f08cf5c\\\\\\\"

        at \\\\\\\"/var/lib/docker/aufs/mnt/7cd32a29938e9f70a727723f550474cb5b41c0966f45ad0c323360779f08cf5c/usr/share/filebeat/filebeat.yml\\\\\\\"

        caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n"'

....

$ docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:38:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:38:45 2017
 OS/Arch:      linux/amd64

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-rancher3", GitCommit:"772c4c54e1f4ae7fc6f63a8e1ecd9fe616268e16", GitTreeState:"clean", BuildDate:"2017-11-27T19:51:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

3 Answers 3

2

caused "not a directory" is kind of self explanatory. What is the exact volume and volumeMount definition you use ? do you use subPath in your declaration ?

EDIT: change

- name: filebeat-conf
  hostPath:
    path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml

to

- name: filebeat-conf
  hostPath:
    path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/

and add subPath: filebeat.yml to volumeMount

2
  • The deployment config is added in the post. Dec 20, 2017 at 11:38
  • 1
    Why dont you use configmaps ? Jun 1, 2018 at 19:48
1

SELinux may also be the culprit here. Log on to the node and execute sestatus. If the policy is disabled, you will see the output as SELINUX=disabled else it will be something similar to this:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             mcs
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

First Option: You can either disable the selinux by editing the /etc/selinux/config file and update the SELINUX=permissive to SELINUX=disabled. Once done, reboot the machine and deploy to see if fixed. However, this is not the recommonded way and can be seen as a temporary fix.

Second Option: Log on to the node and execute ps -efZ | grep kubelet which will give something like this.

system_u:system_r:kernel_t:s0   root      1592     1  2 May23 ?        09:58:18 /usr/local/bin/kubelet --anonymous-auth=false

Now, from this output capture the string system_u:system_r:kernel_t:s0 which can be changed to security context as below in your deployment.

securityContext:
    seLinuxOptions:
      user: system_u
      role: system_r
      type: spc_t
      level: s0

Deploy your application and check the logs if it is fixed. Do let me know if this works for you or need any further help.

0

Is this a multi-node cluster? If so, the file needs to exist on all Kubernetes nodes since the pod is typically scheduled on a randomly available host machine. In any case, ConfigMaps are a much better way to supply static/read-only files to a container.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.