17

Problem

I'd like to issue certs to many different developers (different subjects) all within the dev group, and have them all have access to create and modify things within the dev namespace, but not touch anything outside it, and definitely not see secrets outside it. I suspect the roles, role bindings, etc. I'm creating in step 2 below are not correct, can anyone suggest corrections?

Attempt

  1. Deployed Kubernetes with API Server flags to support "RBAC,AlwaysAllow" authorization modes, set RBAC super user, and enable RBAC API via --runtime-config.
  2. Created a namespace, role, and role binding with the intent that (a) service accounts and system components can effectively still have "AlwaysAllow" access, and (b) any entity in group dev can access anything in namespace dev using this YAML file. NOTE: contents of this link have changed, see YAML files I got working at bottom of question.
  3. Updated Kubernetes to only allow "RBAC" authorization mode.
  4. Generated client TLS data where the certificate subject flag (for openssl) was -subj "/[email protected]/O=dev".
  5. Generated a kubeconfig file following this template.

Actual Result

I get the following errors when running: kubectl -v 8 --kubeconfig=/tmp/dev-kube-config.yml create -f /tmp/busybox.yml:

I1219 16:12:37.584657   44323 loader.go:354] Config loaded from file /tmp/dev-kube-config.yml
I1219 16:12:37.585953   44323 round_trippers.go:296] GET https://api.kubernetes.click/api
I1219 16:12:37.585968   44323 round_trippers.go:303] Request Headers:
I1219 16:12:37.585983   44323 round_trippers.go:306]     Accept: application/json, */*
I1219 16:12:37.585991   44323 round_trippers.go:306]     User-Agent: kubectl/v1.5.1+82450d0 (    darwin/amd64) kubernetes/82450d0
I1219 16:12:38.148994   44323 round_trippers.go:321] Response Status: 403 Forbidden in 562     milliseconds
I1219 16:12:38.149056   44323 round_trippers.go:324] Response Headers:
I1219 16:12:38.149070   44323 round_trippers.go:327]     Content-Type: text/plain; charset=utf-    8
I1219 16:12:38.149081   44323 round_trippers.go:327]     Content-Length: 17
I1219 16:12:38.149091   44323 round_trippers.go:327]     Date: Tue, 20 Dec 2016 00:12:38 GMT
I1219 16:12:38.149190   44323 request.go:904] Response Body: Forbidden: "/api"
I1219 16:12:38.149249   44323 request.go:995] Response Body: "Forbidden: \"/api\""
I1219 16:12:38.149567   44323 request.go:1151] body was not decodable (unable to check for     Status): Object 'Kind' is missing in 'Forbidden: "/api"'
...
I1219 16:12:38.820672   44323 round_trippers.go:296] GET https://api.kubernetes.    click/swaggerapi/api/v1
I1219 16:12:38.820702   44323 round_trippers.go:303] Request Headers:
I1219 16:12:38.820717   44323 round_trippers.go:306]     User-Agent: kubectl/v1.5.1+82450d0 (    darwin/amd64) kubernetes/82450d0
I1219 16:12:38.820731   44323 round_trippers.go:306]     Accept: application/json, */*
I1219 16:12:38.902256   44323 round_trippers.go:321] Response Status: 403 Forbidden in 81     milliseconds
I1219 16:12:38.902306   44323 round_trippers.go:324] Response Headers:
I1219 16:12:38.902327   44323 round_trippers.go:327]     Content-Type: text/plain; charset=utf-    8
I1219 16:12:38.902345   44323 round_trippers.go:327]     Content-Length: 31
I1219 16:12:38.902363   44323 round_trippers.go:327]     Date: Tue, 20 Dec 2016 00:12:38 GMT
I1219 16:12:38.902456   44323 request.go:904] Response Body: Forbidden: "/swaggerapi/api/v1"
I1219 16:12:38.902512   44323 request.go:995] Response Body: "Forbidden:     \"/swaggerapi/api/v1\""
F1219 16:12:38.903025   44323 helpers.go:116] error: error validating "/tmp/busybox.yml": error validating data: the server does not allow access to the requested resource; if you choose to ignore these errors, turn validation off with --validate=false

Expected Result

Expected to create busybox pod in dev namespace.

Additional details:

  • $ kubectl version

    Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+82450d0", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"not a git tree", BuildDate:"2016-12-14T04:09:31Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
    Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"e569a27d02001e343cb68086bc06d47804f62af6", GitTreeState:"clean", BuildDate:"2016-11-12T05:16:27Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
    
  • GitHub issue: https://github.com/kubernetes/kubernetes/issues/38997

  • Mailing list post: https://groups.google.com/forum/#!topic/kubernetes-dev/6TBTu1AC2L8

EDIT: Working solution based on answer and comments

Based on Jordan's answer below, I upgraded to Kubernetes v1.5.1 and then got the following two YAML files to construct the namespace and all the correct RBAC resources so that everything works as desired:

system-access.yml (because the out-of-the-box cluster roles and cluster role bindings didn't seem to work):

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: system:node--kubelet
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node
subjects:
- kind: User
  name: kubelet
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: cluster-admin--kube-system:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: system:node-proxier--kube-proxy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:node-proxier
subjects:
- kind: User
  name: kube-proxy

dev-access.yml:

kind: Namespace
apiVersion: v1
metadata:
  name: dev
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  namespace: dev
  name: dev-all
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1alpha1
metadata:
  name: dev-role-dev-all-members
  namespace: dev
subjects:
  - kind: Group
    name: dev
  - kind: Group
    name: system:serviceaccounts:dev
roleRef:
  kind: Role
  name: dev-all
  apiGroup: "rbac.authorization.k8s.io"

1 Answer 1

8

First, you need to allow access to the URLs kubectl uses for API discovery and validation (swagger, listings of API groups and resource types, etc).

The easiest way to do that is to load the default bootstrap cluster roles and cluster role bindings:

kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-roles.yaml
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/cluster-role-bindings.yaml

That will create a system:discovery ClusterRole and bind all users (authenticated and unauthenticated) to it, allowing them to access swagger and API group information.

Second, you shouldn't include the dev service account in the all cluster role binding. That would allow that service account (and anyone with access to secrets in the dev namespace containing the dev service account credentials) cluster wide access

7
  • 1
    Note that github.com/kubernetes/kubernetes/pull/39219 is open to include access to the /swaggerapi/* urls in that role Dec 24, 2016 at 17:38
  • Thanks Jordan. Where do all the system:... groups come from? Do I need to be on some minimum version of Kubernetes? Also, I noticed there's a handful of roles mentioned in those files which aren't used in any bindings, so do I actually need those roles? Dec 25, 2016 at 0:16
  • as of 1.5, they get auto-created when starting with an empty etcd. some of the roles get bound to service accounts for use by controllers (see github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/…) Dec 25, 2016 at 1:27
  • If I do "git log -S'system:unauthenticated'" I see the first real code referencing that is in commit 0dbcad1763, and then looking at github.com/kubernetes/kubernetes/commit/0dbcad1763 it appears the first tag containing that is 1.5.0, meaning this probably won't work for 1.4.6. Is there any other way to give unauthenticated access to discovery APIs, how did this work for people prior to 1.5.0? I can try to upgrade to 1.5.0 but I'm basing my cluster off of the kops tool which hasn't shipped 1.5.0 support yet. Thanks for all your help. Dec 25, 2016 at 3:39
  • Thanks, this ended up getting me most of the way there. I upgraded to 1.5.1, added a role to access everything in the dev namespace and bound it to the dev and system:serviceaccounts:dev groups, but then I needed to add a few additional cluster role bindings to get things to work, namely: kubelet user to system:node, kube-system::default service account to cluster-admin, and kube-proxy user to system:node-proxier. Dec 26, 2016 at 19:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.