147

I have been trying to follow the getting started guide to EKS. When I tried to call kubectl get service I got the message: error: You must be logged in to the server (Unauthorized) Here is what I did:
1. Created the EKS cluster.
2. Created the config file as follows:

apiVersion: v1
clusters:
- cluster:
    server: https://*********.yl4.us-west-2.eks.amazonaws.com
    certificate-authority-data: *********
  name: *********
contexts:
- context:
    cluster: *********
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: heptio-authenticator-aws
      args:
        - "token"
        - "-i"
        - "*********"
        - "-r"
        - "arn:aws:iam::*****:role/******"
  1. Downloaded and installed latest aws cli
  2. Ran aws configure and set the credentials for my IAM user and the region as us-west-2
  3. Added a policy to the IAM user for sts:AssumeRole for the EKS role and set it up as a trusted relationship
  4. Setup kubectl to use the config file

I can get a token when I run heptio-authenticator-aws token -r arn:aws:iam::**********:role/********* -i my-cluster-ame However when I try to access the cluster I keep receiving error: You must be logged in to the server (Unauthorized)

Any idea how to fix this issue?

2
  • 1
    Used the creator's key instead of a secondary IAM account to overcome the Unauthorized issue. It didn't matter the secondary IAM was a member of a group with the same AdministrativeAccess as the creator. Might make sense to create a specific IAM for both cluster creation and management.
    – bvj
    Apr 2, 2020 at 6:32

25 Answers 25

125

When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl.

eks-docs

So to add access to other aws users, first you must edit ConfigMap to add an IAM user or role to an Amazon EKS cluster.

You can edit the ConfigMap file by executing: kubectl edit -n kube-system configmap/aws-auth, after which you will be granted with editor with which you map new users.

apiVersion: v1
data:
  mapRoles: |
    - rolearn: arn:aws:iam::555555555555:role/devel-worker-nodes-NodeInstanceRole-74RF4UBDUKL6
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
  mapUsers: |
    - userarn: arn:aws:iam::111122223333:user/ops-user
      username: ops-user
      groups:
        - system:masters
  mapAccounts: |
    - "111122223333"

Pay close attention to the mapUsers where you're adding ops-user together with mapAccounts label which maps the AWS user account with a username on Kubernetes cluster.

However, no permissions are provided in RBAC by this action alone; you must still create role bindings in your cluster to provide these entities permissions.

As the amazon documentation(iam-docs) states you need to create a role binding on the kubernetes cluster for the user specified in the ConfigMap. You can do that by executing following command (kub-docs):

kubectl create clusterrolebinding ops-user-cluster-admin-binding --clusterrole=cluster-admin --user=ops-user

which grants the cluster-admin ClusterRole to a user named ops-user across the entire cluster.

11
  • 54
    kubectl edit -n kube-system configmap/aws-auth also returns error: You must be logged in to the server (Unauthorized). Basically, if you don't have authority to assume the IAM entity (user or role) that creates the cluster, you're SOL. In my case, I don't have permission to create Access Keys for my user, but I can create EKS clusters. So I created an EKS cluster through the console but as far as I can tell, there is no way for me to access it because I need an access key to assume my user on the CLI.
    – FGreg
    Nov 16, 2018 at 21:21
  • 1
    Same as @FGreg . I get error: You must be logged in to the server (Unauthorized) when trying this Nov 3, 2019 at 10:05
  • 1
    Is there somewhere to see what IAM user or role created the EKS cluster?
    – fuzzi
    Jan 27, 2020 at 21:43
  • @FGreg Then, how did you handle this? not by using AWS Console but by using cli?
    – Jinsu
    Mar 21, 2020 at 13:45
  • 3
    @umi0410 I ended up deleting the cluster through the console and then re-creating it from the command line instead.
    – FGreg
    Mar 22, 2020 at 1:52
72

I am sure issue is resolved but I will be putting more information here so if any other people are still facing the issue related to any of the below setup then they can use the steps below.

When we create the EKS cluster by any method via CloudFormation/CLI/EKSCTL the IAM role/user who created the cluster will automatically binded to the default kubernetes RBAC API group system:masters (https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) and in this way creator of the cluster will get the admin access to the cluster. Although we can always give the access to other IAM user/role using the aws-auth file but for that we must have to use the IAM user/role who created the cluster.

To verify the role/user for the EKS cluster we can search for the CreateCluster" Api call on cloudtrail and it will tell us the creator of the cluster in the sessionIssuer section for field arn (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).

When we create the cluster using the IAM role or IAM user, setting up the access for the EKS cluster will become little tricky when we created the cluster using the role compare to user.

I will put the steps we can follow for each different method while setting up the access to EKS cluster.

Scenario-1: Cluster was Created using the IAM user (For example "eks-user")


Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity

$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

Scenario-2: Cluster was Created using the IAM Role (For example "eks-role")


Mainly there are four different way to setup the access via cli when cluster was created via IAM role.

1. Setting up the role directly in kubeconfig file.

In this case we do not have to make any assume role api call via cli manually, before running kubectl command because that will be automatically done by aws/aws-iam-authenticator set in the kube config file.

Lets say now we are trying to setup the access for the user eks-user the first make sure that user does have permission to assume the role eks-role

Add the assume role permission to the eks-user

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::xxxxxxxxxxx:role/eks-role"
        }
    ]
}

Edit the trust relationship on the role so that it will allow the eks-user to assume the role.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity. Important thing to remember it should show us the IAM user ARN not the IAM assumed ROLE ARN.

$ aws sts get-caller-identity
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name --role-arn arn:aws:iam::xxxxxxxxxxx:user/eks-role

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      - --role
      - arn:aws:iam::xxxxxxx:role/eks-role
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

2. If you have setup the AWS profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) on CLI and if you want to use that with the kube config.

Confirm that profile is set properly so that it can use the credentials for the eks-user

 $ cat ~/.aws/config
[default]
output = json
region = us-east-1
[eks]
output = json
region = us-east-1
[profile adminrole]
role_arn = arn:aws:iam::############:role/eks-role
source_profile = eks

 $ cat ~/.aws/credentials
[default]
aws_access_key_id = xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[eks]
aws_access_key_id =  xxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Once this profile configuration is done please confirm that profile configuration is fine by running the command aws sts get-caller-identity --profile eks

$ aws sts get-caller-identity --profile eks
{
"Account": "xxxxxxxxxxxx",
"UserId": "xxxxxxxxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user"
}

After that update the kubeconfig file using the below command with the profile and please make sure we are not using the role here.

aws eks update-kubeconfig --name devel --profile eks

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

$ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws
      env:
      - name: AWS_PROFILE
        value: eks

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

3. Assume the role by any other way, For example we can attach the IAM role to the instance directly.

If role is directly attached to the instance profile then we can follow the similar steps as we followed while setting up the access for IAM user in Scenario-1

Verify that we have attached the correct role to EC2 instance and as this instance profile will come into least precedence, this step will also verify that there are no any other credentials setup on the instnace.

[ec2-user@ip-xx-xxx-xx-252 ~]$ aws sts get-caller-identity
{
    "Account": "xxxxxxxxxxxx",
    "UserId": "xxxxxxxxxxxxxxxxxxxxx:i-xxxxxxxxxxx",
    "Arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/eks-role/i-xxxxxxxxxxx"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

4. Manually assuming the IAM role via aws sts assume-role command.

Assume the role eks-role manually by running the cli command.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxxxxxx:role/eks-role --role-session-name test

{
    "AssumedRoleUser": {
        "AssumedRoleId": "xxxxxxxxxxxxxxxxxxxx:test",
        "Arn": "arn:aws:sts::xxxxxxxxxxx:assumed-role/eks-role/test"
    },
    "Credentials": {
        "SecretAccessKey": "xxxxxxxxxx",
        "SessionToken": xxxxxxxxxxx",
        "Expiration": "xxxxxxxxx",
        "AccessKeyId": "xxxxxxxxxx"
    }
}

After that set the required environment variable using the value from above output so that we can use the correct credentials generated from the session.

export AWS_ACCESS_KEY_ID=xxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx
export AWS_SESSION_TOKEN=xxxxxxxxxx

After that verify that we assumed the IAM role by running the command aws sts get-caller-identity.

aws sts get-caller-identity

{
    "Account": "xxxxxxxxxx",
    "UserId": "xxxxxxxxxx:test",
    "Arn": "arn:aws:sts::xxxxxxxxxx:assumed-role/eks-role/test"
}

After that update the kubeconfig file using the below command

aws eks --region region-code update-kubeconfig --name cluster_name

Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.

 $ cat ~/.kube/config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERT
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
contexts:
- context:
    cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
    user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
kind: Config
preferences: {}
users:
- name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - eks-cluster
      command: aws

Once above setup is done you should be able to run the kubectl command.

 $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   xxx.xx.x.x   <none>        443/TCP   12d

NOTE:

I have try to cover major use case here but there might be other use case too where we need to setup the access to the cluster.

Also the above tests are mainly aiming at the first time setup of the EKS cluster and none of the above method is touching the aws-auth configmap yet. But once you have given access to other IAM user/role to EKS cluster via aws-auth (https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) file you can use the same set of commands for those users too as mentioned in above answer.

Update: If you are using the SSO then setup will be preety much same but one thing we have to consider is either in case of SSO or while using the role directly if we are trying to update path based role in ConfigMap then we have to get rid of the paths in role for example instead arn:aws:iam::xxxxxxxxxxx:role/path-1/subpath-1/eks-role of this use arn:aws:iam::xxxxxxxxxxx:role/eks-role so basically we are getting rid of the /path-1/subpath-1 becuase when we run kubectl command it will first make AssumeRole api call and if we see assumed role ARN then it will not contians the path so if we include the path then it EKS will deny those requests.

11
  • 3
    same content aws.amazon.com/premiumsupport/knowledge-center/…
    – Andy Wong
    Oct 23, 2020 at 6:25
  • That SSO quirk seems like a bug.
    – shousper
    Apr 23, 2021 at 8:22
  • It looks like a bug but its not actually, Normally assumed role ARN does not have paths in the ARN
    – Mech
    Apr 30, 2021 at 3:09
  • 1
    @VitalyKarasik if you lost the access to the cluster and I am assuming that there is no other user/role who has the access to the cluster than you have locked out your self, in this case you will not be able to do anything but you might connect to AWS support folks to see if they might be able to help.
    – Mech
    Nov 11, 2021 at 13:42
  • 1
    @VincentYin Thank you for all of your comments, Section 2 mainly aims at when you don't want to use default user with the kubectl utility rather you want to use one of the profile which you have set and in this example its "eks" so if we remove the profile from the config file it will use the default credentials and not the profile. Also there are multiple way to decide precedence on CLI but in this use case our goal is use profile with kubectl instead default. Please correct me If I am missing anything here.
    – Mech
    May 20, 2022 at 15:17
23

If you are using eksctl to manage your aws eks deployments you can add the user to the config map with one command:

eksctl create iamidentitymapping --cluster <cluster-name> --arn arn:aws:iam::<id>:user/<user-name> --group system:masters --username ops-user
3
  • 2
    This saved me after official AWS guide about adding additional IAM roles hasn't helped. Thanks a lot! I had a look on the final aws-auth configmap after this command and the only difference is that now there's a duplicative mapUsers record with <user-name>, but without userarn. Not sure that it matters though
    – ololobus
    Sep 24, 2021 at 12:36
  • 1
    super helpful. you saved me! Dec 28, 2021 at 17:02
  • This should be the top answer Aug 1, 2022 at 5:52
13

I commented out the last two lines of the config file

# - "-r"
# - "arn:aws:iam::**********:role/**********"

and it worked though I have no idea why

2
  • This is the only solution for now I have until further investigation. If you already figured this out please post your answer. much appreciated
    – diyoda_
    Aug 18, 2018 at 23:02
  • Works! Frustratingly, don't know why Jul 14, 2020 at 15:40
10

You need to create the cluster under the same IAM profile that you are accessing it from via AWS cli.

Said in another way, inside ~/.aws/credentials, the profile that is accessing kubectl must match exactly the same IAM that was used to create the cluster.

My recommendation is to use AWS cli to create your clusters as creating from the GUI may be more confusing than helpful. The Getting Started guide is your best bet to get up and running.

8
  • 4
    What if I want another user to be able to deploy to the cluster?
    – Itay k
    Jun 18, 2018 at 6:05
  • @Itayk you can edit your aws-auth configMap to add users and roles docs.aws.amazon.com/eks/latest/userguide/add-user-role.html Jun 18, 2018 at 13:20
  • I wanted the other user to assume the role I have in the ConfigMap. That way I can handle it from the IAM console. It didn't work. I will try to use the CopnfigMap instead. thanks.
    – Itay k
    Jun 19, 2018 at 4:51
  • this is not correct -- the documentation provides mechanisms to add other users via the aws-auth configMap on the EKS cluster. The users do not need to be the same as the user who created the cluster.
    – Mr.Budris
    Jun 19, 2018 at 16:34
  • @Mr.Budris Correct- I never stated you couldn't access as another user; I am stating that if you created an EKS cluster, you have to use the same IAM profile to access that cluster in order to add new users to it. You can't add a configMap if you can't access the cluster.
    – Thor
    Jun 19, 2018 at 17:44
6

Also, make sure your users are in the aws-auth k8s ConfigMap:

https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

1
  • I used a role and I am letting users assume the role so there is no need to add an IAM user. However, when I tried to set up another user with the same permissions it couldn't connect, even though it had the same assume role permissions
    – Itay k
    Jun 17, 2018 at 13:06
4

I had the same problem . It's likely that you are using a root account. It appears root accounts are blocked from assuming the required roles. This error can sometimes be cloaked if you are using expired keys.

0
3

I had the same problem, my AWS credentials for CLI change frequently. These steps fixed the problem:

export AWS_ACCESS_KEY_ID="***************"
export AWS_SECRET_ACCESS_KEY="*************"
export AWS_SESSION_TOKEN="************************"
3

If you have created the EKS cluster with kops, Then all you need to do is update your kubecfig file with following kops command

kops export kubecfg --admin

3

If you have exhausted all of the above solutions and are still getting the same error. Make sure kubectl is actually using the AWS credentials you think you are.

I made a very stupid mistake. I have environment variables setup for different accounts and kubectl seems to always pick up the one from AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY. So precedence of options on this page may help https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html

2

This happens also to me with local environment on minikube, independently of EKS. My problem is related to this issue: https://github.com/kubernetes/kubernetes/issues/76774

The solution i adopted is to remove the cache directories of kubectl: rm -rf ~/.kube/{cache,http-cache}. I guess is the only workaround at the time of writing.

0
2

In my case it is the AWS profile issue, be sure to use aws sts get-caller-identity to verify the IAM user.

2

I just debugged this issue. I have a question. Are you running this on a corporate wifi network? If yes, could you create an EC2 instance and then test if you are able to do kubectl get svc?

Also, try if this command works kubectl get svc --insecure-skip-tls-verify

2
  • 2
    Error: bad flag syntax: ---insecure-skip-tls-verify
    – Fabio
    Jun 11, 2019 at 13:43
  • It should be --insecure-skip-tls-verify (just two hyphens at the beginning). Still doesn't work for me though. Dec 31, 2019 at 20:55
2

The issue for me was that I had set up environment variables with different, invalid, AWS credentials (maybe a long time ago, and forgot). I realised this after running aws configure list and seeing that the credentials are different from what I expected with aws configure list --profile default. Finding and deleting those invalid environment variables fixed the issue, now I can run kubectl get svc.

2

I had the same issue. Refer the answers:

  1. Cannot create namespaces in AWS Elastic Kubernetes Service - Forbidden
  2. User cannot log into EKS Cluster using kubectl

The correct way is to:

  1. Create an IAM Group with the following permissions.
    1. AmazonEKSClusterPolicy
    2. AmazonEKSWorkerNodePolicy
    3. AmazonEC2ContainerRegistryReadOnly
    4. AmazonEKS_CNI_Policy
    5. AmazonElasticContainerRegistryPublicReadOnly
    6. EC2InstanceProfileForImageBuilderECRContainerBuilds
    7. AmazonElasticContainerRegistryPublicFullAccess
    8. AWSAppRunnerServicePolicyForECRAccess
    9. AmazonElasticContainerRegistryPublicPowerUser
    10. SecretsManagerReadWrite

  2. Create an IAM user (AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID will be provided) and add the user to the IAM Group created above.

  3. Next, login to AWS Console as the IAM user and create the EKS Cluster.

  4. Next, use the AWS_ACCESS_KEY_ID and AWS_SECRET_KEY_ID to setup the AWS CLI in local machine.

  5. Next, run the following commands in the local machine:
    1. aws sts get-caller-identity
    2. aws eks describe-cluster --name [cluster-name] --region [aws-region] --query cluster.status (To check the status of the Cluster)
    3. aws eks update-kubeconfig --name [cluster-name] --region [aws-region]

  6. After this, you will be able to run the kubectl commands.

If you need to add additional users to the EKS Cluster, create the additional IAM user, add the user to the IAM Group in AWS. Next, log into the EKS cluster as the original IAM user and run: kubectl edit -n kube-system configmap/aws-auth.

Add the following block of code to the existing configuration:

(Make sure to add changes to the ARN and username)

mapUsers: |
- userarn: arn:aws:iam::111122223333:user/admin
  username: admin
  groups:
    - system:masters

Refer the following link to understand this better: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

Hope this become useful. Happy learning!

1

The issue is with the policy added for the roles created. We must have AWSEKSCNI policy.

Better to create eks using the command:

eksctl create cluster --name ekscluster --version 1.19 --with-oidc
--vpc-public-subnets=subnet-08c6b0b0166abc1d1,subnet-02822a142bb5a802a
--vpc-private-subnets=subnet-09bbf4871902ee64c,subnet-0926c224909b5a811

This will create and assign the policy automatically using cloudformation.

1

In my case the scenario was, that I replaced a person who worked as a devops and created all the infrastructure on aws. He had his own IAM user (so yes, no root access case, however could be also applicable) with all AWS permission and I had my own IAM user also with all available AWS permissions. So, he left the company and that was the problem, since I didn't get any access to cluster which by defauly is shared to the creator only ... and in fact all of my approaches to get the access to cluster were failed, even despite a fact that I had all permissions. The good thing was, guy whom I replaced was still had his IAM user available (not removed). And what I did, I simply generated new AWS access pair under his account and set them as my default aws credentials on my ubuntu host (from which I've tried to access the cluster). Important part was to make sure that after running aws sts get-caller-identity command it meant to be HIS account to appear on the output. In that case I've been able to run all the kubectl commands that I wanted without error You must be logged in to the server (Unauthorized) message. So the solution is - be lucky to find cluster creator credentials and use them! (sounds like a crime, however ...)

1

In addition to the great answers that have already been given, I would like to add a good way to troubleshoot issues. In my case, I was trying to run kubectl in an ECS task as part of an AWS pipeline, and kubectl version was failing with the "You must be logged in to the server" message.

What was happening was that the ECS task was assuming a service role, and then aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_DEFAULT_REGION was being run. It turns out the service role, CodeBuildServiceRole, was not mapped to an RBAC user via a clusterrolebinding in the EKS cluster, and the aws-iam-authenticator EKS service was denying access to the AWS service account (or something like that).

What helped me figure out what was going on in this case was to go to CloudWatch → Log Insights and run the following query against the /aws/eks/<cluster-name>/cluster log group:

fields @timestamp, @message
| sort @timestamp desc
| filter @logStream like /authenticator/
| filter @message like /access denied/
| limit 50

It was clear from the log output that the authenticator service was expecting an ARN in configmap/aws-auth that was lowercase: arn:aws:iam::123456789:role/codebuildservicerole. Once I fixed the case of the ARN in the configmap, kubectl version started working in the ECS task.

$ kubectl edit -n kube-system configmap/aws-auth
1

I had the same problem and the fix was to remove an alias I had added when setting up minikube. Their docs say to add this alias:

alias kubectl="minikube kubectl --"

Which breaks some things like Azure integrations, as you would expect.

0

I got this error when I created the eks cluster using the root from the eks console. I recreated the eks cluster using an IAM user and use the access keys to update the aws configure. It worked. Now you can add additional IAM users to issue kubectl commands.

0

I was trying to create an EKS cluster with a private endpoint. Read this thread several times and the thing that worked for me:

  • Created a user: admin (with access to the console)
  • Logged into the console using that user
  • Created the cluster using the admin user
  • Created my kube/config file using aws eks (aws eks --region update-kubeconfig --name )
  • Done (kubectl get ns)

The last 2 commands were executed from an ec2 instance part of the same VPC.

0

I faced the same issue. I tried to configure AWS CLI directly with access key and secret key and it worked.

It should be bug to not able to assume role. Try setup cli and test.

0

I followed these docs. It took some time to understand things. But finally implemented the things easily with full understanding.

https://kubernetes.io/docs/reference/access-authn-authz/rbac/ https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

0

When invoking eksctl with an assumed role through sso, the following steps got me access to the cluster.

  1. eksctl.exe create cluster --name test --without-nodegroup --profile <your_profile>.
  2. Update the kubeconfig :

eksctl utils write-kubeconfig --cluster=test --profile<your_profile>.

  1. Enable Cloudwatch to get the iam role arn:

eksctl utils update-cluster-logging --enable-types=all --cluster=test --approve --profile=<your_profile>.

  1. run "kubectl get pods -A" . server responds with "error: You must be logged in to the server (Unauthorized)" message.
  2. Switch to the console and get the role arn from the cloudwatch group audit log.
  3. Update the aws-auth config map by mapping the derived arn to system:masters group. The username is users:name field in your current kubeconfig context as extracted it below
user_id=` k config view --minify -o json | jq -r '.users[0].name'` 
eksctl create iamidentitymapping --cluster test --group system:masters --username ${user_id} --arn arn:aws:iam::<account_id>:role/<assumed_role> --profile <your_profile>
0

For me adding the user in a single line like below worked

kubectl edit -n kube-system configmap/aws-auth

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::******:role/eksctl-atoa-microservices-nodegro-NodeInstanceRole-346C48Q1W7OB
      username: system:node:{{EC2PrivateDNSName}}
  mapUsers: "- groups:\n  - system:masters\n  userarn: arn:aws:iam::*****:user/<username>
    \ \n  username: <username>\n"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.