63

I have a Terraform configuration targeting deployment on AWS. It applies beautifully when using an IAM user that has permission to do anything (i.e. {actions: ["*"], resources: ["*"]}.

In pursuit of automating the application of this Terraform configuration, I want to determine the minimum set of permissions necessary to apply the configuration initially and effect subsequent changes. I specifically want to avoid giving overbroad permissions in policy, e.g. {actions: ["s3:*"], resources: ["*"]}.

So far, I'm simply running terraform apply until an error occurs. I look at the output or at the terraform log output to see what API call failed and then add it to the deployment user policy. EC2 and S3 are particularly frustrating because the name of the actions seems to not necessarily align with the API method name. I'm several hours into this with easy way to tell how far long I am.

Is there a more efficient way to do this?

It'd be really nice if Terraform advised me what permission/action I need but that's a product enhancement best left to Hashicorp.

2
  • 5
    Note that applying these from a clean slate will not give you the total permissions needed to manage these resources! Consider updating or deleting these resources in the future... you may need additional permissions to do these actions. Jul 12, 2018 at 5:14
  • 2
    That's a very important distinction, @EricJohnson. Thanks for pointing that out. I'd love recommendations on how to account for that, as well.
    – Colin Dean
    Jul 15, 2018 at 19:53

9 Answers 9

67
+100

Here is another approach, similar to what was said above, but without getting into CloudTrail -

  1. Give full permissions to your IAM user.
  2. Run TF_LOG=trace terraform apply --auto-approve &> log.log
  3. Run cat log.log | grep "DEBUG: Request"

You will get a list of all AWS Actions used.

9
  • 1
    This seems like a decent way to go about it because the log would be parseable to extract the permissions automatically. If you couldn't give full permissions for some reason, run terraform apply in an interactive, iterable loop and add permissions through IAM each time something fails, provided that the exact permission text is available in the log output. Neat!
    – Colin Dean
    Mar 6, 2020 at 19:26
  • 4
    @ColinDean Thanks. That's a good direction but please notice that you can't automate this 100%, some actions have permissions with different names, e.g. "s3.GetBucketAccelerateConfiguration" needs the permission "s3:GetAccelerateConfiguration".
    – AvnerSo
    Mar 7, 2020 at 16:42
  • Great solution @AvnerSo! I typically dev the terraform code in a testing account where I am an admin with full permissions. Once ready I create dedicated roles with minimal permissions for the accounts and environments the final infra will be deployed into. Your solution targets the required permissions (or close to them) pretty quickly for my workflow.
    – hulin003
    Aug 26, 2020 at 19:51
  • 1
    this recommendation, and the CloudTrail one, works fine when you are setting up a project for the first time — that is, granting full power (step 1) is feasible. Any delta use case where there is the need to build a new feature for a production system can’t use this method as it’s too dangerous.
    – Joe
    Feb 19, 2021 at 13:32
  • 3
    @AvnerSo Thanks for the great solution! But it seems like the log format has changed in the meanwhile. I looked for occurrences of aws.operation= instead and replaced your last step with: cat log.log | grep -o "aws.operation=[a-zA-Z0-9]*" | uniq which gives me a list of AWS operations used.
    – andi0815
    Feb 8 at 13:00
29

While I still believe that such super strict policy will be a continuous pain and likely kill productivity (but might depend on the project), there is now a tool for this.

iamlive uses the Client Side Monitoring feature of the AWS SDK to create a minimal policy based on the executed API calls. As Terraform uses the AWS SDK, this works here as well.

In contrast to my previous (and accepted) answer, iamlive should even get the actual IAM actions right, which not necessarily match the API calls 1:1 (and which would be logged by CloudTrail).

For this to work with terraform, you should do export AWS_CSM_ENABLED=true

3
  • 1
    This doesn't seem to work with terraform commands. From my testing, it only works with aws cli commands. Any thoughts?
    – alegria
    May 5, 2021 at 9:30
  • 1
    it absolutely does, I tried it myself. Just set up the correct environment variables May 5, 2021 at 14:20
  • 3
    To be clear, the easiest way to get iamlive working with Terraform is to enable the 'AWS_CSM_ENABLED' environment variable via: 'export AWS_CSM_ENABLED=true' Sep 27, 2021 at 13:09
9

Efficient way I followed.

The way I deal with is, allow all permissions (*) for that service first, then deny some of them if not required.

For example

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSpecifics",
            "Action": [
                "ec2:*",
                "rds:*",
                "s3:*",
                "sns:*",
                "sqs:*",
                "iam:*",
                "elasticloadbalancing:*",
                "autoscaling:*",
                "cloudwatch:*",
                "cloudfront:*",
                "route53:*",
                "ecr:*",
                "logs:*",
                "ecs:*",
                "application-autoscaling:*",
                "logs:*",
                "events:*",
                "elasticache:*",
                "es:*",
                "kms:*",
                "dynamodb:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Sid": "DenySpecifics",
            "Action": [
                "iam:*User*",
                "iam:*Login*",
                "iam:*Group*",
                "iam:*Provider*",
                "aws-portal:*",
                "budgets:*",
                "config:*",
                "directconnect:*",
                "aws-marketplace:*",
                "aws-marketplace-management:*",
                "ec2:*ReservedInstances*"
            ],
            "Effect": "Deny",
            "Resource": "*"
        }
    ]
}

You can easily adjust the list in Deny session, if terraform doesn't need or your company doesn't use some aws services.

enter image description here

1
  • 3
    This is certainly a way to do it, but I'm concerned about missing something. I'd end up having to keep up with the action list as it changes, adding a maintenance burden risk I'm not sure I want to accept.
    – Colin Dean
    Jul 15, 2018 at 19:50
8

EDIT Feb 2021: there is a better way using iamlive and client side monitoring. Please see my other answer.

As I guess that there's no perfect solution, treat this answer a bit as result of my brain storming. At least for the initial permission setup, I could imagine the following:

Allow everything first and then process the CloudTrail logs to see, which API calls were made in a terraform apply / destroy cycle.

Afterwards, you update the IAM policy to include exactly these calls.

2
  • 4
    I think if I were to do this process again, I'd do it this way. I wonder if anyone has written a tool that is some kind of "CloudTrail recorder": it starts and filters out all but one user, then records each action that CloudTrail sees during a time period and spits out a list of those actions when complete.
    – Colin Dean
    Jul 15, 2018 at 19:52
  • this recommendation, and the @avnerso trace log one, work fine when you are setting up a project for the first time — that is, granting full power (step 1) is feasible. Any delta use case where there is the need to build a new feature for a production system can’t use this method as it’s too dangerous.
    – Joe
    Feb 19, 2021 at 13:33
4

As an addition to either of TF_LOG=trace / iamlive / CloudTrail approaches suggested before, please also note that to capture a complete set of actions required to manage a configuration (create/update/delete resources) one would need to actually apply three configurations:

  1. Original one, to capture actions required to create resources.
  2. Mutated one with as many resource arguments changed as possible, to capture actions required to update resources inplace.
  3. Empty one (applied last) or terraform destroy to capture actions required to delete resources.

While configurations 1 and 3 are common to consider, configuration 2 is sometimes overlooked and it can be a tedious one to prepare. Without it Terraform will fail to apply changes that modify resources instead of deleting and recreating them.

4

The tracking of minimum permissions is now provided by AWS itself. https://aws.amazon.com/blogs/security/iam-access-analyzer-makes-it-easier-to-implement-least-privilege-permissions-by-generating-iam-policies-based-on-access-activity/.

If you wanted to be picky about the minimum viable permission principle, you could use CloudFormation StackSets to deploy different roles with minimum permissions, so Terraform could assume them on each module call via different providers, i.e. if you have a module that deploys ASGs, LBs and EC2 instances, then:

  • include those actions in a role where the workload lives
  • add an terraform aws provider block that assumes that role
  • use that provider block within the module call.

The burden is to manage possibly quite a few terraform roles, but as I said, if you want to be picky or you have customer requirements to shrink down terraform user's permissions.

You could also download the CloudTrail event history for the last X days (up to 90) and run the following:

cat event_history.json <(echo "]}") | jq '[.Records[] | .eventName] | unique'

The echo thing is due to the file being too big and shrunk (unknown reason) when downloaded from CloudTrail's page. You can see it below:

> jsonlint event_history.json
Error: Parse error on line 1:
...iam.amazonaws.com"}}
-----------------------^
Expecting ',', ']', got 'EOF'
    at Object.parseError (/usr/local/Cellar/jsonlint/1.6.0/libexec/lib/node_modules/jsonlint/lib/jsonlint.js:55:11)
    at Object.parse (/usr/local/Cellar/jsonlint/1.6.0/libexec/lib/node_modules/jsonlint/lib/jsonlint.js:132:22)
    at parse (/usr/local/Cellar/jsonlint/1.6.0/libexec/lib/node_modules/jsonlint/lib/cli.js:82:14)
    at main (/usr/local/Cellar/jsonlint/1.6.0/libexec/lib/node_modules/jsonlint/lib/cli.js:136:14)
    at Object.<anonymous> (/usr/local/Cellar/jsonlint/1.6.0/libexec/lib/node_modules/jsonlint/lib/cli.js:178:1)
    at Module._compile (node:internal/modules/cjs/loader:1097:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
1

Another option in addition to the previous answers is:

  • give broad permissions "s3:*", ... as explained earlier
  • Check the AWS Access Advisor tab in the AWS console for used permission and then trim down your permissions accordingly
1
0

Here is an extension on AvnerSo's answer:

cat log.log | ack -o "(?<=DEBUG: Request )[^ ]*" | sort -u

This command outputs every unique AWS request that Terraform has logged.

  1. The "(?<=DEBUG: Request )[^ ]*" pattern performs a negative lookahead to find the first word after the match.
  2. The -o flag only shows the match in the output.
  3. sort -u selects the unique values from the list and sorts them.
0

Commenting on AvnerSo's Answer. As someone else mentioned further up, log format has since changed.

In any case, I used the following command to find what I was looking for: perl -ne 'm/aws.operation=([\S]*).*aws.service="([\S]*\s[\S]*)"/ && !$seen{"$1:$2"}++ && print "$2: $1\n"' log.log

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.