10

I'm using Terraform to provision some resources on AWS. Running the "plan" step of Terraform fails with the following vague error (for example):

Error: Error loading state: AccessDenied: Access Denied
        status code: 403, request id: ABCDEF12345678, host id: SOMELONGBASE64LOOKINGSTRING===

Given a request id and a host id is it possible to see more in depth what went wrong?

Setting TF_LOG=DEBUG (or some other level) seems to help, but I was curious if there is a CLI command to get more information from CloudTrail or something.

Thanks!

2
  • My issue ended up being with Workspaces and how they were stored on S3: terraform.io/docs/backends/types/s3.html I guess my main question still stands however as that would make debugging much easier.
    – four43
    Commented Mar 27, 2018 at 16:33
  • 1
    I'm affraid there's no general solution to this. The only documented way of debugging terraform is what you actually wrote. However, in cases like this you can try to reproduce a request using AWS cli with the same credentials you used for terraform and a --debug flag. Maybe it gives you more information about the issue.
    – jpact
    Commented Mar 27, 2018 at 18:59

1 Answer 1

4

Terraform won't have any privileged information about the access denial, but AWS does. Because you mentioned S3 was the problem I based my answer on finding the S3 request id. You have a couple options to find the request given a request id in AWS.

  1. Create a trail in AWS CloudTrail. CloudTrail will log the API calls (including request id) at the bucket level by default. If the request was for a specific object, you need to enable S3 data events when you create the trail.
  2. Turn on S3 server access logs.

You can manually search for the request id in the log files in S3 or use Athena. For CloudTrail, you can also configure CloudWatch Logs and search within the Log Group that gets created via the search bar.

CloudTrail records API calls from all services, not just S3. It could be a useful tool for diagnosing issues besides those related to S3. Note that there can be an up to 15-minute delay for logs to appear in CloudTrail.

1
  • Thanks for the answer. I kind of thought this might be the case. It's such a bummer that AWS doesn't have this level of IAM integration so you could see what's wrong.
    – four43
    Commented Mar 28, 2018 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.