26

I am struggling to understand how I should handle deposed resources found in my TF state.

In particular, when running an apply, I am now getting some errors like the below:

Error applying plan:

11 error(s) occurred:

* aws_instance.mongo-replica-01 (deposed #0): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-0f0bdc2c16e922fbc' does not exist
    status code: 400, request id: 71b98708-cb06-4f11-ad14-8d3d160fbc1a
* aws_instance.mongo-replica-01 (deposed #1): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-080ef01dc84c09685' does not exist
    status code: 400, request id: 07c96f82-1e32-4944-a1d6-ab0e6306b82e
* aws_instance.mongo-replica-02 (deposed #1): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-085b997daac742c1e' does not exist
    status code: 400, request id: 20ff2b73-39c9-4d49-af41-f4ec542ec782
* aws_instance.mongo-replica-02 (deposed #0): Error terminating instance: InvalidInstanceID.NotFound: The instance ID 'i-00bc7fd15b04a3688' does not exist
    status code: 400, request id: cdea3c4f-9bec-496a-aedd-bcfbf0a706d2

The AWS EC2 instances in question do not exist indeed, but TF errors out when trying to delete them (presumably because they don't exist) and still keeps them in the state.

I tried using the terraform state command in an attempt to remove the deposed resources, but it only allows me to delete the whole resource, not just the deposed instance of it...

Am I expected to, simply, manually go in the terraform state file and delete the deposed sections??

3
  • I'm not really sure what you mean by "deposed"? You mean they've been deleted by something other than TF? The default -refresh=true parameter to terraform destroy should automatically update the state file with what's in AWS and thus remove the missing instances and then not try to destroy them.
    – ydaetskcoR
    Commented Jan 26, 2017 at 10:29
  • 2
    Apparently deposed is a state. See, for example, this issue: github.com/hashicorp/terraform/issues/9844
    – gsaslis
    Commented Jan 30, 2017 at 19:46
  • Indeed, documentation is somewhat scarce for the "deposed" state, as indicated in the github issue here
    – ILMostro_7
    Commented Jul 1, 2017 at 12:05

5 Answers 5

14

From the GitHub Issue requesting for Documentation about the "deposed" state

"Deposed" is the state a resource goes into when Terraform is dealing with create_before_destroy... since there can only be one "primary" instance of a given resource, Terraform will first depose the existing one (so it's still tracked in state but no longer used for interpolations), then create the replacement instance before finally deleting the deposed instance.

Errors during that process can cause the deposed instance to stick in the state, since Terraform doesn't want to lose track of them before it can delete them. However, this behavior was a little inconsistent in earlier Terraform versions, and also Terraform handled deposed instances silently rather than including them in the plan output.

So all of this is to say: these deposed instances are leftovers from a failure during the replacement of these resources with create_before_destroy. If you look in the Terraform state file (either terraform.tfstate or .terraform/terraform.tfstate depending on whether you have remote state enabled) you should find the records of these by searching for the word deposed; you can use the data Terraform has stored for these to decide if they are safe to delete before letting Terraform proceed here.

So, you could infer that the "deposed" state of those resources is merely indicative of an underlying issue that's caused errors during the destroy phase. As mentioned in the quote, the "deposed" resources should be "resolved", for lack of a better term, the next time you run terraform apply. In the meantime, check the tfstate files for any/all specific information that terraform has stored to determine if it's safe to proceed and/or to, potentially, diagnose where the problem might be.

Additionally, you can try the following to refresh the local state

terraform refresh      Update local state file against real resources

Also,

terraform debug              Debug output management
terraform state              Advanced state management
1
  • terraform destroy and refresh do not do anything for me; terraform state list shows 5 resources but terraform show indicates all 5 are empty and deposed. Do I remove them with terraform state rm? Then there's also a bunch of data objects left. Might just be easier to drop that state (there are no resources left in AWS related to this).
    – Oliver
    Commented Apr 30, 2020 at 12:41
11

terraform state rm followed by terraform import removes the deposed object from state.

1
  • Thanks for your suggestion, after our CI pipelines tried to apply the failing changes multiple times we had: aws_eks_node_group.node_group is tainted, so must be replaced aws_eks_node_group.node_group (deposed object 9f3a96cc) will be destroyed aws_eks_node_group.node_group (deposed object bd3de6ba) will be destroyed aws_eks_node_group.node_group (deposed object e0e04d23) will be destroyed all of which were cleaned up after terraform state rm 'aws_eks_node_group.node_group' terraform import 'aws_eks_node_group.node_group' my-cluster:my-node-group-name
    – cedric
    Commented Oct 12, 2023 at 13:52
4

I had a similar issue with EC2 instances that I removed, but halted the process mid-way, and what I ended up doing was manually editing my terraform tfstate file.

In it, I found the items that were deposed, changed tainted to false and emptied the deposed array.

After running terraform plan, the instances were gone from the output.

enter image description here

0

For my scenario, I wound up disabling create_before_destroy (changing from true to false). Otherwise, I could not figure out where to apply the other solutions and terraform state did not show deposed nor tainted anywhere.

0

I ran into trouble with this when dealing with EC2 capacity reservations. You of course don't want to lose your existing reservation when changing it, so you use create_before_destroy. But, say you try to upgrade an instance type in your configuration, run terraform apply, but then can't get the instance you want because none are "in stock"... Terraform puts the reservation object in a "deposed" state.

Now, say you abort the apply and then try to put it back. If there aren't any additional instances of the current type in stock, Terraform still try to destroy your deposed reservation and get a new one, even though they're identical. Now you're stuck. I found this to be quite common when trying to get instances with local NVMe "instance store" disks.

The easiest way out of this is to simply crack open the active .tfstate file in your favorite text editor (it's just JSON), search for the "deposed" key/value pair on the resource in question, and delete that line. Save the file, re-run terraform, and et viola! You're back in business.

This is by far the simplest and quickest method to get out of this situation, and would work for any case where the creation of any replacement resource fails and you just want to abort that whole operation and go back to the way things were. Unfortuantely there is no terraform state command to "un-depose" an instance (as of v1.9.8).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.