What Terraform Remembers, and Why It's a Single Point of Failure

What the state file really is, why it holds secrets, how locking and remote backends work, and what drift looks like in a homelab.

Every Terraform tutorial spends ten minutes on providers, resources and variables, then mentions a file called terraform.tfstate in one sentence, usually followed by "don't touch it." That is roughly the same advice as "this button is important, please don't press it" printed next to the button that runs your whole infrastructure. I run a small homelab (a Proxmox VM hosting LXC containers and a k3s cluster) managed with Terraform, and the state file is the one piece I would be sorry to lose. So here is what it actually is, and how not to lose it.

What the state is, and what it is not

Your .tf files describe what you want. The real world, a hypervisor, a cloud account, is what exists. The state is the third thing: Terraform's memory of which real object corresponds to which block in your code. It is a JSON file that maps proxmox_virtual_environment_vm.k3s to "VM number 230 on this node", along with every attribute the provider returned when it created it.

That mapping is the whole point. Without it, Terraform has no way to know that the VM it sees is the one you declared. Run plan with an empty state and it will cheerfully propose to create everything again, next to what already exists. It is not a cache you can delete and regenerate: it is the only place that knows which real objects are yours.

Why you must not commit it

The state stores attribute values in clear text, including ones you marked sensitive. Marking a variable sensitive hides it from the plan output, not from the file. Passwords, generated keys, tokens and cloud-init content can all end up in there, readable by anyone who can read the file.

So the first rule is boring and absolute: the state goes in .gitignore. In my repo, *.tfstate and *.tfstate.* are ignored, and the file lives only on the server that runs terraform apply. The second rule follows from the first and is less comfortable: if it is not in git, it is not versioned, and if the disk dies, so does your mapping. A local state is a single point of failure that you chose by omission.

The .backup file is not a backup strategy

Next to terraform.tfstate you will find terraform.tfstate.backup. It is the previous version of the state, written just before each apply. It is handy when an apply goes wrong and you want to inspect what changed. It sits in the same directory, on the same disk, so it protects you from a bad apply and from nothing else. A real backup is a copy somewhere else, taken on a schedule, and restored at least once to check that it works.

Locking: two applies, one state

Terraform reads the state, computes a plan, applies it and writes the new state. If two runs overlap, say a CI job and you at your terminal, the second write silently overwrites the first and the state no longer matches reality. This is the classic way to corrupt it.

The local backend takes a lock for the duration of a run, which is enough while you are the only user on one machine. As soon as there are several people or a pipeline, you need a backend that supports locking across machines. If a run is killed halfway and leaves a stale lock, terraform force-unlock releases it, after you have checked that nothing is actually still running. Releasing a lock someone else holds is how you get the corruption the lock was there to prevent.

Remote backends

The usual answer to "where should the state live" is a remote backend: an object store with versioning turned on, encryption at rest, access limited to the identities that need it, and locking. On AWS that traditionally meant S3 plus a DynamoDB table for the lock. Recent Terraform versions can lock with a lock file in the bucket itself, so the extra table is no longer required. Terraform Cloud, GitLab and a few others offer managed alternatives.

The point of a remote backend is not fashion. It gives you three things the local file does not: a history of versions you can roll back to, a lock shared by everyone, and a state that survives the machine that created it. If you want encryption of the state itself and not just of the storage, OpenTofu supports it natively, which is a fair reason to look at it.

Drift: when reality moves without telling Terraform

Drift is the gap between the state and the real world. Someone resizes a VM in the web UI, a script changes a setting, a resource is deleted by hand. Terraform finds out at the next plan, when it refreshes the state from the provider and compares. terraform plan -refresh-only shows the differences without proposing to change anything, which is what you want when you suspect drift but are not ready to act on it.

In my homelab the virtual machine that hosts everything else is deliberately outside Terraform: I changed its memory and CPU count directly with libvirt tools. That is fine as long as it is a decision and not an accident, and as long as it is written down. Drift you chose is documentation. Drift you forgot is the thing that bites during a rebuild.

Fixing the state without destroying anything

You will eventually rename a resource or move it into a module, and Terraform will propose to destroy the old one and create a new one, because from its point of view the old address has disappeared. Do not accept that plan blindly. The tools for this are:

• moved blocks in your code, which tell Terraform that an address changed without touching the real object. They are reviewable in a pull request, which terraform state mv is not.
• import blocks to bring an existing object under management instead of creating a duplicate.
• removed blocks (or terraform state rm) to stop managing an object without destroying it.

Prefer the declarative blocks over the manual commands: they live in git, and the plan shows what they will do before anything happens.

When a provider waits for something that never comes

One last lesson from the homelab, because state problems are not always about the file. One of my virtual machines does not run the QEMU guest agent, but the provider waits for the agent to report the machine's IP address. The result: every apply that touches that machine hangs for roughly fifteen minutes before giving up. The state was fine, the plan was fine, and the apply was slow for a reason that had nothing to do with either. The fix is either installing the agent at first boot or telling the provider not to wait. It is a good reminder that "Terraform is stuck" has more causes than the state file.

A checklist you can steal

• The state is in .gitignore, and it is not in any screenshot or gist either.
• It is backed up somewhere other than the machine that runs Terraform, and you have restored a copy at least once.
• With more than one person or pipeline, it lives in a remote backend with locking and versioning.
• You run plan before every apply, and read the lines that say "destroy" and "replace."
• Renames and moves go through moved blocks, not through a leap of faith.
• Anything you deliberately manage outside Terraform is written down.

The state file is not glamorous. Nobody ever put "managed the state well" on a conference slide. But it is the difference between "I can rebuild this from a repository" and "I hope the disk survives," and only one of those is a plan.