Infrastructure Drift Explained: Why 'It Worked When I Provisioned It' Breaks Later

Quick Answer

Quick Answer: Drift is what happens when your real, running infrastructure quietly stops matching the configuration files that supposedly describe it — a console click during an incident, an auto-scaler changing a setting, a one-off CLI command. The fix isn't banning every manual change forever; it's having a habit of detecting and deliberately reconciling drift before it causes an outage.

What Drift Actually Is

Infrastructure-as-code works on one assumption: the configuration file is the single source of truth, and the real, running infrastructure matches it. Drift is what happens when that assumption quietly stops being true.

A concrete example: your Terraform configuration declares a security group that allows inbound traffic only on port 443 from a specific CIDR range. During an urgent production issue, someone opens port 22 from 0.0.0.0/0 in the AWS console to debug something fast, intending to close it right after — and then the incident gets resolved, everyone moves on, and nobody reverts the console change or encodes it back into the .tf file.

At that moment, two things disagree: the actual security group (port 22 open to the world) and the configuration file that claims to describe it (port 22 never mentioned). That disagreement, however small, is drift — and it will sit there, invisible, until something forces a comparison between the two.

Where Drift Actually Comes From

Drift rarely comes from carelessness alone — most of the time it's a reasonable action taken under reasonable pressure:

• Emergency console hotfixes during an incident — the most common source, because incident response correctly prioritizes "fix it now" over "update the code describing it."
• A different team's automation touching a shared resource — a security or networking team's separate tooling modifying a resource your IaC also manages, without either side necessarily knowing about the other.
• Provider-side default changes — a cloud provider updating a default value during an API version bump, changing an attribute's reported state even though nobody on your team touched anything.
• One-off CLI commands — someone running a quick aws or gcloud command directly "just this once" to unblock themselves, with every intention of writing it up properly later.
• Legitimate changes from managed services — an Auto Scaling Group resizing itself based on load is drift relative to a static desired-count in your config, even though nothing is actually wrong; the system is doing exactly what it's supposed to.

Why It Bites Weeks Later, Not Immediately

This is the mechanism the title refers to. Drift usually doesn't cause an immediate problem — the port-22 security group rule from the earlier example keeps working fine for weeks, because nobody runs terraform apply against that resource in the meantime.

Then a routine, unrelated change forces an apply — maybe someone's adding a new tag to the same security group. Terraform refreshes state, compares it against the .tf file, and proposes "fixing" the drift it finds: closing port 22 again, because that's what the configuration has always said. If nobody remembers the original emergency change, this looks like a safe, boring diff to approve — and it silently re-introduces whatever problem that port-22 access was keeping patched over, at a moment with zero context about why it mattered.

It gets worse when the drifted attribute is immutable. Some cloud resource properties can't be updated in place — changing them requires destroying and recreating the resource. If drift touches one of those, an innocuous-looking apply can trigger a replacement of a resource everyone assumed was just getting a quiet tag update, with all the downtime or data-loss risk that a destroy-and-recreate implies.

How Terraform Actually Detects Drift

By default, terraform plan (and apply) refreshes its understanding of each managed resource from the real provider before comparing it against your configuration — so a routine plan already reflects any live drift as part of its proposed changes, whether you were looking for it or not.

For deliberately checking for drift without proposing to fix anything yet, Terraform has a dedicated command: terraform plan -refresh-only. It refreshes resource attributes from the provider and reports the differences between that refreshed state and your configuration — without generating a plan that would change any actual infrastructure.

$ terraform plan -refresh-only

# ~ aws_security_group.app_sg
#   ~ ingress {
#       + from_port = 22
#       + to_port   = 22
#       + cidr_blocks = ["0.0.0.0/0"]
#     }

Plan: 0 to add, 0 to change, 0 to destroy.
# (refresh-only: this only updates recorded state, it proposes nothing)

The practical workflow this enables: run -refresh-only on a schedule (a nightly CI job, for example), inspect whatever diff shows up, and then make a deliberate choice — either the drift was a legitimate change that should be adopted into the .tf file (update the code to match reality), or it was an unintended change that should be reverted (run a normal apply to push reality back to match the code). Drift detection is a decision point you create time to make, not something that silently resolves itself.

Detecting Drift Beyond Your IaC Tool

Terraform's refresh mechanism only catches drift on resources Terraform itself manages, checked at the moment you run it. Two complementary approaches fill the gaps:

• Provider-native drift tracking — AWS Config, for example, continuously records configuration changes to supported resource types independent of whatever IaC tool manages them, giving you a change history even for resources no Terraform state file ever touched.
• Continuous drift scanning — dedicated tools exist specifically to diff live cloud infrastructure against your IaC repository on an ongoing basis, rather than only at the moment someone happens to run a plan. The right choice among these depends heavily on your specific cloud footprint and IaC tool, so it's worth evaluating case by case rather than assuming one tool fits every setup.

Neither approach replaces the habit from the previous section — they just mean you're not solely dependent on remembering to run -refresh-only yourself.

Practical Habits That Actually Reduce Drift

• Restrict standing console/CLI write access for anything IaC manages. Break-glass access for genuine emergencies is reasonable; having broad write access as a daily convenience is how "just this once" becomes a weekly habit.
• Route every emergency fix back into code the same day as a follow-up change, while the context for why it was needed is still fresh — not "eventually," which in practice often means "never."
• Run scheduled -refresh-only checks in CI rather than discovering drift by accident during the next unrelated apply, when there's no time pressure to investigate properly.
• Treat an apply that silently reverts a mystery change as a signal to investigate, not just approve. If a plan proposes undoing something nobody remembers changing on purpose, that's exactly the moment to ask why it drifted before clicking approve — not after something breaks.

None of this requires infrastructure to be perfectly, permanently locked down — it requires closing the loop between "someone changed something real" and "the code now says so" quickly enough that nobody has to guess, weeks later, which version of the truth is actually true. For more DevOps and systems-design fundamentals, see the Priodemy blog.

Frequently Asked Questions