Infrastructure as Code: Lessons from Our First Terraform Disaster

Infrastructure as Code (IaC) is a powerful paradigm, allowing teams to manage and provision infrastructure through code rather than manual processes. Terraform, HashiCorp's popular IaC tool, has become a cornerstone for many organizations, including Stacks Horizon, for its ability to define, provision, and manage cloud infrastructure across various providers.

However, with great power comes great responsibility – and the potential for spectacular failures. Today, we're sharing a cautionary tale from our early days with Terraform: our first 'disaster.'

The Day the Environment Vanished

It was a seemingly routine change. A developer needed to refactor a small part of our staging environment's Terraform configuration. The goal was to consolidate a few redundant resource definitions. After making the changes, they ran terraform plan and saw a few resources slated for deletion, which seemed expected given the refactoring. Confident, they executed terraform apply.

What followed was not expected. Moments later, alerts started firing. Our staging environment, a critical testing ground for ongoing development, was gone. Completely gone. Databases, compute instances, load balancers – all deleted.

Panic, naturally, ensued.

What Went Wrong?

Upon immediate investigation, we uncovered a few critical issues that converged to create this disaster:

1. The State File Misconception

The refactoring involved moving a set of resources into a new module. The developer, new to Terraform's intricacies, hadn't properly used terraform state mv or import commands. Instead, they had simply deleted the old resource definitions and created new ones in the module. Terraform interpreted this as a request to destroy the old resources and create entirely new ones, even though logically they were meant to be the same underlying infrastructure.

2. Overly Permissive Permissions

The developer's credentials had full destroy permissions on the staging environment. While necessary for some operations, in this specific context, it allowed a single terraform apply to wipe out the entire environment without an additional layer of approval or protection.

3. Inadequate terraform plan Review

The terraform plan output was reviewed, but not thoroughly enough. The output did show the deletions, but the sheer volume of changes (due to the misinterpretation of the refactoring) was overwhelming, and the critical deletions were missed amidst the noise of planned additions and modifications.

4. Lack of Environment-Specific Protections

While we had some protections for production, our staging environment was treated with less rigor. There were no prevent_destroy = true flags on critical resources, nor was there a robust backup and restore strategy specifically for the IaC-managed state of staging.

The Hard-Earned Lessons and Our New Best Practices

We spent the next few hours frantically restoring the staging environment from backups (thankfully, we had them, albeit not perfectly integrated with IaC). The experience was painful but incredibly enlightening. Here's how we transformed our Terraform practices:

1. Strict State Management Protocols

• terraform state mv for Refactoring: Any renaming or moving of resources must use terraform state mv to inform Terraform that the resource still exists but has a new address in the configuration. If a resource is truly being recreated, terraform import might be needed.
• Remote State Locking: We already used remote state (S3 backend with DynamoDB locking), but we reinforced the importance of ensuring locks are always respected to prevent concurrent state modifications.

2. Granular Permissions and Least Privilege

• Role-Based Access Control (RBAC): We implemented more granular IAM roles. Developers now have limited permissions for apply operations, typically restricted to specific resource types or namespaces. Destructive operations (destroy) require elevated, time-limited permissions, or are routed through CI/CD.
• CI/CD for apply: All terraform apply operations now go through our CI/CD pipeline, requiring pull request reviews and explicit approvals before execution, especially for sensitive environments.

3. Enhanced terraform plan Review Processes

• Automated Plan Analysis: Our CI/CD pipeline now includes steps to parse terraform plan output, specifically looking for destroy actions on critical resources. Tools like terragrunt or custom scripts help to highlight these warnings.
• Peer Review for Plans: For any significant changes, the terraform plan output is now a mandatory part of the pull request review. Reviewers actively look for destroy operations.

4. Robust Environment Protections

• prevent_destroy = true: Critical resources (databases, core networking components) in all environments (including staging) now have lifecycle { prevent_destroy = true } set in their Terraform configuration. This acts as a crucial safety net.
• Regular Backups: We implemented automated, frequent snapshots and backups for all critical data stores, ensuring that even if infrastructure is accidentally destroyed, data can be recovered quickly.
• Drift Detection: Tools were introduced to detect configuration drift, ensuring that manual changes outside of Terraform are flagged, preventing the state file from becoming out of sync with reality.

Conclusion

Our first Terraform disaster was a harsh but invaluable learning experience. It underscored that IaC, while powerful, requires discipline, robust processes, and a deep understanding of the tools involved. By implementing stricter controls, improving our review processes, and leveraging Terraform's built-in safety features, we've significantly hardened our infrastructure management. We hope our story helps you avoid a similar fate and build more resilient IaC practices from the start.