Better Together: Delivering Infrastructure as Code using Git and Terraform

The duties of an operations team supporting software delivery efforts have shifted from managing hardware racks, physical load balancers and data centers, to leveraging cloud platform services where everything is virtualized and can be consumed ad hoc . If you’re familiar with cloud platforms such as AWS, GCP, Azure, you are probably familiar with the inherent value add of shifting responsibility to these types of platforms: they support architecture designs that contain built-in resiliency, security, high-availability, and fault tolerance.

To start consuming these resources, you can click around a GUI in your favorite browser and click away to configure the resources that you need to support your endeavor. Alternatively, you can use the AWS command line interface (CLI). The AWS CLI is robust and allows for building fairly complex commands: this is considered necessary knowledge. With a decent understanding of this tool and the AWS (respective platform) APIs, sophisticated scripts (e.g. bash, python, etc) can be built and executed. Along these same lines, you can consume AWS resources using Software Development Kits (SDK) for a general purpose programming language like Python or Java. The way that one interacts with a cloud API is dependent on the use case of predetermined software/hardware requirements of the platforms being supported, which is a multitude. While the Console, CLI, and SDK work well, today we’re going to make a case for tools that are cloud agnostic: Git and Terraform.

Git and Terraform

Terraform falls into a category of software tooling considered Infrastructure as Code (IaC). At a high level, this tool allows for the management of physical resources that can be used in an IaaS/PaaS/SaaS platforms. How is this valuable you may be thinking? With the ability to create infrastructure such as virtual machines, load balancers, virtual networks, etc., in programmatic definitions provides several advantages. First thing is that we can integrate an IaC configuration (think of a file with programmatic definitions/statements) into an existing Source Code Management Solution (SCM) e.g git, along with a Version Control System (VCS). Since the files that define the infrastructure are placed into a VCS, these files are versioned and can be revisited if necessary. The same methods used for versioning and managing code (in a general sense) can be applied to IaC. The IaC tool to be discussed is called Terraform. The VCS tool to manage the IaC code base is called Git.

Its hard to work in the software development space without at least hearing about Git. There may be other tools that your team/org uses for version control, but the intended purpose of a tool like Git, is to provide functionality for distributed teams to version control software. Git arose from a need to iterate on software quickly, share code between team members, and create process strategies that helps create higher quality code (less bugs, faster development). For example, using the branching features of git allow for teams to work simultaneously on the same codebase.

If you haven’t heard of it before, Terraform is software written in the Go programming language that was created by a company called Hashicorp. The intended purpose of Terraform is to provide its users with a friendly language called HCL (Hashicorp Configuration Language) that defines the desired set of resources that are to be instantiated in the respective cloud platform of your choosing (there are many) and also take on the responsibility for tracking the state of already provisioned infrastructure. Lets take a second to take this in. At a high level terraform provides:

1Desired State Management 2Transparent change management mechanisms 3Can be treated like other source code 4Enables flexible delivery and deployment strategies

So how do these pieces of tech come together? Since Terraform is Infrastructure as Code (IaC) we can use Git to manage it like any other piece of software. Today we’ll cover a simple case on how to organize a Git repository containing Terraform configurations to walk code through development, staging, and production environments.

Terraform Configuration Structure

As shown in this screenshot of the tree command there is a directory called env that contains dev and prod. These are considered the root of the module. This is where we use the Terraform CLI tool to plan and apply infrastructure. Take the dev folder for example. The main.tf file within it references Terraform code stored in modules/generic_team_resources/. The modules directory and its resources can be consumed by any Terraform configuration. The purpose of a module is to provide encapsulation of related resources. This prevents rewriting code. Similar to a function in computing, modules abstract away complexity.

Git Process

Github flow or Git flow are two methods that can be used to manage and modify source code in a transparent way. Today I’ll briefly describe mainline trunk based development. There exists a master branch. This is the source of truth for all working code. Whenever a feature must be created we create a feature branch. A feature branch is tested and if functional can be merged into master after a peer review. After a branch is merged into master, the developer will pull the latest updates from master to their local machine. They can then create another branch from master and continue with feature development. Keep in mind that building Continuous Integration / Delivery / Deployment pipelines depend on a standardized way to make changes to a code base. These methods should increase confidence in code changes, and be scalable enough for a large team.

Releasing

Next we’ll discuss how we can version modules. Git provides the ability to create branches, or point in time snapshots of the code base, a Release Candidate Branch can be created. This is considered a short lived branch that will not be merged back to master. Once a production release is eminent, its time to leverage the git tag command to create an immutable tag off the Release Candidate branch. This now provides a point in time version of the code base that has been tested. Now that the there exists a stable version of the code, we can create a tagged reference to the module that the different environments consume as shown in the screen shot above.

Having a transparent method to version pin code is necessary when walking out changes from a development environment/account to a production setting. Typically releases can be made at the end of sprints and become actions items for any team member to complete in the beginning of the sprint.