If you’re beyond the first-weeks-of-a-startup stage, chances are your application’s architecture is pretty complex. Especially in CI/CD pipelines, things change on a near daily basis, so what existed in your codebase last week might not this week. In such fast-moving, large-scale environments, it would be impossible for development teams to manually keep track of and hold in their heads every line of code, every change, every new piece of functionality. So why would we expect the same for every piece of infrastructure or observability configuration?
Terraform is a popular Infrastructure as Code (IaC) tool created by Hashicorp that solves for this complexity challenge by getting the manual pieces out of our heads and into automation pipelines and version control. In this post, we’ll start with Terraform basics. We’ll walk through the benefits of using Terraform in general, then move into the “whys” behind incorporating Terraform into observability. Finally, we’ll take a look at how to implement the Splunk Terraform provider to get you up and running with Observability as Code.
Terraform: what is it & why use it?
Rather than going into UIs or manually interacting with CLIs to deploy infrastructure like AWS EC2 instances, Kubernetes clusters, or even entire short-lived test environments, Terraform is a software tool that enables the provisioning and managing of infrastructure via code. This allows development teams to treat their infrastructure (and observability for infrastructure and applications) like any other piece of code they create, making it easy to maintain, update, and deploy additions or changes. Resources are defined in human-readable Terraform (*.tf) files that declaratively describe the desired infrastructure end-state. Plugins, known as providers, interact with cloud platforms and services to define pieces of infrastructure. There are currently over 1,000 providers in the Terraform Registry for managing resources on AWS, Azure, GCP, Kubernetes, Splunk, and many other platforms.
So why might you want to use Terraform? As we said, Terraform takes the tedious, manual, error-prone provisioning out of the infrastructure management process. Instead, with Terraform, teams can build, change, and manage infrastructure in a version-controlled, shared, and repeatable way all from a centralized location – a Terraform file. This file can be committed and stored alongside existing code in GitHub, GitLab, etc. for safe and collaborative infrastructure management. Additionally, multiple cloud platforms can be managed from within the same Terraform file, eliminating the need to manually move between different configuration interfaces. Terraform’s human-readable file structure makes it quick and easy to define and update infrastructure, and the state file that’s created with every deployment keeps track of existing infrastructure and changes so developers don’t have to.
It might sound like the process of using Terraform would be complicated, but it’s pretty straightforward. After installing Terraform, all you need to get started is a main.tf file and a few quick commands to initialize your configuration, view your infrastructure plan, and apply or deploy the defined infrastructure.
Terraform & Observability
How does observability relate to Terraform or Infrastructure as Code? Like infrastructure, observability configurations can be managed as code using Terraform. Rather than managing resources like dashboards, charts, detectors, or cloud platform integrations via observability platform UIs, they can be configured and deployed with Terraform. What does this bring to a team’s observability practice?
Standardization & Consistency
Terraform managed resources can be packaged into modules making them reusable and easy to deploy. This also means that resources for different services and different environments can share configurations, making observing the end result in observability platforms more consistent. Consistency for resources like dashboards, charts, or alerts makes it easier to interpret data across services, products, or environments. During an incident, knowing where to find key metrics or being able to compare them across environments means faster insight into root cause and decreased time to incident resolution. True, this consistency could be accomplished by manually creating the same dashboards and charts across services and service environments, but this is tedious and leaves room for errors and configuration drift.
Collaboration & Improved Maintenance
Managing observability configurations in code means version-controlled audit logs or documentation of changes to provide context around why resources were created or updated. Unlike with manual configurations where one person can go into the UI and adjust a setting, followed by another person subsequently reverting that setting, or even two people adjusting the same configuration at once, version-controlled configurations must be checked in and approved through the merge request process just like every other code change. Additionally, if updates don’t work out as expected, there’s a simple rollback process to revert changes.
Increased Speed & Scalability
With Terraform, observability configurations are stored in one place – alongside the codebases they monitor. This means no scrambling to find one of hundreds of alerts in a UI to adjust its thresholds – instead, simply go into the codebase to tweak and redeploy. Changes can also be applied across multiple resources and multiple environments quickly, taking away the need to scroll through all resources, make adjustments, and then rinse and repeat for each additional service or environment.
It might seem over-kill to create dashboards and charts via Terraform; however, in a world of microservices, where each microservice exists in multiple environments (e.g. dev, staging, production), deploying observability resources via code not only speeds up development, it speeds up issue detection and resolution.
Improved Security
Rather than provisioning observability platform credentials to multiple team members, observability as code keeps things like API keys in one place – with the code – so they can be managed alongside other code secrets for improved security.
Splunk Terraform Provider
Let’s see how easy it is to create observability resources using the Splunk Observability Cloud Terraform provider.
Note: If you don’t already have Terraform installed, start there. Terraform documentation is super thorough and includes a ton of great tutorials.
We’ve created a new main.tf file in our application’s root directory, but if you’re already using Terraform, you can add the same configuration additions to your existing Terraform file. Here’s an example of a Splunk Terraform provider configuration:
The first section is our required_providers block and declares the provider dependencies that we’ll be using, in our case the Splunk Observability Cloud aka signalfx provider. Think of the required_providers block like an import statement, while the following provider-specific block configures the signalfx provider.
Note: since our Splunk organization is located in us1 and not in the default of us0, we needed to specify the api_url.
We then define resource blocks. Each resource block describes the observability objects we want to create or update – dashboards, detectors, charts, etc. All available configurable resources can be found in the Splunk Observability Cloud provider docs under the Resources section. In our configuration, we’ve defined two resource blocks: a dashboard group, and a dashboard.
Now from this root directory, we can run the terraform init command to initialize the new Terraform infrastructure and install the Splunk Observability Cloud provider:
We can next run terraform plan to see the actions that will be taken once we apply our configuration:
Two resources will be created, a dashboard group and a dashboard like we expect, so we can next run terraform apply to create these two resources:
It looks like our resources were successfully created, and we can head over to the Splunk Observability Cloud UI to confirm.
Going into our dashboards in Splunk Observability Cloud, we can see our newly created dashboard group and dashboard:
But this isn’t super useful yet because there’s no real information here that can help us on our observability journey. But that’s ok, to add new resources or update existing ones we can simply edit our main.tf file.
We’ve added a chart resource to our Terraform file and updated our current dashboard to include the new chart:
We can then run terraform plan to again view the plan for the updated actions:
That looks good, so we can apply the changes by running terraform apply and view our newly Terraform-created chart within our Terraform-created dashboard and dashboard group:
Wrap Up
Whether you’re building out detectors and alerts, dashboards and charts, or integrating cloud services via Terraform, managing observability configurations as code can improve resource consistency, collaboration, and maintenance while increasing development and incident resolution speed. If you’re new to Terraform, check out some of the additional resources below. If you’re already using Terraform, try out observability as code and add some Splunk Observability Cloud resources to your Terraform configuration. Need Splunk Observability Cloud? We have a free 14-day trial!
Resources
