Tech Talk Recap
From Chaos to Control: Scaling Splunk Cloud with Infrastructure as Code
Managing apps in Splunk Cloud Platform becomes complex quickly, as an average stack of about 50 applications introduces significant configuration, versioning, and operational overhead.
It can be challenging due to the number of applications involved. Even with around 50 apps, teams face considerable configuration, versioning, and operational tasks. Infrastructure as code (IaC) offers a solution by enabling teams to manage infrastructure through files rather than manual actions. This approach shifts management from browser clicks or one-off CLI commands to repeatable definitions that describe the desired end state, providing clearer control over changes in complex cloud environments.
This is particularly important for app management in Splunk, where apps, users, tokens, and indexes must remain aligned. With the right workflow, teams can define these resources once, apply them consistently, and prevent drift across environments.
Why Declarative Workflows Matter
Imperative tools specify how to perform actions, such as creating files or restarting services. Declarative tools specify the desired result, and the tool determines how to achieve that state. The ACS CLI is mostly imperative, requiring explicit actions like restarting a stack. While creating apps or users is closer to declarative intent, the CLI lacks shared state, so independent changes by multiple users can cause inconsistencies. Declarative workflows allow defining the target state once and comparing it with the current state, enabling safer and more predictable management of changes in Splunk Cloud.
What Infrastructure as Code Improves
Storing configuration in CI/CD workflows brings benefits such as:
- Auditability: Changes stored as code (usually in Git) provide visibility into what changed, when, and in what order.
- Compliance and Access Control: Essential in regulated environments (e.g., SOC2, HIPAA), supported by CI/CD platforms like GitHub Actions, GitLab CI, and Jenkins with branch and environment variable management.
- Reproducibility and Rollback: Code-driven definitions allow consistent recreation of environments and easy rollback to previous configurations if issues arise.
These benefits depend on how Terraform manifests are written, emphasizing:
- Separation of concerns through files and modules
- Environmental parity across development, staging, and production
- Reduced risk of storing secrets in source code by adopting safer patterns
Using Terraform for Splunk Cloud App Management
Terraform, developed by HashiCorp (now part of IBM), is a popular IaC solution using its own .tf manifest format. Teams define desired infrastructure and Terraform aligns actual infrastructure accordingly.
For Splunk Cloud Platform, the Terraform provider is available in the Terraform Registry under providers/Splunk/scp. The provider acts like a library with resources functioning as functions within it, supporting:
- HEC tokens
- Indexes
- IP allow lists
- Roles
- Users
- Splunkbase apps
- Private apps
This provides a broad set of building blocks for managing Splunk Cloud through code.
Building a Practical Example with Splunkbase and GitHub
A practical example uses the Splunk Add-on for GitHub (Splunk TA for GitHub) from Splunkbase. The goal is to install the app with Terraform, configure it, and use it to bring GitHub audit activity into Splunk Cloud.
The initial manifest defines the Splunk provider, pins a provider version, and includes connection details for Splunk Cloud and Splunkbase. Hard-coded credentials in the first version are noted as a poor practice due to security risks.
Improvements include using Terraform variables to separate sensitive values from the infrastructure blueprint, allowing loading from .tfvars files. The manifest defines both a Splunkbase app resource and an scp_user resource for a user accessing the app. Using references between resources reduces duplication and lets Terraform infer dependencies automatically.
Reducing Repetition and Managing Dependencies
Terraform allows referencing resource attributes directly to reduce repetition and help understand resource order. Defining locals stores reusable values like app names, simplifying configuration across resources.
Explicit dependency declarations using depends_on ensure correct resource creation order, such as installing the app before creating the user, which is critical in Splunk Cloud.
A distinction exists between Terraform and Splunk naming:
- The label in the resource line is Terraform’s internal name
- The name parameter appears in the Splunk UI
- Changing one does not necessarily change the other
From Manifest to Running Resources
The Terraform workflow includes:
- Terraform init: Downloads required providers and sets up the working directory.
- Terraform plan: Compares current state with desired state, showing planned changes without applying them.
- Terraform apply: Creates or updates resources as defined. Existing resources are refreshed, and unnecessary changes are skipped, ensuring idempotency.
This contrasts with imperative tooling, where repeated runs may recreate actions unnecessarily. Terraform’s declarative approach ensures no changes if the desired state is already met, ideal for CI/CD pipelines.
Extending the Workflow To GitHub
Adding the GitHub provider and github_repository resource, Terraform creates a repository in a GitHub account. The Splunk Add-on for GitHub collects audit data from this repository into Splunk Cloud, closing the loop:
- Terraform installs and configures the Splunk app
- Terraform creates the GitHub resource
- The app collects GitHub activity
- Splunk displays the activity in search
For more Splunk Cloud Application Management in Terraform go here.
Improving Security and Quality in Terraform Definitions
Infrastructure as code enables automated analysis and guardrails. Tools like Checkov scan Terraform manifests for errors and misconfigurations, flagging weak password patterns and other issues.
Snyk can scan Terraform definitions for cloud resources like AWS and GCP, though Splunk-specific Terraform definitions are not currently supported. Nonetheless, automated checks help catch issues earlier than manual UI or CLI management.
Why Remote State Matters for Teams
Local Terraform state files work for simple cases but pose challenges in team environments:
- Multiple administrators managing the same resources can cause conflicts without shared state.
- Local state loss (e.g., lost laptop) risks losing infrastructure state history, complicating recovery.
Terraform supports remote state backends such as S3, Kubernetes, Postgres, Cassandra, and Terraform Cloud. Using Terraform Cloud provides a centralized, durable record of operations with details like:
- Who triggered the run
- Terraform version used
- Tags (e.g., staging)
- Full run history including successes and failures
Practical Takeaways for Splunk Teams
For teams new to managing Splunk Cloud apps with Terraform:
- Start with declarative infrastructure basics
- Use the Splunk Cloud provider from the Terraform Registry
- Store sensitive values in variables and .tfvars files, not directly in manifests
- Reduce duplication with resource references and locals
- Use terraform plan before terraform apply
- Add remote state for team workflows
- Scan manifests with tools like Checkov and Snyk
Explore The Next Step
Terraform offers a practical way to manage Splunk Cloud Platform resources with consistency, auditability, and control. The provider supports operations from app installation to managing users, indexes, and tokens, fitting naturally into CI/CD workflows.
To learn more, explore the Splunk Cloud Terraform provider in the Terraform Registry, browse apps on Splunkbase, and follow the full GitHub-based example shared in the community. For additional technical sessions, join upcoming Splunk Tech Talks and engage right here on community.splunk.com.
Watch the full Tech Talk replay here
- Get more out of Splunk with applications
- Read more about Splunk Cloud Application Management in Terraform
Behind the mic: Gregory Reshetniak, Senior Product Manager, Splunk.
I’m a technical product manager in the cloud / platform engineering domain. I bridge between teams engineering cutting edge solutions using the latest technology to business stakeholders and clients.
Proponent of data driven decision making, I’m very active in strategic discussions, looking to perfect definitions, goals, metrics and missions. More than happy to crunch numbers to better understand the task at hand. I’ve been keenly interested in UX discipline since before such roles existed on a market - I enjoy building and testing prototypes, conducting user interviews, ideating new product features and studying analytics data for insights about the customer.
About Terraform
Terraform is an open-source infrastructure as code software tool created by HashiCorp. It enables users to define and provision data center infrastructure using a declarative configuration language known as HashiCorp Configuration Language (HCL). Terraform manages infrastructure lifecycle through repeatable, version-controlled configuration files, supporting a wide range of providers and platforms. It helps teams automate infrastructure deployment, improve consistency, and reduce manual errors, making it a popular choice for cloud and on-premises infrastructure automation.
