Deployment Architecture

Search Head Clusters in Amazon Web Services Auto Scaling Groups (ASG) - automate cluster removes and joins

New Member


Has anyone got this to work correctly? Is it supported or even a valid idea?

When an ASG instance is terminated, it comes back with a new IP and hostname. The old server IP remains in the cluster and a manual Splunk remove has to be run and a corresponding Splunk add run for the new IP.

We have the Splunk data persisted on a data EBS volume. Presumably, we also need to run a Splunk clean on the newly rebuilt instance before adding it back into the cluster.

My brain is telling me that Splunk is still stuck in the physical tin / Splunk cloud world and this is old school thinking not cloud thinking. Would a single larger standalone deployment with the data on an ebs volume be a better solution? How would this scale and be resilient to failure?

Sorry for the wall of questions but I have reached the end of my Splunk design abilities and we are waiting on raising an official support request.



Tags (1)
0 Karma


With a search head cluster, you're going to have better scalability and high availability, with the added complexities of managing the cluster as it scales up or down as you've pointed out. I don't think you're missing anything there. A single instance would be easier to manage, and you could achieve some resiliency by running an ASG with a min/max of 1 instance. That way if the instance dies, the ASG will automatically replace it. Script the remount of your EBS splunk volume to make sure you still have your configs when the replacement instance comes online. In terms of scaling, increasing the instance size will work, it depends on how many active users and how many concurrent searches you plan on running. Its a totally reasonable solution, especially since you can pretty easily swap out instance types in AWS.

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...