Deployment Architecture

Search Head Clusters in Amazon Web Services Auto Scaling Groups (ASG) - automate cluster removes and joins

New Member


Has anyone got this to work correctly? Is it supported or even a valid idea?

When an ASG instance is terminated, it comes back with a new IP and hostname. The old server IP remains in the cluster and a manual Splunk remove has to be run and a corresponding Splunk add run for the new IP.

We have the Splunk data persisted on a data EBS volume. Presumably, we also need to run a Splunk clean on the newly rebuilt instance before adding it back into the cluster.

My brain is telling me that Splunk is still stuck in the physical tin / Splunk cloud world and this is old school thinking not cloud thinking. Would a single larger standalone deployment with the data on an ebs volume be a better solution? How would this scale and be resilient to failure?

Sorry for the wall of questions but I have reached the end of my Splunk design abilities and we are waiting on raising an official support request.



Tags (1)
0 Karma


With a search head cluster, you're going to have better scalability and high availability, with the added complexities of managing the cluster as it scales up or down as you've pointed out. I don't think you're missing anything there. A single instance would be easier to manage, and you could achieve some resiliency by running an ASG with a min/max of 1 instance. That way if the instance dies, the ASG will automatically replace it. Script the remount of your EBS splunk volume to make sure you still have your configs when the replacement instance comes online. In terms of scaling, increasing the instance size will work, it depends on how many active users and how many concurrent searches you plan on running. Its a totally reasonable solution, especially since you can pretty easily swap out instance types in AWS.

Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...