Search Head Clusters in Amazon Web Services Auto Scaling Groups (ASG) - automate cluster removes and joins

[KaneKennedyNHSD]

Hi,

Has anyone got this to work correctly? Is it supported or even a valid idea?

When an ASG instance is terminated, it comes back with a new IP and hostname. The old server IP remains in the cluster and a manual Splunk remove has to be run and a corresponding Splunk add run for the new IP.

We have the Splunk data persisted on a data EBS volume. Presumably, we also need to run a Splunk clean on the newly rebuilt instance before adding it back into the cluster.

My brain is telling me that Splunk is still stuck in the physical tin / Splunk cloud world and this is old school thinking not cloud thinking. Would a single larger standalone deployment with the data on an ebs volume be a better solution? How would this scale and be resilient to failure?

Sorry for the wall of questions but I have reached the end of my Splunk design abilities and we are waiting on raising an official support request.

Cheers,

Kane.

[Jeremiah]

With a search head cluster, you're going to have better scalability and high availability, with the added complexities of managing the cluster as it scales up or down as you've pointed out. I don't think you're missing anything there. A single instance would be easier to manage, and you could achieve some resiliency by running an ASG with a min/max of 1 instance. That way if the instance dies, the ASG will automatically replace it. Script the remount of your EBS splunk volume to make sure you still have your configs when the replacement instance comes online. In terms of scaling, increasing the instance size will work, it depends on how many active users and how many concurrent searches you plan on running. Its a totally reasonable solution, especially since you can pretty easily swap out instance types in AWS.