Getting Data In

How to integrate AWS Autoscale with Splunk indexers to automate high availability without an admin redeploying configurations

Path Finder

Last year we had great luck with our Splunk configuration and I'm trying to adapt it to use multisite clustering for this year for a better HA story. There is one place where I'm getting stuck though.

There would be two indexers per AWS region in our setup. Ideally, these are set up to come up with an ASG in case one dies, they can automatically heal. I am not seeing a way for this to work without reconfiguring the forwarders with a new IP addresses when it comes up and it seems using an ELB in front of the indexers is frowned upon. Is there a known way to get this behavior so Splunk heals itself automatically without an admin going in and bringing up a new box and redeploying configurations?

0 Karma

Splunk Employee
Splunk Employee

Here is a .conf2015 talk that my colleagues and I did on deploying a highly available Splunk Enterprise architecture on AWS. We talk about how to leverage DNS entries instead of hardcoding IP addresses in your forwarders. Also, in Splunk 6.3 we introduced the new feature, indexer discovery, which allows the forwarders to get the full list of indexers from the master node.

Indexer Discovery Overview and Setup

Slidedeck from .conf2015 - Deploying Splunk on Amazon Web Services

Recording from .conf2015 - Deploying Splunk on Amazon Web Services

0 Karma


You're probably going to need to figure out some orchestration here. (And might already have some)

One thing that comes to mind is that you shouldn't be configuring your forwarders with IP addresses for each indexer. Instead, create a DNS listing with all of your indexer IP's as A records within it. Then you just point your forwarder at the DNS record, and it'll load-balance across all the IP's found.

When you need to add/remove indexers, you simply update the DNS listing. The forwarders will pick up on that change and forward to the new indexers automatically.

Relevant documentation

Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...