How to integrate AWS Autoscale with Splunk indexers to automate high availability without an admin redeploying configurations

[agoebel]

Last year we had great luck with our Splunk configuration and I'm trying to adapt it to use multisite clustering for this year for a better HA story. There is one place where I'm getting stuck though.

There would be two indexers per AWS region in our setup. Ideally, these are set up to come up with an ASG in case one dies, they can automatically heal. I am not seeing a way for this to work without reconfiguring the forwarders with a new IP addresses when it comes up and it seems using an ELB in front of the indexers is frowned upon. Is there a known way to get this behavior so Splunk heals itself automatically without an admin going in and bringing up a new box and redeploying configurations?

[nkwong_splunk]

Here is a .conf2015 talk that my colleagues and I did on deploying a highly available Splunk Enterprise architecture on AWS. We talk about how to leverage DNS entries instead of hardcoding IP addresses in your forwarders. Also, in Splunk 6.3 we introduced the new feature, indexer discovery, which allows the forwarders to get the full list of indexers from the master node.

Indexer Discovery Overview and Setup
http://docs.splunk.com/Documentation/Splunk/6.3.1/Indexer/indexerdiscovery

Slidedeck from .conf2015 - Deploying Splunk on Amazon Web Services
http://conf.splunk.com/session/2015/conf2015_SYep_Splunk_Cloud_DeployingSplunkOnAmazon.pdf

Recording from .conf2015 - Deploying Splunk on Amazon Web Services
http://conf.splunk.com/session/2015/recordings/2015-splunk-126.mp4

[emiller42]

You're probably going to need to figure out some orchestration here. (And might already have some)

One thing that comes to mind is that you shouldn't be configuring your forwarders with IP addresses for each indexer. Instead, create a DNS listing with all of your indexer IP's as A records within it. Then you just point your forwarder at the DNS record, and it'll load-balance across all the IP's found.

When you need to add/remove indexers, you simply update the DNS listing. The forwarders will pick up on that change and forward to the new indexers automatically.

Relevant documentation