Indexer cluster autoscaling, how to configure universal forwarders with the new indexer endpoints.

[sent2020]

We are using a splunk indexer cluster in AWS using autoscaling to increase the cluster size. Universal forwarders are configured with the indexers IPs. When a new indexer gets launched how to update the Universal forwarders. Is it recommended to use the ELB for the indexers? Can we use deployment server to handle this?

[esix_splunk]

Splunk Cloud uses ELB quite extensively, so there really is no reason you couldnt go that route, and on the back end you can map the ELB to the indexers. When you add indexers, you can add them to the ELB and not worry about it.Do note, that you're using the ELB as more of a front end mapping to the indexers and not the real load balancing functionality of them (use Splunk's LB.)

[Jeremiah]

How does using an elb impact distribution of data across indexers?

[ericlaan]

We'd also like to use an ELB to distribute load across our indexers. Currently we have a few intermediate forwarders but the amount is lower than the amount of indexers which leads to some indexers doing nothing while others are being blown out of the water.

Can you comment on the use of the ELB and share experiences? Thanks in advance!

[khourihan_splun]

I'd go with Jeremiah's suggestion. LB is built into the forwarders. Splunk Cloud uses native forwarder LB similar to what Jeremiah describes with a single DNS entry resolving to all of the indexers. When more indexers are added, just add them to the DNS entry.

[Jeremiah]

I would not use elb, instead have the forwarders connect directly to the indexers. You have several options:

Yes you could script the update of an outputs.conf file that you deliver from the deployment server. The update interval would depend on how frequently your forwarders check in to the deployment server.

You could switch from using a list of ip addresses to a single DNS entry that resolves to a list of all of your indexers. They you can script the update of the DNS entry. Here the update interval would depend on how long the forwarders hang onto the cached DNS entry.

Another option is to use the indexer discovery feature available in 6.3. You can point your forwarders at your cluster master, and the CM will distribute the list of indexers to the forwarders. Keep in mind that this does make the CM a single point of failure; the docs do a good job explaining forwarder behavior when the CM is offline.

How often do you add new indexers? And how do you handle instance termination in the asg without losing data?