Deployment Architecture

Indexer cluster autoscaling, how to configure universal forwarders with the new indexer endpoints.

sent2020
Explorer

We are using a splunk indexer cluster in AWS using autoscaling to increase the cluster size. Universal forwarders are configured with the indexers IPs. When a new indexer gets launched how to update the Universal forwarders. Is it recommended to use the ELB for the indexers? Can we use deployment server to handle this?

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

Splunk Cloud uses ELB quite extensively, so there really is no reason you couldnt go that route, and on the back end you can map the ELB to the indexers. When you add indexers, you can add them to the ELB and not worry about it.Do note, that you're using the ELB as more of a front end mapping to the indexers and not the real load balancing functionality of them (use Splunk's LB.)

0 Karma

Jeremiah
Motivator

How does using an elb impact distribution of data across indexers?

0 Karma

ericlaan
New Member

We'd also like to use an ELB to distribute load across our indexers. Currently we have a few intermediate forwarders but the amount is lower than the amount of indexers which leads to some indexers doing nothing while others are being blown out of the water.

Can you comment on the use of the ELB and share experiences? Thanks in advance!

0 Karma

khourihan_splun
Splunk Employee
Splunk Employee

I'd go with Jeremiah's suggestion. LB is built into the forwarders. Splunk Cloud uses native forwarder LB similar to what Jeremiah describes with a single DNS entry resolving to all of the indexers. When more indexers are added, just add them to the DNS entry.

0 Karma

Jeremiah
Motivator

I would not use elb, instead have the forwarders connect directly to the indexers. You have several options:

Yes you could script the update of an outputs.conf file that you deliver from the deployment server. The update interval would depend on how frequently your forwarders check in to the deployment server.

You could switch from using a list of ip addresses to a single DNS entry that resolves to a list of all of your indexers. They you can script the update of the DNS entry. Here the update interval would depend on how long the forwarders hang onto the cached DNS entry.

Another option is to use the indexer discovery feature available in 6.3. You can point your forwarders at your cluster master, and the CM will distribute the list of indexers to the forwarders. Keep in mind that this does make the CM a single point of failure; the docs do a good job explaining forwarder behavior when the CM is offline.

How often do you add new indexers? And how do you handle instance termination in the asg without losing data?

Get Updates on the Splunk Community!

Financial Services Industry Use Cases, ITSI Best Practices, and More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Splunk Federated Analytics for Amazon Security Lake

Thursday, November 21, 2024  |  11AM PT / 2PM ET Register Now Join our session to see the technical ...

Splunk With AppDynamics - Meet the New IT (And Engineering) Couple

Wednesday, November 20, 2024  |  10AM PT / 1PM ET Register Now Join us in this session to learn all about ...