What is the best way to manage universal forwarders and indexers at scale?
By this I mean, If I have 10 indexers today, I specify those on the universal forwarders configuration (outputs.conf). If I add 3 new indexers will have to update every universal forwarder in my site?
This could be upwards to 200 machines running the universal forwarders that would need an update. I've read a few times that using a loadbalancer is not a good solution to reduce the overhead of managing the indexers list.
What is the best way to manage the indexers list and configuration on the forwarders?
Probably best to use a Deployment Server to manage outputs.conf across all your indexers. Just build it as part of a deployed app.
Lots of documentation on using a deployment server:
[http://docs.splunk.com/Documentation/Splunk/5.0.3/Deploy/Aboutdeploymentserver][1]