Indexer Tuning Best Practices: How to decide which apps or add-ons are not needed?

[hartfoml]

I want to clean up the indexers and remove unnecessary Apps that could be using up unnecessary CPU and memory. I have three indexers and they all have a different set of apps on each of the three indexers. I am on Splunk version 6.2.3

How can I tell if an app is needed on the indexer?
For instance the Windows app is on only one indexer.
Do I need this on all three or none?
I also have S.o.S - Splunk on Splunk on all three indexers, one has the TA-splunk and the Splunk app/add-on for *nix.
Are all three TA-s needed? Don't they all run scripted inputs?
Is there some where or some one that has addressed indexer tuning best practices?

[niemesrw]

There are a few things you should do:

How can I tell if an app is needed on the indexer?
- Generally you can find out if the documentation for the app says it has index-time operations. You'll have to examine each app and see if there are any transforms or props stanzas that would apply at index-time.

Specifically, the windows app contains entries in props.conf that modify sourcetype, which is an index-time operation. So you'll need it on the indexers. You only need it on the indexers where you're sending the windows logs, which is probably all of them.

For the SoS app I'm not sure what the requirements are, but you probably need them all running on all of the indexers to collect information from them.

You might consider setting up a "heavy forwarder" layer where all of your apps are installed, and then removing all or most of the apps from the indexers. That way the tasks of index-time operations can all be done on the heavy forwarders instead of the indexers.

You might find this useful as well: http://wiki.splunk.com/Things_I_wish_I_knew_then