Where to install apps in a distributed environment?

[JdeFalconr]

We have a distributed environment of one search head, one indexer and one deployment server + license master. I'm working on resolving CPU utilization issues right now related to too many scheduled searches running during the day and towards that end, I'm trying to prune extraneous applications. I've noticed that I have a number of applications installed on my Indexer as well as my Search Head and I'm concerned that they are causing scheduled searches to be executed extraneously. On which of those servers do I need to install each application? Both Search Head and Indexer or only the Search Head?

[richgalloway]

It depends on the app. Add-ons (also called TAs) should be installed on indexers. Apps with a UI go on the search head(s). Apps that have a UI and also have inputs should be installed on both the search head and indexers. In that case, be sure to disable the inputs on the search head so you don't run up your license usage; also disable any scheduled searches on the indexers.

[JdeFalconr]

Thanks, that's extremely helpful. So what's a good way to tell if an app has inputs? Simply whether there's an inputs.conf present? Also should I assume that I should go through and remove any SavedSearch.conf files present for apps on my Indexer?

[richgalloway]

The presence of inputs.conf is a good clue. Don't delete it if it's in the 'default' directory because it may be restored the next time you update the app. Use the GUI to disable the inputs or copy inputs.conf to 'local' and add a disabled=1 attribute for each stanza.
Removing savedsearch.conf will work (after restarting the indexer), but don't delete it from the 'default'. See above.

[Runals]

I'm not sure there is a good way through the GUI (might be wrong). Otherwise though look for inputs.conf in the app default directory as you were thinking. The other thing you might want to explore at some point is having a 'job' server where you try to move your scheduled searches to. That is really just a SH with a more dedicated role not something Splunk specific like a DS, LM, etc.

[Runals]

Also I wouldn't use your DS to push apps with UI elements unless you want them to have kiosk like limitations. Otherwise when users make changes they will be blown away the next time the SH checks in with the DS.

[mreynov_splunk]

1. enabled entries in inputs.conf are a definite way to tell if there are inputs 😄
2. While in general @richgalloway is correct, I would also add the following: since you do not have forwarders and some add-ons may have modular inputs, I would suggest splitting the processing between SH and indexer.
3. I would not remove savedsearches (to avoid human error) from indexers though, since they are not going to run there.

[JdeFalconr]

So I'm a bit confused now. My goal is to avoid saved searches executing on both my indexer as well as my search head. But are you saying that they are not ever going to execute from my indexer and thus my concern is unfounded?

We do have universal forwarders active in our environment but they're all pointed to the Search Head. otherwise I'm not too concerned about local or remote data inputs being duplicated.

[jeremiahc4]

I agree on the confusion. I am in the process of reworking how I deploy an app to a large clustered environment for the very reason that all my indexers were executing my saved searches when I deployed my custom built app to both the SHC and Index cluster. My first attempt to fix it was to change all my saved searches to "enableSched=0" on the savedsearches.conf file that was deployed to my index cluster. I'm looking at potential removal as a next step because all my accelerated searches are filling up the "Report Acceleration Summaries" table on my indexers even though I don't want them there (they are currently summarizing partial data).