In our Splunk environment we have two data centers with one indexer each and one heavy forwarder each, and then we have one distributed search head. My lab environment is my home where I install and test Splunk apps. Since my home/lab box is collapsed, that is to say, the indexer, forwarder, and search head are all one box, it is obvious where I install the apps. However, in our enterprise/production environment, this is far less obvious. One app in particular that we want to run is the Palo Alto Networks App for Splunk 5.0.0. It works fine in the lab, however, we are not sure where to install it in our distributed environment? The Search Head, the indexers, the forwarders, all five boxes, we just aren't sure. Any guidance on this would be appreciated.
The problem is that many apps were built just like your lab environment: with all the parts in one. In the case of apps, it means that all of the configuration files are stored in one app, regardless of whether the configurations apply at input time, parsing time, index time or search time.
You could simply install the app on every indexer, search head and heavy forwarder. That might work for many apps, but it would really be better to split the app into the needed parts. For an XYZ app, you might need 4 new apps:
XYZsh - contains search time configs, such as dashboards, fields, eventtypes, etc. needed by search heads
XYZidx - contains parsing and index time configs such as props.conf and transforms.conf, which are needed by indexers
XYZhf - contains inputs and parsing configs such as inputs.conf props.conf and transforms.conf, which are needed by heavy forwarders
XYZuf - contains input configs such as inputs.conf and props.conf
But you could probably combine the hf and uf versions in most cases. Some apps have a separate add-on in Splunkbase, usually called XYZ_TA that should be installed on forwarders.
This wiki page has helped many people: Where do I configure my Splunk settings?
Although I find your comment very interesting, I do not find a solution to my problem from your answer. You seem to be implying that I have to take someone elses work and dice it up appropriately for my environment. It occurs to me that this should be the burden of the App developer, not the end user. I find it hard to believe that very few enterprises are working in a distributed environment and have to modify all the apps to make them work in their respective environments. Maybe I'm wrong...
Yes, you will need to dice up the app to make it fit your environment. And yes, that's not great. But most of the apps in Splunkbase are free. For free, you get apps that were packaged as standalone apps. For free, you will find many apps that ought to be split into multiple parts, but have not been. But a lot of apps have "add-ons" that at least split-out the parts that need to be placed on forwarders.
Much of this work is underway for apps that are written and supported by Splunk. For the paid apps, this work has been done. For the free apps, it is going to take a while - but you can see that many of the Splunk-supported apps have been split:
Splunk for Cisco ASA + Splunk Add-on for Cisco ASA
Splunk for Unix and Linux + Splunk Add-on for Unix and Linux
There is both an app and an add-on for Palo Alto networks; neither were written by Splunk.
In our distributed environment (index cluster of 12 nodes and search head cluster of 6 nodes) we install based on what the app does. If it is primarily UI/saved search based, then it likely belongs on the search head. If it is mostly extracts & lookups based, then it goes on the indexers. One way to handle if not sure, is to install on both indexers and search heads, but I would urge you to go through the saved searches on the indexers and disable any scheduled searches to prevent alerts and reports being run on the indexer assuming your search head distributes to both indexers.
disclaimer: I've not used the specific app you mentioned. I have an app I wrote, so I know very specifically what it does and therefore where it should exist.
According to the documentation for the Palo Alto app, located here http://pansplunk.readthedocs.org/en/latest/getting_started.html, it seems you should install it on forwarders, search heads and indexers.
The Palo Alto Networks App and Add-on
must be installed on all Searchheads,
Indexers, and Heavy Forwarders.
As per "Note The Palo Alto Networks App and Add-on must be installed on all Searchheads, Indexers, and Heavy Forwarders." Now I feel a bit like Charlie Brown when Lucy often says to him: "You block head Charlie Brown". I know you didn't intend that. However, THANK YOU for pointing out the obvious 🙂 Very Respectfully, Peter.