I just installed the SplunkforF5 app. I installed it on the indexer and the search head. The app has many scheduled searches, including some that feed the summary index. It seems to me that having both the search head and the indexer run the scheduled searches and si-related commands is a waste.
Just disable the scheduled jobs on the indexer? Best practices?
I'd recommend just installing it on the Search head. Set up the search head as a forwarder so that the summary indexes are populated on the indexers (for redundancy and licensing measures).
I don't use the application myself, but those would be my suggestions off the top of my head.
For future Splunkers, here's what I did. Comments welcome. On the indexer, I removed the F5 app, and moved the input I had defined for the LTM logs into .../etc/system/local/inputs.conf. Restarted the indexer. On the search head I enabled the Forwarder app (splunk enable app SplunkForwarder -auth admin:whatevs), restarted, added the indexer as a destination (splunk add forward-server 10.1.176.114:7903 -auth admin:whatevs), and finally restarted again. Badabing? Thanks to Brian and GK.
By doing so, where are your indexes configured ? shouldn't you keep indexes.conf (additionally to inputs.conf) on the indexer ?
Yes. FYI, you can globally disabled all scheduled searches on a Splunk indexer (or forwarder; this is set in LWF inside the SplunkLightForwarder app) by putting into
[pipeline:scheduler] disabled_processors = LiveSplunks
This works in 4.1+, but I'm not sure about 4.0.
I don't know exactly how the F5 app is. If it's just search-time things, you can just put in on the search head only. However, some apps contain index-time configuration.
There are certain best practices that I recommend for people creating apps that might be deployed in a distributed Splunk system (i.e., pretty much all of them), and my recommendations are in an answer to this:
Basically, I think that large monolithic apps are a bad idea, and they should be decomposed into the components as I outlined for proper deployment and configuration. It would be nice if you could pack everything together, deploy a single app, and have the Splunk instance figure out it's own "role" and therefore what settings to ignore or pay attention to. However, we're not yet there in Splunk, so what I'm suggesting works. Even in cases where the Splunk server can selectively use parts of an app, I'd suggest that development of an app benefits from the breakdown, and usage of an app by another party over time will also work better with this kind of role decomposition of app configs.