To be clear, I've already received official responses from Splunk support on this topic (just in case they feel inclined to follow-up on here), but what I'm looking for now is verification of these processes. Likewise, what's their experience in migrating, and how much of a hassle has it been whenever new "apps" are installed/modified on your installation? Thanks.
We are currently using a single splunk indexer/search head that reads its inputs via NFS mounts on a remote log server. To improve performance, we're moving to a distributed search architecture (load balanced), and so the data will now be sent out from a light-weight forwarder (LWF) instance running directly on the log server itself. One potential problem that we've identified, and don't really see addressed in the docs anywhere, is that any separate inputs.conf files from installed apps need to be addressed somehow on the LWF.
Will those apps have to also be installed on the LWF just so that it's aware of the inputs defined in those files? Or should the inputs.conf files be somehow merged when placing them on the LWF. As an example:
- Splunk has the existing SPLUNK_HOME/etc/system/local/inputs.conf file... which will be moved to the LWF
- Splunk also has the app "OSSEC" installed under SPLUNK_HOME/etc/apps/ossec, which within contains the file SPLUNK_HOME/etc/apps/ossec/local/inputs.conf. Does this entire app need to be installed on the LWF to work properly, or should the contents of the ossec/local/inputs.conf be manually merged with system/local/inputs.conf prior to it getting migrated?
If it's the latter, that could become a bit of a hassle (but obviously still doable) to keep up with.
Anyone care to share if this is also their experience? And what the best method is when migrating and then keeping everything in sync going forward? Thanks in advance.
