I have the case that I am sharing the UFs with the Splunk SIEM solution, however I work for another project collecting the Unix / Database log details. I have no access to the SIEM and there is basically little chance to reuse the data from there for our purpose.
So, I would like to collect for example the /var/log/messages from the unix/vm machines and send it to my own indexer. I thought I would create a custom app, say called VARLOG, which would consist of the inputs.conf and outputs.conf and forward the var/log/messages to my Splunk. Now, the questions that come to my mind are:
- how does it work actually when there multiple inputs/outputs.conf in different apps on the forwarder?
- is it possible to have it that way at all? Would my inputs/outputs.conf be valid only for my VARLOG app as it is in the corresponding app folder on the fowarder? Or will the inputs/outputs files be joined by the forwarder based on the precedence rules and then I really need to be careful what goes where?
Shortly speaking, how would I take the //messages and forward it somewhere else in case it is already being collected by other app?
You need to be really careful about what goes where. Splunk apps are not fully independent entities. Instead, all inputs.conf files are merged to define the inputs for the UF. Likewise for outputs.conf.
--- If this reply helps you, an upvote would be appreciated.
As you haven't any control for SIEM then it maybe better to install additional UF to that host? If you do that then you must use separate folder e.g. /opt/splunkforwarder_2 and also you must update startup files and/or those names something else than splunk / splunkd.service or otherwise there will be some challenges later on.
Another option as @richgalloway and you already said, is use separate inputs.conf and outputs.conf on that host. BUT that must agreed with SIEM group or otherwise you could be sure that time by time you will lost your logs. In this case btool is your friend. And you must agree proper change management with test with SIEM group!