This is typically a problem in larger companies where the larger Security Group operates independently (for monitoring purposes) of the smaller Dev groups,so most other people would ask "Why do you need a second Forwarder on the same machine?"
Ideally, we'd (Devs) get access to the repository that the Security Group manages, and we'd be able to use the same instance of the Windows Splunk Forwarder.
Since that's not a reality here at this point, and Splunk has not (officially) enabled multiple instances the Windows Splunk Forwarder, I'll have to look at other means of getting the Windows (WMI) metrics I need, possibly using tools from SolarWinds or Dynatrace, if I don't end up writing my own code to get at that data.
Given the above comments what is the correct method in this scenario?
I have a server which has a splunk forwarder installed for the purpose of collecting security logs and sending those to the security team's Indexing cluster.
This server also is using splunk forwarder for a different purpose by a different group and needs their data sent to another cluster.
Can this be done within one app?
Can this be done without the 2nd groups needing to modify the pushed configs from security?
If it is the same data then send it two different instances using the UF:
If its different source types:
0) Ideally, discuss about centralising Splunk in the organisation as various teams are utilising Splunk.
1) Create a new Indexer with two different indexes and set permissions accordingly.
2) Install two UF's based on other recommendations below
3) send the data to both indexers and use transforms to send security data to null queue for other team (tedious and will open up security data to other team, if they change things in the conf file -- conduct your risk analysis)
i remembered this blogpost, where a configuration for your usecase is given. But this definitly is an unsupported configuration and not for use in production.
Why exactly do you want to run multiple forwarders on a single machine, because there are only a few cases where this is needed? What about virtualization?
RE: "Why exactly do you want to run multiple forwarders on a single machine, because there are only a few cases where this is needed?"
Security Group pre-configures the forwarder settings for their specific use on all hosts.
The DevOps people just want some performance metrics, and maybe some custom application-log Splunking.
The Devs will never be granted access to the Security Group's repository.
A second Splunk Forwarder forwarding to another indexer managed by DevOps would solve the problem, without any mods to the original Security Group's Forwarder.
The problem is management of the conf files.
I work in security which has global configurations deployed for forwarders.
When we deployed the forwarder we ran into other unknown users of splunk within our environment.
They also have splunk forwarder deployed and are managing their own conf files. They also send their data to their indexers.
All in All I would rather them not have the ability to mess with the security deployment of splunk or have to sub manage the deployment of their conf files.