We have a requirement to send audit logs from our host servers (/var/log/audit/audit.log) to both our indexers and to a 3rd party host syslog server. I am testing with a host gary-test2.ussl.uhs with audit logs in /var/log/audit/audit.log.
I have configured the universal forwarder host gary-test2.ussl.uhs to redirect all it's logs to the heavy forwarder. I would like to have the heavy forwarder send it's logs to the indexers but also a copy of all audit events to the syslog server syslogp01.ussl.uhs
Here is the architecture involved with the routing. Universal Forwarder gary-test1.ussl.uhs
Heavy Forwarder ussl-splkhfwt01.ussl.uhs
Indexers splkidxt01.ussl.uhs splkidxt02.ussl.uhs
Syslog server syslogp01.ussl.uhs (10.17.8.206)
Here is how I configured the Heavy forwarder... splkhfwt01.ussl.uhs (heavy forwarder host)
[tcpout:default-autolb-group] disabled = false server = splkidxt01.ussl.uhs:9997,ussl-splkidxt02.ussl.uhs:9997
[syslog:Myroutingrsa] server = 10.17.8.206:514 sendCookedData = false type = udp disabled = false
What I am seeing is that the /var/log/audit/audit.log logs from host gary-test2.ussl.uhs are appearing in search queries on Splunk. Also those same logs are appearing on the syslog server. Here is the problems I found. Logs other than the audit.log logs from the host gary-test2.ussl.uhs are also appearing on the syslog server. I suspected that the props.conf and the transforms.conf were not doing their job so I remarked out all the settings in props.conf and transforms.conf then restarted splunk. I found that the logs continued to be sent to the syslog server which says the props.conf and transforms.conf files are having no effect. Just to be sure, in the outputs.conf, I removed the "[syslog:Myroutingrsa]" and it's settings. That then made the logs stop forwarding to the syslog server. Does anyone see what is wrong with my forwarding configuration settings?
Since the defaultGroup is set to the non-existent group "noforward" (meaning that there is no defaultGroup), the forwarder only forwards data that has been routed to explicit target groups in inputs.conf. It drops all other data.