Getting Data In

How to Forward events to both our indexers and a syslog server?

giga444
Engager

We have a requirement to send audit logs from our host servers (/var/log/audit/audit.log) to both our indexers and to a 3rd party host syslog server. I am testing with a host gary-test2.ussl.uhs with audit logs in /var/log/audit/audit.log.

I have configured the universal forwarder host gary-test2.ussl.uhs to redirect all it's logs to the heavy forwarder. I would like to have the heavy forwarder send it's logs to the indexers but also a copy of all audit events to the syslog server syslogp01.ussl.uhs

Here is the architecture involved with the routing.
Universal Forwarder
gary-test1.ussl.uhs

Heavy Forwarder
ussl-splkhfwt01.ussl.uhs

Indexers
splkidxt01.ussl.uhs
splkidxt02.ussl.uhs

Syslog server
syslogp01.ussl.uhs (10.17.8.206)

Here is how I configured the Heavy forwarder...
splkhfwt01.ussl.uhs (heavy forwarder host)

/opt/splunk/etc/apps/forwarder_syslog/local/props.conf
[source::/var/log/audit/audit.log]
TRANSFORMS-routing=troutingrsa

/opt/splunk/etc/apps/forwarder_syslog/local/transforms.conf
[troutingrsa]
REGEX=.
DEST_KEY=_SYSLOG_ROUTING
FORMAT=Myroutingrsa

/opt/splunk/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group
indexAndForward = 0

[tcpout:default-autolb-group]
disabled = false
server = splkidxt01.ussl.uhs:9997,ussl-splkidxt02.ussl.uhs:9997

[syslog:Myroutingrsa]
server = 10.17.8.206:514
sendCookedData = false
type = udp
disabled = false

What I am seeing is that the /var/log/audit/audit.log logs from host gary-test2.ussl.uhs are appearing in search queries on Splunk. Also those same logs are appearing on the syslog server. Here is the problems I found. Logs other than the audit.log logs from the host gary-test2.ussl.uhs are also appearing on the syslog server. I suspected that the props.conf and the transforms.conf were not doing their job so I remarked out all the settings in props.conf and transforms.conf then restarted splunk. I found that the logs continued to be sent to the syslog server which says the props.conf and transforms.conf files are having no effect. Just to be sure, in the outputs.conf, I removed the "[syslog:Myroutingrsa]" and it's settings. That then made the logs stop forwarding to the syslog server. Does anyone see what is wrong with my forwarding configuration settings?

0 Karma

ruman_splunk
Splunk Employee
Splunk Employee

When you create the [syslog:Myroutingrsa] stanza, all data is sent to that group.

In order to avoid default sending data to that group, you could try something like

[syslog]
defaultGroup = bogusGroup

where bogusGroup does not exist.

this is analogous to the tcp routing instructions here: https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Index_one_input_l...

Since the defaultGroup is set to the non-existent group "noforward" (meaning that there is no defaultGroup), the forwarder only forwards data that has been routed to explicit target groups in inputs.conf. It drops all other data. 
0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...