Cisco ESA add-on (email security) extracting logs from a common syslog file

[fahmed11]

I'm using the Cisco ESA add-on (https://splunkbase.splunk.com/app/1761/) 

The documentation references files which need to be monitored by adding monitor stanzas to inputs.conf (example pasted below).

[monitor://<Cisco_Ironport_LOG_PATH>\mail.@20130712T172736.s]
sourcetype = cisco:esa:textmail

 However, I do not have a single file with only Cisco ESA logs. I have a syslog server which receives logs from numerous devices (routers, switches, firewalls) and puts them into a single log file. This log file is then processed by a Splunk UF and sent to Splunk Cloud.

How do I extract and parse Cisco ESA logs from this file using the ESA add-on? I'm assuming that creating a monitor stanza with the common log file with categorize everything as "cisco:esa:textmail".

[richgalloway · Accepted solution]

IMO, it was a mistake to dump all of the syslog events into a single file.  It creates the very situation in which you find yourself.

Better practice is to use the features of the syslog server to separate the events into different files based on the source.  Both rsyslog and syslog-ng can do this.  This makes it simple for a UF to read a file and associate a single sourcetype with it.  See https://www.splunk.com/en_us/blog/tips-and-tricks/using-syslog-ng-with-splunk.html

Another option is the new Splunk Connect for Syslog product.  It's a free syslog container you can deploy to take most of the work out of managing syslog-ng.  See https://www.splunk.com/en_us/blog/tips-and-tricks/splunk-connect-for-syslog-turnkey-and-scalable-sys...

Finally, if can can't or won't do either of the above then you'll have to change sourcetype on the fly as events are read from the syslog file.  That has to be done on the indexers or in a heavy forwarder, not a UF.  It can be challenging because you have to design regular expressions to identify each sourcetype among all of the similiar-looking syslog events.  See https://docs.splunk.com/Documentation/Splunk/8.0.4/Data/Advancedsourcetypeoverrides

[fahmed11]

Appreciate the detailed response. I think separating log files is the way to go. However, it looks like the add-on wants to assign sourcetypes to events by referencing separate logfiles in multiple monitor stanzas. I can separate the ESA logs into another file using a host-based filter, but that means that I may still have to do some regex processing on rsyslog to separate them further into individual file types. I feel like the approach this add-on is taking is a bit odd / restrictive.