Splunk Search

How to extract mail logs (clearswift) and link across multiple lines

lmaclean
Path Finder

Hi,

I have searched and haven't really found anything to parse Clearswift mail logs. The issue is that one email may be on 10+ log lines because each part of the email header is on it's own line. On top of that I need to filter the logs based upon the log type which is the 9th field within the log if the mail system or 6th if another part of the clearswift system. Though for the email logs there is the transaction ID (example below "t6T5CBOe007210") which can be used to link the lines together but I am just not sure on how to do this.

An example of the logs:

Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: TLS_Requested=0 (none)
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: Not using TLS to deliver to smtp server: gmail-smtp-in.l.google.com
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: >>> MAIL From:<email> SIZE=274311
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: <-- 250 2.1.0 OK fl3si6695494pad.107 - gsmtp
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:29Z <mailserver> mail - - - sm-outbound[8461]: t6T5CBOe007210: >>> RCPT To:<email>
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:31Z <mailserver> mail - - - appspam[18234]: perconnection_bExemptTrustManagerBad: true
Jul 29 15:15:33 <host> 1 2015-07-29T05:12:31Z <mailserver> mail - - - appspam[18234]: perconnection_bExemptAntiSpoofing: true

Jul 29 15:27:53 <host> 1 2015-07-29T05:26:49Z <mailserver> pmm - - - INFO : RemoteActionWatcherTask: 1 action files copied to <mailserver>

skoelpin
SplunkTrust
SplunkTrust

Can you explain what you mean by filtering. Do you mean linebreaking an event?

0 Karma

lmaclean
Path Finder

I mean creating different tags or sourcetypes based upon if it is part of the "mail" system then for "sm-outbound" events group them together by the transaction ID, if "sm-inbound" same thing, etc... While if it was for another system like "pmm" then having a sourcetype to read those lines differently.

0 Karma

domenico_perre
Path Finder

Hi lmaclean,

Creating sourcetypes can be simple but it depends on how you are logging and where you are logging to.

Are you logging to a file or are you sending straight syslog to your splunk indexer. Do you have any intermediate forwarders?

I can give you a hand with creating an app if you want that will set correct sourcetypes etc. Its not too hard once you do a couple :).

A props.conf could create a transaction id field on the fly for you. Can you post what your sourcetype is looking like at the moment?

props.conf (on your search head) could be something like this if your sourcetype was [clearswift]
[clearswift]
EXTRACT-transaction-id = (?:\s+)(?[\d\w]{14})

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...