Getting Data In

Can I line break some events and not others coming from the same source/sourcetype?

Path Finder

Hi,

I've struggled with this log file for a while and can't seem to come up with a way to make it very usable.

I have a log file that contains these types of events (examples):

---- SMTPR log entry made at 12/29/2010 17:37:37
Incoming SMTP call from A.B.C.D at 17:37:37.
Message B0290088118@msgid.server.com received at 17:37:37 from external.server.com (unverified [A.B.C.D]).
Size: 1943 bytes
Return-path: user@server.com
Recipients: me@myserver.com, 
Incoming SMTP call from A.B.C.D completed at 17:37:37.


---- SMTPD log entry made at 12/29/2010 13:59:34
*** Log is continued from intermediate LogID [13b014c8] ***
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERA.COM\B0288541202.RCP queued for remote delivery to domain customera.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERA.COM\B0288541209.RCP queued for remote delivery to domain customera.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERA.COM\B0288541210.RCP queued for remote delivery to domain customera.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP queued for remote delivery to domain customerb.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERC.COM\B0288541211.RCP sent to 1 out of 1 recipient(s) in domain customerc.com: 250 2.0.0 oBTIxERa017308 Message accepted for delivery\r\n
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERC.COM\B0288541211.RCP delivered to recipient server@customerc.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERC.COM\B0288541211.RCP deleted for recipient(s) in domain customerc.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP sent to 1 out of 1 recipient(s) in domain customerb.com: 250 2.0.0 oBTIxE5C020605 Message accepted for delivery\r\n
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP delivered to recipient Joel@customerb.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP deleted for recipient(s) in domain customerb.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERD.COM\B0288541229.RCP queued for remote delivery to domain customerd.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERD.COM\B0288541230.RCP queued for remote delivery to domain customerd.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERD.COM\B0288541243.RCP queued for remote delivery to domain customerd.com (.LCK).
*** Intermediate LogID [13b00aec] will be continued later. ***

The SMTPR events are easy to handle as I can just treat it as a multiline event and get what I need out of it. The SMTPD events are harder as in theory I would want to break each line in the entire event up into its own event.

Can I use Line Breaking to break up the one event type and not the other if they are both coming from the same source/sourcetype?

Thanks!

Tags (1)
1 Solution

Motivator

The best thing would be to try and get your SMTPR and SMTPD logs into different files, and to assign different sourcetypes to the different files.

If you do have everything in one file and want to try linebreaker, it should be doable. Something like this might work:

LINE_BREAKER = ([\r\n]+)((----\s)|(\*\*\*\s)|(Message [A-Za-z]:))

to break on either a line starting with either dashes, asterisks, or the "Message E:" string.

View solution in original post

0 Karma

Motivator

The best thing would be to try and get your SMTPR and SMTPD logs into different files, and to assign different sourcetypes to the different files.

If you do have everything in one file and want to try linebreaker, it should be doable. Something like this might work:

LINE_BREAKER = ([\r\n]+)((----\s)|(\*\*\*\s)|(Message [A-Za-z]:))

to break on either a line starting with either dashes, asterisks, or the "Message E:" string.

View solution in original post

0 Karma

Path Finder

Was able to get it to work after updating the ()'s as in the new one. I also had to turn line merge off.

Thanks!

0 Karma

Motivator

Maybe, or maybe it's a bug. What version of Splunk are you running? I seem to remember seeing doing something like that once before, on an older version.

Or try the revised breaker string above - I gave that one a (very quick) test using your sample data and it worked fine on 4.1.6.

0 Karma

Path Finder

I've tried several variations of that regex but it keeps crashing splunkd with "Assertion `end > start' failed". Is it the | that it doesn't like?

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!