Getting Data In

Can I line break some events and not others coming from the same source/sourcetype?

Derek
Path Finder

Hi,

I've struggled with this log file for a while and can't seem to come up with a way to make it very usable.

I have a log file that contains these types of events (examples):

---- SMTPR log entry made at 12/29/2010 17:37:37
Incoming SMTP call from A.B.C.D at 17:37:37.
Message B0290088118@msgid.server.com received at 17:37:37 from external.server.com (unverified [A.B.C.D]).
Size: 1943 bytes
Return-path: user@server.com
Recipients: me@myserver.com, 
Incoming SMTP call from A.B.C.D completed at 17:37:37.


---- SMTPD log entry made at 12/29/2010 13:59:34
*** Log is continued from intermediate LogID [13b014c8] ***
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERA.COM\B0288541202.RCP queued for remote delivery to domain customera.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERA.COM\B0288541209.RCP queued for remote delivery to domain customera.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERA.COM\B0288541210.RCP queued for remote delivery to domain customera.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP queued for remote delivery to domain customerb.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERC.COM\B0288541211.RCP sent to 1 out of 1 recipient(s) in domain customerc.com: 250 2.0.0 oBTIxERa017308 Message accepted for delivery\r\n
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERC.COM\B0288541211.RCP delivered to recipient server@customerc.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERC.COM\B0288541211.RCP deleted for recipient(s) in domain customerc.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP sent to 1 out of 1 recipient(s) in domain customerb.com: 250 2.0.0 oBTIxE5C020605 Message accepted for delivery\r\n
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP delivered to recipient Joel@customerb.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERB.COM\B0288541215.RCP deleted for recipient(s) in domain customerb.com.
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERD.COM\B0288541229.RCP queued for remote delivery to domain customerd.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERD.COM\B0288541230.RCP queued for remote delivery to domain customerd.com (.LCK).
Message E:\PROGRAM FILES\SPOOL\DOMAINS\CUSTOMERD.COM\B0288541243.RCP queued for remote delivery to domain customerd.com (.LCK).
*** Intermediate LogID [13b00aec] will be continued later. ***

The SMTPR events are easy to handle as I can just treat it as a multiline event and get what I need out of it. The SMTPD events are harder as in theory I would want to break each line in the entire event up into its own event.

Can I use Line Breaking to break up the one event type and not the other if they are both coming from the same source/sourcetype?

Thanks!

Tags (1)
1 Solution

southeringtonp
Motivator

The best thing would be to try and get your SMTPR and SMTPD logs into different files, and to assign different sourcetypes to the different files.

If you do have everything in one file and want to try linebreaker, it should be doable. Something like this might work:

LINE_BREAKER = ([\r\n]+)((----\s)|(\*\*\*\s)|(Message [A-Za-z]:))

to break on either a line starting with either dashes, asterisks, or the "Message E:" string.

View solution in original post

0 Karma

southeringtonp
Motivator

The best thing would be to try and get your SMTPR and SMTPD logs into different files, and to assign different sourcetypes to the different files.

If you do have everything in one file and want to try linebreaker, it should be doable. Something like this might work:

LINE_BREAKER = ([\r\n]+)((----\s)|(\*\*\*\s)|(Message [A-Za-z]:))

to break on either a line starting with either dashes, asterisks, or the "Message E:" string.

0 Karma

Derek
Path Finder

Was able to get it to work after updating the ()'s as in the new one. I also had to turn line merge off.

Thanks!

0 Karma

southeringtonp
Motivator

Maybe, or maybe it's a bug. What version of Splunk are you running? I seem to remember seeing doing something like that once before, on an older version.

Or try the revised breaker string above - I gave that one a (very quick) test using your sample data and it worked fine on 4.1.6.

0 Karma

Derek
Path Finder

I've tried several variations of that regex but it keeps crashing splunkd with "Assertion `end > start' failed". Is it the | that it doesn't like?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...