topic Re: breaking log events in Getting Data In

breaking log events

JeffTanYH — Mon, 28 Sep 2020 11:49:01 GMT

I have several text format log files in which I need help in linebreaking them into the appropriate events that I need. This is an exmaple of what my file looks like:

====================================================

LOGS_BEGIN: DO_LOG

====================================================

-rw------- 1 root root 3505 Apr 23 12:58 /var/adm/sulog

====================================================

SU 02/12 16:53 - pts/1 hkim-root

SU 02/12 16:53 + pts/1 hkim-root

SU 02/13 13:10 + ??? root-sys

SU 02/14 13:10 + ??? root-sys

SU 02/15 10:28 + pts/1 hkim-root

SU 02/15 13:10 + ??? root-sys

SU 02/16 13:10 + ??? root-sys

SU 02/17 13:10 + ??? root-sys

SU 02/18 13:10 + ??? root-sys

SU 02/19 13:10 + ??? root-sys

SU 02/19 16:42 + pts/1 hkim-root

SU 02/20 13:10 + ??? root-sys

SU 02/20 15:06 + pts/1 hkim-root

SU 02/21 09:54 + pts/1 hkim-root

SU 02/21 13:10 + ??? root-sys

SU 02/21 17:22 - pts/1 hkim-root

SU 02/21 17:22 + pts/1 hkim-root

SU 02/22 09:56 + pts/1 hkim-root

SU 02/22 10:03 + pts/1 hkim-hkim

SU 02/22 10:03 + pts/1 hkim-sybmp

SU 02/22 13:10 + ??? root-sys

====================================================

I need to be able to filter out the top part of the log,which consist of the 1st line till the 6th line (shown below)

====================================================

LOGSBEGIN: DOLOG

====================================================

-rw------- 1 root root 3505 Apr 23 12:58 /var/adm/sulog

====================================================

After which,I will need each and every line to be a seperate event (shown below)

SU 02/15 10:28 + pts/1 hkim-root

SU 02/15 13:10 + ??? root-sys

I would also have to filter out the last part of the log(shown below)

====================================================

I have tried several props.conf be it doesn't to be working!! Please help me!

Re: breaking log events

lguinn2 — Mon, 14 May 2012 05:42:35 GMT

Try this

props.conf

[yoursourcetypename]
SHOULD_LINEMERGE=false
MAX_TIMESTAMP_LOOKAHEAD=15
TRANSFORMS-d1=delete_header_and_footer

transforms.conf

[delete_header_and_footer]
REGEX=(?:^=====|^LOGSBEGIN|^[-rw][-rw][-rw])
DEST_KEY=queue
FORMAT=nullQueue

What it does: tells Splunk that your data is one event per line, with the timestamp in the first 15 characters. It also takes each event and checks it against the regular expression in transforms.conf. Events that match are discarded.

I checked the regular expression at http://gskinner.com/RegExr/ but you should check it too.

And BTW, the "d1" is just an arbitrary identifier, but it needs to be unique within props.conf.

Re: breaking log events

JeffTanYH — Mon, 14 May 2012 06:57:31 GMT

Hi,thanks for the help!
I hav another questn,my file also includes these lines:

====================================================

-rw------- 1 root sys 2984 Apr 11 2006 /var/adm/loginlog

====================================================

vkaliya:/dev/pts/1:Wed May 4 11:14:34 2005

vkaliaya:/dev/pts/1:Wed May 4 11:14:43 2005

Your above prop.conf and transform.conf works with the earlier example,but it would not correctly timestamp the events (in the 2nd example). Can you assist me pls?

Re: breaking log events

lguinn2 — Mon, 28 Sep 2020 11:49:37 GMT

You could simply remove this line

MAX_TIMESTAMP_LOOKAHEAD=15

I don't think it is really necessary