Getting Data In

unable to break multi line events in splunk

luv
Explorer

10:32:21,453 INFO [2212] abcdxyz <-|
10:32:21,112 INFO [2212] abcdxyz |
10:32:22,409 INFO [1121] abcdxyz |

10:32:23,003 INFO [1121] abcdxyz |
10:32:23,453 INFO [2212] abcdxyz |
|
............ |
............ |
............ | <---this whole block is taken as the 1st event by splunk
some HTML codes |
............ |
............ |
............ |
<-|
11:09:30,103 INFO [1008] abcdxyz <-|
11:09:30,409 INFO [4560] abcdxyz |
11:10:31,092 INFO [2212] abcdxyz |
11:10:31,470 INFO [2212] abcdxyz | <- this whole block is taken as the 2nd event by splunk

|
............ |
............ |
............ |
Some HTML codes |
............ |
............ |
............ |
<-|
12:09:20,125.....and so on <-|
| <- 4th event
.
.
<-.
. <- 5th event
.
<-.

I don't know why splunk is not breaking the lines when it finds the timestamp instead of that it is combining few lines as if they are single event though they have different timestamps
Is there any way i can split these lines according to the timestamp?

FYI- I am using splunk universal forwarder(Windows) and splunk indexer(Linux)

Thanks 🙂

0 Karma
1 Solution

lguinn2
Legend

I like @mloven_splunk's answer, but I always prefer BREAK_ONLY_BEFORE = <regular expression> to LINE_BREAKER. LINE_BREAKER has potentially nasty side-effects.

[sourcetypenamehere]
BREAK_ONLY_BEFORE =\d{1,2}:\d{2}:\d{2},\d{3}

You may also want to specify the TIME_FORMAT. Here are some links to the documentation:

How timestamp assignment works

Configure timestamp recognition

Finally, if you get the timestamp recognition right, you might not need to specify the linebreaking. Splunk generally breaks on timestamp by default...

View solution in original post

0 Karma

lguinn2
Legend

I like @mloven_splunk's answer, but I always prefer BREAK_ONLY_BEFORE = <regular expression> to LINE_BREAKER. LINE_BREAKER has potentially nasty side-effects.

[sourcetypenamehere]
BREAK_ONLY_BEFORE =\d{1,2}:\d{2}:\d{2},\d{3}

You may also want to specify the TIME_FORMAT. Here are some links to the documentation:

How timestamp assignment works

Configure timestamp recognition

Finally, if you get the timestamp recognition right, you might not need to specify the linebreaking. Splunk generally breaks on timestamp by default...

View solution in original post

0 Karma

luv
Explorer

Thanks Iguinn that worked 🙂

0 Karma

lguinn2
Legend

Sorry about the typo - I made the correction in my answer. Thanks for the catch.

0 Karma

mataharry
Communicator

yes they were removed by the html formating

"d" shoud be "backslashd"

0 Karma

lukejadamec
Super Champion

I'm still learning regex, is the regex you posted missing \ before the d's?

0 Karma

lukejadamec
Super Champion

Can you post your configuration for this input from inputs.conf?
See if mloven's line breaker works, and then we'll know if you need help with the timestamp.

0 Karma

somesoni2
Revered Legend

By default Splunk looks for timestamp with date (your just have time) to automatically split the events based on timestamp. Since, your file shows just the time part, splunk need an extra option to break the events on just time. you would have to add "BREAK_ONLY_BEFORE=\d{2}:\d{2}:\d{2},\d{3}" in props.conf to enable splunk to do eventbreak per your requirement.

0 Karma

mloven_splunk
Splunk Employee
Splunk Employee

plop this in a props.conf file

[sourcetypenamehere]
LINE_BREAKER = ([\r\n]+)(?\d{1,2}:\d{2}:\d{2},\d{3}

luv
Explorer

It is a java code used to perform banking transactions
Is there anyway i can split it?
this "10:32:21,453" time stamp doesn't count as a valid time by splunk?

0 Karma

lukejadamec
Super Champion

This log looks like one of those logs that Splunk needs a little help with.
What program is generating this log?

0 Karma

luv
Explorer

no i don't have props.conf configured for this in the indexer.
splunk is supposed to split it automatically right? as there is no need to tell splunk explicitly where to split the events because many lines have timestamps

0 Karma

lukejadamec
Super Champion

How do you have this sourcetype configured in props.conf?

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!