Getting Data In

unable to break multi line events in splunk

luv
Explorer

10:32:21,453 INFO [2212] abcdxyz <-|
10:32:21,112 INFO [2212] abcdxyz |
10:32:22,409 INFO [1121] abcdxyz |

10:32:23,003 INFO [1121] abcdxyz |
10:32:23,453 INFO [2212] abcdxyz |
|
............ |
............ |
............ | <---this whole block is taken as the 1st event by splunk
some HTML codes |
............ |
............ |
............ |
<-|
11:09:30,103 INFO [1008] abcdxyz <-|
11:09:30,409 INFO [4560] abcdxyz |
11:10:31,092 INFO [2212] abcdxyz |
11:10:31,470 INFO [2212] abcdxyz | <- this whole block is taken as the 2nd event by splunk

|
............ |
............ |
............ |
Some HTML codes |
............ |
............ |
............ |
<-|
12:09:20,125.....and so on <-|
| <- 4th event
.
.
<-.
. <- 5th event
.
<-.

I don't know why splunk is not breaking the lines when it finds the timestamp instead of that it is combining few lines as if they are single event though they have different timestamps
Is there any way i can split these lines according to the timestamp?

FYI- I am using splunk universal forwarder(Windows) and splunk indexer(Linux)

Thanks 🙂

0 Karma
1 Solution

lguinn2
Legend

I like @mloven_splunk's answer, but I always prefer BREAK_ONLY_BEFORE = <regular expression> to LINE_BREAKER. LINE_BREAKER has potentially nasty side-effects.

[sourcetypenamehere]
BREAK_ONLY_BEFORE =\d{1,2}:\d{2}:\d{2},\d{3}

You may also want to specify the TIME_FORMAT. Here are some links to the documentation:

How timestamp assignment works

Configure timestamp recognition

Finally, if you get the timestamp recognition right, you might not need to specify the linebreaking. Splunk generally breaks on timestamp by default...

View solution in original post

0 Karma

lguinn2
Legend

I like @mloven_splunk's answer, but I always prefer BREAK_ONLY_BEFORE = <regular expression> to LINE_BREAKER. LINE_BREAKER has potentially nasty side-effects.

[sourcetypenamehere]
BREAK_ONLY_BEFORE =\d{1,2}:\d{2}:\d{2},\d{3}

You may also want to specify the TIME_FORMAT. Here are some links to the documentation:

How timestamp assignment works

Configure timestamp recognition

Finally, if you get the timestamp recognition right, you might not need to specify the linebreaking. Splunk generally breaks on timestamp by default...

0 Karma

luv
Explorer

Thanks Iguinn that worked 🙂

0 Karma

lguinn2
Legend

Sorry about the typo - I made the correction in my answer. Thanks for the catch.

0 Karma

mataharry
Communicator

yes they were removed by the html formating

"d" shoud be "backslashd"

0 Karma

lukejadamec
Super Champion

I'm still learning regex, is the regex you posted missing \ before the d's?

0 Karma

lukejadamec
Super Champion

Can you post your configuration for this input from inputs.conf?
See if mloven's line breaker works, and then we'll know if you need help with the timestamp.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

By default Splunk looks for timestamp with date (your just have time) to automatically split the events based on timestamp. Since, your file shows just the time part, splunk need an extra option to break the events on just time. you would have to add "BREAK_ONLY_BEFORE=\d{2}:\d{2}:\d{2},\d{3}" in props.conf to enable splunk to do eventbreak per your requirement.

0 Karma

mloven_splunk
Splunk Employee
Splunk Employee

plop this in a props.conf file

[sourcetypenamehere]
LINE_BREAKER = ([\r\n]+)(?\d{1,2}:\d{2}:\d{2},\d{3}

luv
Explorer

It is a java code used to perform banking transactions
Is there anyway i can split it?
this "10:32:21,453" time stamp doesn't count as a valid time by splunk?

0 Karma

lukejadamec
Super Champion

This log looks like one of those logs that Splunk needs a little help with.
What program is generating this log?

0 Karma

luv
Explorer

no i don't have props.conf configured for this in the indexer.
splunk is supposed to split it automatically right? as there is no need to tell splunk explicitly where to split the events because many lines have timestamps

0 Karma

lukejadamec
Super Champion

How do you have this sourcetype configured in props.conf?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...