Getting Data In

How to edit my configuration to line break events at every "= ID:" in my sample log file?

sshres5
Communicator

Some of the events are not being broken down. It works most of the time, but will not break lines couple of times, each time the log gets ingested.

Moreover, the config works fine in my test environment. And I repeat, there is no issue over there. However, when I deploy it on prod, it is failing couple of times in each log.

Log sample

= ID: 453608, XXXXXXXXX: **MonitorAll YYYYYYYYYYYYYYY YYYYYY aYYYYYYYYY: N/A, Target: N/A, Filename: N/A, Blocked: XXXXX, Endpoint: ??????????????? = ID: 453604, XXXXXXXXX: **MonitorAll -YYYYYYYY YYYYY vYYYYvYYYY N/A, Target: N/A, Filename: N/A, Blocked: XXXXX, Endpoint: ????????????? = ID: 453605, XXXXXXXX: **MonitorAll -YYYYYYY eYYYYYYY CYYYYYYYYYYY N/A, Target: N/A, Filename: N/A, Blocked: XXXXX, Endpoint: ????????????????

I have been trying to start a new line every time, I see = ID:

Both the configs work most of the time, but there is always some event, just like above, that has hiccups.

KV_MODE = none
SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ^\=\sID:\s 

KV_MODE = none
SHOULD_LINEMERGE = false
LINE_BREAKER=([\n\r]+(\=\sID:\s+))
0 Karma
1 Solution

somesoni2
Revered Legend

Try this

[Yoursourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)(?=\=\sID:\s)

OR

[Yoursourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\=\sID:\s)

View solution in original post

0 Karma

somesoni2
Revered Legend

Try this

[Yoursourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+)(?=\=\sID:\s)

OR

[Yoursourcetype]
SHOULD_LINEMERGE=false
LINE_BREAKER=(\=\sID:\s)

View solution in original post

0 Karma

sshres5
Communicator

I have already tried both of them, does not work on my production system. Works great on my test box though. And it is just few lines on the log that it skips, but rest of them works fine.

0 Karma

somesoni2
Revered Legend

Then there is must be something different between your test box and Production. Where are you putting this props.conf for line breaking (it should be on Indexer OR heavy forwarder whichever comes first in the data flow from source). Often test boxes are standalone Splunk (acts as both Search Head and Indexer) so when migrating to PROD with distributed environment, it should be configured on Indexer/HF and Splunk should be restarted.

0 Karma

sshres5
Communicator

I have it under Indexer.

0 Karma

sshres5
Communicator

So yes, the location of the props.conf was the issue. Once I moved it to the forwarder TA in HF, it works as a charm.

Thanks @somesoni2

This document has details on it.
http://wiki.splunk.com/Where_do_I_configure_my_Splunk_settings%3F

So if you are using HF, parsing needs to be done on HF.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.