topic Multiline Event in Monitoring Splunk

Multiline Event

nikhilmehra79 — Mon, 28 Sep 2020 14:54:20 GMT

HI,

I am new to Splunk but have questions which people should have experienced:

I am currently setting up a POC environment in multi DC environment, env is simple right now:

In one DC we have a Single instance of Splunk which contains (i think every component), we have a universal forwarder system looking at 5 different log files of (same type)

Till now i am able to go to Universal Forwarder - :\Program Files\SplunkUniversalForwarder\etc\system\local\ and made an entry for - crcSalt =

eg:

[monitor://D:\Notfier.out]

crcSalt =

[monitor://D:\Notfier2.out]

crcSalt =

This creates 2 sources in Splunk (server) and i see all logs are directed - so i think at this point it is working fine.

Now when we are getting events they are multiline and splubnk is confused how to split them, after reading i figured out that i need props.conf to learn splunk how to split each event

Sample event i am getting:

Subject: CR

%customer_name=Xys

%zrepby=xys

%group=ABC.DOC
%priority=3

%summary= Alert: 10/05/2013 16:01:17 A CRITICAL alarm has occurred on Host Abcs.Domain

%CATEGORY=Mon.Xyz.Abc

%DESCRIPTION=Alert: DEVICE HAS STOPPED RESPONDING TO POLLS - CONDITION PERSISTS FOR 10 MINUTES SYMPTOMS: Device has stopped responding to polls. PROBABLE CAUSES: 1) Device Hardware Failure. 2)

Status: CRITICAL

Customer:

Device Type: Host

Primary Engineer:

*** From UpdateScript ***

I created Props.conf on Splunk Server ( i created this MyApp to customize dashobaords and views)
D:\Splunkv5\etc\apps\MyApp\local\props.conf

[monitor://D:\Notifier.OUT]

SHOULD_LINEMERGE = false

LINE_BREAKER = ^Subject: (.*)$

I was thinking to break each event before Line "Subject: CR" starts.

Unfortunately my events are not breaking and i have no idea what i am missing.

Any help will be appreciated.

Thanks,
Nik

Re: Multiline Event

gkanapathy — Sat, 05 Oct 2013 23:40:57 GMT

Good try, but the important thing to remember is that ^ in regular expression refers to the beginning of the string. This is irrelevant and never set because you are looking at a whole stream of bytes. In other words ^ only works after lines have been broken, so you can't use it to break lines. You should use:

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)(?=Subject:)

Also, it's not clear to me why you're using crcSalt = <SOURCE>.

Re: Multiline Event

nikhilmehra79 — Mon, 28 Sep 2020 14:54:28 GMT

Hi Thanks. "Also, it's not clear to me why you're using crcSalt = ." - reason why i am using it is because we have 5 - log files which i am reading "Notifier.OUT" "NOTIFIER1.OUT" to 5...they all are having content different but when i tried installing universal installer on server where i have these log files the universal forwarder only read "Notifier.OUT" and not rest 4, when i did more reserach in log files it refered that Splunk Forwarder is not reading rest of 4 log files and will need CRC Salt beacaue apparently few lines at top of log files are same - (possible may be header of log files) and splunk do not index complete log files to see if one log file is unique vs another. Hence i did following configuration on Universal Forwarded (source server for these log files)

[monitor://D:Notfier.out]

crcSalt =

[monitor://D:Notfier2.out]

crcSalt =

Let me know if you think i am doing something stupid here....Now, going back to problem of splitting (since we have a universal forwarder and Splunk server only setup). I added your lines at props.conf at Splunk server

Here is what i have in props.conf :

[My Alert]

SHOULD_LINEMERGE = false

LINE_BREAKER = ([\n\r]+)(?=Subject:)

The path of my props.conf is under a custom application on Splunk Server (not universal forwarder) :
D:\Splunkv5\etc\apps\MyApp\local

is this the right path....do i need to put this somewhere else?

The events are still not splitting at before "Subject: CR".

Re: Multiline Event

luv — Mon, 28 Sep 2020 15:16:51 GMT

this props.conf must be changed in your indexer
and make sure you have the same sourcetype used in this stanza which you have used in your inputs.conf of the forwarder

your inputs.conf (forwarder)

[monitor://D:Notfier.out]
crcSalt =
sourcetype=abc

[monitor://D:Notfier2.out]
crcSalt =
sourcetype=abc

props.conf (indexer)

[abc]

SHOULD_LINEMERGE = false
LINE_BREAKER = ([\n\r]+)(?=Subject:)