topic Re: loglevel detection incorrect - How do you re-align/transform it? in Getting Data In

loglevel detection incorrect - How do you re-align/transform it?

Splunkdoobiest — Wed, 12 Mar 2014 13:37:54 GMT

Hi,

I'm a relative newbie at this stuff so please bear with me if I am asking a stupid question.
I have an index that has inputs from two logfiles in different formats:

Logfile 1:

00:00:01|14|Debug|<Message Text>|

Logfile2:

2014-03-11 00:00:00,085 [alert1] INFO <Message Text>

loglevel is generally assigned correctly for Logfile2, but never for Logfile1, except when INFO, DEBUG, etc are contained in the message text (ie: not the actual log level as listed after the second pipe in the given example above.)

Basically I would like to assign the correct Loglevel to messages from both sourcetypes such that should I query the index for a report against total loglevel messages for a given period, for example, I would actually end up with accurate results.

I'm entirely certain my confusion is simple lack of experience, as I can happily generate simple queries and reports, but trying to align the data as per my requirement above is totally defeating me.

Any suggestions would be greatly appreciated.

Re: loglevel detection incorrect - How do you re-align/transform it?

bshuler_splunk — Wed, 12 Mar 2014 14:01:32 GMT

I suspect you have both logs set as the same sourcetype, and so your field extractions are colliding. Here is a workaround for that.

You need to create 2 field extractions. Name the first loglevel, and the second loglevel2. The manage your field extractions, and change the ?P to ?P

This will let you define multiple extractions with the same name, and allow you to support both logs.

http://d.pr/i/KOME

Re: loglevel detection incorrect - How do you re-align/transform it?

Splunkdoobiest — Wed, 12 Mar 2014 14:44:47 GMT

Thanks bshuler_splunk!

That certainly gets all the correct log-levels listed against the index.
I defined the extractions as follows:
Logfile1: (?i)^[^|]|\d+(?P|\w+|)
Logfile2: (?i)^[^]]]\s+(?P[^ ]+)

Unfortunately I now end up with a slightly stranger mish-mash, owing to my extractions. I now get loglevel duplicates like this:
|Debug|
DEBUG
|Warn|
WARN

I just need to find a way to tell splunk that "|Warn| = WARN" etc...

Re: loglevel detection incorrect - How do you re-align/transform it?

bshuler_splunk — Wed, 12 Mar 2014 15:52:44 GMT

Tags: http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Abouttagsandaliases

Re: loglevel detection incorrect - How do you re-align/transform it?

Splunkdoobiest — Wed, 12 Mar 2014 15:55:59 GMT

Many thanks bshuler_splunk 🙂
That's exactly what I'm looking for (and a classic case of RTFM!)
Thanks for your patience and assistance - Much appreciated!