topic Re: How to sort different error strings in one log file? in Splunk Search

How to sort different error strings in one log file?

Bergans — Wed, 20 Aug 2014 07:07:14 GMT

Hi,
I'm currently importing log-files into Splunk, to monitor the different kind of Errors that passes through the system that are monitored.
Up to now I've only searched for the string 'ERROR' in each log file. Since a log file may contain many different kind of errors, the result is that many kind of Errors are presented together. I would like to sort/group the different kind of errors in one diagram.

The search as today is as follows (for one of the log-files)

ERROR source="/home/logs/DataTransferService.log"

The result would then consist of many different ERROR messages, similar to these three (as an example):

2014-08-19 12:00:00,394 [pool-1-thread-1] ERROR - Unexpected error com.eMeter.PIPe.datatransferservice.exception.DTSRuntimeException:…

2014-08-19 11:20:01,815 [pool-1-thread-4] ERROR - Invalid date. java.lang.NullPointerException:…

2014-08-19 11:20:01,814 [pool-1-thread-4] ERROR - SDP lookup failed [null]: id to load is required for loading [Additional Information:…

Re: How to sort different error strings in one log file?

strive — Wed, 20 Aug 2014 08:41:38 GMT

Can we consider the following as error messages
Unexpected error com.eMeter.PIPe.datatransferservice.exception.DTSRuntimeException
Invalid date. java.lang.NullPointerException
SDP lookup failed [null]

Means, all error messages start after - and end before :

Re: How to sort different error strings in one log file?

kristian_kolb — Wed, 20 Aug 2014 08:48:45 GMT

Well, it's not entirely clear what you want to achieve. Sorting and grouping is usually performed on fields. Say that you want to group all errors on the type of error ("Unexpected error", "Invalid date" or "SDP lookup failed" in your example), you need to extract this part of the message as a field (let's call it errType) and whatever comes after it is extracted as errMsg. This can be done in config files or inline in the search query. We'll do the latter here.

ERROR source="/home/logs/DataTransferService.log" 
| rex "\sERROR\s-\s(?<errType>[^\.\r\n:]+)(?<errMsg>.*)"
| eval niceTime = strftime(_time, "%F %T")
| stats list(niceTime) list(errMsg) by errType

This should result in an output like

Unexpected Error        2014-06-20 11:22:23     com.eMeter.PIPe.datatransferblahblahblahb
                        2014-06-21 12:23:34     some.other.errormessage
Invalid date            2014-06-23 22:32:21     Blah Blah Blah error

and so forth. Maybe something like that is what you're looking for?

Re: How to sort different error strings in one log file?

Bergans — Wed, 20 Aug 2014 09:52:54 GMT

No, the Errors may be identified by 'ERROR - '*

Re: How to sort different error strings in one log file?

Bergans — Wed, 20 Aug 2014 10:33:48 GMT

Thanks a lot @kristian.kolb,
I see that I was a bit unclear about what really wanted to achieve. I changed your last part:

| stats list(niceTime) list(errMsg) by errType"

with:

| timechart span=1d count by errType usenull=f

Then I got exactly what I needed.
Now I got a sorted list of the different kind of errors, and a visualized graphical view.

Re: How to sort different error strings in one log file?

kristian_kolb — Wed, 20 Aug 2014 12:10:50 GMT

btw, if you're NOT doing the stats list(niceTime) part, you can skip the preceding eval niceTime strftime(_time, "%F %T") as well.

EDIT : missing "NOT"
/k