Splunk Search

Finding an Error in the middle of a Transaction

tyronetv
Communicator

I have a unique ID (RID) for a each client click. A single click can execute between 3 and, sometimes, over 100 log entries. Sometimes, in the middle of all of that is a situation where an error occurs.

I currently have a search that generates a list of actions by userid/RID but the report creates every line as an "INFO" response (I guess because the first and last line of the transaction are always INFO log level). What I need to do is set the corresponding report to acknowledge the transaction as an ERROR level based upon the ERROR existing someone in the middle of the transaction.

How would I do that?

2014-02-04 05:58:14,473 INFO [ndler-HTTPThreadGroup-289] RID=1391522013732-458538

2014-02-04 05:58:14,462 ERROR [ndler-HTTPThreadGroup-289] RID=1391522013732-458538

2014-02-04 05:53:33,776 DEBUG [ndler-HTTPThreadGroup-289] RID=1391522013732-458538

2014-02-04 05:53:33,760 DEBUG [ndler-HTTPThreadGroup-289] RID=1391522013732-458538

2014-02-04 05:53:33,760 INFO [ndler-HTTPThreadGroup-289] RID=1391522013732-458538

The above is for a client search request. The request timed out after 5 minutes. When I report on this transaction as a summary of actions taken by the client I currently report on the action and the duration. I would like to include the "Error_Level" as a column with a drill down to the actual transaction(RID) experiencing the error. But first, I just want to be able to identify it as an 'error' transaction.

Tags (2)
0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

I am assuming the log_level is the field containing INFO/ERROR/DEBUG, try this.

..your base search..| ..your transaction command on RID..| eval failFlag=mvfind(log_level,"ERROR") | eval HasError=if(isnull(failFlag),"No","Yes") |...your other reporting command including HasError field..

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

I am assuming the log_level is the field containing INFO/ERROR/DEBUG, try this.

..your base search..| ..your transaction command on RID..| eval failFlag=mvfind(log_level,"ERROR") | eval HasError=if(isnull(failFlag),"No","Yes") |...your other reporting command including HasError field..

kristian_kolb
Ultra Champion

Maybe not exactly what you're asking for, but a general optimization tip, especially if there are a large amount of transactions, and only a few of them has errors.

Instead of creating transactions and then looking for possible ERRORs, it might be better to do it the other way round:

index=blah sourcetype=bleh [index=blah sourcetype=bleh ERROR | dedup RID | fields + RID] | transaction RID

The subsearch will find the RIDs where there are ERRORs, and the outer search will build transactions based on them.

0 Karma

tyronetv
Communicator

Yeah. I understand this as a solution but more often then not (and shame on me training them as such) the users are used to seeing a sequential listing of the user's activity and I've been asked to mark the one that Error'd as part of the report. So, I find the user, then their RID's, and then generate a report. I need the report to show the ERROR on the line item of the users activity.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...