Splunk Search
Highlighted

How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

New Member

I have a lookup csv file which contains for each error code:

  • interval
  • threshold
  • some additional informational fields

I would like to have a real-time search with a rolling window of the past day e.g.

  • Start time = rt-24h
  • Finish time = rt

That search will display as a list:

  • error that has exceeded its limits (interval and threshold)
  • time that this has happened
  • some of the additional informational fields
  • descending order based on the time this has happened

That list should not display distinct errors.

Every time an error exceeds its limits, it must be included in the result.

I have configured properly the lookup source and I am able to use it in searches.

I have trouble in creating the syntax of the search that it will produce the above result as I am new in using Splunk.

Any help would be highly appreciated.

Regards

Tags (2)
0 Karma
Highlighted

Re: How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

Splunk Employee
Splunk Employee

Having a hard time understanding the outcome you are looking for. Is this a crude sample of your lookup table? do you want the search to use the interval as a earliest= or latest= only if the threshold is surpassed?

error_code,interval,threshold,someaddtionalfiled
err404,20,5,bar
err202,10,6,foo

0 Karma
Highlighted

Re: How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

New Member

Yes, sample lookup table is correct.
Apologies but I didn't understand about Search using interval as earliest or latest, can you please clarify?
I want to build a functionality for a support engineer that he will be able to see the errors as they are happening and limits are exceeding.

0 Karma
Highlighted

Re: How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

Splunk Employee
Splunk Employee

error_code,intervalValue,thresholdValue,someaddtionalfield
err404,20,5,bar
err202,10,6,foo

error that has exceeded its limits (interval and threshold)

sourcetype=error_codes earliest=rt-24h latest=rt err404 |where interval>intervalValue AND threshold>threshholdValue |table _time someadditionalfield

IntervalValue and thresholdValue exist in the lookup table

0 Karma
Highlighted

Re: How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

New Member

Thank you very much for the answer but I believe this is not going to work as the log files have different errors where for some of the errors (as not every error should be monitored) each one of them has different interval and threshold in the lookup table. I know that it is confusing as when the starting point is always rolling but realistically I am actually interested in the snapshot of the events from rt and backwards. Not really interested if the snapshot close to rt-24h (e.g. -23h55m) "is not accurate" compared to what I was monitoring 2m before

0 Karma
Highlighted

Re: How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

New Member

Also, please keep in mind that the interval will realistically make sure that the results are "normalised" as they age out and most of the time constant results (but most of all, meaningful to assist the support engineer) will keep being displayed as the time window rolls.

0 Karma
Highlighted

Re: How to create a real-time search with rolling window for errors in log files, based on interval and threshold defined per error in a lookup source

Splunk Employee
Splunk Employee

You can specify time modifiers in-line with earliest= and latest= . like I did below.

0 Karma