Splunk Search

Use of predict command for alerting

ksharma7
Path Finder

Well , I want to create an alert which alert me whenever there is spike in Errors. Currently we are comparing say past 30m count with last 2 week same time same date and comparing with 2w average. But I want to create a near real time alert as it can be false positive this way. 

My errors are like some are trending some come only at time of issues and some are like more during peak business hours and less during off business hours but I want to capture the real spikes like avoiding it to trigger when we move from non business to business hours. I was hoping if I can use predict command to do that but not clear with all algos and if that is right thing to use here.

 

 

 

index=rxc sourcetype="rxcapp" (level=ERROR) earliest=-30m@m latest=@m|rex "Id:\s*(?<Id>\d+)," | search  [| inputlookup abc.csv | rename id as Id | fields Id]| lookup abc.csv id As Id OUTPUT site|bucket _time span=5m| stats count by _time error_msg site| predict lower95=lower upper95=upper algorithm=LLP5 count as predict| where count>'upper(predict)'|stats latest(count) by error_msg site

 

 

 

will this be helpful or this is wrong ? Can predict be used this way with stats command ?or any other suggestion on approach. 

Labels (1)
Tags (1)
0 Karma

DalJeanis
SplunkTrust
SplunkTrust

I'm skeptical that predict would be the right way to do that. 

It seems like the right thing to do would be, each night off peak, to calculate the next day's boundaries once  for each 5, 10 or 15 minute increment, and output those times and limits to a lookup table.

Then, you'd just have to calculate the current errors and read the lookup table to get the limits for whatever _time and site you are running and test the compliance. 

 

0 Karma

ksharma7
Path Finder

Yeah , then I think it is good the way I am using it currently like comparing with two week average count

Tags (1)
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!