topic Use of predict command for alerting in Splunk Search

Use of predict command for alerting

ksharma7 — Tue, 23 Jun 2020 09:01:04 GMT

Well , I want to create an alert which alert me whenever there is spike in Errors. Currently we are comparing say past 30m count with last 2 week same time same date and comparing with 2w average. But I want to create a near real time alert as it can be false positive this way.

My errors are like some are trending some come only at time of issues and some are like more during peak business hours and less during off business hours but I want to capture the real spikes like avoiding it to trigger when we move from non business to business hours. I was hoping if I can use predict command to do that but not clear with all algos and if that is right thing to use here.

will this be helpful or this is wrong ? Can predict be used this way with stats command ?or any other suggestion on approach.

Re: Use of predict command for alerting

DalJeanis — Tue, 23 Jun 2020 19:07:59 GMT

I'm skeptical that predict would be the right way to do that.

It seems like the right thing to do would be, each night off peak, to calculate the next day's boundaries once for each 5, 10 or 15 minute increment, and output those times and limits to a lookup table.

Then, you'd just have to calculate the current errors and read the lookup table to get the limits for whatever _time and site you are running and test the compliance.

Re: Use of predict command for alerting

ksharma7 — Wed, 24 Jun 2020 05:05:41 GMT

Yeah , then I think it is good the way I am using it currently like comparing with two week average count