Well , I want to create an alert which alert me whenever there is spike in Errors. Currently we are comparing say past 30m count with last 2 week same time same date and comparing with 2w average. But I want to create a near real time alert as it can be false positive this way.
My errors are like some are trending some come only at time of issues and some are like more during peak business hours and less during off business hours but I want to capture the real spikes like avoiding it to trigger when we move from non business to business hours. I was hoping if I can use predict command to do that but not clear with all algos and if that is right thing to use here.
index=rxc sourcetype="rxcapp" (level=ERROR) earliest=-30m@m latest=@m|rex "Id:\s*(?<Id>\d+)," | search [| inputlookup abc.csv | rename id as Id | fields Id]| lookup abc.csv id As Id OUTPUT site|bucket _time span=5m| stats count by _time error_msg site| predict lower95=lower upper95=upper algorithm=LLP5 count as predict| where count>'upper(predict)'|stats latest(count) by error_msg site
will this be helpful or this is wrong ? Can predict be used this way with stats command ?or any other suggestion on approach.
I'm skeptical that predict would be the right way to do that.
It seems like the right thing to do would be, each night off peak, to calculate the next day's boundaries once for each 5, 10 or 15 minute increment, and output those times and limits to a lookup table.
Then, you'd just have to calculate the current errors and read the lookup table to get the limits for whatever _time and site you are running and test the compliance.