I hope that your experience with Splunk can help me.
Actually, I'm working on making an alert. The case is putting an alert on the "server message rate". And the goal is to put an alert on the moving thresholds. I did a prediction (LLP5) and generated the upper and lower bound, then i put an alert on it.
There are few points to be mentioned: 1- we have data just from 2nd of Dec 2020 (less than 2 months), 2- we saw a behaviour that alerts of this week are not anymore alerts when we run the query the next week.
Some questions:
1- Can LLP5/LLP be the best possible MLTK algorithm to try when we wanna make alerts? I treid StateSpace Forecast and DecisionTreeRegressor as well, but in these algorithms, we should create lower and upper bounds ourselves.
2- Can we create alerts on the base of comparing week by week data? Prediction week by week and then comparing for example Mondays, Tuesdays, etc. with each other. Normally I know that it's possible, but in prediction and alerting I don't know.
3- Is it normal that an alert that we saw last week, this week gets disappeared? Does it mean, that data learns from the historical data, then an alert of last week can get disappeared this week?
4- In MLTK, In the time picker, when i choose 30 days in relative time, the result is different from the preset 30 dayst, why?? It's weird for me.
5- Finally, less than 2 months data, is it enough?
The simplified query is as follows:
| tstats sum(msg) as msg where `sws_logs_indexes` sourcetype=sws* host="p*" mp_name="xxx" sag_instance="*xxx" by _time sag_instance span=10m | streamstats current=false latest(msg) as previous_msg by sag_instance | eval rate=msg-previous_msg | timechart span=10m avg(rate) as "Server msg rate"
