Alert Set up

smuderasi — Tue, 14 Jul 2020 03:45:08 GMT

I want to set up an alert when the server status is not "HEALTH_OK" for three consecutive times in a row.

Any pointers on how to schedule such search. I receive data every two minutes.

Capture.PNG

Send Alert If	Comments/Alert Frequency
Status is not "RUNNING" or State is not "HEALTH_OK" for 3 times consecutively	1 alert every 2 min, for first 10 min, then 1 alert every 30 min

Re: Alert Set up

rnowitzki — Tue, 14 Jul 2020 07:55:02 GMT

Hi @smuderasi ,

This is not the finished meal, but should give you an idea of how you can do it:

| noop | makeresults | eval id = "1 2 3 4 5 6 7 8 9" | makemv id | mvexpand id | eval server = case(id=1,"hostA",id=2,"hostB",id=3,"hostC",id=4,"hostA",id=5,"hostC",id=6,"hostB",id=7,"hostB",id=8,"hostA",id=9,"hostC") | eval _time = case(id=1,_time+10,id=2,_time+20,id=3,_time+40,id=4,_time+40,id=5,_time+50,id=6,_time+60,id=7,_time+70,id=8,_time+80,id=9,_time+90) | eval state = case(id=1,"health_ok",id=2,"health_not_ok",id=3,"health_ok",id=4,"health_ok",id=5,"health_ok",id=6,"health_not_ok",id=7,"health_not_ok",id=8,"health_not_ok",id=9,"health_not_ok") | fields - id | sort server, _time | streamstats count as alert_counter by server, state

The first block, until including the | fields - id is just to make up some sample data - you don't need that as you have data 🙂

The streamstats will create the field alert_counter which adds 1 every time the server and state is the same as in the line above. So your alert could trigger when alert_counter>=3 AND state!=health_ok
You can also add your status field like that.

I don't have a good idea right now for your alert conditions (every 2 minutes, later every 30 minutes), might be possible with 2 different alerts somehow.

Hope I could at least give you some inspiration.

topic Alert Set up in Alerting

Alert Set up

Re: Alert Set up