topic Re: Dynamically updated alert search in Alerting

Dynamically updated alert search

hartfoml — Tue, 29 Nov 2011 21:56:45 GMT

I am running a port monitoring app using NMAP that record system availability

If this stops it makes my system availability stats wrong

I would like to create and alert that will tell me if this is not working.

Not too hard but I would like the alert to be dynamic and adjust for the number of systems I am monitoring.

Right now I have 60 systems monitored every minute so ( 60*60 ) or 3600 events per hour

So I can set an alert for less than 3600 event but if I add a system, the alert should go up or ( 61*60 ) trigger = 3660 per hour

How do I write the alert to have dynamic triggers?

Re: Dynamically updated alert search

kristian_kolb — Wed, 30 Nov 2011 09:46:49 GMT

If your nmap application writes one event per monitored port per minute, and you'd like to find out if the port monitoring app itself has crashed, you could simply have a scheduled search that will detect if the logs stop coming in completely.

<your source/sourcetype> earliest=-1m

with the alert triggering set on the condition "number of events = 0"

Have I understood your problem correctly? Is this a viable solution?

/kristian

Re: Dynamically updated alert search

hartfoml — Wed, 30 Nov 2011 17:34:36 GMT

Thanks for answering I really appreciate your participation and trying to help. I can't monitor the log inputs because some of the clients are quit for longer than a minute and I wanted to get a one minute interval to monitor availability.

Re: Dynamically updated alert search

kristian_kolb — Wed, 30 Nov 2011 18:43:28 GMT

Sorry, I thought you wanted to monitor availability in splunk, and also monitor the status of the monitoring application itself - and that it was the latter problem you wanted help with. I still don't know really what you want to perform.

Re: Dynamically updated alert search

hartfoml — Fri, 02 Dec 2011 18:57:29 GMT

I'm sorry I was looking for one kind of answer and you gave me the right one. It was not what I expected so I thought it must be wrong. I was looking for a way to alert on the input not the monitoring. I have setup an alert like you have suggested above and this will do what I wanted. Thanks for the help I really appreciate it.

Re: Dynamically updated alert search

kristian_kolb — Fri, 02 Dec 2011 20:27:12 GMT

You are most welcome