Dynamically updated alert search

[hartfoml]

I am running a port monitoring app using NMAP that record system availability

If this stops it makes my system availability stats wrong

I would like to create and alert that will tell me if this is not working.

Not too hard but I would like the alert to be dynamic and adjust for the number of systems I am monitoring.

Right now I have 60 systems monitored every minute so ( 60*60 ) or 3600 events per hour

So I can set an alert for less than 3600 event but if I add a system, the alert should go up or ( 61*60 ) trigger = 3660 per hour

How do I write the alert to have dynamic triggers?

[kristian_kolb · Accepted solution]

If your nmap application writes one event per monitored port per minute, and you'd like to find out if the port monitoring app itself has crashed, you could simply have a scheduled search that will detect if the logs stop coming in completely.

<your source/sourcetype> earliest=-1m 

with the alert triggering set on the condition "number of events = 0"

Have I understood your problem correctly? Is this a viable solution?

/kristian

[kristian_kolb]

You are most welcome

/k

[hartfoml]

I'm sorry I was looking for one kind of answer and you gave me the right one. It was not what I expected so I thought it must be wrong. I was looking for a way to alert on the input not the monitoring. I have setup an alert like you have suggested above and this will do what I wanted. Thanks for the help I really appreciate it.

[kristian_kolb]

Sorry, I thought you wanted to monitor availability in splunk, and also monitor the status of the monitoring application itself - and that it was the latter problem you wanted help with. I still don't know really what you want to perform.

/k

[hartfoml]

Thanks for answering I really appreciate your participation and trying to help. I can't monitor the log inputs because some of the clients are quit for longer than a minute and I wanted to get a one minute interval to monitor availability.