topic How to create a Splunk alert to trigger when Tomcat is down? in Splunk Search

How to create a Splunk alert to trigger when Tomcat is down?

splunker9999 — Tue, 19 Jul 2016 22:32:22 GMT

Hi,

We have scenario to create an alert for tomcat to trigger an alert when tomcat is down.

Based on our tomcat logs, it gives PID for every 30secs when ever it is up.
If it is down it wont trigger any event for that 30 sec interval.

We need to set up an alert to trigger by host ,if any of the host has not have an entry for 30secs period.

Below is basic search for it:

 index=index1  source=ps host=host1* OR host=host2* apache-tomcat|table host pid _time

Re: How to create a Splunk alert to trigger when Tomcat is down?

twinspop — Tue, 19 Jul 2016 22:39:27 GMT

Find hosts down for 5 minutes:

index=index1  source=ps host=host1* OR host=host2* apache-tomcat | 
stats latest(_time) as latest by host | 
eval age=now()-latest | where age>300

Re: How to create a Splunk alert to trigger when Tomcat is down?

splunker9999 — Tue, 19 Jul 2016 22:50:34 GMT

Hi, This would works when host is down.

Here we are checking for pid in the events: We will get events for every 30secs when ever Tomcat is up and running.

When ever server is down, it wont trigger any events to splunk
Ex:
If Tomcat is stopped around 3.47.30.000, we wont get any events from 3.47.30.000 to until it is up.

So we need to customise our search , in such away that if search find no events from any particular host ..we need to give status as down. By default when ever search returns events which have pid, then status should be up.

If we have search query for this , we can give condition like|where status=down

Thanks

Re: How to create a Splunk alert to trigger when Tomcat is down?

twinspop — Wed, 20 Jul 2016 00:48:04 GMT

Change the stats command to include pid: ... as latest by host pid ?