Splunk Search

How to write a search to alert if our Tomcat server is down?

splunker9999
Path Finder

Hi,

We need to create an alert to check if tomcat is up and running. This we could identify using pid.

If tomcat is up and running, then we would receive tomcat events (logs) with pid.

If tomcat is down, we wont receive any event.

Can you please help us with the search for this?

Base query below, but this does not work because whenever Tomcat is down, we won't get any events. We may need to use something like fillnull?

index=idne1 source=ps host=host1 OR host=host2* apache-tomcat
|stats count(pid) as Count by host _time
|where count <1
0 Karma

sundareshr
Legend

Try this

index=idne1 source=ps host=host1 OR host=host2* apache-tomcat pid="*" earliest=-10m | stats latest(_time) as latest_event | where latest_event<relative_time(now(), "-5m@m")
0 Karma

lycollicott
Motivator

If you want to do this sort of thing in Splunk, you might be better off trying something like this:

| tstats latest(_time) as lastTime by host 
| search host=host1 OR host=host2*
| eval age=now()-lastTime 
| search age > 120
| sort age d 
| convert ctime(lastTime) 
| fields age,host,lastTime 
| sort host

This will tell you if those hosts have sent no events to any index within the past 120 seconds.

woodcock
Esteemed Legend

If I assume that every running Tomcat host will have at least 1 log every 5 minutes and that you need to be alerted within an hour if anything goes down, then try this running the every hour over last 24 hours:

index=idne1 source=ps host=host1 OR host=host2* apache-tomcat pid="*"
| dedup host
| search _time > (5*60)
0 Karma

splunker9999
Path Finder

We are actually taking these from Source =ps, where we ger 1 log for every 30 secs

0 Karma

woodcock
Esteemed Legend

So just change it to fit.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...