How do you set up an alert for when an application process is not running or hung?

[bsaujla131984]

There are a number of application processes in our environment which either go down or stop responding. I am trying to setup an alert in the event if process is down or hung in Unix/Linux.

Can anyone assist with this please?

[ddrillic]

It's interesting in the context of Splunk processes which can be hung, forwarders or indexers...

[bsaujla131984]

This is not related to Splunk process. It is related to application process in Linux/Unix servers.

[mstjohn_splunk]

Hi @bsaujla131984 - Did @inventsekar 's answer below help provide a solution to your question? If yes, please click “Accept” below the best answer to resolve this post and upvote anything that was helpful. If no, please leave a comment with more feedback. Thanks.

[bsaujla131984]

Hi @mstjohn... I would need more info. The answer is given is mainly related to commands, however I would like to do it through Splunk...

[mstjohn_splunk]

Gotcha, thanks bud! Good luck with your issue.

[inventsekar]

from a linux forum... "ps auxww |grep 'process_ID' " you can have current process' status like S (Sleep), R (Running) , Z (zombie) etc..

[root@server]# cat /proc/2324/status
Name: kmpathd/1
State: S (sleeping)
SleepAVG: 98%
Tgid: 2324
Pid: 2324
PPid: 13
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 32

Also, by using top command, you can see if the running process is taking high cpu(probably, hung process)

then you can create an alert accordingly..

[bsaujla131984]

Actually I want something which can be done entirely through Splunk instead of logging to UNIX/Linux serves..

[jkat54]

Do you have a way of knowing if the process is hung?

ie: curl to see if port is open, or run service $name status