How to monitor Server hung state on both Windows and Linux using Splunk?

[ansif]

Can anyone help me to know the possibility of monitoring server hung state using Splunk?

[woodcock · Accepted solution]

This app was just posted yesterday and probably does everything you need (and if not the author will surely be very responsive):
Unified Forwarder Monitoring App for Splunk: https://splunkbase.splunk.com/app/3805/

[ansif]

This is working and it has the module what @niketnilay suggetsed.

Thanks @woodcock and @niketnilay

[niketn]

@ansif, we had an issue on our Windows servers which used to hang during scheduled reboot and would not shut down. While servers used to respond to ping request, almost all services including the Remote Login used to stop functioning. Obviously Splunk's splunkd service also used to stop. Which implied that Splunk Universal Forwarder on such Windows machines (Splunk Deployment Client) would stop pinging Deployment Server. We could do either one of the following to check for such events:

1) Check the Deployment Console in DMC to identify the deployment clients which have not pinged recently.

2) Use REST API to get the deployment clients which have not pinged recently (following example if for last 5 min):

| rest /services/deployment/server/clients
| table name lastPhoneHomeTime
| where lastPhoneHomeTime>300

PS: DMC also uses the same REST API: https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTdeploy#deployment.2Fserver.2Fclients

If you are already indexing some stats from these Windows and/or Linux machines(even if Splunk's internal logs from these hosts) there may be several other options as well. Refer to the following Answer thread: https://answers.splunk.com/answers/592278/query-for-splunkd-status.html#answer-593319

[ansif]

@niketnilay : Could you please help me with a solution on below:

Our deployment server is at customer site,is there any option to create an inputs.conf definition?

[woodcock]

Why do you post so many answers as comments, @niketnilay?

[niketn]

@woodcock, 😄 for several reasons but I mostly try to keep questions as unanswered so that others can assist with their inputs as well. Other cases when I require further clarification or am not sure of the solution I provide then also I post as a comment to hear back. Keeping as unanswered leaves the question open for others in case the poster responds back and I miss.

This is complete different case though 🙂 Ansif and I work at same company, so I can follow up with him at work 🙂