topic Re: wanted to use Splunk to get notification/alerts whenever service goes down or hung in Alerting

wanted to use Splunk to get notification/alerts whenever service goes down or hung

sachindarade — Wed, 19 Feb 2020 15:25:41 GMT

Hi All,

I am new to Splunk. I have few windows services in our environment. Sometime those services get hung or stopped automatically.
I wanted to use Splunk to get notification/alerts whenever service goes down or hung. If somebody can share any steps that would really appreciated.

Thanks in Advance!

Re: wanted to use Splunk to get notification/alerts whenever service goes down or hung

gcusello — Wed, 30 Sep 2020 04:14:52 GMT

Hi @sachindarade,
I think that you already have Splunk Universal Forwarder on these servers, otherwise you have to install it on all.

The first step is (if you haven't yet) to download Splunk_TA_Windows ( https://splunkbase.splunk.com/app/742/ ).
then you have to edit inputs.conf file changing in [WinHostMon://Process] room the option disabled = 0
Then you have to deploy this modified TA to all your servers.
In this way you should receive a list of active processes on your servers with the frequency you have in [WinHostMon://Process] room (default 600 seconds).

Now you should create a lookup (called e.g. services.csv) containing all the processes of each host (two columns: host Name) that you want to monitor.

At least you have to run a search like this:

index=windows sourcetype=WinHostMon Type=Process host="BKKLT00067" Name="*"
| eval host=lower(host), Name=lower(name)
| stats count BY host Name
| append [ | inputlookup services.csv | eval host=lower(host), Name=lower(name), count=0 | fields host Name count ]
| stats sum(count) As total BY host Name
| where total=0

In this way you'll have the list of all services for each host never found.

Ciao.
Giuseppe

Re: wanted to use Splunk to get notification/alerts whenever service goes down or hung

sachindarade — Mon, 02 Mar 2020 14:07:28 GMT

Hi,

Similarly, i have another requirement.where I have pass log directory to forwarder. Now i want to read the logs and generate the alerts when log file contains "file(s) count is 2" or greater than 1. (condition is : File(s) count is greater than 1)

your help would be really appreciated.

Thanks in Advance.