I am new to Splunk. I have few windows services in our environment. Sometime those services get hung or stopped automatically.
I wanted to use Splunk to get notification/alerts whenever service goes down or hung. If somebody can share any steps that would really appreciated.
I think that you already have Splunk Universal Forwarder on these servers, otherwise you have to install it on all.
The first step is (if you haven't yet) to download SplunkTAWindows ( https://splunkbase.splunk.com/app/742/ ).
then you have to edit inputs.conf file changing in [WinHostMon://Process] room the option disabled = 0
Then you have to deploy this modified TA to all your servers.
In this way you should receive a list of active processes on your servers with the frequency you have in [WinHostMon://Process] room (default 600 seconds).
Now you should create a lookup (called e.g. services.csv) containing all the processes of each host (two columns: host Name) that you want to monitor.
At least you have to run a search like this:
index=windows sourcetype=WinHostMon Type=Process host="BKKLT00067" Name="*"
| eval host=lower(host), Name=lower(name)
| stats count BY host Name
| append [ | inputlookup services.csv | eval host=lower(host), Name=lower(name), count=0 | fields host Name count ]
| stats sum(count) As total BY host Name
| where total=0
In this way you'll have the list of all services for each host never found.
Similarly, i have another requirement.where I have pass log directory to forwarder. Now i want to read the logs and generate the alerts when log file contains "file(s) count is 2" or greater than 1. (condition is : File(s) count is greater than 1)