We have a report which helps us to trigger an alert when the Indexer is down.
Is there a way we can monitor if the forwarder is stopped on the server which can send an email alert?
Forwarders can be not forwarding for many reasons other than shutdown including a crash, which would not have a shutdown event or a network problem. Try this:
| metadata index=* type=hosts | eval latencySeconds=(recentTime-lastTime) | eval quietSeconds=(now()-recentTime) | fieldformat firstTime=strftime(firstTime, "%m/%d/%Y %H:%M:%S") | fieldformat lastTime=strftime(lastTime, "%m/%d/%Y %H:%M:%S") | eval indexTime=strftime(recentTime, "%m/%d/%Y %H:%M:%S")
The field quietSeconds tells you how long it has been since that forwarder sent any data to any indexer.
forward the _internal logs of the forwarder to the indexer and search like this:
This will list all shutdown events.
Basics about _internal forwarding can be found here http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata
Yes, you can do the same on your forwarder.
Hope this helps ...