How to create an alert to trigger an email when a forwarder is stopped on a server?

[athorat]

We have a report which helps us to trigger an alert when the Indexer is down.
Is there a way we can monitor if the forwarder is stopped on the server which can send an email alert?

[woodcock]

Forwarders can be not forwarding for many reasons other than shutdown including a crash, which would not have a shutdown event or a network problem. Try this:

| metadata index=* type=hosts | eval latencySeconds=(recentTime-lastTime) | eval quietSeconds=(now()-recentTime) | fieldformat firstTime=strftime(firstTime, "%m/%d/%Y %H:%M:%S") | fieldformat lastTime=strftime(lastTime, "%m/%d/%Y %H:%M:%S") | eval indexTime=strftime(recentTime, "%m/%d/%Y %H:%M:%S")

The field quietSeconds tells you how long it has been since that forwarder sent any data to any indexer.

[MuS]

Hi athorat,

forward the _internal logs of the forwarder to the indexer and search like this:

index=_internal component=ShutdownHandler

This will list all shutdown events.

Basics about _internal forwarding can be found here http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata
Yes, you can do the same on your forwarder.

Hope this helps ...

cheers, MuS