https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271869#M52228 <P>Hi athorat,</P> <P>forward the <CODE>_internal</CODE> logs of the forwarder to the indexer and search like this:</P> <PRE><CODE>index=_internal component=ShutdownHandler </CODE></PRE> <P>This will list all <CODE>shutdown</CODE> events.</P> <P>Basics about <CODE>_internal</CODE> forwarding can be found here <A href="http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata">http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata</A> <BR /> Yes, you can do the same on your forwarder.</P> <P>Hope this helps ...</P> <P>cheers, MuS</P> Tue, 13 Oct 2015 23:56:41 GMT MuS 2015-10-13T23:56:41Z https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271868#M52227 <P>We have a report which helps us to trigger an alert when the Indexer is down.<BR /> Is there a way we can monitor if the forwarder is stopped on the server which can send an email alert?</P> Tue, 13 Oct 2015 23:32:58 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271868#M52227 athorat 2015-10-13T23:32:58Z https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271869#M52228 <P>Hi athorat,</P> <P>forward the <CODE>_internal</CODE> logs of the forwarder to the indexer and search like this:</P> <PRE><CODE>index=_internal component=ShutdownHandler </CODE></PRE> <P>This will list all <CODE>shutdown</CODE> events.</P> <P>Basics about <CODE>_internal</CODE> forwarding can be found here <A href="http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata">http://docs.splunk.com/Documentation/Splunk/6.3.0/DistSearch/Forwardsearchheaddata</A> <BR /> Yes, you can do the same on your forwarder.</P> <P>Hope this helps ...</P> <P>cheers, MuS</P> Tue, 13 Oct 2015 23:56:41 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271869#M52228 MuS 2015-10-13T23:56:41Z https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271870#M52229 <P>Forwarders can be not forwarding for many reasons other than shutdown including a crash, which would not have a shutdown event or a network problem. Try this:</P> <PRE><CODE>| metadata index=* type=hosts | eval latencySeconds=(recentTime-lastTime) | eval quietSeconds=(now()-recentTime) | fieldformat firstTime=strftime(firstTime, "%m/%d/%Y %H:%M:%S") | fieldformat lastTime=strftime(lastTime, "%m/%d/%Y %H:%M:%S") | eval indexTime=strftime(recentTime, "%m/%d/%Y %H:%M:%S") </CODE></PRE> <P>The field <CODE>quietSeconds</CODE> tells you how long it has been since that forwarder sent any data to any indexer.</P> Wed, 14 Oct 2015 13:23:48 GMT https://community.splunk.com/t5/Getting-Data-In/How-to-create-an-alert-to-trigger-an-email-when-a-forwarder-is/m-p/271870#M52229 woodcock 2015-10-14T13:23:48Z