Some updates,linux aside,I just did an upgrade for one of our old splunk forwarder(to universal forwarder 4.2.4) on windows server.I've noticed the following:
1) By searching the _internal index method,if the forwarder is shutdown,the indexer will not receive the shutdown event until the forwarder has restarted.(This way we probably will not be alerted when its down?)
2)By monitoring the windows system event log for forwarder shutdown events,when the service is shut down an event will be logged to windows event but the forwarder will not send this event to indexer.Even if the forwarder service has been restarted,the duration when the forwarder is down will not be captured at indexer.(I'm not sure why but seems that the older version of splunk able to do so)
... View more