During the SPLUNK users conference, I attended a session presented by Vishal Patel where they showed a Beta version of a "SPLUNK Deployment Monitor", it was basically a dashboard for watching your SPLUNK forwarders. I was wondering what stage this App is in, and if there's any way to get a hold of the beta version of the App.
Nice to hear this feature is going to be introduced.
We currently have Splunk version 4.1.5 deployed across 100+ windows servers and a few linux servers in production. One problem we have been having recently is that occasionally some data inputs stop showing up (stop being sent from the forwarder?). Restarting the forwarder fixes the problem, but data is lost from the time this happens. I have opened a case: https://www.splunk.com/page/issue_detail?case_id=5004000000DvUe6AAF
This might be something in configuration or something in the tool (we have noticed this on both Windows and Linux machines occasionally).
In any case, based on this experience here's what I would like (if you haven't considered this already): Some sort of heartbeat mechanism which has some predictive capabilities. i.e. say input X from forwarder Y stops for sometime, if we can get an alert or some other notification, we can use that to know (and eventually automate), the action to be taken - say restart the forwarder. If this can happen from within Splunk itself it would be great. I would prefer this over a dashboard type mechanism for a large number of machines (one more thing to monitor).
No such thing is of course required where there are no Splunk forwarders installed e.g. syslog, where we can check the syslog mechanism easily to know if something is being written to the UDP port. If forwarders dont work correctly, I would like to be able to known without running searches or after we lose events.
It will be part of 4.2. If you are in the beta program for 4.2, you can copy the app into a 4.1 instance. It will mostly work, but some views will show no data as 4.1 does not record all the stats needed for all views.