in my clustered environment i have a search head which is dedicated to a special user group and should serve a very simple dashboard view for system status overview (just some icons like "Indexers OK" " Search Heads Not OK" 🙂 ). It is connected to the indexer cluster which has all the internal data from every connected Splunk environment. The MC (Monitoring Console) is not configured on it.
i assume all the necessary data is in the _* indexes like _internal. How do i identify the "system role" within this data?
I can identify forwarders with "index=_internal source=*metrics.log group=tcpin_connections..." but how about other "roles"?
Any ideas are very welcome.
Check out the "Inherit a Splunk Enterprise Deployment" manual (http://docs.splunk.com/Documentation/Splunk/6.6.0/InheritedDeployment/Introduction).
take the hosts from the search above and create a search:
index= _internal host =1 or host=2 ... or host=n | timechart span=5m count by host
this will tell you how many events are indexed every 5 minutes by your splunk instances.
save as an alert if count = 0