How to identify Splunk Instance role by internal logs?

[swe]

Hi there,

prologue:
in my clustered environment i have a search head which is dedicated to a special user group and should serve a very simple dashboard view for system status overview (just some icons like "Indexers OK" " Search Heads Not OK" 🙂 ). It is connected to the indexer cluster which has all the internal data from every connected Splunk environment. The MC (Monitoring Console) is not configured on it.

question:
i assume all the necessary data is in the _* indexes like _internal. How do i identify the "system role" within this data?

I can identify forwarders with "index=_internal source=*metrics.log group=tcpin_connections..." but how about other "roles"?

Any ideas are very welcome.

thanks,
swe

[richgalloway]

Check out the "Inherit a Splunk Enterprise Deployment" manual (http://docs.splunk.com/Documentation/Splunk/6.6.0/InheritedDeployment/Introduction).

[adonio]

link doesnt work for me

[richgalloway]

Clicking on the link doesn't work for me, either, but copy-and-paste into a new browser tab worked. Odd.

[koshyk]

Easiest way is to setup DMC in any of the system. If not, setup in your local laptop and you can get all the queries for DMC

[adonio]

enjoy
| rest/services/server/info | table host server_roles
btw, what do you want to be reported on when you say Indexer "OK" ?

[swe]

hi, thanks. ill check that. 🙂

ok would be something like last know log entry is younger than 1 minute, or so 🙂

[adonio]

take the hosts from the search above and create a search:
index= _internal host =1 or host=2 ... or host=n | timechart span=5m count by host
this will tell you how many events are indexed every 5 minutes by your splunk instances.
save as an alert if count = 0