topic Re: How to identify Splunk Instance role by internal logs? in Deployment Architecture

How to identify Splunk Instance role by internal logs?

swe — Tue, 29 Sep 2020 13:58:15 GMT

Hi there,

prologue:
in my clustered environment i have a search head which is dedicated to a special user group and should serve a very simple dashboard view for system status overview (just some icons like "Indexers OK" " Search Heads Not OK" 🙂 ). It is connected to the indexer cluster which has all the internal data from every connected Splunk environment. The MC (Monitoring Console) is not configured on it.

question:
i assume all the necessary data is in the _* indexes like _internal. How do i identify the "system role" within this data?

I can identify forwarders with "index=_internal source=*metrics.log group=tcpin_connections..." but how about other "roles"?

Any ideas are very welcome.

thanks,
swe

Re: How to identify Splunk Instance role by internal logs?

koshyk — Fri, 05 May 2017 16:54:19 GMT

Easiest way is to setup DMC in any of the system. If not, setup in your local laptop and you can get all the queries for DMC

Re: How to identify Splunk Instance role by internal logs?

adonio — Fri, 05 May 2017 16:56:55 GMT

enjoy
| rest/services/server/info | table host server_roles
btw, what do you want to be reported on when you say Indexer "OK" ?

Re: How to identify Splunk Instance role by internal logs?

swe — Fri, 05 May 2017 17:05:18 GMT

hi, thanks. ill check that. 🙂

ok would be something like last know log entry is younger than 1 minute, or so 🙂

Re: How to identify Splunk Instance role by internal logs?

adonio — Fri, 05 May 2017 17:12:47 GMT

take the hosts from the search above and create a search:
index= _internal host =1 or host=2 ... or host=n | timechart span=5m count by host
this will tell you how many events are indexed every 5 minutes by your splunk instances.
save as an alert if count = 0

Re: How to identify Splunk Instance role by internal logs?

richgalloway — Fri, 05 May 2017 20:44:36 GMT

Check out the "Inherit a Splunk Enterprise Deployment" manual (http://docs.splunk.com/Documentation/Splunk/6.6.0/InheritedDeployment/Introduction).

Re: How to identify Splunk Instance role by internal logs?

adonio — Fri, 05 May 2017 20:52:10 GMT

link doesnt work for me

Re: How to identify Splunk Instance role by internal logs?

richgalloway — Mon, 08 May 2017 14:06:12 GMT

Clicking on the link doesn't work for me, either, but copy-and-paste into a new browser tab worked. Odd.