Monitoring Splunk

Would I be able to learn Splunk architecture by looking at various _internal log files?

Path Finder


I have been given a fully built Splunk (Enterprise) environment. I was not given the architecture/any docs related to the setup.

Will I be able to learn the connectivity by looking at the various log files? (splunkd, metrics etc.)

Note: I think I will be able to figure out the connectivity by looking at the conf files. I am awaiting access to the boxes. In the meantime, can I figure out at least the basic connectivity of forwarder -> Indexer -> Search Head from the _internal log files?


0 Karma


Hi Deepak,
yes internal indexes can give you information about the architecture ... but it gets easier with apps like S.o.S

0 Karma

Path Finder

Thankyou. Unfortunately it is a prod system, hence I cannot install SOS.

Can you tell me which _internal files to start with? I specifically need to identify the connections between the search heads, indexers and forwarders.

0 Karma


SoS is dated. I would recommend use of the DMC instead. Trying to map out the connectivity via internal logs will be a rabbit hole. I'd just recommend looking at the config files. The rest api can present those too,

0 Karma

Path Finder

@starcher - Thankyou very much. I will certainly take a look at the REST configurations.

I am awaiting access for DMC, and it looks like it will take sometime to get it. I want to start learning the architecture as soon as possible.

Are there any other options to learn the architecture, apart from the below?

  1. DMC - awaiting access, not feasible at the moment
  2. REST API - will check it out
  3. Configuration files on server boxes - awaiting access, not feasible at the moment
  4. _internal logs - time consuming
0 Karma

Path Finder

I found this by browsing:
To get list of all forwarders connecting to Splunk, run

 index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp

Is the above query correct? Is there a similar query I can run to find out the list of search heads and indexers?

0 Karma

Ultra Champion
| rest /services/server/info | table splunk_server

Based on how to get list of search peers via rest api

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes and swag!