Would I be able to learn Splunk architecture by looking at various _internal log files?

[deepak02]

Hi,

I have been given a fully built Splunk (Enterprise) environment. I was not given the architecture/any docs related to the setup.

Will I be able to learn the connectivity by looking at the various log files? (splunkd, metrics etc.)

Note: I think I will be able to figure out the connectivity by looking at the conf files. I am awaiting access to the boxes. In the meantime, can I figure out at least the basic connectivity of forwarder -> Indexer -> Search Head from the _internal log files?

Thanks,
Deepak

[Yasaswy]

Hi Deepak,
yes internal indexes can give you information about the architecture ... but it gets easier with apps like S.o.S

[deepak02]

Thankyou. Unfortunately it is a prod system, hence I cannot install SOS.

Can you tell me which _internal files to start with? I specifically need to identify the connections between the search heads, indexers and forwarders.

[starcher]

SoS is dated. I would recommend use of the DMC instead. Trying to map out the connectivity via internal logs will be a rabbit hole. I'd just recommend looking at the config files. The rest api can present those too,

http://docs.splunk.com/Documentation/Splunk/latest/RESTTUT/RESTconfigurations

[deepak02]

@starcher - Thankyou very much. I will certainly take a look at the REST configurations.

I am awaiting access for DMC, and it looks like it will take sometime to get it. I want to start learning the architecture as soon as possible.

Are there any other options to learn the architecture, apart from the below?

1. DMC - awaiting access, not feasible at the moment
2. REST API - will check it out
3. Configuration files on server boxes - awaiting access, not feasible at the moment
4. _internal logs - time consuming

[deepak02]

I found this by browsing:
To get list of all forwarders connecting to Splunk, run

 index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp

Is the above query correct? Is there a similar query I can run to find out the list of search heads and indexers?

[ddrillic]

| rest /services/server/info | table splunk_server

Based on how to get list of search peers via rest api