Monitoring Splunk

Would I be able to learn Splunk architecture by looking at various _internal log files?

deepak02
Path Finder

Hi,

I have been given a fully built Splunk (Enterprise) environment. I was not given the architecture/any docs related to the setup.

Will I be able to learn the connectivity by looking at the various log files? (splunkd, metrics etc.)

Note: I think I will be able to figure out the connectivity by looking at the conf files. I am awaiting access to the boxes. In the meantime, can I figure out at least the basic connectivity of forwarder -> Indexer -> Search Head from the _internal log files?

Thanks,
Deepak

0 Karma

Yasaswy
Contributor

Hi Deepak,
yes internal indexes can give you information about the architecture ... but it gets easier with apps like S.o.S

0 Karma

deepak02
Path Finder

Thankyou. Unfortunately it is a prod system, hence I cannot install SOS.

Can you tell me which _internal files to start with? I specifically need to identify the connections between the search heads, indexers and forwarders.

0 Karma

starcher
SplunkTrust
SplunkTrust

SoS is dated. I would recommend use of the DMC instead. Trying to map out the connectivity via internal logs will be a rabbit hole. I'd just recommend looking at the config files. The rest api can present those too,

http://docs.splunk.com/Documentation/Splunk/latest/RESTTUT/RESTconfigurations

0 Karma

deepak02
Path Finder

@starcher - Thankyou very much. I will certainly take a look at the REST configurations.

I am awaiting access for DMC, and it looks like it will take sometime to get it. I want to start learning the architecture as soon as possible.

Are there any other options to learn the architecture, apart from the below?

  1. DMC - awaiting access, not feasible at the moment
  2. REST API - will check it out
  3. Configuration files on server boxes - awaiting access, not feasible at the moment
  4. _internal logs - time consuming
0 Karma

deepak02
Path Finder

I found this by browsing:
To get list of all forwarders connecting to Splunk, run

 index=_internal source=*metrics.log* tcpin_connections | stats count by sourceIp

Is the above query correct? Is there a similar query I can run to find out the list of search heads and indexers?

0 Karma

ddrillic
Ultra Champion
| rest /services/server/info | table splunk_server

Based on how to get list of search peers via rest api

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...