Getting Data In

How do you identify if a box is an indexer or a search head?

makincerdas
Explorer

Hi,

Splunk were installed on 2 boxes by previous admin.
I can browse to port 8000 on both boxes, and get the 'Search and Reporting' UI.

How do you identify accurately if a box is an indexer and another box is a search head?

Thank you.

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

A box can be both.
Use btool to check :

  • Being an indexer usually means that you keep a copy of the indexed data locally.
    do a btool of outputs
    if you see an outputs setting, you are a forwarder not an indexer.
    And if you see an outputs and the setting indexAndForward =true, they you are an indexer AND a heavy forwarder.

  • to be search-head means that you search on remote search-peers
    do a btool on distsearch.conf, look for a list of servers
    do a btool on server.conf, and look is you are a search-head in an indexingcluster
    If you see any of those, you are a search-head.

Remarks :
- in a standalone mode, you are both an indexer and a search-head of yourself.
- usually it's recommended to configure the search-head as a forwarder.

View solution in original post

yannK
Splunk Employee
Splunk Employee

A box can be both.
Use btool to check :

  • Being an indexer usually means that you keep a copy of the indexed data locally.
    do a btool of outputs
    if you see an outputs setting, you are a forwarder not an indexer.
    And if you see an outputs and the setting indexAndForward =true, they you are an indexer AND a heavy forwarder.

  • to be search-head means that you search on remote search-peers
    do a btool on distsearch.conf, look for a list of servers
    do a btool on server.conf, and look is you are a search-head in an indexingcluster
    If you see any of those, you are a search-head.

Remarks :
- in a standalone mode, you are both an indexer and a search-head of yourself.
- usually it's recommended to configure the search-head as a forwarder.

makincerdas
Explorer

Do you have sample on how to use the btool command to check?
Thanks.

0 Karma

prakash007
Builder
0 Karma

yannK
Splunk Employee
Splunk Employee
cd /opt/splunk/bin

then

./splunk cmd btool outputs list
# to check  if you see a defaultgroup destination under the stanza [tcpout]

./splunk cmd btool distsearch list
# to see if you see a list or searchpeers 

./splunk cmd btool server list
# to see under the stanza [clustering] if you you are in an indexer cluster, and your role (search-head, cluster-master, indexers)
0 Karma

makincerdas
Explorer

Thank you.

0 Karma

lukejadamec
Super Champion

You can check Splunk_Home\var\lib\splunk for folders that match your index names.
You can check your index names from the Splunk UI by going to Settings > Indexes.

You can also check your deployed outputs.conf files to see where the data is being sent.

0 Karma

lukejadamec
Super Champion

Well, it looks like it was set up at one point to do load balancing. At that time I would expect to see two IP's on the server line for the default-autolb-group, but since there is only one load balancing must have stopped sometime in the past. Or, it was never configured correctly.

Do all of the Universal Forwarders have the same outputs.conf?

On each server, look in server.conf for the [clustering] stanza, and check the value for mode =
If it is a search head, then it should point to a master_uri. If it is a search head, then the server with server.conf mode=searchhead probably has a different IP than the one in outputs.conf, and the master_uri for that one will match the IP in outputs.conf.
Keep in mind that an Enterprise Splunk installation (search head or indexer) will have indexes, and it looks like the indexes you have are all defaults, so the indexes will match between the two, but the larger one is probably the one with the IP found in outputs.conf.

0 Karma

lukejadamec
Super Champion

Like I said, these look like default index that will exist on all Splunk Enterprise installations, and will be used at minimum by Splunk itself. So, no, it is not unusual to have these indexes on indexers that are not targets of the forwarders.

Can you confirm that the indexer with the 10GB size is the 10.200.2.30:9997 system?

0 Karma

makincerdas
Explorer

@sundareshr,

I found the value of splunk_server after doing the recommended search.

I am thinking in the environment where there are more than 2 boxes.
Is it safe to assume if it is not displayed on 'splunk_server' field, then it is NOT an indexer box, so it must be a search head box?

Thanks.

0 Karma

makincerdas
Explorer

@lukejadamec,
I login to both boxes using putty. Both boxes have all folders shown on Splunk UI (Settings > Indexes)

However, on one box the folder size is 10 GB , on the other box the folder size is 1 GB of data.
Why both boxes have exact folders name shown on Splunk UI?
Should only the Indexer show exact folders name as shown on Splunk UI?

ie.
[root /opt/splunk/var/lib/splunk]# ls -l
total 80
drwx------ 6 root root 4096 Oct 6 19:20 audit
-rw------- 1 root root 1 Oct 6 20:06 _audit.dat
drwx------ 2 root root 4096 Oct 6 19:20 authDb
drwx------ 6 root root 4096 Oct 6 19:20 defaultdb
drwx------ 8 root root 4096 Oct 7 13:42 fishbucket
drwx------ 2 root root 4096 Oct 6 19:20 hashDb
-rw------- 1 root root 1 Oct 6 20:06 history.dat
drwx------ 6 root root 4096 Oct 6 19:20 historydb
-rw------- 1 root root 1 Oct 6 20:06 _internal.dat
drwx------ 6 root root 4096 Oct 6 19:20 _internaldb
drwx------ 6 root root 4096 Oct 6 19:20 _introspection
-rw------- 1 root root 1 Oct 6 20:06 _introspection.dat
drwx------ 3 root root 4096 Oct 6 19:20 kvstore
-rw------- 1 root root 1 Oct 6 20:06 main.dat
drwx------ 3 root root 4096 Oct 7 13:40 persistentstorage
-rw------- 1 root root 1 Oct 6 20:06 summary.dat
drwx------ 6 root root 4096 Oct 6 19:20 summarydb
drwx------ 6 root root 4096 Oct 6 19:20 _telemetry
-rw------- 1 root root 1 Oct 6 20:06 _telemetry.dat
-rw------- 1 root root 1 Oct 6 20:06 _thefishbucket.dat

The following is the content of outputs.conf of Universal Forwarder
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.200.2.30:9997

[tcpout-server://10.200.2.30:9997]

So, the indexer must be on box with IP address 10.200.2.30, correct?

Thanks.

0 Karma

sundareshr
Legend

Run a search in the UI (index=_internal | head 100), look at the value in splunk_server field. That will show the name of the indexer.

0 Karma

gcusello
SplunkTrust
SplunkTrust

you could also verify in [Settings -- Distributed Search] which one shows a Search Peer.
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

Industry Solutions for Supply Chain and OT, Amazon Use Cases, Plus More New Articles ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In November, the Splunk Threat Research Team had one release of new security content via the Enterprise ...