How do you identify if a box is an indexer or a search head?

[makincerdas]

Hi,

Splunk were installed on 2 boxes by previous admin.
I can browse to port 8000 on both boxes, and get the 'Search and Reporting' UI.

How do you identify accurately if a box is an indexer and another box is a search head?

Thank you.

[yannK · Accepted solution]

A box can be both.
Use btool to check :

• Being an indexer usually means that you keep a copy of the indexed data locally.
  do a btool of outputs
  if you see an outputs setting, you are a forwarder not an indexer.
  And if you see an outputs and the setting indexAndForward =true, they you are an indexer AND a heavy forwarder.

• to be search-head means that you search on remote search-peers
  do a btool on distsearch.conf, look for a list of servers
  do a btool on server.conf, and look is you are a search-head in an indexingcluster
  If you see any of those, you are a search-head.

Remarks :
- in a standalone mode, you are both an indexer and a search-head of yourself.
- usually it's recommended to configure the search-head as a forwarder.

[makincerdas]

Do you have sample on how to use the btool command to check?
Thanks.

[prakash007]

Here's the sample....

./splunk cmd btool transforms list 

https://docs.splunk.com/Documentation/Splunk/6.5.0/Troubleshooting/Usebtooltotroubleshootconfigurati...

[yannK]

cd /opt/splunk/bin

then

./splunk cmd btool outputs list
# to check  if you see a defaultgroup destination under the stanza [tcpout]

./splunk cmd btool distsearch list
# to see if you see a list or searchpeers 

./splunk cmd btool server list
# to see under the stanza [clustering] if you you are in an indexer cluster, and your role (search-head, cluster-master, indexers)

[makincerdas]

Thank you.

[lukejadamec]

You can check Splunk_Home\var\lib\splunk for folders that match your index names.
You can check your index names from the Splunk UI by going to Settings > Indexes.

You can also check your deployed outputs.conf files to see where the data is being sent.

[lukejadamec]

Well, it looks like it was set up at one point to do load balancing. At that time I would expect to see two IP's on the server line for the default-autolb-group, but since there is only one load balancing must have stopped sometime in the past. Or, it was never configured correctly.

Do all of the Universal Forwarders have the same outputs.conf?

On each server, look in server.conf for the [clustering] stanza, and check the value for mode =
If it is a search head, then it should point to a master_uri. If it is a search head, then the server with server.conf mode=searchhead probably has a different IP than the one in outputs.conf, and the master_uri for that one will match the IP in outputs.conf.
Keep in mind that an Enterprise Splunk installation (search head or indexer) will have indexes, and it looks like the indexes you have are all defaults, so the indexes will match between the two, but the larger one is probably the one with the IP found in outputs.conf.

[lukejadamec]

Like I said, these look like default index that will exist on all Splunk Enterprise installations, and will be used at minimum by Splunk itself. So, no, it is not unusual to have these indexes on indexers that are not targets of the forwarders.

Can you confirm that the indexer with the 10GB size is the 10.200.2.30:9997 system?

[makincerdas]

@sundareshr,

I found the value of splunk_server after doing the recommended search.

I am thinking in the environment where there are more than 2 boxes.
Is it safe to assume if it is not displayed on 'splunk_server' field, then it is NOT an indexer box, so it must be a search head box?

Thanks.

[makincerdas]

@lukejadamec,
I login to both boxes using putty. Both boxes have all folders shown on Splunk UI (Settings > Indexes)

However, on one box the folder size is 10 GB , on the other box the folder size is 1 GB of data.
Why both boxes have exact folders name shown on Splunk UI?
Should only the Indexer show exact folders name as shown on Splunk UI?

ie.
[root /opt/splunk/var/lib/splunk]# ls -l
total 80
drwx------ 6 root root 4096 Oct 6 19:20 audit
-rw------- 1 root root 1 Oct 6 20:06 _audit.dat
drwx------ 2 root root 4096 Oct 6 19:20 authDb
drwx------ 6 root root 4096 Oct 6 19:20 defaultdb
drwx------ 8 root root 4096 Oct 7 13:42 fishbucket
drwx------ 2 root root 4096 Oct 6 19:20 hashDb
-rw------- 1 root root 1 Oct 6 20:06 history.dat
drwx------ 6 root root 4096 Oct 6 19:20 historydb
-rw------- 1 root root 1 Oct 6 20:06 _internal.dat
drwx------ 6 root root 4096 Oct 6 19:20 _internaldb
drwx------ 6 root root 4096 Oct 6 19:20 _introspection
-rw------- 1 root root 1 Oct 6 20:06 _introspection.dat
drwx------ 3 root root 4096 Oct 6 19:20 kvstore
-rw------- 1 root root 1 Oct 6 20:06 main.dat
drwx------ 3 root root 4096 Oct 7 13:40 persistentstorage
-rw------- 1 root root 1 Oct 6 20:06 summary.dat
drwx------ 6 root root 4096 Oct 6 19:20 summarydb
drwx------ 6 root root 4096 Oct 6 19:20 _telemetry
-rw------- 1 root root 1 Oct 6 20:06 _telemetry.dat
-rw------- 1 root root 1 Oct 6 20:06 _thefishbucket.dat

The following is the content of outputs.conf of Universal Forwarder
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.200.2.30:9997

[tcpout-server://10.200.2.30:9997]

So, the indexer must be on box with IP address 10.200.2.30, correct?

Thanks.

[sundareshr]

Run a search in the UI (index=_internal | head 100), look at the value in splunk_server field. That will show the name of the indexer.

[gcusello]

you could also verify in [Settings -- Distributed Search] which one shows a Search Peer.
Bye.
Giuseppe