Which instance or configuration file in my Splunk environment contains cluster master details?

[Hemnaath]

Currently in our environment, we have 5 indexer instances, four search heads, a scheduled search Job instance, 2 Heavy Forwarders, and a deployment manager/ License master running on the same instance?

Kindly let me know in which instance/configuration file can I find out the cluster-master details.

[maciep]

It sounds like you're new to this environment? Are you sure you have an indexer cluster? If you do, your indexers would know which server the master is. And like renjith said, it would be in the clustering stanza in server.conf. So maybe run btool on one of your indexers and see if it's a slave to a master.

This should return any non-default settings for the clustering stanza, and which files those settings are coming from.

/opt/splunk/bin/splunk btool server list clustering --debug | grep -v system/default

[Hemnaath]

Yes I am new to this environment and the person who had worked in the organization had left the company and there is no an documentation. In Architecture diagram I could see there are 5 indexer instance running and had assumed that its a cluster environment.

I had executed the command in one of the indexer server and but there no result popped out.
/opt/splunk/bin
./splunk btool server list clustering --debug | grep -v system
./splunk btool server list clustering --debug | grep -v default

Currently we are using splunk 6.2 version and 6.0.3 version in our environment. As you aware coming 21 July 2016 the default root certificate will expiry and we had planned to stick with version by executing the renewcert.sh script provided by splunk. So I was checking for the details.

One more question regarding the default certificate ?

I could see 1098 UF agent running in our environment, to determine default certificate used by the forwarder, I had ran the below query and the result indicated that SSL is false, does this means there is no SSL configured in the forwarders? if not then should I need to execute the query individually to all the agents? Kindly clarify this

index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl

thanks in advance.

[maciep]

As I understand it, if your forwarders are not using ssl to communicate with the indexers, then you don't have to worry about the certificates expiring.

Out of the box, forwarders don't use ssl. And I think that search looks right as well. So likely, you won't have to take any action against your forwarders for the expiring certificates.

But you may want to get a second opinion to be sure 🙂

[renjith_nair]

You can find the details from server.conf

For master, you will have the below entries under [clustering] stanza

[clustering]
mode = master --> tells the master
replication_factor = 4
search_factor = 3
pass4SymmKey = whatever
cluster_label = cluster1

and for peers

[clustering]
master_uri = https://<ip>:8089 -->gives you master URI
mode = slave  -->tells its's a peer
pass4SymmKey = whatever

You can find the license master details also from server.conf under [license] stanza

Hope this helps

[Hemnaath]

Thanks Renjith, I have checked the server. Conf file in the indexers under the following paths /opt/splunk/etc/apps/xxxx/default/server.conf and I could see only below stanza

[httpServer]
maxThreads = 1000

Similarly under this path /opt/splunk/etc/system/local - I could see this stanza

[sslConfig]
sslKeysfilePassword = xxxxxxxxxx

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[license]
master_uri = https://:8089 -->
[general]
pass4SymmKey = xxxxxxxx
serverName = xxxxxxxxxx

incase of License /deployment master instance under this path
/opt/splunk/etc/shcluster/apps/XXX-ADMIN-all_indexers/default/server.conf

[httpServer]
maxThreads = 1000

/opt/splunk/etc/shcluster/apps/XXX-ADMIN-hvy_forwarders/default/server.conf

[queue]
maxSize = 200MB

/opt/splunk/etc/shcluster/apps/XXX-ADMIN-searchheadcluster/default/server.conf

[license]
master_uri = https://:8089 --> 

In all the above mentioned path I could not see the stanza called clustering, so kindly let me is this is the correct path to validate recoding the master node details.

Thanks in Advance.

[renjith_nair]

Check with btool in all server configuration.

 ./splunk cmd btool server list --debug clustering

If it returns only default values from /etc/system/default/server.conf, then most likely your environment is not clustered but a distributed environment

http://docs.splunk.com/Documentation/Splunk/6.4.1/Indexer/Keydifferences

Similarly run this for finding out the search head clustering details from a search head

 ./splunk cmd btool server list --debug shclustering

[Hemnaath]

thanks Ranjith, after executing the above command in all the indexer, It returned only the default values from /etc/system/default/server.conf, from this it was clear that ours is Distributed Splunk Environment not a clustered.
Similarly when I ran the command on the search heads (Four search heads / one Scheduled Job search severs) got the same default result. But as per the architecture diagram, I could see two search heads are connected to file sharing pools where it share the information about the search's. Kindly guide why its not showing the clustered results or whether its clustered one but currently its not working as clustered.
thanks in Advance.

[renjith_nair]

Which version of splunk you are using? File share was used in 5.x search head pooling and it's deprecated and changed to search head clustering in 6.x

[Hemnaath]

thank Renjith, We are using the 6.0.3 and 6.2.1 version in our environment. As per the architecture diagram, the two searched heads are clustered along with this file sharing server. I had ran the below command to identify whether the two search head are in clustered or not, but I did not get any output after executing the CLI.
./splunk cmd btool server list --debug shclustering

I had even stopped one of the search head to test for clustering behavior, but it was not getting switched to another server. It was still pointing to the same server, I had verified this by going to setting --> system details --> General setttings --> splunk server name.

Kindly guide me to fix this problem. thanks in advance

[renjith_nair]

Hello Hemnaath,

If you are using different versions of splunk in the same environment, that itself is a matter of concern. From all the discussions above, most probably, you have a distributed system with search head pooling

http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Configuresearchheadpooling
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Overviewofconfiguration

To verify that run btool and look for the stanza [pooling]

Also look for the file distsearch.conf (http://docs.splunk.com/Documentation/Splunk/6.4.1/admin/Distsearchconf) which will have details of the distributed search

[Hemnaath]

thanks Renjith , I am not sure why they had configured like this but we wanted to make all to run with same version, so let me check the details give in the links , but mean while can you guide me on how to reconfigure the search heads to behave as clustered. Is it possible to get me a step by step procedure to do this. thanks in advance.

[renjith_nair]

For SHC , please refer to http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/SHCdeploymentoverview

For migration from existing standalone to search head cluster, please refer to

http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Migratefromstandalonesearchheads
http://docs.splunk.com/Documentation/Splunk/6.4.1/DistSearch/Migratefromsearchheadpooling
https://answers.splunk.com/answers/242518/migrating-separate-environments-to-search-head-clu.html

[gcusello]

If you go in one of the clustered indexers and see at "Settings - Indexers Clustering" you can see the Master's URL.
After, you can see configurated peers in Master Node at "Settings - Indexers Clustering".
Bye.
Giuseppe

[Hemnaath]

Thanks, Giuseppe, I had tried login into one of the search head --> settings --> Distributed Environment --> clustering. I could not see any configuration related to clustering, I could see Enable Clustering option. Correct if this is not the correct place to look into. Similarly under settings --> Distributed Environment --> Distributed search --> search peers --> I could see all the indexer are listed. Kindly let me is any other way I can find the master node details.

[gcusello]

If you are speaking about Indexers Cluster, to find the Master Node you have to see in one of the indexers.
Bye.
Giuseppe