How to configure and distribute KVstore in a Splunk 6.2.2 Search Head Pooling environment?

[Glenn]

We are upgrading our environment (including search head pools) from 5.x to 6.2.2, and would like to take advantage of kvstore. From what I can work out, by default kvstore is setup in a standalone mode, since there is no process to make the mongodb aware of the other node in its cluster (ie. the other search heads in the pool). I would have thought that a benefit of using a distributed db like MongoDB would be so the same info can be made available across all nodes (search heads) in the pool, but can't work out how this is done in Splunk.

I can't find much information about administering kvstore in a multi-search-head environment in the docs, but something is mentioned in the server.conf doc http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Serverconf

[kvstore]
replication_host = <host>
* The host name to access the KV Store.
* In search head pooling, this host value is a requirement for using KV Store. 

Is this relevant? What should it be set to? Or does anyone have general guidance about how to get kvstore distributed?

I suspect we need to nominate one of the search heads in the pool as a "primary", and set the hostname of this to the replication_host setting on all pool members.

[dgladkikh_splun · Accepted solution]

If you want to take advantage of KVStore in Search Head Pooling environment you need
a) Make all nodes of Search Head Pool to be visible to each other on the network.
b) On each member of Search Head Pool specify the key you discovered (kvstore/replication_host) to the hostname, which can be used on other member to discover this member on network.
c) Verify that kvstore/port is open and other member can get access to this port.
d) Just start all members on by one. KVStore automatically will be configured with Cluster.
e) You can use this REST endpoint to verify that KVStore is in Cluster now http://docs.splunk.com/Documentation/Splunk/6.2.2/RESTREF/RESTintrospect#server.2Fintrospection.2Fkv... or just query introspection index as it will be populated with introspection data including information from this url. Or use Distributed Management Console to check your configuration.

[dsmc_adv]

I think is a must to have 3 members to have mongo replication work, can anyone confirm it?

[dmr195]

I can confirm that the KV store can be made to work in a search head pooling environment using these steps. However, it's disappointing that the main KV store documentation doesn't make clearer that manually setting the replication_host setting in the [kvstore] stanza is a requirement when using search head pooling. This crucial piece of information is somewhat hidden in the monospaced text of server.conf.spec.

Another thing is that if you run in a search head pooling environment without setting replication_host then mongod.log fills up with messages like:

2015-03-17T11:27:29.454Z [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

logged once per second.

[ykpramodh]

Hi,

we got KV Store replication to work in search head without specifying replication_host parameter in [kvstore] stanza.

Version: 6.5.0

regards
Pramodh

[VEkhardt]

Hi ykpramodh

How did you configured KV Store replication ? Can you share your experience please?

Thank you 

regards

Vladimir

[philip_wong]

I had replication_host set but still getting the same error.
kvstore becomes unusable...