Knowledge Management

How to configure and distribute KVstore in a Splunk 6.2.2 Search Head Pooling environment?

Glenn
Builder

We are upgrading our environment (including search head pools) from 5.x to 6.2.2, and would like to take advantage of kvstore. From what I can work out, by default kvstore is setup in a standalone mode, since there is no process to make the mongodb aware of the other node in its cluster (ie. the other search heads in the pool). I would have thought that a benefit of using a distributed db like MongoDB would be so the same info can be made available across all nodes (search heads) in the pool, but can't work out how this is done in Splunk.

I can't find much information about administering kvstore in a multi-search-head environment in the docs, but something is mentioned in the server.conf doc http://docs.splunk.com/Documentation/Splunk/6.2.2/Admin/Serverconf

[kvstore]
replication_host = <host>
* The host name to access the KV Store.
* In search head pooling, this host value is a requirement for using KV Store. 

Is this relevant? What should it be set to? Or does anyone have general guidance about how to get kvstore distributed?

I suspect we need to nominate one of the search heads in the pool as a "primary", and set the hostname of this to the replication_host setting on all pool members.

1 Solution

dgladkikh_splun
Splunk Employee
Splunk Employee

If you want to take advantage of KVStore in Search Head Pooling environment you need
a) Make all nodes of Search Head Pool to be visible to each other on the network.
b) On each member of Search Head Pool specify the key you discovered (kvstore/replication_host) to the hostname, which can be used on other member to discover this member on network.
c) Verify that kvstore/port is open and other member can get access to this port.
d) Just start all members on by one. KVStore automatically will be configured with Cluster.
e) You can use this REST endpoint to verify that KVStore is in Cluster now http://docs.splunk.com/Documentation/Splunk/6.2.2/RESTREF/RESTintrospect#server.2Fintrospection.2Fkv... or just query introspection index as it will be populated with introspection data including information from this url. Or use Distributed Management Console to check your configuration.

View solution in original post

dgladkikh_splun
Splunk Employee
Splunk Employee

If you want to take advantage of KVStore in Search Head Pooling environment you need
a) Make all nodes of Search Head Pool to be visible to each other on the network.
b) On each member of Search Head Pool specify the key you discovered (kvstore/replication_host) to the hostname, which can be used on other member to discover this member on network.
c) Verify that kvstore/port is open and other member can get access to this port.
d) Just start all members on by one. KVStore automatically will be configured with Cluster.
e) You can use this REST endpoint to verify that KVStore is in Cluster now http://docs.splunk.com/Documentation/Splunk/6.2.2/RESTREF/RESTintrospect#server.2Fintrospection.2Fkv... or just query introspection index as it will be populated with introspection data including information from this url. Or use Distributed Management Console to check your configuration.

dsmc_adv
Path Finder

I think is a must to have 3 members to have mongo replication work, can anyone confirm it?

0 Karma

dmr195
Communicator

I can confirm that the KV store can be made to work in a search head pooling environment using these steps. However, it's disappointing that the main KV store documentation doesn't make clearer that manually setting the replication_host setting in the [kvstore] stanza is a requirement when using search head pooling. This crucial piece of information is somewhat hidden in the monospaced text of server.conf.spec.

Another thing is that if you run in a search head pooling environment without setting replication_host then mongod.log fills up with messages like:

2015-03-17T11:27:29.454Z [rsStart] replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)

logged once per second.

0 Karma

ykpramodh
Engager

Hi,

we got KV Store replication to work in search head without specifying replication_host parameter in [kvstore] stanza.

Version: 6.5.0

regards
Pramodh

0 Karma

VEkhardt
Engager

Hi ykpramodh

How did you configured KV Store replication ? Can you share your experience please?

Thank you 

 

regards

Vladimir

Tags (2)
0 Karma

philip_wong
Communicator

I had replication_host set but still getting the same error.
kvstore becomes unusable...

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...