In Splunk 6.2 the KV store is automatically replicated between search heads that are in a search head cluster.
Also, it is possible to get the KV store to replicate between search heads in a search head pool by using the instructions here: http://answers.splunk.com/answers/231043/how-to-configure-and-distribute-kvstore-in-a-splun.html#ans...
However, is it possible to get the KV store to replicate between two search heads that are neither in a pool nor a cluster?
The situation where this would be useful is as follows:
I understand that it is usual practice to run Splunk Enterprise security on a separate search head, hence the suggestion of enabling search head clustering would not be very helpful.
The Answers post you mention should work in a non-clustered environment. KVstore has it's own replication abilities, so setting the replication_host on the nodes in question should build a KVstore cluster. You will have to be careful not to have different searchheads that are not "coordinated" by a cluster, writing duplicate data to the KVstore.
Also, it is possible to have a separate SHC for ES, although clearly that still leaves you with the same issue of uncoordinated KVstores between the ES SHC and the non ES searchheads.
Thanks for the answer but I'm not convinced it's as simple as you make out.
The replication_host setting in the [kvstore] stanza of server.conf seems to tell the instance which local IP address to accept connections on. However, another piece of information is needed to form the KV store replica set, namely which remote IP address to attempt to connect to. In Splunk 6.2 I cannot see any config setting to tell an arbitrary Splunk instance that it should communicate with some other arbitrary Splunk instance for KV store replication. (Obviously for search head pooling or clustering each search head knows the IP addresses of the others in the pool/cluster and can try to join the replica set consisting of the KV stores on all of these.)
Also, the documentation for replication_host in server.conf (http://docs.splunk.com/Documentation/Splunk/6.2.3/Admin/Serverconf) states, "This setting has no effect on a single Splunk instance."
I have done some testing trying to get two independent search heads on the same subnet to replicate their KV stores and they never attempt to talk to each other on port 8191. This is not due to firewalls: I can telnet from each of the search heads to port 8191 on the other one and get a TCP connection.
So, as far as I can see, the ability to do what I want is tantalisingly close - it just needs one extra config setting to specify a remote IP address to connect to for KV store replication - but not possible in 6.2. Maybe such a setting has been added to the development version of Splunk that you have access to but I don't...
The information you are looking for is in etc/pooling/pooling.ini
guid = 80D0C74D-0819-4AA4-B5D8-C5EB5e41029B
kvstore_secret = 5cdae94f7015f55698abb975R2c7818e3fabaa9e3001f61b3942ee3bfd854338
kvstore_active_members = 99145D4E-6AF2-4D0C-8DCB-269027785CAD,B76C11A2-49F1-458A-9EA2-7C877B06CAF4
kvstore_sync_master = B76C11A2-49F1-458A-9EA2-7C877B06CAF4
host1 = host1.mylocaldomain:8191
host2 = host2.mylocaldomain:8191
So, I suppose that must be part of head pooling in order to read this configuration and sync the mongo db