However, is it possible to get the KV store to replicate between two search heads that are neither in a pool nor a cluster?
The situation where this would be useful is as follows:
First search head runs apps that generate data that is forwarded to an indexer layer, and also KV store entries that are stored on the search head
Second search head runs Splunk Enterprise security app, is set up to search indexes on the same indexer layer as the first search head and would ideally also be able to access the same KV store data as the first search head
I understand that it is usual practice to run Splunk Enterprise security on a separate search head, hence the suggestion of enabling search head clustering would not be very helpful.
The Answers post you mention should work in a non-clustered environment. KVstore has it's own replication abilities, so setting the replication_host on the nodes in question should build a KVstore cluster. You will have to be careful not to have different searchheads that are not "coordinated" by a cluster, writing duplicate data to the KVstore.
Also, it is possible to have a separate SHC for ES, although clearly that still leaves you with the same issue of uncoordinated KVstores between the ES SHC and the non ES searchheads.
Thanks for the answer but I'm not convinced it's as simple as you make out.
The replication_host setting in the [kvstore] stanza of server.conf seems to tell the instance which local IP address to accept connections on. However, another piece of information is needed to form the KV store replica set, namely which remote IP address to attempt to connect to. In Splunk 6.2 I cannot see any config setting to tell an arbitrary Splunk instance that it should communicate with some other arbitrary Splunk instance for KV store replication. (Obviously for search head pooling or clustering each search head knows the IP addresses of the others in the pool/cluster and can try to join the replica set consisting of the KV stores on all of these.)
I have done some testing trying to get two independent search heads on the same subnet to replicate their KV stores and they never attempt to talk to each other on port 8191. This is not due to firewalls: I can telnet from each of the search heads to port 8191 on the other one and get a TCP connection.
So, as far as I can see, the ability to do what I want is tantalisingly close - it just needs one extra config setting to specify a remote IP address to connect to for KV store replication - but not possible in 6.2. Maybe such a setting has been added to the development version of Splunk that you have access to but I don't...