Why are there conflicting statements in documentation about KV store in Search Head Clustering?

[arubi2]

There appears to be conflicting statements about this feature.

First we find a potential warning in the docs about using/deploying the kv store:

The KV store files reside on search
heads.

In a search head cluster, if any node
receives a write, the KV store
delegates the write to the KV store
captain. The KV store keeps the reads
local, however.

From the Splunk dictionary:

KV store captain noun

The KV store captain is the single
instance in the app key value store
that receives write operations. Other
KV store instances replicate data from
the captain.

In a search head cluster, when a node
receives a write request, the KV store
delegates the write to the KV store
captain, while the KV store keeps read
requests local.

However, this page says:

Search head clustering and KV store KV
store can reside on a search head
cluster. However, the search head
cluster does not coordinate
replication of KV store data or
otherwise involve itself in the
operation of the KV store. For
information on KV store, see "About KV
store" in the Admin Manual.

I'm looking to keep state between search heads in a cluster with durability. The docs seem contradictory. Not sure if it's old docs which have failed to update or what. Also, the concept of using a captain seems a bit mysterious. Is this the SH Captain? What happens to durability during the timeframe when a new captain is being re-elected?

Is it maybe better to just use redis instead? With this app?

Anybody out there using the kv store in a SH cluster who might be able to chime in?

[jlaw]

The KV store and search head clustering operate pretty much independently.

The KV store captain is elected by its peers, just like the search head captain, although it is different code that runs those two elections. It's possible that one node would end up hosting both captains, but not necessary.

The potential warning you point out is not a warning, but rather information about how kv store handles reads and writes. And where exactly do you think the docs are in conflict? KV store runs on a search head cluster, but search head clustering doesn't interact with the KV store.

[arubi2]

The conflict is between the definition of the KV Captain and the last part. (i.e. between the second and third excerpt). One seems to indicate that data will be replicated (i.e. having a captain which handles writes) and the other seems to contra-indicate that (i.e. 'does not handle replication').

[jlaw]

Like the docs you've pointed out say, the KV store captain handles writes to the KV store. The search head cluster doesn't care which of its nodes is doing what with KV store. KV store should work out of the box. Here's the small section on docs.splunk.com that outlines using KV store, with links to the docs on dev.splunk.com, where more of the usage docs live:

http://docs.splunk.com/Documentation/Splunk/6.2.4/Admin/AboutKVstore#Use_the_KV_store

[starcher]

Technically the SHC cluster replication function is not handling the KV store. The KV store captain is coordinating that. Which could be any node in the SHC. Technically KV store is mongoDB.

[arubi2]

So, I think the confusion in the docs is that it's considering the KV Store (which operates over the very same cluster) to be a different cluster than the SH cluster itself? Maybe I'm missing some intermediate document which describes how to set up the KV Store cluster? Or does it work out of the box to implement a cluster based upon any/all SH's in the cluster which also have the same KV Store setup?

[starcher]

Is there any method for determining the KV captain? I have yet to find it. Maybe an undocumented rest api call or cli command? Best I understand it, the members delegate their writes to the captain so I expect they cache them while a new one is elected.

[gabriel_vasseur]

On the command line you can do:
splunk show kvstore-status
In a similar vein that you can check the SH cluster status:
splunk show shcluster-status
http://docs.splunk.com/Documentation/Splunk/7.0.1/Admin/TroubleshootKVstore

[jlaw]

I don't know of one, but that doesn't mean it doesn't exist. Why do you want to do this?

Here are the docs for the KV store endpoints:
http://docs.splunk.com/Documentation/Splunk/6.2.4/RESTREF/RESTkvstore

I don't see one in there for this purpose.