Where should the kvstore be deployed in a distributed Splunk environment

[bandit]

The kvstore appears to be a database version of the traditional lookup table, however, it's a bit of a black box to me in how I should deploy to Splunk best practices and handle management and backups, etc.

The kvstore is up and running on all Splunk instances out of the box, however I couldn't find any architecture diagrams for how the kvstore should be deployed.

Should it be deployed to the search tier and the indexing tier?

What are the implications of shutting it off on the indexing tier? (if i'ts not used I'd rather disable it)

The kvstore seems to have it's own mongodb peer-to-peer clustering.
Does this clustering span horizontally across the search tier?
Does this clustering span horizontally across the indexing tier?
Does this cluster span horizontally and vertically across both search and indexing tiers?
Does this change when deploying clusters?

Thanks,

Rob

[jwelch_splunk · Accepted solution]

The KVStore, can be disabled on indexers.

The KVStore should be running on all SH's.

By default splunk has a couple of collections that using KVStore:
[SavedSearchHistory]
type = internal_cache

This is responsible for for keeping track of things like Continuous Scheduled Searches.

Short of that if you are not running a premium app like Enterprise Security or ITSI, most likely you have no other collections in KVStore.

You can tell if you just look for files called collections.conf.

IF you were to use the kvstore for lookups, and the lookups were configured for remote, E.G gonna be done on the peers vs the SH, then in your collections.conf you would configure that collection to replicate = true It would then dump the contents of that collection add it to your search bundle and push it over to the peers and they would use a dumped csv file to perform the remote lookups... meaning you still don't need kvstore running on your indexers.

As far as backing up... There are apps / and processes behind this, but at present we say. Stop splunk / tar it up / start splunk

If you are running SHC, then you will have a primary kvstore member which could be different than the SHC Captain, and then secondaries, they all replicate kvstore data between themselves. If however you have 3 independent SH's they know nothing about each others KVStore.

This is a very high level look... If you are going to be running a premium app that heavily relies on this.... I would suggest more research.

[nawazns5038]

After replication to the indexers, where does the data store in the indexers,

what is path /opt/splunk/var/lib/splunk/kvstore/mean in the indexers ? we have changed
SPLUNK_DB=/local/hot/ and there are two KVstore folders one in /local/hot and other in /opt/splunk/var/lib/splunk/kvstore/ ??

Is it safe to delete /opt/splunk/var/lib/splunk/kvstore/

[jwelch_splunk]

you can disable kvstore on your indexers...... don't delete the directory..... although I guess the one in /opt/splunk/var/lib/splunk would be okay, but the other one in your new SPLUNK_DB will most likely get re-created all the time..

The flat file dumps are in var/run/searchpeers in the bundles that are sent over.

[bandit]

Great summary, John. Just what I was looking for.

Rob

[nawazns5038]

After replication to the indexers, where does the data store in the indexers,

what is path /opt/splunk/var/lib/splunk/kvstore/ mean in the indexers ? we have changed
SPLUNK_DB=/local/hot/ and there are two KVstore folders one in /local/hot and other in /opt/splunk/var/lib/splunk/kvstore/ ??

Is it safe to delete /opt/splunk/var/lib/splunk/kvstore/