KVStore - is it used in our installation?

[Dabbsy]

I'm looking into upgrading Splunk Enterprise from 9.0.4 to 9.3.0.

following the upgrade docs, there's a step to backup the KV Store.

Check the KV store status

To check the status of the KV store, use the show kvstore-status command:

./splunk show kvstore-status

When I run this command, it's asking me for a splunk username and password.  this was handed over by a project team, but nothing was handed over about what the splunk password might be, or also if we actually  use a KV store.  I've tried the admin password, but that's not worked.

I've found some splunk documents advising the KV store config would be in $SPLUNK_HOME/etc/system/local/server.conf, under [kvstore]

There is nothing in our server.conf under kvstore.

I've also found some notes talking about KVStore not starting if there's a $SPLUNK_HOME\var\lib\splunk\kvstore\mongo\mongod.lock file present

We have 2 splunk servers - one of these has a lock file dated Oct 2022, and the other dated July 19th.  So based on this, I suspect it's not used otherwise we'd have hit issues with it before?

That's just a guess, but this is my first foray into splunk, so I thought I'd ask if, based on the above scenarios whether I need to back up the KV store or not, or are there any other checks to confirm definitively if we have a KV store that's used?

thanks in advance

[dural_yyz]

Run the following on a single instance server or the distributed installation Monitoring Console instance.  The rest call SPL can be a massive help if the the CMD line option is not authenticating you.

| rest splunk_server=* /services/kvstore/status

In my experience anything that is a search head or search cluster you do want to have a KVStore backup in case of any corruptions.  A lot of apps are switching from lookup tables and opting for a better performing KVStore instance. 

[Dabbsy]

Thanks dural_yyz

that's saying current.backupRestoreStatus = ready

current.status=failed

so looks it's in use but not happy, and that's probably why the command isn't working.

i've found a hit about an old mongod.lock being present, so I'm going to arrange an outage to restart after removing the lock file.  

Will report back how this goes....

thanks

[JohnEGones]

@Dabbsy to add a proviso to Rich's response, make sure that you are coordinating with the system owner or backend admin or support team, to make sure that if they need those credentials that you are getting the right approvals and documenting things.

[richgalloway]

When a CLI command asks for credentials, it expects a Splunk local account name with admin privileges.   If you do not have the admin password then you should reset it.

Except for indexers and universal forwarders, just about any Splunk instance may be using KVStore.  It's also possible a new app will be installed that uses KVStore so it should be running.

[Dabbsy]

Thanks for the reply Rich.  I have an splunk admin account (that I log onto the splunk console with), but that password doesn't work.

When you say local account - is this different and if it is, how would I set one of these up?

[richgalloway]

A local account is one that does not use SAML or LDAP for authentication.  It's the default if you have not configured SSO.

The account you log onto the Splunk console with may not be available on all instances.  The account is know to the search heads, but not to indexers or universal forwarders and probably not to heavy forwarders.  Each could have had an admin account created when Splunk was installed.  If it was not created or if you don't have the password, reset the account using the instructions at https://community.splunk.com/t5/Security/How-to-Reset-the-Admin-password/m-p/10622

[JohnEGones]

The local admin account is not just an account with an admin role, it is the built-in local admin account that was created when you installed Splunk.

See (for example): Install on Windows - Splunk Documentation