How to identify active KVstore collections on all Splunk deployment instances?

[Glasses2]

In order to upgrade Splunk from 8.1.3 to 9.0.4, I need to migrate/upgrade the KVstore engine from MMAPv1 to WiredTiger and then upgrade the KVstore (mongoDB) version to 4.2.

I believe that I only need to upgrade the KVstore on instances with active KVstore collections.   And I should stop scheduled searches writing to collections or wait until they are finished before running the migration.

By default the KVstore is enabled on all instances which is a bit confusing, BUT the monitoring console (MC)  KVstore dashboard gives me some indication that only the instances with active kvstore collections are the SHC and some HFs...  but not my MC/license master or the Deployment Server or the Index Cluster master.

Splunk docs mention that you can disable kvstore and ignore certain collections.

My question is how to verify and identify the active collections on my instances?
Where is the kvstore files location? (apparently the directory seems to be /opt/splunk/var/lib/splunk/kvstore/mongo)  but the files are not human-readable clear... 

Is there a query to see the individual KVstore collection file sizes with human-readable names?

This gives me I think most of them... not sure

| rest /servicesNS/-/-/data/transforms/lookups  splunk_server=local 
| table eai:acl.app title collection 
| search collection!=""

Any advice appreciated.

Thank you

[richgalloway]

Please don't upgrade only the instances with active collections because the next day an un-upgraded instance will want to load a collection and fail.

Upgrade all instances that can have a collection - search heads and heavy forwarders.

[Glasses2]

Thank you for the response. 

You answered me before in another post >> https://community.splunk.com/t5/Deployment-Architecture/In-a-distributed-and-clustered-environ-which...

 upgrade all nodes. 

So I presume there is no harm upgrading  a kvstore engine and version on an instance w/ or w/out any active collections?

I am scrambled and support won't even give me a definite answer.

[richgalloway]

There is never a need to upgrade the KVStore on indexers because indexers do not use the KVStore.  Upgrade only Search Heads and Heavy Forwarders.

The determining factor is not if the instance has a collection - it's if the instance has a running KVStore.

Support won't help because this is not a break/fix matter.

[Glasses2]

Thanks for the clarification (and persevering thru my questions).  I really do appreciate it!

My concerns, re: IDXCluster and KVstore, are due to all the non-sensical configurations the previous admins left for me....

It would not surprise me to see something unusual here, as I have seen some really goofy stuff.   For instance , I am concerned they did some sort of remote kvstore replication/sync to the indexer peers etc... maybe that doesn't matter, not 100% about that.

Would you advise that I "disable" the KVstore on the indexers? for safety? per Splunk docs>>> https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/AboutKVstore

RE: "upgrade only Search Heads and Heavy Forwarders" ,  I see collections (probably default) on my MC/LM instance and Deployment Server instance as well.  So I am thinking upgrade those as well? IDK.

when I run this on my MC/LM instance >>> 

| rest splunk_server="local" "/services/search/distributed/peers/" 
| search disabled=0 
| rename peerName AS splunk_server 
| fields host splunk_server 
| join type=left splunk_server 
    [| rest splunk_server="*" "/services/kvstore/status" 
    | table splunk_server current.status current.replicationStatus current.storageEngine ] 
| sort - current.storageEngine

I see every instance (including indexers)  has a default enabled kvstore in "ready"... 

So now, knowing this information, does that change your advice?

RE: support, IDK, I do feel support should be able to answer these questions.  I get support is "break/fix" but do I really have to break it first to get help or a couple definitive answers?

I am sure Support will kick it to our Sales Rep and they will say we need to engage PS for 2 week$ 🤣...  

Thank you!

[richgalloway · Accepted solution]

Indexers definitely do not use KVStore.  If you have collections on your indexers then you have other problems.  😀  Disabling KVStore on indexers isn't a bad idea and can help save resources.

I'm not sure why the MC and DS have collections (could be "goofy stuff"), but it can't hurt to upgrade them.

[Glasses2]

POA

1) disable KVstore on the indexer instances only

2) update KVstore all other Splunk instances with a "ready" enabled KVstore to be safe.  **In my case, all instances are default enabled **

Hopefully everything goes well! I will post if it goes sideways.

[Glasses2]

Well, in line with your recommendations...

I am being advised by Internal Splunk Personnel, to upgrade everything except indexers...

My feeling is to disable the kvstore on the indexers but not the other nodes like IDXcluster master etc...

Would you agree? (you can conclude that I value your opinion with equal weight as support 😉, maybe more)