Splunk Enterprise

How to identify active KVstore collections on all Splunk deployment instances?

Glasses2
Communicator

In order to upgrade Splunk from 8.1.3 to 9.0.4, I need to migrate/upgrade the KVstore engine from MMAPv1 to WiredTiger and then upgrade the KVstore (mongoDB) version to 4.2.

I believe that I only need to upgrade the KVstore on instances with active KVstore collections.   And I should stop scheduled searches writing to collections or wait until they are finished before running the migration.

By default the KVstore is enabled on all instances which is a bit confusing, BUT the monitoring console (MC)  KVstore dashboard gives me some indication that only the instances with active kvstore collections are the SHC and some HFs...  but not my MC/license master or the Deployment Server or the Index Cluster master.

Splunk docs mention that you can disable kvstore and ignore certain collections.

Glasses2_0-1678232807360.png

My question is how to verify and identify the active collections on my instances?
Where is the kvstore files location? (apparently the directory seems to be /opt/splunk/var/lib/splunk/kvstore/mongo)  but the files are not human-readable clear... 

Is there a query to see the individual KVstore collection file sizes with human-readable names?

This gives me I think most of them... not sure

| rest /servicesNS/-/-/data/transforms/lookups  splunk_server=local 
| table eai:acl.app title collection 
| search collection!=""

Any advice appreciated.

Thank you

 

Labels (1)
Tags (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Indexers definitely do not use KVStore.  If you have collections on your indexers then you have other problems.  😀  Disabling KVStore on indexers isn't a bad idea and can help save resources.

I'm not sure why the MC and DS have collections (could be "goofy stuff"), but it can't hurt to upgrade them.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

Please don't upgrade only the instances with active collections because the next day an un-upgraded instance will want to load a collection and fail.

Upgrade all instances that can have a collection - search heads and heavy forwarders.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Glasses2
Communicator

Thank you for the response. 

You answered me before in another post >> https://community.splunk.com/t5/Deployment-Architecture/In-a-distributed-and-clustered-environ-which...

 upgrade all nodes. 

So I presume there is no harm upgrading  a kvstore engine and version on an instance w/ or w/out any active collections?

I am scrambled and support won't even give me a definite answer.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

There is never a need to upgrade the KVStore on indexers because indexers do not use the KVStore.  Upgrade only Search Heads and Heavy Forwarders.

The determining factor is not if the instance has a collection - it's if the instance has a running KVStore.

Support won't help because this is not a break/fix matter.

---
If this reply helps you, Karma would be appreciated.
0 Karma

Glasses2
Communicator

Thanks for the clarification (and persevering thru my questions).  I really do appreciate it!

My concerns, re: IDXCluster and KVstore, are due to all the non-sensical configurations the previous admins left for me....

It would not surprise me to see something unusual here, as I have seen some really goofy stuff.   For instance , I am concerned they did some sort of remote kvstore replication/sync to the indexer peers etc... maybe that doesn't matter, not 100% about that.

Would you advise that I "disable" the KVstore on the indexers? for safety? per Splunk docs>>> https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/AboutKVstore

Glasses2_0-1678285929089.png

 

RE: "upgrade only Search Heads and Heavy Forwarders" ,  I see collections (probably default) on my MC/LM instance and Deployment Server instance as well.  So I am thinking upgrade those as well? IDK.

when I run this on my MC/LM instance >>> 

 

 

 

| rest splunk_server="local" "/services/search/distributed/peers/" 
| search disabled=0 
| rename peerName AS splunk_server 
| fields host splunk_server 
| join type=left splunk_server 
    [| rest splunk_server="*" "/services/kvstore/status" 
    | table splunk_server current.status current.replicationStatus current.storageEngine ] 
| sort - current.storageEngine

 

 

 

 

I see every instance (including indexers)  has a default enabled kvstore in "ready"... 

So now, knowing this information, does that change your advice?

RE: support, IDK, I do feel support should be able to answer these questions.  I get support is "break/fix" but do I really have to break it first to get help or a couple definitive answers?

I am sure Support will kick it to our Sales Rep and they will say we need to engage PS for 2 week$ 🤣...  

Thank you!

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Indexers definitely do not use KVStore.  If you have collections on your indexers then you have other problems.  😀  Disabling KVStore on indexers isn't a bad idea and can help save resources.

I'm not sure why the MC and DS have collections (could be "goofy stuff"), but it can't hurt to upgrade them.

---
If this reply helps you, Karma would be appreciated.

Glasses2
Communicator

POA

1) disable KVstore on the indexer instances only

2) update KVstore all other Splunk instances with a "ready" enabled KVstore to be safe.  **In my case, all instances are default enabled **

Hopefully everything goes well! I will post if it goes sideways.

0 Karma

Glasses2
Communicator

Well, in line with your recommendations...

I am being advised by Internal Splunk Personnel, to upgrade everything except indexers...

My feeling is to disable the kvstore on the indexers but not the other nodes like IDXcluster master etc...

Would you agree? (you can conclude that I value your opinion with equal weight as support 😉, maybe more)

 

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...