Knowledge Management

How to clean KVstore on search head cluster nodes

Splunk_rocks
Path Finder

Happy Splunking,

We have a situation on our search head cluster nodes and one of the peer node KVstore is filling up, so not sure we can clean manually or is there any process to backup and clean?
What are the limitations if we clean all the data?

/opt/splunk/var/lib/splunk/kvstore -190GB

Labels (1)
0 Karma

badrinath_itrs
Communicator

You need to identify which KVStore collection is taking lot of space and decide if that is required or not . Don't just remove all KVStore collections in your SHC environment unless you understand the impact.

I would start with Checking the KVStore collection size in DMC and find out which collections are taking more size and go from there.

If you need to remove one collection individually in SHC, you can use | outputlookup

to remove it.

If you decide to backup your kvstore and restore it later you can refer below documentation .

https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/BackupKVstore

To delete the KVStore on members you can refer below documentation.

Refer documentation https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/ResyncKVstore

0 Karma

Splunk_rocks
Path Finder

I have found way to clean KV store app and collection

You can run following command to see which app and collection taking more space

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "(?.).(?.)"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

Then based on size you can use below command to clean as your required on nodes

/opt/splunk/bin/splunk clean kvstore -app -collection

I hope this will help folks

0 Karma

diconium
Explorer

@Splunk_rocks  Your search isn't completely working as the rex command it doesn't extract the values. So the search should look as follows:

 

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "^(?<App>.+)\.(?<Collection>.+)$"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

 

0 Karma

starcher
SplunkTrust
SplunkTrust
0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.