How to clean KVstore on search head cluster nodes

[Splunk_rocks]

Happy Splunking,

We have a situation on our search head cluster nodes and one of the peer node KVstore is filling up, so not sure we can clean manually or is there any process to backup and clean?
What are the limitations if we clean all the data?

/opt/splunk/var/lib/splunk/kvstore -190GB

[badrinath_itrs]

You need to identify which KVStore collection is taking lot of space and decide if that is required or not . Don't just remove all KVStore collections in your SHC environment unless you understand the impact.

I would start with Checking the KVStore collection size in DMC and find out which collections are taking more size and go from there.

If you need to remove one collection individually in SHC, you can use | outputlookup

to remove it.

If you decide to backup your kvstore and restore it later you can refer below documentation .

https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/BackupKVstore

To delete the KVStore on members you can refer below documentation.

Refer documentation https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/ResyncKVstore

[Splunk_rocks]

I have found way to clean KV store app and collection

You can run following command to see which app and collection taking more space

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "(?.).(?.)"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

Then based on size you can use below command to clean as your required on nodes

/opt/splunk/bin/splunk clean kvstore -app -collection

I hope this will help folks

[diconium]

@Splunk_rocks  Your search isn't completely working as the rex command it doesn't extract the values. So the search should look as follows:

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "^(?<App>.+)\.(?<Collection>.+)$"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

[starcher]

https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/BackupKVstore