topic Re: How to clean KVstore on search head cluster nodes in Knowledge Management

How to clean KVstore on search head cluster nodes

Splunk_rocks — Sat, 06 Jun 2020 17:42:24 GMT

Happy Splunking,

We have a situation on our search head cluster nodes and one of the peer node KVstore is filling up, so not sure we can clean manually or is there any process to backup and clean?
What are the limitations if we clean all the data?

/opt/splunk/var/lib/splunk/kvstore -190GB

Re: How to clean KVstore on search head cluster nodes

starcher — Fri, 13 Dec 2019 20:33:29 GMT

https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/BackupKVstore

Re: How to clean KVstore on search head cluster nodes

Splunk_rocks — Wed, 08 Jan 2020 21:17:32 GMT

I have found way to clean KV store app and collection

You can run following command to see which app and collection taking more space

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "(?.).(?.)"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

Then based on size you can use below command to clean as your required on nodes

/opt/splunk/bin/splunk clean kvstore -app -collection

I hope this will help folks

Re: How to clean KVstore on search head cluster nodes

badrinath_itrs — Wed, 08 Jan 2020 21:24:09 GMT

You need to identify which KVStore collection is taking lot of space and decide if that is required or not . Don't just remove all KVStore collections in your SHC environment unless you understand the impact.

I would start with Checking the KVStore collection size in DMC and find out which collections are taking more size and go from there.

If you need to remove one collection individually in SHC, you can use | outputlookup

to remove it.

If you decide to backup your kvstore and restore it later you can refer below documentation .

https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/BackupKVstore

To delete the KVStore on members you can refer below documentation.

Refer documentation https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/ResyncKVstore

Re: How to clean KVstore on search head cluster nodes

diconium — Mon, 18 Jan 2021 10:13:12 GMT

@Splunk_rocks Your search isn't completely working as the rex command it doesn't extract the values. So the search should look as follows:

| rest /services/server/introspection/kvstore/collectionstats | mvexpand data | spath input=data | rex field=ns "^(?<App>.+)\.(?<Collection>.+)$" | eval dbsize=round(size/1024/1024, 2) | eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2) | stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection