Knowledge Management

How to clean KVstore on search head cluster nodes

Splunk_rocks
Path Finder

Happy Splunking,

We have a situation on our search head cluster nodes and one of the peer node KVstore is filling up, so not sure we can clean manually or is there any process to backup and clean?
What are the limitations if we clean all the data?

/opt/splunk/var/lib/splunk/kvstore -190GB

Labels (1)
0 Karma

badrinath_itrs
Communicator

You need to identify which KVStore collection is taking lot of space and decide if that is required or not . Don't just remove all KVStore collections in your SHC environment unless you understand the impact.

I would start with Checking the KVStore collection size in DMC and find out which collections are taking more size and go from there.

If you need to remove one collection individually in SHC, you can use | outputlookup

to remove it.

If you decide to backup your kvstore and restore it later you can refer below documentation .

https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/BackupKVstore

To delete the KVStore on members you can refer below documentation.

Refer documentation https://docs.splunk.com/Documentation/Splunk/8.0.1/Admin/ResyncKVstore

Splunk_rocks
Path Finder

I have found way to clean KV store app and collection

You can run following command to see which app and collection taking more space

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "(?.).(?.)"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

Then based on size you can use below command to clean as your required on nodes

/opt/splunk/bin/splunk clean kvstore -app -collection

I hope this will help folks

0 Karma

diconium
Explorer

@Splunk_rocks  Your search isn't completely working as the rex command it doesn't extract the values. So the search should look as follows:

 

| rest /services/server/introspection/kvstore/collectionstats
| mvexpand data
| spath input=data
| rex field=ns "^(?<App>.+)\.(?<Collection>.+)$"
| eval dbsize=round(size/1024/1024, 2)
| eval indexsize=round(totalIndexSize/1024/1024, 2),dbsizegb=round(dbsize/1024, 2)
| stats first(count) AS "Number of Objects" first(nindexes) AS Accelerations first(indexsize) AS "Acceleration Size (MB)" first(dbsize) AS "Collection Size (MB)" first(dbsizegb) as "Collection Size (GB)" by App, Collection

 

starcher
Influencer
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...