Kvstore taking more space

[nawazns5038]

Hi,

/opt/splunk/var/lib/splunk/kvstore/mongo is getting occupying more space in the search heads. How can we reduce the space ?
It is around 25 GB. Is it normal ?

Thanks

[nawazns5038 · Accepted solution]

Problem was solved by relocating the KVstore to a different location.

This isn’t something you can push out via the deployer because the kvstore files must be manually moved. If you can deal with a few minutes of downtime on the SH I would recommend the following:

1) Stop all of the search heads.
2) Move the kvstore directory to a new volume.
3) Modify the server.conf file to point to the new location.
4) Start all of the search heads.

[nnimbe1]

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance

[hdbang_splunk]

Splunk will create one database per collection and one db file will have one collection.
(collections are specified in collections.conf)

MongoDB Documents are restricted to 16MB in size.
Thus each Splunk collection will have two db files (.0 and .ns) in 16MB in size respectively

The db path is defined in server.conf.
[kvstore]
dbPath = $SPLUNK_DB/kvstore

Hope this helps.

[hdbang_splunk]

You can change dbPath in local/server.conf.
Here is another answers
https://answers.splunk.com/answers/389298/relocate-kvstore-files-on-filesystem.html.

[spodda01da]

I checked the server.conf on the following path

I did not find any reference of kvstore under the path ./etc/system/local/server.conf

I did find the default path dbPath = $SPLUNK_DB/kvstore at ./etc/system/default/server.conf dbPath

The kvstore is consuming around 40GB of space which as the default location. I need to relocate it to another location which enough space and need some help on how do you do it in a cluster.

[p_gurav]

What splunk version are you using?

[nawazns5038]

6.5.3 version

[spodda01da]

Was this issue resolved ? Can you please share the details.

[nawazns5038]

Yes, we have relocated KVstore to another drive.

[spodda01da]

Great, can you please suggest the command/process to get it done. Since my kvstore has reached to 40GB, I might also have to do the same without breaking or losing any data.

[nawazns5038]

I have posted the steps involved. you can accept it or make it +1 so that others would take help of it.

[spodda01da]

Thanks, I will work on my environment and confirm.