topic Re: Kvstore taking more space in Splunk Dev

Kvstore taking more space

nawazns5038 — Tue, 08 May 2018 20:56:57 GMT

Hi,

/opt/splunk/var/lib/splunk/kvstore/mongo is getting occupying more space in the search heads. How can we reduce the space ?
It is around 25 GB. Is it normal ?

Thanks

Re: Kvstore taking more space

p_gurav — Wed, 09 May 2018 13:59:04 GMT

What splunk version are you using?

Re: Kvstore taking more space

nawazns5038 — Thu, 10 May 2018 18:16:31 GMT

6.5.3 version

Re: Kvstore taking more space

spodda01da — Thu, 17 Jan 2019 09:39:13 GMT

Was this issue resolved ? Can you please share the details.

Re: Kvstore taking more space

nawazns5038 — Thu, 17 Jan 2019 22:07:33 GMT

Yes, we have relocated KVstore to another drive.

Re: Kvstore taking more space

spodda01da — Fri, 18 Jan 2019 03:42:47 GMT

Great, can you please suggest the command/process to get it done. Since my kvstore has reached to 40GB, I might also have to do the same without breaking or losing any data.

Re: Kvstore taking more space

hdbang_splunk — Fri, 18 Jan 2019 04:29:49 GMT

Splunk will create one database per collection and one db file will have one collection.
(collections are specified in collections.conf)

MongoDB Documents are restricted to 16MB in size.
Thus each Splunk collection will have two db files (.0 and .ns) in 16MB in size respectively

The db path is defined in server.conf.
[kvstore]
dbPath = $SPLUNK_DB/kvstore

Hope this helps.

Re: Kvstore taking more space

spodda01da — Fri, 18 Jan 2019 08:13:30 GMT

I checked the server.conf on the following path

I did not find any reference of kvstore under the path ./etc/system/local/server.conf

I did find the default path dbPath = $SPLUNK_DB/kvstore at ./etc/system/default/server.conf dbPath

The kvstore is consuming around 40GB of space which as the default location. I need to relocate it to another location which enough space and need some help on how do you do it in a cluster.

Re: Kvstore taking more space

hdbang_splunk — Fri, 18 Jan 2019 11:41:56 GMT

You can change dbPath in local/server.conf.
Here is another answers
https://answers.splunk.com/answers/389298/relocate-kvstore-files-on-filesystem.html.

Re: Kvstore taking more space

nawazns5038 — Fri, 18 Jan 2019 22:35:40 GMT

Problem was solved by relocating the KVstore to a different location.

This isn’t something you can push out via the deployer because the kvstore files must be manually moved. If you can deal with a few minutes of downtime on the SH I would recommend the following:

1) Stop all of the search heads.
2) Move the kvstore directory to a new volume.
3) Modify the server.conf file to point to the new location.
4) Start all of the search heads.

Re: Kvstore taking more space

nawazns5038 — Fri, 18 Jan 2019 22:37:04 GMT

I have posted the steps involved. you can accept it or make it +1 so that others would take help of it.

Re: Kvstore taking more space

spodda01da — Mon, 21 Jan 2019 04:18:52 GMT

Thanks, I will work on my environment and confirm.

Re: Kvstore taking more space

nnimbe1 — Mon, 10 Jan 2022 14:02:42 GMT

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance