Splunk Dev

Kvstore taking more space

nawazns5038
Builder

Hi,

/opt/splunk/var/lib/splunk/kvstore/mongo is getting occupying more space in the search heads. How can we reduce the space ?
It is around 25 GB. Is it normal ?

Thanks

Tags (2)
1 Solution

nawazns5038
Builder

Problem was solved by relocating the KVstore to a different location.

This isn’t something you can push out via the deployer because the kvstore files must be manually moved. If you can deal with a few minutes of downtime on the SH I would recommend the following:

1) Stop all of the search heads.
2) Move the kvstore directory to a new volume.
3) Modify the server.conf file to point to the new location.
4) Start all of the search heads.

View solution in original post

0 Karma

nawazns5038
Builder

Problem was solved by relocating the KVstore to a different location.

This isn’t something you can push out via the deployer because the kvstore files must be manually moved. If you can deal with a few minutes of downtime on the SH I would recommend the following:

1) Stop all of the search heads.
2) Move the kvstore directory to a new volume.
3) Modify the server.conf file to point to the new location.
4) Start all of the search heads.

0 Karma

nnimbe1
Path Finder

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance

0 Karma

hdbang_splunk
Splunk Employee
Splunk Employee

Splunk will create one database per collection and one db file will have one collection.
(collections are specified in collections.conf)

MongoDB Documents are restricted to 16MB in size.
Thus each Splunk collection will have two db files (.0 and .ns) in 16MB in size respectively

The db path is defined in server.conf.
[kvstore]
dbPath = $SPLUNK_DB/kvstore

Hope this helps.

0 Karma

hdbang_splunk
Splunk Employee
Splunk Employee

You can change dbPath in local/server.conf.
Here is another answers
https://answers.splunk.com/answers/389298/relocate-kvstore-files-on-filesystem.html.

0 Karma

spodda01da
Path Finder

I checked the server.conf on the following path

I did not find any reference of kvstore under the path ./etc/system/local/server.conf

I did find the default path dbPath = $SPLUNK_DB/kvstore at ./etc/system/default/server.conf dbPath

The kvstore is consuming around 40GB of space which as the default location. I need to relocate it to another location which enough space and need some help on how do you do it in a cluster.

0 Karma

p_gurav
Champion

What splunk version are you using?

0 Karma

nawazns5038
Builder

6.5.3 version

0 Karma

spodda01da
Path Finder

Was this issue resolved ? Can you please share the details.

0 Karma

nawazns5038
Builder

Yes, we have relocated KVstore to another drive.

0 Karma

spodda01da
Path Finder

Great, can you please suggest the command/process to get it done. Since my kvstore has reached to 40GB, I might also have to do the same without breaking or losing any data.

0 Karma

nawazns5038
Builder

I have posted the steps involved. you can accept it or make it +1 so that others would take help of it.

0 Karma

spodda01da
Path Finder

Thanks, I will work on my environment and confirm.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...