Archive

Kvstore taking more space

nawazns5038
Builder

Hi,

/opt/splunk/var/lib/splunk/kvstore/mongo is getting occupying more space in the search heads. How can we reduce the space ?
It is around 25 GB. Is it normal ?

Thanks

Tags (2)
1 Solution

nawazns5038
Builder

Problem was solved by relocating the KVstore to a different location.

This isn’t something you can push out via the deployer because the kvstore files must be manually moved. If you can deal with a few minutes of downtime on the SH I would recommend the following:

1) Stop all of the search heads.
2) Move the kvstore directory to a new volume.
3) Modify the server.conf file to point to the new location.
4) Start all of the search heads.

View solution in original post

0 Karma

nawazns5038
Builder

Problem was solved by relocating the KVstore to a different location.

This isn’t something you can push out via the deployer because the kvstore files must be manually moved. If you can deal with a few minutes of downtime on the SH I would recommend the following:

1) Stop all of the search heads.
2) Move the kvstore directory to a new volume.
3) Modify the server.conf file to point to the new location.
4) Start all of the search heads.

View solution in original post

0 Karma

hdbang_splunk
Splunk Employee
Splunk Employee

Splunk will create one database per collection and one db file will have one collection.
(collections are specified in collections.conf)

MongoDB Documents are restricted to 16MB in size.
Thus each Splunk collection will have two db files (.0 and .ns) in 16MB in size respectively

The db path is defined in server.conf.
[kvstore]
dbPath = $SPLUNK_DB/kvstore

Hope this helps.

0 Karma

hdbang_splunk
Splunk Employee
Splunk Employee

You can change dbPath in local/server.conf.
Here is another answers
https://answers.splunk.com/answers/389298/relocate-kvstore-files-on-filesystem.html.

0 Karma

spodda01da
Explorer

I checked the server.conf on the following path

I did not find any reference of kvstore under the path ./etc/system/local/server.conf

I did find the default path dbPath = $SPLUNK_DB/kvstore at ./etc/system/default/server.conf dbPath

The kvstore is consuming around 40GB of space which as the default location. I need to relocate it to another location which enough space and need some help on how do you do it in a cluster.

0 Karma

p_gurav
Champion

What splunk version are you using?

0 Karma

nawazns5038
Builder

6.5.3 version

0 Karma

spodda01da
Explorer

Was this issue resolved ? Can you please share the details.

0 Karma

nawazns5038
Builder

Yes, we have relocated KVstore to another drive.

0 Karma

spodda01da
Explorer

Great, can you please suggest the command/process to get it done. Since my kvstore has reached to 40GB, I might also have to do the same without breaking or losing any data.

0 Karma

nawazns5038
Builder

I have posted the steps involved. you can accept it or make it +1 so that others would take help of it.

0 Karma

spodda01da
Explorer

Thanks, I will work on my environment and confirm.

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!