Deployment Architecture

Why are my 3 search heads in a search head clustering environment filling up the directory "/opt/splunk/var/lib/splunk/kvstore/mongo"?

harrymclaren
Explorer

I'm currently building out a Splunk environment and could do with some help.

The three search heads (clustered) are all filling up the following directory /opt/splunk/var/lib/splunk/kvstore/mongo. I haven’t configured anything to do with kvstore or mongo as far as I know.

Searching via documentation, I can't see why this would be the case.

Help is appreciated.

harrymclaren
Explorer

Removed the local. files and the boxes came back up.

Still not sure what is filling up these mongo data files.

Anyone got an idea? They are setup as a Search Head Cluster.

0 Karma

alacercogitatus
SplunkTrust
SplunkTrust

Do you have ES or PCI? Those use KVStore. Check for any collections.conf files in all the directories to see if any kvstores are configured. You may also want to drill down in that folder to see what actually is the cause. There could be a configuration error that is causing the mongodb to spit errors, and that could be filling up the folder.

0 Karma

harrymclaren
Explorer

No apps are installed.

Files are:
local.0 (.1, .2 , .ns)

The are no conf files in the directory (/opt/splunk/var/lib/splunk/kvstore/mongo)

That SH now won't start as displays the error:
Operation "fclose" failed in /home/build/build-src/6.2.3/src/libzero/conf-mutator-locking.c:336, conf_mutator_lock(); No space left on device.

What are those 'local.' files used for? I don't even have any data inputs configured yet, only thing that has been done is, standard install, configure connection to license server, configure cluster and configure LDAP for login.

Any help is appreciated, thanks.

0 Karma
Get Updates on the Splunk Community!

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...