Why are my 3 search heads in a search head clustering environment filling up the directory "/opt/splunk/var/lib/splunk/kvstore/mongo"?

[harrymclaren]

I'm currently building out a Splunk environment and could do with some help.

The three search heads (clustered) are all filling up the following directory /opt/splunk/var/lib/splunk/kvstore/mongo. I haven’t configured anything to do with kvstore or mongo as far as I know.

Searching via documentation, I can't see why this would be the case.

Help is appreciated.

[harrymclaren]

Removed the local. files and the boxes came back up.

Still not sure what is filling up these mongo data files.

Anyone got an idea? They are setup as a Search Head Cluster.

[alacercogitatus]

Do you have ES or PCI? Those use KVStore. Check for any collections.conf files in all the directories to see if any kvstores are configured. You may also want to drill down in that folder to see what actually is the cause. There could be a configuration error that is causing the mongodb to spit errors, and that could be filling up the folder.

[harrymclaren]

No apps are installed.

Files are:
local.0 (.1, .2 , .ns)

The are no conf files in the directory (/opt/splunk/var/lib/splunk/kvstore/mongo)

That SH now won't start as displays the error:
Operation "fclose" failed in /home/build/build-src/6.2.3/src/libzero/conf-mutator-locking.c:336, conf_mutator_lock(); No space left on device.

What are those 'local.' files used for? I don't even have any data inputs configured yet, only thing that has been done is, standard install, configure connection to license server, configure cluster and configure LDAP for login.

Any help is appreciated, thanks.