We have a six search-heads cluster and kv store is frequently crashing on one of them. The error we get is "KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details". And when we look at the mongod.log, we had noticed the following disk full error: 2019-07-10T14:32:21.049Z I CONTROL [journal writer] LogFile::synchronousAppend failed with 8192 bytes unwritten out of 8192 bytes; b=0x55ebc4348000 errno:28 No space left on device
2019-07-10T14:32:21.050Z I - [journal writer] Fatal Assertion 13515
The disk utilization on the host is for the root partition / is at 61%. We did try cleaning up kvstore locally and resyncing it. The servers holds it for around 12 hours after every instance of cleanup and then kvstore crashes again.
Any help in this would be appreciable.
Thanks.
