KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

[omprakash9998]

I have the following message regarding an indexer in my environment (Splunk 6.6.5). :

Search peer indexer has the following message: KV Store process terminated abnormally (exit code 100, status exited with code 100). See mongod.log and splunkd.log for details.

splunkd.log

04-27-2018 13:29:07.622 -0400 WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event (Fri Apr 27 13:29:06 2018). Context: source::/opt/splunk/var/log/splunk/mongod.log|host::olpidx01|mongod|50

mongodb.log

 2018-04-25T18:07:11.837Z W -        [initandlisten] Detected unclean shutdown - /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock is not empty.
 2018-04-25T18:07:11.845Z I STORAGE  [initandlisten]
 2018-04-25T18:07:11.845Z I STORAGE  [initandlisten] ** WARNING: Readahead for /opt/splunk/var/lib/splunk/kvstore/mongo is set to 4096KB
 2018-04-25T18:07:11.845Z I STORAGE  [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
 2018-04-25T18:07:11.845Z I STORAGE  [initandlisten] **          http://dochub.mongodb.org/core/readahead
 2018-04-25T18:07:11.845Z I STORAGE  [initandlisten] **************
 old lock file: /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock.  probably means unclean shutdown,
 but there are no journal files to recover.
 this is likely human error or filesystem corruption.
 please make sure that your journal directory is mounted.
 found 5 dbs.

The permissions on files are also same

I tried this to make sure

 chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key

I even restarted my server and splunk service. Nothing seems to work please help.

The mongod.log says empty jounal. My journal folder is empty on that indexer. Can i copy the files on another indexer and place them there.

[pkiripolsky]

Hey there, have you tried chmod'ing the permissions to 600 and deleting the lock file ( /opt/splunk/var/lib/splunk/kvstore/mongo/mongod.lock )?

In my experience whenever this happens, the solution for me has been to chmod the mongo splunk key to 600, delete the lock file, and then reboot the whole server (not just the splunk service).

Additionally, if you're still having issues check the permissions to make sure that the proper account owns/has access to mongo.