Splunk is not coming up due to MongoDB exit code 100

[keio_splunk]

Splunk is not starting up when performing an upgrade to Splunk 7.1.6 on the indexer.

Error message when starting up Splunk:
mongod exited abnormally(exit code 100, status: exited with code 100) - look at mogod.log to investigate. 
ERROR: Failed to fix non-voting members Priority value during mogod version migration from 3.0.x to 3.4.x.
Reason: ERROR while running mogod-fix-voting-priority migration. 

[keio_splunk · Accepted solution]

KVStore and MongoDB was down before the upgrade was initiated.

Repair the KV Store to bring up MongoDB and restart Splunk:
1. Backup $SPLUNK_DB/kvstore/mongo/. 
2. Execute: $SPLUNK_HOME/bin/splunk clean kvstore --local
3. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal 
4. Manually remove $SPLUNK_DB/kvstore/mongo/mongod.lock 
5. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 --repair 
6. Restart Splunk after the KV Store is repaired.