Splunk is not starting up when performing an upgrade to Splunk 7.1.6 on the indexer.
Error message when starting up Splunk:
mongod exited abnormally(exit code 100, status: exited with code 100) - look at mogod.log to investigate.
ERROR: Failed to fix non-voting members Priority value during mogod version migration from 3.0.x to 3.4.x.
Reason: ERROR while running mogod-fix-voting-priority migration.
KVStore and MongoDB was down before the upgrade was initiated.
Repair the KV Store to bring up MongoDB and restart Splunk:
1. Backup $SPLUNK_DB/kvstore/mongo/.
2. Execute: $SPLUNK_HOME/bin/splunk clean kvstore --local
3. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal
4. Manually remove $SPLUNK_DB/kvstore/mongo/mongod.lock
5. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 --repair
6. Restart Splunk after the KV Store is repaired.
KVStore and MongoDB was down before the upgrade was initiated.
Repair the KV Store to bring up MongoDB and restart Splunk:
1. Backup $SPLUNK_DB/kvstore/mongo/.
2. Execute: $SPLUNK_HOME/bin/splunk clean kvstore --local
3. Manually delete folder: $SPLUNK_DB/kvstore/mongo/journal
4. Manually remove $SPLUNK_DB/kvstore/mongo/mongod.lock
5. Run: $SPLUNK_HOME/bin/mongod --dbpath $SPLUNK_DB/kvstore/mongo --port=8191 --repair
6. Restart Splunk after the KV Store is repaired.