Upgrade from 7.2.1 to 7.2.3 fails

[rapheal]

When I run the installer it fails and rolls back

I see errors about MongoDb lockfile

2019-01-14T18:45:28.291Z W - [initandlisten] Detected unclean shutdown - /DB/kvstore/mongo/mongo\mongod.lock is not empty.
2019-01-14T18:45:28.320Z I STORAGE [initandlisten] **************
old lock file: /DB/kvstore/mongo/mongod.lock. probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
please make sure that your journal directory is mounted.
found 7 dbs.
see: http://dochub.mongodb.org/core/repair for more information

I ran ./mongod --dbpath /DB/kvstore/mongo it repaired and emptied the mongod.lock file. But then when I run the installer again I get the same error.

When it fails I can manually start the splunkd service and it runs 7.2.1

I also see in the migration.log.2019-01-14.16-32-00
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 100, waited for 0 seconds.
ERROR while running mongod-fix-voting-priority migration.

[nikita_p]

Hi,
This can be permissions issue or you may also have to delete the mongod.lock file under $SPLUNKHOME/var/lib/splunk/kvstore/mongo/ and run the mongod --repair command to as well
PFB link in splunk answers, this might help you:
https://answers.splunk.com/answers/689196/can-you-help-me-with-the-following-kv-store-error.html

Let me know if this helps.

[dkeck]

HI,

did you try the solution mentioned here?

https://answers.splunk.com/answers/689196/can-you-help-me-with-the-following-kv-store-error.html

OR try to open a case with splunk for that.

Kind Regards