topic upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration." in Deployment Architecture

upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

sylim_splunk — Tue, 08 Jan 2019 19:33:24 GMT

Splunk start produces error below and stopped running. So far it failed on 2 servers in a row.
Any suggestions?

Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
license with stackid=enterprise found in stack download-trial
license with stackid=enterprise found in stack download-trial
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 14, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.

ERROR while running mongod-fix-voting-priority migration.

There is no splunkd.log messages put in the file since the upgrade and even 'splunk diag' fails with the same error.

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

sylim_splunk — Tue, 08 Jan 2019 19:40:42 GMT

We have been using default certs and were able to work around it by re-create the expired server.pem by following the steps below;

$ splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n SplunkServerDefaultCert
$ mv server.pem server.pem.orig
$ mv SplunkServerDefaultCert.pem server.pem

The Mongod.log has had error messages against expired server.pem file since last year so this was the last thing we suspected - some changes in 7.2 might have started to cause the behavior.

To check the expire date;
$ openssl x509 -noout -enddate -in $SPLUNK_HOME/etc/auth/server.pem

If it is still the same then check the below;
i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
ii) expired cert (KV store was not used at all)

iii) unsupported ciphers
iv) mongod port taken by other process,

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

MoniM — Thu, 10 Jan 2019 08:53:12 GMT

Hi Sylim,
i also faced the same issue. i followed the above steps but the issue still persists.
[root@localhost ~]# openssl x509 -noout -enddate -in /opt/splunk/etc/auth/server.pem
notAfter=Jan 9 08:43:08 2022 GMT
the above is my output after creating new certificate. so when i am trying to run this command ./splunk start --accept-license ,the output is below:-
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
ERROR while running mongod-fix-voting-priority migration.

can you please help me out here?

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

sylim_splunk — Thu, 10 Jan 2019 19:34:18 GMT

Could you check the below?

i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
ii) expired cert (KV store was not used at all)

iii) unsupported ciphers
iv) mongod port taken by other process,

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

witski — Tue, 15 Jan 2019 16:12:15 GMT

I have encountered the same ERROR when I tried to upgrade my cluster from 7.1.4 to 7.1.5.
I have a two-site indexer-cluster. Several indexers in site2 produced "mongod exited abnormally (exit code 14, ..." and "ERROR: Failed to running mongod-fix-voting-priority migration." during starting procedure.
These indexers have one thing in common. The modtime of the server.pem is in May 2016, and now is in Jan 2019.
I tried to create new server.pem with commands provided by your answer and it worked.
After I replaced the old server.pem file with the new one, Splunk started normally.

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

marrette — Sat, 06 Apr 2019 05:05:58 GMT

I also had this same error when upgrading from Splunk 6.5 to 7.3.2. I tried Sylim's fixes and they didn't seem to help. But then I found the following solution which fixed it immidately (though I have a feeling I would have needed to have applied Sylim's fix also):

The permissions on $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key were -rw-rw----
I've seen it documented somewhere that this file must be set to owner read only (-r--------)
I ran the following to fix: chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
Then re-ran the Splunk upgrade successfully.

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

333mendyp — Tue, 23 Jul 2019 17:57:30 GMT

Mine was a permission issue, the above steps from Marrette fixed our issue and I was able to upgrade and restart my IDX.

PS: In my environment, our "splunk.key" was located @ /splunk/kvstore/mongo/splunk.key.

Thank you.

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

saramamurthy_sp — Fri, 26 Jul 2019 07:58:13 GMT

I had a similar case where the client himself found the solution, he performed the below steps and it fixed the issue.

While upgrading from 6.6 to 7.2.3 if you are not able to start the Splunk and if it keeps on prompting to accept license follow the below steps

Go to /opt/splunk
mv ftr /tmp
cd /etc/init.d
./splunk start

If the above steps are followed we can get rid of MongoDB error and we can upgrade to the latest version.

Re: upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

arrangineni — Wed, 20 Nov 2019 22:44:09 GMT

Worked like a charm. Thank you @marrette