Archive

upgrade to 7.2 fails with "ERROR while running mongod-fix-voting-priority migration."

Splunk Employee
Splunk Employee

Splunk start produces error below and stopped running. So far it failed on 2 servers in a row.
Any suggestions?


Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
license with stackid=enterprise found in stack download-trial
license with stackid=enterprise found in stack download-trial
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 14, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.

ERROR while running mongod-fix-voting-priority migration.

There is no splunkd.log messages put in the file since the upgrade and even 'splunk diag' fails with the same error.

Tags (1)
1 Solution

Splunk Employee
Splunk Employee

We have been using default certs and were able to work around it by re-create the expired server.pem by following the steps below;

$ splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n SplunkServerDefaultCert
$ mv server.pem server.pem.orig
$ mv SplunkServerDefaultCert.pem server.pem

The Mongod.log has had error messages against expired server.pem file since last year so this was the last thing we suspected - some changes in 7.2 might have started to cause the behavior.

To check the expire date;
$ openssl x509 -noout -enddate -in $SPLUNK_HOME/etc/auth/server.pem

If it is still the same then check the below;
i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
ii) expired cert (KV store was not used at all)

iii) unsupported ciphers
iv) mongod port taken by other process,

View solution in original post

Splunk Employee
Splunk Employee

I had a similar case where the client himself found the solution, he performed the below steps and it fixed the issue.

While upgrading from 6.6 to 7.2.3 if you are not able to start the Splunk and if it keeps on prompting to accept license follow the below steps

Go to /opt/splunk
mv ftr /tmp
cd /etc/init.d
./splunk start

If the above steps are followed we can get rid of MongoDB error and we can upgrade to the latest version.

0 Karma

Splunk Employee
Splunk Employee

We have been using default certs and were able to work around it by re-create the expired server.pem by following the steps below;

$ splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n SplunkServerDefaultCert
$ mv server.pem server.pem.orig
$ mv SplunkServerDefaultCert.pem server.pem

The Mongod.log has had error messages against expired server.pem file since last year so this was the last thing we suspected - some changes in 7.2 might have started to cause the behavior.

To check the expire date;
$ openssl x509 -noout -enddate -in $SPLUNK_HOME/etc/auth/server.pem

If it is still the same then check the below;
i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
ii) expired cert (KV store was not used at all)

iii) unsupported ciphers
iv) mongod port taken by other process,

View solution in original post

Path Finder

I also had this same error when upgrading from Splunk 6.5 to 7.3.2. I tried Sylim's fixes and they didn't seem to help. But then I found the following solution which fixed it immidately (though I have a feeling I would have needed to have applied Sylim's fix also):

The permissions on $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key were -rw-rw----
I've seen it documented somewhere that this file must be set to owner read only (-r--------)
I ran the following to fix: chmod -R 400 $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/splunk.key
Then re-ran the Splunk upgrade successfully.

Path Finder

Worked like a charm. Thank you @marrette

0 Karma

Loves-to-Learn

Mine was a permission issue, the above steps from Marrette fixed our issue and I was able to upgrade and restart my IDX.

PS: In my environment, our "splunk.key" was located @ /splunk/kvstore/mongo/splunk.key.

Thank you.

0 Karma

Explorer

I have encountered the same ERROR when I tried to upgrade my cluster from 7.1.4 to 7.1.5.
I have a two-site indexer-cluster. Several indexers in site2 produced "mongod exited abnormally (exit code 14, ..." and "ERROR: Failed to running mongod-fix-voting-priority migration." during starting procedure.
These indexers have one thing in common. The modtime of the server.pem is in May 2016, and now is in Jan 2019.
I tried to create new server.pem with commands provided by your answer and it worked.
After I replaced the old server.pem file with the new one, Splunk started normally.

Communicator

Hi Sylim,
i also faced the same issue. i followed the above steps but the issue still persists.
[root@localhost ~]# openssl x509 -noout -enddate -in /opt/splunk/etc/auth/server.pem
notAfter=Jan 9 08:43:08 2022 GMT

the above is my output after creating new certificate. so when i am trying to run this command ./splunk start --accept-license ,the output is below:-
ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
ERROR while running mongod-fix-voting-priority migration.

can you please help me out here?

0 Karma

Splunk Employee
Splunk Employee

Could you check the below?

 i) delete old mongod.lock file in $SPLUNK_HOME/var/lib/splunk/kvstore/mongo
 ii) expired cert (KV store was not used at all)

 iii) unsupported ciphers 
 iv) mongod port taken by other process,

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!