Knowledge Management
Highlighted

In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

Hello,

I am giving the Splunk Enterprise 7.1.3 to 7.2.0 upgrade a try in my test environment, and I am currently stuck on the Search Cluster upgrade. I first attempted to do one node at a time, which failed, and then took the entire search cluster offline to do the upgrade. Now, I cannot get the Splunk service to start back up and am getting the following error message.

The Search Deployer had a similar issue with the upgrade, but it was resolved with a simple reboot of the instance. I tired the same thing on the search node, along with killing one leftover mongodb process, neither helped.

I also attempted to run the 'splunk migrate migrate-kvstore' command based on other Splunk Answers posts, which also failed with the same reason.

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade. 

"/opt/splunk/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal
"/opt/splunk/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal
Clustering migration already complete, no further changes required.

Generating checksums for datamodel and report acceleration bucket summaries for all indexes.
If you have defined many indexes and summaries, summary checksum generation may take a long time.
Processed 2 out of 22 configured indexes.
Processed 4 out of 22 configured indexes.
Processed 6 out of 22 configured indexes.
Processed 8 out of 22 configured indexes.
Processed 10 out of 22 configured indexes.
Processed 12 out of 22 configured indexes.
Processed 14 out of 22 configured indexes.
Processed 16 out of 22 configured indexes.
Processed 18 out of 22 configured indexes.
Processed 20 out of 22 configured indexes.
Processed 22 out of 22 configured indexes.
Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
ERROR: Failed to migrate mongo feature compatibility version:
ERROR while running migrate-kvstore migration.

I looked in the splunkd.log and mongo.log files, but there are no new events that have been created since I shutdown the service prior to starting the 'rpm' upgrade. They both end with the related shutdown event as shown below.

[root@ip-10-2-31-134 ~]# tail -n 10 /opt/splunk/var/log/splunk/splunkd.log
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Queue"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_CallbackRunner"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_HttpClient"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_DmcProxyHttpClient"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_Duo2FAHttpClient"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_ApplicationLicenseChecker"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_S3ConnectionPoolManager"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - shutting down level "ShutdownLevel_TelemetryMetricBuffer"
10-11-2018 18:48:34.583 +0000 INFO  ShutdownHandler - Shutdown complete in 36.05 seconds
10-11-2018 18:48:35.581 +0000 INFO  loader - All pipelines finished.

[root@ip-10-2-31-134 ~]# tail -n 10 /opt/splunk/var/log/splunk/mongod.log
 2018-10-11T18:48:02.886Z I JOURNAL  [signalProcessingThread] old journal file /opt/splunk/var/lib/splunk/kvstore/mongo/journal/j._0 will be reused as /opt/splunk/var/lib/splunk/kvstore/mongo/journal/prealloc.0
 2018-10-11T18:48:02.887Z I JOURNAL  [signalProcessingThread] Terminating durability thread ...
 2018-10-11T18:48:02.986Z I JOURNAL  [journal writer] Journal writer thread stopped
 2018-10-11T18:48:02.986Z I JOURNAL  [durability] Durability thread stopped
 2018-10-11T18:48:02.986Z I STORAGE  [signalProcessingThread] shutdown: closing all files...
 2018-10-11T18:48:02.986Z I STORAGE  [signalProcessingThread] closeAllFiles() finished
 2018-10-11T18:48:02.986Z I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
 2018-10-11T18:48:02.986Z I CONTROL  [signalProcessingThread] now exiting
 2018-10-11T18:48:02.986Z I CONTROL  [signalProcessingThread] shutting down with code:0
 2018-10-11T18:48:02.986Z I CONTROL  [initandlisten] shutting down with code:0

Thanks,
Erik

Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

Not an answer but I have the same issue upgrading from 7.1.1 to 7.2

Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
ERROR: Failed to migrate mongo feature compatibility version:
ERROR while running migrate-kvstore migration.

and again nothing in the logs. Reboot did not fix it.

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

I opened up a Enterprise Support ticket for this issue and will update this post when we have figured out the issues with the fix.

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

@capilarity, would you mind posting what Splunk Base apps you have installed on your Splunk instance that also failed the upgrade from 7.1.1 to 7.2.0, along with your setup (Search Cluster, Indexer Cluster, etc.)?

Splunk Support and I are trying to track down the issues and I recently found that our base installs, nothing special setup, upgrades from v6.6.5 to v7.1.3 to v7.2.0 and from v6.6.5 straight to v7.2.0 without issues. I am thinking there is an issue with the specific settings on each instance to make them a Search Cluster, Indexer Cluster, Deployer, Master Node, etc. and/or there is an issue with one of the Splunk Base apps that is installed in my Splunk environment that is having issues with the upgrade.

I would like to provide them a comparison of two setups having the same issues. Thanks!

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

We have no splunk base apps installled on this instance, and only one home grown app that monitors changes to the config.
This is our master node, in addtion we have two indexers in site 1 site 2 configuration, two non clustered search head and deployment server all seperate.

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

Thanks, I will relay the information to Splunk Support.

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

Thanks all, this has now been resolved with the help of support. We had defined tls 1.2 for our splunk to splunk comms and this was forcing mongodb to use the same.
hash out the config, upgrade completed fine.
Once upgrade complete, revert to original tls configuration and restart.

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Engager

Can you indicate what specific changes you made to which specific files to resolve the problem?

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

Hi,
we are having the same issues - a support case is opened, but now result until now.

Alex

0 Karma
Highlighted

Re: In upgrading Splunk Enterprise from 7.1.3 to 7.2.0, why is the Mongo Migration failing?

Path Finder

We already updated your Linux Kernel to Version
Linux 4.4.156-94.57-default #1 SMP Tue Oct 2 06:43:37 UTC 2018 (82521a6) x8664 x8664 x86_64 GNU/Linux
but the segfault is still going on with that version.

0 Karma