topic Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise in Installation

Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Tue, 29 Sep 2020 23:06:02 GMT

Hello everyone, actually we have Splunk version 6.5.0 and we need upgrade to latest version (7.2.3) but it throws follow error:

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.
"/u01/splunk/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal
"/u01/splunk/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal
Clustering migration already complete, no further changes required.
Generating checksums for datamodel and report acceleration bucket summaries for all indexes.
If you have defined many indexes and summaries, summary checksum generation may take a long time.
Processed 1 out of 8 configured indexes.
Processed 2 out of 8 configured indexes.
Processed 3 out of 8 configured indexes.
Processed 4 out of 8 configured indexes.
Processed 5 out of 8 configured indexes.
Processed 6 out of 8 configured indexes.
Processed 7 out of 8 configured indexes.
Processed 8 out of 8 configured indexes.
Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
**ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
ERROR while running mongod-fix-voting-priority migration.*

Upgrade 6.3.3 to 6.5.0 without problems but now to upgrade to 7.2.3 we have problems.

Thank you in advance

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

maraman_splunk — Wed, 06 Feb 2019 21:12:52 GMT

Hi,

does splunk show kvstore-status from 6.5 says kvstore is fine ?
what is your os with version ?
you could be in one of the case in known issues / Unsorted issues here : https://docs.splunk.com/Documentation/Splunk/7.2.3/ReleaseNotes/Knownissues (especially a very old glibc or the elision one)

any meaningfull info in mongod logs ?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

vishaltaneja070 — Thu, 07 Feb 2019 08:03:49 GMT

@rjfv8205
Check this:
https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/TroubleshootKVstore#KV_store_migration_message

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

maraman_splunk — Fri, 08 Feb 2019 09:18:19 GMT

You may be interested into retrying with 7.2.4
see https://docs.splunk.com/Documentation/Splunk/7.2.4/ReleaseNotes/Fixedissues
Topic Upgrade
there's 1 entry about upgrade failing du to a kvstore migration issue

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

AKG1_old1 — Fri, 08 Feb 2019 09:29:14 GMT

I had the same issue. I was trying to upgrade from 7.0.3 to 7.2.3. I first upgraded to intermidiate version(7.2.2) and then upgraded to 7.2.3. It worked for me.

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Fri, 08 Feb 2019 13:33:41 GMT

Hi @vishaltaneja07011993

Steps mentioned in this topic.

splunk migrate mongod-fix-voting-priority
splunk migrate migrate-kvstore

I apply this commands with version 6.5.0? I mean, before upgrade to 7.2.3 or after?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

vishaltaneja070 — Fri, 08 Feb 2019 13:45:46 GMT

@rjfv8205 correct please run these commands before upgrade.

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Fri, 08 Feb 2019 13:47:15 GMT

@vishaltaneja07011993 what do it do exactly this commands?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

vishaltaneja070 — Fri, 08 Feb 2019 14:00:37 GMT

@rjfv8205
It will just migrate and fix any issue related to KV store before upgrade like certificate expired etc.

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Fri, 08 Feb 2019 16:28:59 GMT

@vishaltaneja07011993

Thanks, another questions. Apply comands with splunk up or down?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Fri, 08 Feb 2019 16:33:31 GMT

@agoyal Actually we have 6.5.0. What version do you recommend upgrade from 6.5.0 until a version before 7.2.3?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

AKG1_old1 — Fri, 08 Feb 2019 16:49:52 GMT

hi,
Actually mine was a workaround. I think you should first try commands mentioned in other comment regarding KV stone. if that doen't work you can try this workaround. you can take any stable version in between these vesions. Also I assume you are not going from 6.5 to 7.2.3 in production without proper testing. As there is major release between these version and may impact functionality. Telling with personal experience.

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

lhanich1 — Fri, 08 Feb 2019 16:59:01 GMT

We had this issue a few days ago.

Make sure that no mongod.lock exists in the $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/ dir.
Please also check the permissions of the splunk.key file in that dir are "chmod 0600 splunk.key".

In our case, splunk.key was too permissive which I was told it was a bug in 7.2.3.

Cheers

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

klaxdal — Mon, 11 Feb 2019 19:19:01 GMT

with splunk down

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Tue, 12 Feb 2019 14:28:20 GMT

Thanks @lhanich1 for your answer. Could you tell me mongod.lock use please? Before delete it

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

lhanich1 — Tue, 12 Feb 2019 14:49:41 GMT

This file indicates that mongo did not shut down gracefully and deleting the file will have no ramifications.

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Tue, 12 Feb 2019 14:53:45 GMT

Delete it with splunk down right?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

lhanich1 — Tue, 12 Feb 2019 15:32:46 GMT

Either should be fine.

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

lhanich1 — Tue, 19 Feb 2019 16:57:24 GMT

Did this get resolved?

Re: Problem with upgrade to 7.2.3 versión Splunk Enterprise

rjfv8205 — Fri, 01 Mar 2019 13:09:08 GMT

@lhanich1 Yes It was a problem with splunk.key permissions

Thank you !