Problem with upgrade to 7.2.3 versión Splunk Enterprise

[rjfv8205]

Hello everyone, actually we have Splunk version 6.5.0 and we need upgrade to latest version (7.2.3) but it throws follow error:

It seems that the Splunk default certificates are being used. If certificate validation is turned on using the default certificates (not-recommended), this may result in loss of communication in mixed-version Splunk environments after upgrade.
"/u01/splunk/etc/auth/ca.pem": already a renewed Splunk certificate: skipping renewal
"/u01/splunk/etc/auth/cacert.pem": already a renewed Splunk certificate: skipping renewal
Clustering migration already complete, no further changes required.
Generating checksums for datamodel and report acceleration bucket summaries for all indexes.
If you have defined many indexes and summaries, summary checksum generation may take a long time.
Processed 1 out of 8 configured indexes.
Processed 2 out of 8 configured indexes.
Processed 3 out of 8 configured indexes.
Processed 4 out of 8 configured indexes.
Processed 5 out of 8 configured indexes.
Processed 6 out of 8 configured indexes.
Processed 7 out of 8 configured indexes.
Processed 8 out of 8 configured indexes.
Finished generating checksums for datamodel and report acceleration bucket summaries for all indexes.
**ERROR: [App Key Value Store migration] Service(36) terminated before the service availability check could complete. Exit code 1, waited for 0 seconds.
[App Key Value Store migration] Checking if migration is needed. Upgrade type 1. This can take up to 600 seconds.
ERROR while running mongod-fix-voting-priority migration.*

Upgrade 6.3.3 to 6.5.0 without problems but now to upgrade to 7.2.3 we have problems.

Thank you in advance

[lhanich1 · Accepted solution]

We had this issue a few days ago.

Make sure that no mongod.lock exists in the $SPLUNK_HOME/var/lib/splunk/kvstore/mongo/ dir.
Please also check the permissions of the splunk.key file in that dir are "chmod 0600 splunk.key".

In our case, splunk.key was too permissive which I was told it was a bug in 7.2.3.

Cheers

[rjfv8205]

Thanks @lhanich1 for your answer. Could you tell me mongod.lock use please? Before delete it

[lhanich1]

This file indicates that mongo did not shut down gracefully and deleting the file will have no ramifications.

[rjfv8205]

Delete it with splunk down right?

[lhanich1]

Either should be fine.

[lhanich1]

Did this get resolved?

[rjfv8205]

@lhanich1 Yes It was a problem with splunk.key permissions

Thank you !

[AKG1_old1]

I had the same issue. I was trying to upgrade from 7.0.3 to 7.2.3. I first upgraded to intermidiate version(7.2.2) and then upgraded to 7.2.3. It worked for me.

[rjfv8205]

@agoyal Actually we have 6.5.0. What version do you recommend upgrade from 6.5.0 until a version before 7.2.3?

[AKG1_old1]

hi,
Actually mine was a workaround. I think you should first try commands mentioned in other comment regarding KV stone. if that doen't work you can try this workaround. you can take any stable version in between these vesions. Also I assume you are not going from 6.5 to 7.2.3 in production without proper testing. As there is major release between these version and may impact functionality. Telling with personal experience.

[maraman_splunk]

You may be interested into retrying with 7.2.4
see https://docs.splunk.com/Documentation/Splunk/7.2.4/ReleaseNotes/Fixedissues
Topic Upgrade
there's 1 entry about upgrade failing du to a kvstore migration issue

[vishaltaneja070]

@rjfv8205
Check this:
https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/TroubleshootKVstore#KV_store_migration_mess...

[rjfv8205]

Hi @vishaltaneja07011993

Steps mentioned in this topic.

splunk migrate mongod-fix-voting-priority
splunk migrate migrate-kvstore

I apply this commands with version 6.5.0? I mean, before upgrade to 7.2.3 or after?

[vishaltaneja070]

@rjfv8205 correct please run these commands before upgrade.

[rjfv8205]

@vishaltaneja07011993 what do it do exactly this commands?

[vishaltaneja070]

@rjfv8205
It will just migrate and fix any issue related to KV store before upgrade like certificate expired etc.

[rjfv8205]

@vishaltaneja07011993

Thanks, another questions. Apply comands with splunk up or down?

[klaxdal]

with splunk down

[maraman_splunk]

Hi,

does splunk show kvstore-status from 6.5 says kvstore is fine ?
what is your os with version ?
you could be in one of the case in known issues / Unsorted issues here : https://docs.splunk.com/Documentation/Splunk/7.2.3/ReleaseNotes/Knownissues (especially a very old glibc or the elision one)

any meaningfull info in mongod logs ?