topic Update from 8.2 to 9.0.4- How to resolve problems kvstore upgrade to wiredtiger? in Splunk Enterprise

Update from 8.2 to 9.0.4- How to resolve problems kvstore upgrade to wiredtiger?

osakachan — Wed, 15 Mar 2023 16:36:30 GMT

Hello Folks,

I am having a bit of trouble finishing an update.

I have this message in the update:

¿Where is the migration log? mongod.log? I do not see anything to work with.

Now kvstored is disabled and I can not manually update to wiredTiger.

Thanks in advance.

Re: Update from 8.2 to 9.0.4, problems kvstore upgrade to wiredtiger

isoutamo — Wed, 15 Mar 2023 11:15:22 GMT

all logs are on $SPLUNK_HOME/var/log/splunk/ directory.

Which kind of instance you are upgrading to 9.0.4 (Indexer, SH or all in one box)?

This error said that you haven't sslPassword for kvstore. Just check from $SPLUNK_HOME/etc/system/local/server.conf what you have and if/when needed use

splunk btool server list sslConfig --debug

That should show your sslPassword and where it has set.

r. Ismo

Re: Update from 8.2 to 9.0.4, problems kvstore upgrade to wiredtiger

osakachan — Wed, 15 Mar 2023 12:16:05 GMT

all logs are on $SPLUNK_HOME/var/log/splunk/ directory.

Ah, yes, I saw it now.. but looks like is the same information than you can see as the update is running.

Which kind of instance you are upgrading to 9.0.4 (Indexer, SH or all in one box)?

Is master/DS with SH capabilities. Right now is not a problem because of that, but as you can expect, I would liketo understand what is happening for when we update the SH.

Because we encountered this problem, I think disabling kvstore in IDX and HF levels is an option. I understand that disabling it at that nodes is not problem because it only works with "lookup" data. ¿?¿?

This error said that you haven't sslPassword for kvstore. Just check from $SPLUNK_HOME/etc/system/local/server.conf what you have and if/when needed use

splunk btool server list sslConfig --debug

That should show your sslPassword and where it has set.

Yeah, I know, but certs are passwordless, so sslPassword is blank. Testing with random password results in this:

And finishing, it is like an instance of mongodb keeps alive and using the default port:

Trying to do the upgrade to wiredtiger happens this:

mongod.log

Kind regards.

Re: Update from 8.2 to 9.0.4, problems kvstore upgrade to wiredtiger

osakachan — Thu, 16 Mar 2023 14:05:15 GMT

We did not have any kind of problem to update kvstore in HF level. If disabled, SplunkDBConnect stoped working.

Re: Update from 8.2 to 9.0.4- How to resolve problems kvstore upgrade to wiredtiger?

osakachan — Wed, 22 Mar 2023 12:51:37 GMT

Looks like it has nothing to do with wiredTiger at all.

The problem the kvstore does not initiate in 9.0.4.

This is with --verbose

An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [connection closed calling ismaster on '...<domain>...:8191']

in mongd.log we can se a lot of connections and disconects.

We tried infinity of things arround TLS certificates without results.