Storage Engine did not migrate to WiredTiger after upgrade to Splunk 9.0.2?

[SplunkNinja]

Hello everybody!

My standalone SH Storage Engine did not migrate to WiredTiger after upgrade to Splunk 9.0.2.  I then followed these steps: https://docs.splunk.com/Documentation/Splunk/8.1.3/Admin/MigrateKVstore

Migrate the KV store after an upgrade to Splunk Enterprise 8.1.* or 8.2.* in a single-instance deployment

Stop Splunk Enterprise. Do not use the -f option.
Open server.conf in the $SPLUNK_HOME/etc/system/local/ directory.
Edit the storageEngineMigration setting to match the following example:

[kvstore]
storageEngineMigration=true

Save the server.conf file.
To begin the migration, use the following command:

splunk migrate kvstore-storage-engine --target-engine wiredTiger

Starting KV Store storage engine upgrade:
Phase 1 (dump) of 2:
..ERROR: Failed to migrate to storage engine wiredTiger, reason=

How can I troubleshoot this further?  Thanks.

[richgalloway]

TBH, I'm more concerned about the "connection rejected" errors than the "connecting with a certificate" warning, but either way I don't have a solution for you. I suggest double-checking all of your KVstore settings. Contact Splunk Support if you have an entitlement.

[SplunkNinja]

Thanks Rich.

I have opened a Support case to further investigate.

[SplunkNinja]

Hello Rich,

I followed the 9.0.2 steps with the same results.  My mongodb.log is showing issues with SSL.  Not sure why we see - connecting with a certificate with cluster membership

This is supposed to be a standalone SH.  Not sure what to try next.  Thanks.

2022-11-29T19:21:34.576Z W ACCESS [conn1] Client isn't a mongod or mongos, but is connecting with a certificate with cluster membership
2022-11-29T19:21:34.576Z I ACCESS [conn1] Successfully authenticated as principal CN=a.b.c.d,OU=xxx,O=yyy on $external from client 127.0.0.1:37278
2022-11-29T19:21:34.576Z I NETWORK [conn1] end connection 127.0.0.1:37278 (0 connections now open)
mongodump 2022-11-29T19:21:35.588+0000 WARNING: --sslAllowInvalidCertificates and --sslAllowInvalidHostnames are deprecated, please use --tlsInsecure instead
2022-11-29T19:21:35.593Z I NETWORK [listener] connection accepted from 127.0.0.1:37284 #2 (1 connection now open)
2022-11-29T19:21:35.593Z I NETWORK [listener] connection accepted from 127.0.0.1:37286 #3 (2 connections now open)
2022-11-29T19:21:35.605Z E NETWORK [conn2] no SSL certificate provided by peer; connection rejected
2022-11-29T19:21:35.606Z I NETWORK [conn2] Error receiving request from client: SSLHandshakeFailed: no SSL certificate provided by peer; connection rejected. Ending connection from 127.0.0.1:37284 (connection id: 2)
2022-11-29T19:21:35.606Z I NETWORK [conn2] end connection 127.0.0.1:37284 (1 connection now open)
2022-11-29T19:21:35.606Z E NETWORK [conn3] no SSL certificate provided by peer; connection rejected
2022-11-29T19:21:35.606Z I NETWORK [conn3] Error receiving request from client: SSLHandshakeFailed: no SSL certificate provided by peer; connection rejected. Ending connection from 127.0.0.1:37286 (connection id: 3)

[richgalloway]

Try again using the documentation for the version you are running.  https://docs.splunk.com/Documentation/Splunk/9.0.2/Admin/MigrateKVstore