Accidentally deleted search head instance and I'm not getting Mongodb error - kvstore?

[ze271021]

Hello,

My architecture is the following: cluster of indexers (3 indexers), 1 Master , 1 Search Head.

I removed by mistake the SErach head instance form the cluster and I re add it and now I am receiving this error:

Failed to start KV Store process. See mongod.log and splunkd.log for details.

KV Store changed status to failed. KVStore process terminated.

KV Store process terminated abnormally (exit code 6, status PID 3339006 killed by signal 6: Aborted). See mongod.log and splunkd.log for details.

mongod.log :

2023-04-27T14:41:33.317Z I CONTROL [initandlisten] MongoDB starting : pid=3339006 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=sv-na-splhead
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] db version v3.6.17-linux-splunk-v4
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] git version: 226949cc252af265483afbf859b446590b09b098
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2zf-fips 21 Jun 2022
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] allocator: tcmalloc
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] modules: none
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] build environment:
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] distarch: x86_64
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] target_arch: x86_64
2023-04-27T14:41:33.317Z I CONTROL [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, ssl: { PEMKeyFile: "/opt/splunk/etc/auth/server.pem", PEMKeyPassword: "<password>", allowInvalidHostnames: true, disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireSSL", sslCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." }, unixDomainSocket: { enabled: false } }, replication: { oplogSizeMB: 200, replSet: "4F365B7A-622E-4121-82B0-CF2933199661" }, security: { javascriptEnabled: false, keyFile: "/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "/opt/splunk/var/lib/splunk/kvstore/mongo", engine: "mmapv1", mmapv1: { smallFiles: true } }, systemLog: { timeStampFormat: "iso8601-utc" } }
2023-04-27T14:41:33.320Z I STORAGE [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /opt/splunk/var/lib/splunk/kvstore/mongo created by the 'wiredTiger' storage engine, but the specified storage engine was 'mmapv1'., terminating
2023-04-27T14:41:33.320Z F - [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272
2023-04-27T14:41:33.320Z F - [initandlisten]
***aborting after invariant() failure
2023-04-27T14:41:33.331Z F - [initandlisten] Got signal: 6 (Aborted).

0x55fed413bde1 0x55fed413aff9 0x55fed413b4dd 0x7fb5cae3d140 0x7fb5cac8dce1 0x7fb5cac77537 0x55fed27f5330 0x55fed2aa2d18 0x55fed3fe07f1 0x55fed3fdc8d7 0x55fed286c93c 0x55fed4137195 0x55fed27f6513 0x55fed28762f0 0x55fed287431a 0x55fed27f7159 0x7fb5cac78d0a 0x55fed285b665
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55FED1E7B000","o":"22C0DE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55FED1E7B000","o":"22BFFF9"},{"b":"55FED1E7B000","o":"22C04DD"},{"b":"7FB5CAE2A000","o":"13140"},{"b":"7FB5CAC55000","o":"38CE1","s":"gsignal"},{"b":"7FB5CAC55000","o":"22537","s":"abort"},{"b":"55FED1E7B000","o":"97A330","s":"_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j"},{"b":"55FED1E7B000","o":"C27D18","s":"_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj"},{"b":"55FED1E7B000","o":"21657F1","s":"_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE"},{"b":"55FED1E7B000","o":"21618D7","s":"_ZN5mongo6Client20makeOperationContextEv"},{"b":"55FED1E7B000","o":"9F193C"},{"b":"55FED1E7B000","o":"22BC195"},{"b":"55FED1E7B000","o":"97B513","s":"_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE"},{"b":"55FED1E7B000","o":"9FB2F0","s":"_ZN5mongo10StatusWithISt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_ESaIS9_EEEC1ERKSC_"},{"b":"55FED1E7B000","o":"9F931A","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55FED1E7B000","o":"97C159","s":"main"},{"b":"7FB5CAC55000","o":"23D0A","s":"__libc_start_main"},{"b":"55FED1E7B000","o":"9E0665"}],"processInfo":{ "mongodbVersion" : "3.6.17-linux-splunk-v4", "gitVersion" : "226949cc252af265483afbf859b446590b09b098", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.10.0-18-amd64", "version" : "#1 SMP Debian 5.10.140-1 (2022-09-02)", "machine" : "x86_64" }, "somap" : [ { "b" : "55FED1E7B000", "elfType" : 3 }, { "b" : "7FFF14ABA000", "path" : "linux-vdso.so.1", "elfType" : 3 }, { "b" : "7FB5CB31C000", "path" : "/opt/splunk/lib/libdlwrapper.so", "elfType" : 3 }, { "b" : "7FB5CB2FE000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3 }, { "b" : "7FB5CB019000", "path" : "/opt/splunk/lib/libcrypto.so.1.0.0", "elfType" : 3 }, { "b" : "7FB5CAFA2000", "path" : "/opt/splunk/lib/libssl.so.1.0.0", "elfType" : 3 }, { "b" : "7FB5CAF9C000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3 }, { "b" : "7FB5CAF92000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3 }, { "b" : "7FB5CAE4C000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3 }, { "b" : "7FB5CAE2A000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3 }, { "b" : "7FB5CAC55000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3 }, { "b" : "7FB5CB321000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3 }, { "b" : "7FB5CAC38000", "path" : "/opt/splunk/lib/libz.so.1", "elfType" : 3 } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55fed413bde1]
mongod(+0x22BFFF9) [0x55fed413aff9]
mongod(+0x22C04DD) [0x55fed413b4dd]
libpthread.so.0(+0x13140) [0x7fb5cae3d140]
libc.so.6(gsignal+0x141) [0x7fb5cac8dce1]
libc.so.6(abort+0x123) [0x7fb5cac77537]
mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x55fed27f5330]
mongod(_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj+0x158) [0x55fed2aa2d18]
mongod(_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE+0x41) [0x55fed3fe07f1]
mongod(_ZN5mongo6Client20makeOperationContextEv+0x27) [0x55fed3fdc8d7]
mongod(+0x9F193C) [0x55fed286c93c]
mongod(+0x22BC195) [0x55fed4137195]
mongod(_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE+0x36E) [0x55fed27f6513]
mongod(_ZN5mongo10StatusWithISt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_ESaIS9_EEEC1ERKSC_+0x0) [0x55fed28762f0]
mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x87A) [0x55fed287431a]
mongod(main+0x9) [0x55fed27f7159]
libc.so.6(__libc_start_main+0xEA) [0x7fb5cac78d0a]
mongod(+0x9E0665) [0x55fed285b665]
----- END BACKTRACE ----

Can I have any advise on how to resolve it ?

Thank you!